Statement of Dan Nagala, UTSI International

MR. NAGALA: Thank you. It's a pleasure to be here.

I must say that over the last few days I've seen a lot of similarities between different transportation industries and the pipeline industry, and I'm going to talk about some of them as we continue. As I begin, I'd like to make one important distinction between pipelines and other modes of transportation, and that is that a pipeline is a fixed asset. It's a very long asset, and the product that's transported moves through the pipeline. The pipeline itself doesn't move. Fortunately it doesn't move! Because of this, we have a much better base for acquiring information, and for being exactly sure about where that information is acquired and what it's telling us while we're analyzing it in real time.

We still have a tremendously difficult problem because of the fluid dynamics that exist within a pipeline system. And of course, it's impossible to put instrumentation at every possible mile or inch or centimeter of a pipeline system. It's cost prohibitive and it just can't happen. So in the pipeline industry, especially in the control systems business, it's always been a challenge to manage the expectations of the customers within the operating companies. Vendors and software developers of control systems and of simulation systems have struggled with this for a long time.

I'm sure that all of us have seen the movie cliches of Star Trek and other sorts of science fiction programs where the computer does everything and knows everything. If they want to know what happened last week or yesterday or in the last 10 minutes, they just say, "Tell us what happened, and give us an analysis." Well, of course we'd all like to be able to do this, but most of us also realize that it's not completely practical. Although there are some advances in the industry that are moving toward being able to do a lot of proactive analysis, and which can also provide better mechanisms for incident analysis after an incident occurs.

So as I talk today, I'm going to spend a little bit of time speaking about pipeline system data recording techniques, and of what we are doing today. Also, I will highlight what is done when an incident occurs, and how we go about dealing with all of the disparate data sources. Then I will talk about some future vision, and some architectures that are on the forefront of our technology in this industry.

During the past couple of days, my colleagues from Teledyne, Brown Engineering and Colorado Interstate Gas have spent quite a lot of time talking about SCADA systems and how those SCADA systems apply to pipelines, so I'm not going to belabor this point. But there are a couple of important points I'd like to make.

First of all, we should think of a SCADA system as consisting of six principal components. At the low end we have field devices which consist of instrumentation, pressure transducers, temperature transducers, flow rate measurement devices, metering, and actuators that give us device control to open and close valves, start and stop pump units, compressors or whatever. Moving up from there, usually at every physical location, we have remote terminal units. Remote terminal units connect to the field devices and provide the conduit, or the principal data acquisition source, for where information is acquired from the field.

Our SCADA systems normally are in continuous communication with this equipment, and in different kinds of applications, for example in the gas industry, we typically maintain certain historical data at the level of the remote unit. However, in the liquids industry, that's not so popular.

Another other important difference at the remote terminal unit level is that the communication methods used in the gas industry typically acquire information every two to five minutes, which is generally accepted practice in this gas industry. This is because the dynamics of a gas pipeline system are fairly slow. It's a very compressible product, and changes don't happen that fast. So, there hasn't been a lot of push to increase the data acquisition rates up until now.

In the liquids industry, when you're dealing with thousands of miles of columnar flow that's somewhat compressible, but generally not a very compressible product, we require much faster data rates. Typical data rates here are within five to fifteen seconds, depending on the operating company and the type of pipeline.

We're also seeing some operating requirements now where certain kinds of information is acquired every two seconds. Communication systems play a very important role in this whole process, and communication still continues to be the big link and the big problem, in the event there is an upset and you use lose communication to your field devices. Communications integrity is very important, and in the liquids industry it's common to have backup communication systems. If you are communicating via satellite, leased lines, or some other medium, and you have a failure of your primary mechanism, these systems will typically fail-over to a backup, or redundant, mode of communication that provides connectivity at least the most important sites.

At the upper end or the mid level here, we have our server components, which consist of redundant SCADA systems. These are hot fail-over systems so you don't lose control in the event of an equipment or software failure at this level. Of course we also have our communication termination equipment and application systems at this level. These might consist of various kinds of historical archive systems, and might also include various real time models and simulation systems.

In the gas industry, predictive modeling and optimization is very prevalent today. In the liquids industry, leak detection and real time modeling associated with leak detection and tracking different product batches as they move through the line are very prevalent today.

We generally provide connectivity within a control center, and within the enterprise through a network. This includes a myriad of user interface devices, along with the control center server components. Of course, our user interface devices provide the main link between the operations personnel, and what's actually going on in the field. There are a lot of layers in these systems where the propagation of data can cause latency, and contribute to response time in the event that an incident does occur.

So what is the industry doing today in terms of data recording? In most cases today there are only selected variables kept in history. Up until now, it's been fairly impractical to record every piece of acquired data when you've got hundreds of thousands of independent variables being acquired in real time, maybe down to the five second level. Because of the volumes of information, companies have taken a stance at looking at the most important variables, and they typically take selected pressures, temperatures, and volumetric information to be saved in history.

Alarm and event information, is of course very important to a pipeline system, and that too, is kept in history. In fact, I believe that in most cases alarm and event information is a mandated requirement, so every alarm that occurs, and every event that occurs is kept in some sort of short-term, and additionally a long-term archive system.

The problem we have here is that typically the information that's kept in real time historical data files, or in independent archive systems, are typically different systems from those systems that keep the alarm and event information. Alarm and event information is text information. It's time stamped and it's usually saved in chronological order. The historical telemetered data is normally kept as compressed information, which may be sampled at the time of change, or via some exception sort of reporting scheme. It is usually saved in a proprietary or customized sort of historical archive system. Since the alarm/event and data items are typically saved in different systems, these tend to cause a little bit of a problem when you're trying to reassemble information after an incident, and when you are looking at it to try to figure out exactly what happened.

Currently, if we have an incident or an anomalous operational event, which may not necessarily mean that we've had a pipeline breech or that we've lost any product, typically what pipeline management and investigators will do is come into the pipeline control center and say, "Okay, I want all of your historical data for the time leading up to this incident, including the time during and immediately after this incident. I want all of your alarm and event records, along with any operator notes." Generally speaking, the investigator wants to have everything that you have that's going to help him figure out what went wrong.

Pipeline companies traditionally have had some problem assembling all of this information, and keeping it correct with respect to time and sequence. These are the challenges that exist today. And really, as I've listened to the presentations in the last couple of days, I've noticed that there are a lot of other similarities in the other industries. Things like correlation of independent variables with respect to time, and not having enough information. Maybe the selected variables that we decided we were going to keep weren't the right ones for this particular kind of an incident. So either we don't have enough data or we don't have the correct data. Maybe we didn't sample it fast enough, or we have limitations in storage capacity. I've heard talk about eight independent variables on a data recorder moving up to 20, to 200, to maybe to thousands of variables in an aircraft, or in a train, or a ship, or some other mode of transportation. In pipelines, we're dealing with hundreds of thousands of data points, and we have the same problems, just on a different scale.

We are also challenged with resistance to change. We have seen the same kind of problems in the pipeline control center that I've heard about in the airline industry, where the pilots don't want to have their privacy in the workplace infringed upon. Still, there has been a lot of talk about recording voice conversations of operators, and today some pipeline companies actually do keep records of incoming calls. There has been talk about keeping more and more information regarding the day-to-day operations of the pipeline control center, and pipeline operators are a little concerned about that. So the change process is a big challenge that pipeline companies will have to deal with as we move forward into the future.

The physical pipeline is managed proactively pretty well. Pipeline companies have a lot of programs that are targeted at eliminating, or reducing, corrosion damage, and identifying it before it causes a breech. This done by inspecting the pipeline through smart pigging and other sorts of processes to ensure that the pipe is still intact, and is remains in good secure order. However, third party damage continues to be the major cause of pipeline accidents, and unreported third party damage might happen a year or more before an actual incident happens as a result of that damage.

What our task comes down to is that we need to shift from a concept of just doing what we need to get by to operate the pipeline, and provide better recognition and response protocols for pipeline operators. When I talk about recognition and response protocols, I mean better software, better simulation systems, better leak detection and better training for our operators, providing us the ability to recognize incidents shortly [quickly and accurately] after they happen. Unfortunately, we can't recognize a leak before it happens, but we can before too much collateral damage has occurred as a result of a breech.

Mark Westhoff yesterday talked about leak detection in gas pipelines, and the difficulties with simulation systems being able to, by software means, recognize a leak quicker than it can be identified by physical observation. This is because a rancher out in the middle of West Texas somewhere, is going to hear and/or see it long before the software system can really come around to saying, "hey, I've got an anomaly here that looks like it's a leak." The software system cannot do this until it's satisfied some persistence requirement allowing it to announce that it is certain there's a leak. Gas leak detection is real difficult. And even though we understand the technology very well, it is still very hard to detect a gas pipeline leak with a software based leak detection system before it is discovered in the field by direct observation.

In the liquids industry however, it is a lot different. This is because a seepage or a small leak can go undetected for a long time at a minimal leak flow rate, and nobody is likely to see it unless they happen to be driving along and catch recognize some indication.

What we're going to do in the future is to integrate simulation systems, real time playback, and a much more comprehensive historical data acquisition and archival types of systems. These applications will all be integrated together into what I call "Full Scope In-Line Simulation" systems. These systems will be tied together through common communication techniques employed within the control system environment, as well as through common techniques for defining the physical pipeline and its attributes for simultaneous use by the simulation and the SCADA systems. They will also provide the operator information in a seamless environment, and they will allow him to not only access simulated views of information, but to perform "what-if" simulations in-line with the real-time environment, but to also review past performance and operations for analysis and training.

So, here is a conceptual architecture for this type of system, similar to the SCADA environment, except without all the physical components shown. This architecture assumes that there is a common base containing a definition of a pipeline. Every detail about the pipeline has to be defined here so that all of the applications within the system can leverage off of exactly the same configuration details. This contains physical pipe characteristics, instrumentation characteristics, and data acquisition details. It describes how the SCADA system works, and it describes how all of the real-time and predictive modeling systems will function together.

The key components we have here are a real time data acquisition system, a historical archive system capable of keeping every change for every calculated and telemetered variable, in a highly optimized and compressed form. This allows for rapid recall so that it can be used for various kinds of post-operational analysis and training. We also have simulation systems, real time models, leak detection, predictive and look ahead models, and also a fully integrated training system.

Another key factor in all of this is that all of these entities coexist in a highly distributed operating environment - of course it's impractical to think that they could all run on the same computer. All of these types of applications require different kinds of processing capabilities, and different kinds of resources. It's impractical to think that they could run all on one box, and therefore, we need a common mechanism for each of them to present their information to the dispatching personnel.

This includes a common messaging bus, something like CORBA. We've seen CORBA standards being talked about quite a lot recently, and there may be new standards in the forefront as well. Microsoft's COM and DCOM technologies could prove be a player here as well. But the idea is to be able to have all of these applications working within some standard interfacing technique to obtain their configuration information, and within some publishing standard to provide information to user interfaces. This is important so that users are able to see information views, via the same mechanism, on the same CRT, from all of these different sources in a seamless manner.

Those of you who know me, know that I've been a proponent of the common user interface in pipeline control systems for maybe eight or 10 years, and now more and more we're beginning to see this happen. Vendors are starting to come out with these capabilities, and there have been a few third party products on the market for a few years, but they haven't really taken hold yet.

But I tell you, I hate to walk into a pipeline control center and see 10 different CRTs, each of which is dedicated to different kind of application that an operator has to look at. He has to assimilate the information the way that it's presented. And the way that he interacts with any one source is different from that of all the other systems in the control center. We are expecting him to operate several thousand miles of pipeline while he's dealing with all these different systems, and it's just too much. All of this information needs to come together so that our operators are able to work more effectively.

Common configuration repositories, historical archive and retrieval systems capable of keeping every piece of information, and historical playback systems which allow access to historical data, and provide playback in the same way that it was viewed in real time, are all components of the future architecture. We've seen some examples of these related to SCADA information, but we're still looking to the future for them to be integrated together as a common seamless function that includes the simulation systems as well. These capabilities will require transparent switching between real time history and simulation functions, and the common user interface will provide the window through which these functions managed and viewed.

Data requirements. Well, I talked about every piece of data earlier, but it's not just telemetered data that I'm concerned about. It's not just what you're seeing from the field that is going to lead you to the conclusions you have to come to when evaluating the current state of a pipeline system. There's a lot of calculated information too. It might be directly shown in field measurements, but it might not be obvious without some additional calculated enhancements within the master station. Therefore, calculated information remains a very important component here.

Operator actions. Currently, very few control centers keep all operator actions as historical entries. They keep selected items like alarm acknowledgements, control requests, various kinds of parameter entries and whatnot, but they don't keep any display navigation information. They don't keep any sort of "just looking around in the system to see what's going on" types of information, including pop-ups of different kinds of windows, and various ad-hoc advisories. I believe that this is very important, because if you have an incident, you can find out exactly what was going on in the control console at the time of the incident. Let's say we had a leak in a liquid pipeline that flowed for 15 or 20 minutes. The control center had not seen or at least reacted to it, and by looking at the operator's records you find all of the appropriate information to indicate some action should have been taken, but no action was taken. However, by reviewing the operator's records, you might find that he was busy dealing with another event at the time, which might change the tone of an incident investigation. If you can see what he was doing at the time, you can either eliminate him from fault, or you can have a better idea of where he might not have been trained well enough to recognize a given type of condition. Improper or incomplete training can lead to missing subtle indications of a problem, and can contribute to slow reaction times.

Historical alarms and events must be presented in the correct sequence. Of course, we try to keep them in chronological order, but they also need to be in the correct sequence with respect to the data. When we are forced to deal with disparate systems, which is the case many times right now, the assembly process can a big problem. So in the future we will be able to keep all of the alarm and event information in exactly the right sequence with respect to time, just as we do with the telemetered and calculated data.

Of course, every time you make a configuration change, an update, add a new device to the field, or any type of change to the infrastructure, this also must be recorded. Furthermore, these changes must be propagated at the same time, to all of the systems that are involved in the pipeline control system.

User interfaces. I talked a little bit about common user interfaces already with reference to the simultaneous access of multiple forms of data. These also have to be better integrated in the future. They have to be integrated to provide the maximum benefits to the control system, to the users, and to allow them to deal with more information in an efficient and effective manner. The real problem is that we are continually coming up with more and more information for our operators. Therefore, we need to also be able to provide this seamless integration functionality between the real time, simulation and training functions. This will allow an operator, an operations manager, or a supervisor to say, "Okay, let's do a training scenario using yesterday's operational anomaly as an example." "We are going to look at this data, and at some point we're going to deviate from the playback and switch to an in-line simulation." The benefit of this is that we can say, "what if we had done something different at this time? Would the result have been different? Or even better?" Training operators to deal with real situations through the use of real data, augmented by simulation will provide a valuable base for better response and recognition in the future.

Let's say that we had an event, and it turns out to have been a leak. Maybe it ran for an hour before it was detected, but there were some minor indications. If we can train our operators to recognize those minor indications, especially new operators, our losses and collateral damage will be reduced. Many times seasoned operators, that have looked at the same pipeline for years, and understood its hydraulics, can see things that a new operator will not. One objective of using this recorded data for training is to pass on some of this experience to new operators, and to reduce the time and cost associated with their training. As a result of this training, if they have a minor incident, even if the simulation or leak detection systems haven't quite given a definitive warning, they might start to notice that something that's going wrong.

Many pipeline companies tell me that it takes as much as a year from the time they hire a new operator, to the time they will allow him to operate a shift by himself, or at least to have that responsibility. That's a tremendous investment. And if we can improve that to allow him to recognize his key indicators much quicker, then our response gets quicker and our damage to the environment and our potential risk to public safety is significantly reduced.

The benefits of this kind of system are that it provides availability of accurate and precise details for incident analysis, and allows us to come to conclusions much quicker. It will allow us to train our operators to recognize and respond quicker, and it will also improve our business operations.

This information is a valuable resource. Up until about 10 years ago, pipeline companies operated as if there was a wall between the management of the company and the operations. The physical system that operated the pipeline was viewed as more of a tool than a resource. It kept the asset running, allowed the product to flow and the company made money, and of course that was good. In the last eight or 10 years, companies have realized that this is a great resource and they can use this information to optimize their business. They can use it for marketing, planning and strategizing: for example study where to make infrastructure improvements, or add more customers.

In summary, I'd like to say that advances in computer technology will allow this kind of a system to become a reality, probably within the next six to eight years. We're already seeing some companies playing with these kinds of technologies but still the integration of the components isn't quite there.

Several of the larger pipeline companies in North America are looking seriously at how we can better deal with the information from our simulation systems, predictive models, leak detection systems and our SCADA systems. One company in particular that John Wallace from Teledyne mentioned, is Koch Industries. Koch has been sort of a leader in this area in the last few years, and I think John mentioned that they have an exhibit now in the Smithsonian about how they are dealing with technology, and what advances they're employing in their pipeline control center. In their particular case, in fact, they are one of very few companies archiving all of their real time telemetered information in a long-term historical archive system. There are still some difficulties in dealing with this much data, but time will fix that. Koch is also taking results from their simulation systems, and keeping them in the same historical archive so that they can view all of this information together, and analyze it as a whole as required.

There are many improvements already in progress, and vendors appear to be in support of the change. However, the vision I have presented here will take time, and money to realize.

That concludes my talk. Unfortunately, my written paper didn't make it into the conference publication. I had a little confusion over the submission dates, and I apologize for that. However, if any of you would like a copy of my printed paper, I'd be happy to e-mail it to you. My e-mail address is on this last slide, along with my telephone number. Please feel free to contact me if you wish.

Thanks.