Industrial process control system with back-up data processors to take over from failed primary data processors

Claims

I claim:

1. An industrial control system for managing a plurality of input/output devices comprising a primary data processor having a memory and operable to receive signals from said input/output devices, said primary data processor including a central processing unit operable to carry out an applications program and apply signals to said input/output devices in accordance with determinations made in said applications program to control said input/output devices, said primary processor maintaining in said memory of said primary processor a record of status data including a status data component representing a plurality of parameters of the current operation of said applications program, said central processing unit in carrying out said applications program operating on and in response to said status data in said current record maintained in the memory of said primary processor, a back-up data processor having a memory and having an active mode of operation and a back-up mode of operation, said back-up processor being operable in said active mode of operation to receive signals from said input/output devices, carry out said applications program and apply signals to said input/output devices in accordance with the determinations made in the applications program carried out by said back-up processor to control said input/output devices, said back-up processor maintaining a record of status data in the memory of said back-up processor corresponding to the record of status data maintained in the memory of said primary processor, said back-up processor, when carrying out said applications program, operating on and in response to the status data in the record thereof maintained in the memory of said back-up processor, said primary processor and said back-up processor operating in said back-up mode comprising means to periodically transfer a copy of the status data in the memory of said primary processor to a buffer memory section of the memory of said back-up processor, said means to transfer status data from the memory of said primary processor to a buffer memory section of the memory of said back-up processor including a dual ported memory connected between said primary processor and said back-up processor, said primary processor operating to periodically transfer a copy of the status data from the record of status data maintained by said primary processor in the memory of said primary processor to said dual ported memory, said back-up processor, when operating in said back-up mode, operating to periodically transfer a copy of the status data in said dual ported memory to the buffer memory section of the memory of said back-up processor, said central processing unit generating for each copy of status data transferred to said back-up processor information from which the validity of the status data can be determined, said information being included with the corresponding copy of status data transmitted to said back-up processor, said back-up processor including means operable when said back-up processor is in said back-up mode to determine from the entire content of said status data component and the corresponding information in each periodic copy of status data received in said buffer memory section whether said status data component is valid and, if the copy of said status data component is determined to be valid, to update the record of status data maintained in the memory of said back-up processor in accordance with the copy of the status data in said buffer memory section, means to monitor the condition of said primary processor independently of said status data and to generate a fail over signal when said primary processor fails, and means to switch said back-up processor from said back-up mode of operation to said active mode of operation in response to said fail over signal.

2. An industrial process control system as recited in claim 1, wherein said primary data processor sets a predetermined byte in said dual ported memory each time it stores a copy of the status data in said dual ported memory, said back-up processor clearing said predetermined byte in said dual ported memory each time it reads a copy of status data out of said dual ported memory and stores the status data copy in the buffer memory section of the memory of said back-up processor, said primary processor storing a copy of the status data in said dual ported memory only when said predetermined byte in said dual ported memory is clear and said back-up processor transferring the copy of the status data from the dual ported memory to the memory of the back-up processor only when said predetermined byte in said dual ported memory is set.

3. An industrial process control system as recited in claim 1, wherein said primary data processor comprises at least one processing unit to carry out said applications program, the memory of said primary data processor, communication means to apply signals to said input and output devices and a communications bus connected between said processing unit of said primary data processor, said memory of said primary data processor and said communication means of said primary data processor, said back-up data processor comprising at least one processing unit to carry out said applications program, the memory of said back-up processor, communication means to apply signals to said input/output devices, and a communications bus connected between the processing unit of said back-up processor, the memory of said back-up processor, and the communication means of said back-up processor, said dual ported memory connected between the communications bus of said primary processor and the communications bus of said back-up processor.

4. An industrial control system as recited in claim 1, wherein said information is a multibit check word and said primary processor transfers said multibit check word to said dual ported memory along with each copy of the status data periodically transferred by said primary processor to said dual ported memory, said multibit check word having a value depending upon the bits in the status data, the copy of which is stored in said dual ported memory, said backup processor when operating in said backup mode, operating to transfer the check word stored in said dual ported memory along with the copy of the status data in said dual ported memory to the memory of said backup processor, said backup processor carrying out a validity check on the copy of said status data received in said buffer memory section by recomputing the check word from the copy of the status data in the buffer memory section of the memory of said backup processor and comparing the recomputed check word with the check word transferred from said dual ported memory to the memory of said backup processor along with the copy of the status data.

5. An industrial control system for managing a plurality of input/output devices comprising a primary data processor having a memory and operable to receive signals from said input/output devices, said primary data processor including a plurality of processing units each operable to execute a corresponding applications program to control said input/output devices, each applications program comprising a set of instructions executed in sequence by the corresponding processing unit, said primary processor maintaining in said memory of said primary processor a record of status data including a status data component representing a plurality of parameters of the current operation of said applications programs, said each of said processing units in carrying out the corresponding applications program acting on and in response to the status data in said record of status data in the memory of said primary processor, a back-up data processor having a memory and having an active mode of operation and a back-up mode of operation, said back-up processor being operable in said active mode of operation to receive signals from said input/output devices, carry out said applications programs and apply signals to said input/output devices in accordance with the determinations made in the applications programs carried out by said back-up processor to control said input/output devices, said back-up processor maintaining a record of status data in the memory of said back-up processor corresponding to the record of status data maintained in the memory of said primary processor, said back-up processor, when carrying out said applications programs, operating on and in response to the status data in the record thereof maintained in the memory of said back-up processor, said primary processor and said back-up processor operating in said back-up mode comprising means to periodically transfer a copy of the status data in the memory of said primary processor to a buffer memory section of the memory of said back-up processor, each of the applications programs being executed by said processing units including a predetermined instruction at which a copy of the status data is to be transferred to the back-up processor, said means to transfer a copy of status data to the back-up processor operating to transfer the copy of status data when all of said central processing units have reached said predetermined instruction in the in the sequence of execution of the corresponding applications program, each of said processing units waiting in the sequence of execution of the corresponding applications program at said predetermined instruction until said means to transfer a copy of said status data has completed the readout of a copy of the status data from the memory of said primary processor, said central processing unit generating for each copy of status data transferred to said back up processor information from which the validity of the status data can be determined, said information being included with the corresponding copy of status data transmitted to said back-up processor, said back-up processor including means operable when said back up processor is in said back-up mode to determine from the entire content of said status data component and the corresponding information in each periodic copy of status data received in said buffer memory section whether said status data component is valid and, if the copy of said status data component is determined to be valid, to update the record of status data maintained in the memory of said back-up processor in accordance with the copy of the status data in said buffer memory section, means to monitor the condition of said primary processor independently of said status data and to generate a fail over signal when said primary processor fails, and means to switch said back-up processor from said back-up mode of operation to said active mode of operation in response to said fail over signal.

6. An industrial control system for managing a plurality of input/output devices comprising a primary data processor having a memory and operable to receive signals from said input/output devices, said primary data processor including a central processing unit operable to carry out an applications program and apply signals to said input/output devices in accordance with determinations made in said applications program to control said input/output devices, said primary processor maintaining in said memory of said primary processor a record of status data including a status data component representing a plurality of parameters of the current operation of said applications program, said central processing unit in carrying out said applications program operating on and in response to said status data in said current record maintained in the memory of said primary processor, a back-up data processor having a memory and having an active mode of operation and a back-up mode of operation, said back-up processor being operable in said active mode of operation to receive signals from said input/output devices, carry out said applications program and apply signals to said input/output devices in accordance with the determinations made in the applications program carried out by said back-up processor to control said input/output devices, said back-up processor maintaining a record of status data in the memory of said back-up processor corresponding to the record of status data maintained in the memory of said primary processor, said back-up processor, when carrying out said applications program, operating on and in response to the status data in the record thereof maintained in the memory of said back-up processor, said primary processor and said back-up processor operating in said back-up mode comprising means to periodically transfer a copy of the status data in the memory of said primary processor to a buffer memory section of the memory of said back-up processor, said applications program comprising a series of arithmetic functions performed in sequence with each function producing a variable output value as a function of one or more input values, said status data including the variable output value produced by each arithmetic function performed in said applications program, said means to transfer a copy of said status data to the memory of said back-up processor operating to read out from the memory of said primary processor the output value of each arithmetic function at the time said applications program completes said arithmetic function to initiate the transfer of this portion of the copy of the status data to the memory of the back-up processor, said central processing unit generating for each copy of status data transferred to said back up processor information from which the validity of the status data can be determined, said information being included with the corresponding copy of status data transmitted to said back-up processor, said back-up processor including means operable when said back up processor is in said back-up mode to determine from the entire content of said status data component and the corresponding information in each periodic copy of status data received in said buffer memory section whether said status data component is valid and, if the copy of said status data component is determined to be valid, to update the record of status data maintained in the memory of said back-up processor in accordance with the copy of the status data in said buffer memory section, means to monitor the condition of said primary processor independently of said status data and to generate a fail over signal when said primary processor fails, and means to switch said back-up processor from said back-up mode of operation to said active mode of operation in response to said fail over signal.

7. An industrial control system for managing a plurality of input/output devices comprising a primary data processor having a memory and operable to receive signals from said input/output devices, said primary data processor including a central processing unit operable to carry out an applications program and apply signals to said input/output devices in accordance with determinations made in said applications program to control said input/output devices, said primary processor maintaining in said memory of said primary processor a record of status data including a status data component representing a plurality of parameters of the current operation of said applications program, said central processing unit in carrying out said applications program operating on and in response to said status data in said current record maintained in the memory of said primary processor, a back-up data processor having a memory and having an active mode of operation and a back-up mode of operation, said back-up processor being operable in said active mode of operation to receive signals from said input/output devices, carry out said applications program and apply signals to said input/output devices in accordance with the determinations made in the applications program carried out by said back-up processor to control said input/output devices, said back-up processor maintaining a record of status data in the memory of said back-up processor corresponding to the record of status data maintained in the memory of said primary processor, said back-up processor, when carrying out said applications program, operating on and in response to the status data in the record thereof maintained in the memory of said back-up processor, said primary processor and said back-up processor operating in said back-up mode comprising means to periodically transfer a copy of the status data in the memory of said primary processor to a buffer memory section of the memory of said back-up processor, said central processing unit generating for each copy of status data transferred to said back-up processor information from which the validity of the status data can be determined, said information being included with the corresponding copy of status data transmitted to said back-up processor, said back-up processor including means operable when said back up processor is in said back-up mode to determine from the entire content of said status data component and the corresponding information in each periodic copy of status data received in said buffer memory section whether said status data component is valid and, if the copy of said status data component is determined to be valid, to update the record of status data maintained in the memory of said back-up processor in accordance with the copy of the status data in said buffer memory section, means to monitor the condition of said primary processor independently of said status data and to generate a fail over signal when said primary processor fails, and means to switch said back-up processor from said back-up mode of operation to said active mode of operation in response to said fail over signal, said status data including an instruction pointer designating the next instruction to be executed by said applications program, said back-up processor beginning operation to carry out said applications program upon failure of said primary processor at the instruction designated in the copy of said instruction pointer transferred to the memory of said back-up processor by said means to transfer a copy of status data from the memory of said primary processor to the memory of said back-up processor.

8. An industrial control system as recited in claim 6, wherein said means to transfer a copy of the status data to the memory of the back-up processor comprises a dual ported memory connected between said primary processor and said back-up processor, said primary processor storing in said dual ported memory each output value of an arithmetic function of the applications program as it is read out from the memory of said primary processor, storing successive output values of the arithmetic functions in successive memory slots in said dual ported memory, said back-up processor reading out the output values from said successive memory slots and storing the values in the buffer memory section in the memory of said back-up processor in the same sequence that said output values are stored in said dual ported memory.

BACKGROUND OF THE INVENTION

This invention relates to a data processing system used to control an industrial process wherein a back-up data processor or processors are used to automatically assume control over the industrial process when the primary processor controlling the industrial process fails.

Many system type industrial installations, such as those related to industrial process-type manufacturing and electrical power generation, often employ a large number of physically distributed control devices and associated sensors for effecting coordinated operation of the overall system. One such system is disclosed in Michael E. Cope U.S. Pat. No. 4,304,001 and assigned to the assignee of this application. In the Cope application, a plurality of remote stations are connected to various control devices and sensors and communicate with one another through a communications link. Each of the remote stations will have a data processor and, at most of the remote stations, these data processors will operate to receive signals from sensors and control process parameters of the industrial process. One of the remote stations would include a control panel to provide operator access to and control of the overall system. In the above described system, as disclosed in the Cope patent, if one of the data processors at a given remote station fails, this will not mean that the entire process control system will fail because the other remote stations will continue to function receiving information from the sensors and controlling the output parameters assigned thereto. Nevertheless, it is important to keep all of the remote stations functioning to maintain efficient automatic operation of the industrial process. To achieve this purpose, the present invention provides at each of the remote stations a back-up processor to take over the input/output operations when the primary processor at the remote station fails.

In the past, it has been proposed to employ redundant or back-up processors to take over from a primary processor when the primary processor fails. However, in such systems, a process upset often occurs when the primary processor fails because of problems in getting the back-up processor to operate on valid data concerning the current status of the process. Proposals have been made in the past to periodically have the primary processor transfer status data to the back-up processor. The problem of process upset still exists because when the primary processor begins to fail, the status data is often contaminated by the problem which caused the primary processor to fail.

SUMMARY OF THE INVENTION

In accordance with the present invention, the primary processor, while carrying out its applications program, periodically transfers a copy of its status into the memory of the back-up processor. The back-up processor, which maintains its own correct record of the status data, does a validity check on the status data and, only if the data is valid, updates the record of the status data. The transfer of data to the memory of the back-up processor is by way of a dual ported memory in which both the primary processor and the back-up processor can store and read out data. Both the primary processor and the back-up processor are provided with a processing unit called a real time executor which controls the communication of the processor with the input and output devices. The real time executor of the primary processor will be connected to other processing units of the primary processor over the primary systems bus and control the input and output devices under the control of the one or more processing units which actually carry out the applications programs for those input/output devices. The back-up processor will also have a real time executor capable of communicating with the input and output devices and connected to the other processing units of the back-up processor over the back-up processor systems bus. When one of the processing units of the primary processor fails, this fact will be detected by the real time executor, which will apply a failover signal to a processor selector module, whereupon the primary processor will cease to exercise control over the input and output devices. The processor selector module will signal the back-up processor that it is now the active processor in response to the failover signal. The back-up processor will then begin exercising control over the input/output devices in accordance with the last valid status data received from the dual ported memory.

In the system of the present invention, because the back-up processor only takes over operation making use of periodically updated status data, which is validated by an error checking system, process upset occurring when the back-up processor takes over from a failed primary processor is minimized.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an industrial control system in which the system of the present invention is employed;

FIG. 2 is a schematic block diagram of one of the remote stations shown in FIG. 1 and schematically illustrating the system of the present invention;

FIGS. 3 and 4 are flow charts of program routines employed in the system of the present invention to copy status data from a primary data processor into a dual ported memory at the remote station shown in FIG. 2; and

FIG. 5a and 5b illustrate a flow chart of a program routine employed in the system of the present invention to copy status data from the dual ported memory into the memory of a back-up data processor at the remote station shown in FIG. 2.

DESCRIPTION OF THE PREFERRED EMBODIMENT

An industrial control system in accordance with the present invention is shown in schematic form in FIG. 1 and includes a communications link CL having a plurality of remote stations R1 through R8 connected thereto. While, for purposes of illustration, only eight remote stations are shown in FIG. 1, it is to be understood that the system is designed to be used with a much larger number of remote stations. The remote stations R2 through R8 control an industrial processor system through input/output devices represented by I/O.sub.2 through I/O.sub.8, respectively. Each of these remote stations may control a large number of output devices and respond to a large number of input devices and the blocks labeled I/O in FIG. 1 each represent many input and output devices.

The remote station R1 represents the remote station at which a control panel is provided in order to provide operator access and control of the overall system. The remote station R1 is shown without input/output devices, but the remote station R1 may also control input/output devices, if desired. The control panel may be like that disclosed in the copending application Ser. No. 159,599, entitled "Industrial Process Control System", filed June 16, 1980 invented by Billy R. Slater and Dennis Simpson. Alternatively, the control panel may be like that disclosed in copending application Ser. No. 253,964 entitled "Combined Mode Supervisory Program-Panel Controller Method for an Industrial Process Control System", filed April 13, 1981, invented by Billy R. Slater. The remote stations communicate with each other over the communications link CL in the manner disclosed in Michael E. Cope U.S. Pat. No. 4,304,001.

In accordance with the present invention, each of the remote stations R2 through R8 is provided with a primary data processor, which receives information from the input devices and controls the output devices, and a back-up data processor, which is operable to take over the control of the input/output devices from the primary processor in the event the primary processor fails. A back-up processor may also be provided at remote station R1 to take over interaction with the control panel should the primary processor at the remote station R1 fail.

The block diagram of FIG. 2 illustrates an example of a remote station having a primary data processor and a back-up data processor. As shown in FIG. 2, the primary processor at the remote station comprises a modem 10, a communications protocol controller 12, a real time executor 14, one or more central processing units 16, a random access memory 18, and a communications bus 22 which provides addressing, control and information transfer between the devices of the primary data processor. The communications protocol controller 12 comprises a data processing unit and, via the modem 10, interfaces the primary data processor with the communications link CL. The real time executor 14 applies signals to the output devices to control their states in accordance with the determinations made by the primary data processor and receives signals from the input/output devices indicating the states of these input and output devices and, in other words, manages the flow of signals to and from the input/output devices for the primary data processor. The central processing units 16 receive data from the real time executor 14 regarding the status of the input/output devices and also information, directions, or commands from other remote stations via the communications controller 12. Using sequential logic, the central processing units 16, process the data and commands making use of the random access memory 18 and data stored therein and send commands to the real time executor 14 to direct changes in the output devices controlled by the real time executor 14. The sequential logic programs by which the central processing units 16 respond to and control the output devices connected to the real time executor are referred to as applications programs. The details of how protocol controller 12 communicates with the other remote stations via the modem 10 and the communications link CL and the operation of the primary processor in controlling and receiving information from the input and output devices is essentially the same as that described in the above-mentioned Cope U.S. Pat. No. 4,304,001.

The input/output devices normally will include some devices which merely have two states, such as an on and off state or an open or closed state and some remote stations may only have these kind of input and output devices. Some output devices, such as a valve, for example, may have a range of positions and some input signals from input devices, such as a flow meter, may vary over a range of values. The applications programs which control and receive status signals from the two-state devices, for convenience, are referred to as CQ3 applications programs. The applications programs to control the devices which can be set through a range of positions or values and which operate on input signals which can vary through a range of values, for convenience, are referred to as the CQ4 applications programs. When both CQ3 and CQ4 applications programs are implemented at a remote station, then the CQ3 applications program or programs will be carried out on one or more central processing units 16 and the CQ4 applications program or programs will be carried out on one or more separate central processing units 16. In addition to these programs, the primary data processor will also carry out housekeeping functions by means of a program called a system program and the system program will be implemented on one of the central processing units 16, which is called the system processing unit and which is specifically designated 16a. The system processing unit will also usually be programmed to execute one of the CQ3 programs or it may be programmed to execute a CQ4 program.

The details of a CQ4 program and how it operates are disclosed in the