Broadcast data distribution system - Patent Review 4868866

What is claimed is:

1. A broadcast data distribution system for acquiring, editing, formatting and distributing real-time and non-real-time data collected from a plurality of data sources being received in a number of different formats comprising:

data collection means for receiving, and editing the data, and for reformatting said data for further processing in the broadcast data distribution system;

central database means for receiving and storing said data in a plurality of data files, said central database means including means for updating data already stored in the central database means from data received from said data collection means;

message generation means for preparing data messages from the data stored in the central database means for further processing, said message generation means further comprising:

dynamic message generation means for preparing real-time data messages from current data values stored in the central database means;

recap message generation means for preparing periodic summaries of real-time data messages from current data stored in the central database means;

controlled message generation means for preparing controlled data messages from non-real-time data stored in the central database means; and

entitlement message generation means for preparing entitlement messages from entitlement data stored in the central database means, said entitlement messages being used to control subscriber access to broadcast messages;

message queue processing management means responsive at least tothe message generation means for controlling operation of said message generation means;

message broadcast means responsive at least to said message generation means for preparing broadcast messages for broadcast transmission from data messages prepared by said message generation means; and

data receiving means for receiving said broadcast messages from the message broadcast means, said data receiving means including means for preparing local data messages from the broadcast messages, local database means for receiving and storing data in a plurality of data files, and means for updating data already stored in the local database means for data messages contained in broadcast messages received by said receiving means,

whereby said real-time and non-real-time data are broadcast without substantially continuous repetition of said broadcast messages.

2. A broadcast data distribution system as set forth in claim 1, wherein the data collection means further comprises:

reasonability checking means for checking the reasonability of at least real-time data collected by the broadcast data distribution system; and

reformatting means for reformatting data collected in each of said different formats into a smaller set of more efficiently formatted data for further processing.

3. A broadcast data distribution system as set forth in claim 1, wherein the central database means further comprises:

real-time data file means for storing current values of real-time data:

historical data file means for storing past values of real-time data; and

update flag means for indicating an updating of real-time data stored in the real-time data file means by the broadcast data distribution system.

4. A broadcast data distribution system as set forthin claim 3, the central database means further comprising:

static data file means for storing non-real-time data collected by the broadcast data distribution system; and

entitlement level file means for storing entitlement data, said entitlement data including identification of subscribers entitled to receive broadcast messages from the broadcast data distribution system and service level indicating means for controlling the level of service provided to each of said subscribers.

5. A broadcast data distribution system as set forth in claim 1, wherein the dynamic message generation means further comprises:

dynamic message generation control means responsive to said message queue processing management means for controlling the dynamic message generation means to (a) prepare real-time data messages for all real-time data updated by the central database updating means, (b) prepare real-time data messages for some but not all real-time data updated by said central database updating means, or (c) prepare no real-time data messages.

6. A broadcast data distribution system as set forth in claim 5, wherein the recap message generation means further comprises:

update data detection means for scanning the real-time data stored in the central database means in order to detect real-time data which is updated from that which was in the central database means at the immediately preceding scan of said detection means; and

recap message generation management means responsive to said message queue processing management means for controlling the generation of recap messages either in groups at predetermined intervals of time or continuously.

7. A broadcast data distribution system as set forth in claim 6, wherein the controlled message generation means further comprises:

controlled message generation controlling means for preparing a predetermined number of identical controlled messages for each controlled message generated.

8. A broadcast data distribution system as set forth in claim 7, wherein the entitlement message generation means further comprises:

entitlement message generation controlling means for preparing entitlement messages both periodically and upon the instruction of a system operator.

9. A broadcast data distribution system as set forth in claim 1, wherein the message queue processing management means further comprises:

message queue examination means for determining the depths of a plurality of message queues associated with each of said varous message generation means;

queue depth alert means for determining when any of the message queue depths exceeds a predetermined value; and

system mode management means responsive to said queue depth alert means and the current system mode for controlling the dynamic message generation means to (a) prepare real-time data messages for all real-time data updated by the central database updating means, (b) prepare real-time data messages for some but not all real-time data updated by said central database updating means, or (c) to prepare no real-time data messages.

10. A broadcast data distribution system as set forth in claim 9, wherein the system mode management means further comprises means for controlling the generation of recap meassages either in groups at predetermined intervals of time or continuously.

11. A broadcast data distribution system as set froth in claim 10, wherein the message queue processing management means further comprises:

a queue depth alert timer for recording the time at which the queue depth alert means detects that a predetermined queue depth limit has been exceeded, said system mode management means being further responsive to said queue depth alert timer for controlling the broadcast data distribution system to maintain the real-time status of each subscriber's local database under increased data traffic volumes.

12. A broadcast data distribution system as set forth in claim 1, wherein the message broadcast means further comprises:

message dequeuing means for removing data messages from the various message queues; and

transmission block generation means for preparing broadcast messages, each said broadcast message comprising at least one data message and having a predetermined format for broadcast transmission over a broadcast medium.

13. A broadcast data distribution system as set forth in claim 1, wherein the data receiving means further comprises:

entitlement checking means for determining whether said data receiving means is entitled to receive and further process a particular received broadcast message.

14. A broadcast data distribution system as set forth in claim 13, wherein the local data base updating means further comprises:

message type determination means for distinguishing entitlement update messages from dynamic and controlled data messages;

local entitlement file means for storing a subscriber's service level;

local entitlement file updating means for updating the local entitlement file whenever an entitlement update message is received;

controlled message logging means for storing the first transmission of a group of controlled messages;

controlled message comparison means for comparing subsequent identical groups of controlled messages to the group of controlled messages logged, and for correcting errors detected in the logged group of controlled messages; and

controlled broadcast message completion detecting means for detecting the end of a controlled message so that the local database means can be updated with the controlled data messages.

15. A method of broadcasing real-time and non-real-time data in the same data stream to subscribers comprising the steps:

collecting data from a plurality of data sources, the data from each source having a potentially unique format;

editing and converting the received data into a smaller set of message formats;

selectively checking the value of the data for reasonableness;

storing the data in a central database if the data is a type not already stored in said central database;

updating the data stored in the central database with new incremental value pertaining to such data;

flagging each new or updated data value as being an updated data value;

removing previous values of updated data from the central database to temporary storage locations for further reference;

preparing real-time data messages for broadcast during periods of normal data traffic volume received;

preparing real-time data messages for broadcast only for updated data values which differ from the then current data by incremental values of a predetermined amount during periods of heavier data traffic volume received;

preparing no real-time messages for broadcast during stil heavier data traffic volume received;

periodically preparing update message summaries for broadcast by scanning the central database for data values which have been updated since the last central database scan during both normal and heavier data traffic volumes received;

continuously preparing update message summaries for broadcast by continuously scanning the central database for updated data values during periods of still heavier data traffic volume received;

preparing a predetermined number of identical groups of non-real-time messages for broadcast to ensure accurate reception of these messages;

preparing entitlement messages for broadcast both periodically and upon request of an operator, said entitlement messages being used to enable and disable particular subscribers' receiving apparatus;

preparing broadcast messages from the previously prepared data messages; and

transmitting the broadcast messages to an unlimited number of subscribers over a broadcast medium, each subscriber then decoding the broadcast message and updating its local database with the individual data messages contained in the broadcast message received, if entitled to receive the data message.

Description Submit all comments and votes

DISCLOSURE OF THE INVENTION

This invention relates to a method and apparatus for acquiring, editing, verifying, formatting and distributing data from multiple data sources, In particular, the invention is directed to an information distribution system for providing both real-time data and periodic (non-real-time) data to an unlimited number of subscribers each having receiving apparatus via a one-way, non-interactive broadcast data feed.

Currently available architecture for data distribution systems may take several forms. The first of these can be described as a acknowledge/negative-acknowledge (Ack/Nack) system, where data is distributed from a central database and the subscriber then verifies the data and acknowledges or negatively acknowledges its receipt back to the central database system. While this traditional form of data distribution provides extremely high reliability of the data received by the subscriber, it is also expensive and cumbersome, since each subscriber must be provided with a method of returning its Ack/Nack message (most typically by a two-way telephonic link), the data must be transmitted individually to each subscriber who has a need for the data, and the central system computer must have resources proportional to the number of active subscribers.

Another form of a currently available data distribution system provides a complete data picture to each subscriber on a periodic or "Loop" basis. This "Loop" architecture overcomes the limitation described in the Ack/Nack system in that data is uniformly broadcast to an unlimited set of subscribers. In the "Loop" architecture, central site resources are not a function of the number of subscribers. However, a major limitation of this system is the amount of data that can be delivered since each new data element added to the database introduces an additional "Loop" delay (i.e. the time required for all data to be distributed once). The "Loop" delay is directly proportional to the size of the database. Therefore, these systems can distribute only a limited amount of data and still maintain real-time status.

A third architecture for electronic data distribution is simple broadcast of transaction/free-wheeling data, i.e., data transmitted in one direction from a source to a receiving station which does not depend on the receiving station's readiness to receive the data. The most well-known examples of this method of data distribution are news wire services and stock exchange tickers. These data distribution systems also provide for the uniform broadcast of data to an unlimited number of subscribers. They suffer, however, from a serious reliability problem in that data is sent only once. Therefore, if any data is lost by a subscriber, it cannot be recovered. Without some mechanism for recovery of missed or lost data, these systems have very limited applications in a comprehensive distributed data processing or data display environment.

The financial industry provides an illustrative example of an industry requiring vast amounts of data in order to successfully engage in trading in the many financial markets which exist worldwide. Just in the case of North Amercian exchanges, more than thirty distinct data streams each different, covering tens of thousands of issues generate more than seventy update transactions per second. Dozens of issues are added daily, and dozens more are deleted; at varying monthly intervals, hundreds of options expire and hundreds of others are created. With respect to other issues such as dividends, earnings, and capitalization, descriptive material is modified daily. Additionally, each of the exchanges limits access to their data to their authorized receivers. Due to exchange regulations and exchange fees each subscriber system is permitted to access only the set of exchanges it chooses and for which it is authorized. Prior to this invention, maintenance and control of these databases has required their location at a central site or at a very limited number of distributed sites requiring two-way communications.

It is therefore an object of this invention to provide a data distribution system which can provide a broad range of data through a one-way broadcast medium to an unlimited number of subscribers at the lowest possible delivery cost.

It is another object of this invention to provide a data distribution system that can provide highly reliable data suitable for comprehensive distributed data processing without the requirement of acknowlegement of receipt by the subscriber.

It is yet a further object of this invention to provide a broadcast data distribution system that eliminates the necessity of continually rebroadcasting the entire database.

It is still another object of this invention to provide a broadcast data distribution system which can transmit both real-time and non-real-time data either of which can be formatted or unformatted.

It is yet another object of this invention to provide a comprehensive means of controlling access to the broadcast data by the receiving systems without the requirement for two-way communications.

The above and other objects of the present invention are realized in a specific illustrative embodiment relating to the financial industry. In the system of the present invention, financial data having multiple input protocols is collected from securities exchanges worldwide and decoded, and the central database is updated. Depending on the current volume of data traffic, the incoming data may simultaneously be reformatted for broadcast and then transmitted on a broadcast data bus. Periodic summaries, called recaps, of recently received data are broadcast to ensure accuracy of all current information at the subscriber's location. Queue processing techniques are employed to provide continuous summaries when the volume of data traffic precludes broadcast of individual data messages. Particular data messages which must be accurately received by all subscribers are distributed by controlled transmissions and repeated a predetermined number of times sufficient to ensure high reliability of reception. Data is broadcast over existing common carriers, including satellite, leased line, voice-grade line and others.

Subscriber access is controlled by transmitted "entitlement" messages which enable the receiving means of authorized subscribers to receive and process the broadcast data. Entitlement messages are used to provide limited users with selected portions of data while providing full broadcast service to others, over the same broadcast data feed. Each subscriber has means for receiving, maintaining and processing a local data base tailored to its specific requirements. The local database is being constantly updated by data broadcast transmissions. User requests, in the event of broadcast failure, are serviced via backup transmission modes.

The foregoing and additional features and advantages of the instant invention will become more readily apparent from the following detailed description of a specific illustrative embodiment of particular features thereof, presented hereinbelow in conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram depicting the principal functional blocks in accordance with the principles of the present invention for an improved broadcast data distribution system;

FIG. 2 is a schematic flow chart depicting detailed data processing for the data collection block 1 and the central database block 2 of FIG. 1;

FIG. 3 is a schematic flow chart depicting detailed data processing for the dynamic message generation block 3 of FIG. 1;

FIG. 4 is a schematic flow chart representing detailed data processing for the recap message generation block 4 of FIG. 1;

FIG. 5 is a schematic flow chart depicting data processing for the controlled message (static or non-real-time data) generation block 5 of FIG. 1;

FIG. 6 is a schematic flow chart representing data processing for the entitlement message generation block 6 of FIG. 1;

FIG. 7 is a schematic flow chart depicting data processing for the queue processing and management block 7 of FIG. 1;

FIG. 8 is a schematic flow chart depicting the message broadcast block 8 of FIG. 1; and

FIG. 9 is a schematic flow chart depicting detailed data processing for the data receiving block 9 of FIG. 1.

Referring now to FIG. 1, there is shown in overall scope a data processing and system organizational block diagram for implementing an improved broadcast data distribution system incorporating the principles of the present invention. After briefly describing each of the major functional blocks of FIG. 1, the data processing function and structure of each functional block will be described in detail.

At the beginning of the data distribution chain is the data collection block 1. In this illustrative embodiment relating to the financial industry, data enters the system in a number of ways, including communication lines from financial exchanges, online operator terminals, external online real-time systems, offline data entry terminals, and offline data entry systems. Depending upon the particular data source, the data may have to be edited, verified and/or converted into a machine readable format compatible with whatever internal specifications the system designer chooses.

Once data verification methods have determined and ensured the accuracy of collected data, the data is stored in central database files containing, for example, real-time financial pricing information. The database update functional block 2 represents the accurate updating and maintenance of a large number of different database files, each designed for different types of data.

Each of the next four blocks (dynamic message generation 3, recep message generation 4, controlled message generation 5, and entitlement message generation 6) describe portions of the system which are continuously operating to format received data from the central database so that it may be further transmitted to subscribers.

The messages produced by each of the the four message generation functional blocks 3, 4, 5, 6 are next queued for broadcast in queue management block 7. Data processing in this block prioritizes the messages to be broadcast. In addition, queue monitoring functions are employed to determine the actual rate of traffic flow into the data collection block 1 to the overall system. This traffic flow rate is used to control the several modes of operation of the dynamic message generation block 3 and the recap message generation block 4.

From the queue management block 7, data messages are passed to the data broadcast block 8 where they are transmitted via any of a number of broadcast media to an unlimited number of subscribers.

Finally, the data messages are received by structure and associated data processing of the data receiving functional block 9. This portion of the system receives all of the data which a particular subscriber is entitled to receive, decodes the messages if necessary, alerts the subscriber when data to which he is entitled has not been properly received, and stores the data in the subscriber's local database. Not included within the definition of the broadcast data distribution system is the subscriber-specific hardware/software which the subscriber uses to access and manipulate the local database. Such apparatus is within the state of the art, and may be provided in the form, for example only, of a programmed "personal" computer.

These foregoing functional blocks will now be described in detail.

Referring now to FIG. 2, the various elements of the data collection functional block 1 are shown, as required in this specific illustrative embodiment relating to the financial industry. Transaction/free-wheeling data 30 includes real-time pricing and administrative information concerning current trading and trading-related activities. This free-wheeling data is received via communications lines from the floors of the various stock option, bond and commodity exchanges. Data is transmitted from the exchanges in one direction only, from the exchange to the free-wheeling data receiver 30, and the data transmission occurs regardless of the preparedness of data receiver 30 to receive such data.

Free-wheeling data may be in asynchronous or synchronous format, and is transmitted using a variety of baud rates, record formats and parity checking specifications. This specific embodiment of the present invention handles many hundreds of different transaction types and formats. Illustrative of the possible variations between input protocols are Chart 1, Ticker Line Specifications, and Chart 1A, Commodity Line Specifications, shown below. Of course, the present broadcast data distribution system receives data from many more sources than the eight input sources shown on the two charts. Each of the ticker lines provides a set of messages having unique formatting requirements, thus making up a total of more than 600 different formats in the present system. Of course, the number of unique input formats will depend upon the specific application of the present invention.

CHART 1 - TICKER LINE SPECIFICATIONS

______________________________________ TICKER [1] CTS NASD-Q NYB-T TSE ______________________________________ BROADCAST yes yes yes yes BAUD RATE 9600 2400 134.5 4800 SYNC/ASYNC S S A A BITS/CHAR 8 8 9 9 START BITS 0 0 1 1 DATA BITS 7 7 6 7 PARITY BITS odd odd none even STOP BITS 0 0 2 1 LRC yes-1 yes-2 no yes-3 BLOCKING yes no no yes SEQ # yes no no yes RETRANSMIT yes yes no yes FIXED/VAR F V V F CHAR CODE ASCII ASCII ticker ASCII ______________________________________ [1] CTS is the Consolidated Tape System maintained by Securities Industry Automation Corp., providing data on all U.S. stocks. NASDQ is the quote line from the National Ass'n of Securities Dealers. NYBT is the trade line for New York Stock Exchange bonds. TSE is the Toronto Stock Exchange.

CHART 1A - COMMODITY LINE SPECIFICATIONS

______________________________________ COMMODITY [2] CBT NYFE KCBT MACE ______________________________________ BROADCAST yes yes yes yes BAUD RATE 2400 2400 1200 1200 SYNC/ASYNC A S A A BITS/CHAR 10 8 9 11 START BITS 1 9 1 1 DATA BITS 7 7 6 7 PARITY BITS even odd none always 1 STOP BITS 1 0 2 2 LRC none yes none none BLOCKING no yes none none SEQ # yes yes none yes RETRANSMIT yes yes none yes FIXED/VAR F F V F CHAR CODE ASCII ASCII ticker ASCII ______________________________________ [2] CBT is the Chicago Board of Trade. NYFE is the New York Futures Exchange. KCBT is the Kansas City Board of Trade. MACE is the MidAmerica Commodity Exchange.

For the sake of clarity, the terms in the left column of Charts 1 and 1A will be briefly defined, although the meaning of each of the terms is known to those of ordinary skill in the art:

BROADCAST - In this context, broadcast indicates that the data feed from the exchange is a one-way communication of data.

BAUD RATE - The unit of signalling speed. It is the number of signal elements sent and received per second. Since a signal element can represent more than one bit, baud rate is not the same as bits per second.

SYNC/ASYNC