 |
Description  |
|
|
TECHNICAL FIELD
This invention relates, in general, to data communication and, in particular, to controlling the flow of information between senders of information and receivers of that information, when the information is being forwarded across links being used
as channels.
BACKGROUND ART
Communication between computer systems usually involves a sending system (sender) sending a command to a receiving system (receiver) over a link used to couple the sender and the receiver. The receiving system then, typically, sends a response
back over the link to the sending system.
One example of a link used by International Business Machines Corporation to communicate between a sender and a receiver is an intersystem channel link. In particular, an intersystem channel link couples an intersystem channel adapter on one
system (e.g., a central processor) that may be used for sending or receiving messages with an intersystem channel adapter on another system (e.g., a coupling facility that contains data shared by any central processor coupled thereto) that may also be
used for sending or receiving messages.
An intersystem channel link supports connections up to approximately 20 kilometers, and the propagation time on the link (at 20 kilometers), when using fiber optic conductors, is approximately 100 microseconds in each direction. To minimize
message latency, a minimum number of round trip acknowledgements is used, and thus, large data buffers at each end of the link are needed for flow control. In one example, a message exchange requires only one single round trip over the link.
As more and more data is desired to be included with the message, larger data buffers are needed to maintain the single round trip flow control. However, eventually, the expense of large buffers becomes prohibitive and the flow control is
modified to add intermediate acknowledgement exchanges to throttle the data transfer preventing the buffers from overrunning. These extra exchanges, though, significantly increase message latency.
Based on the foregoing, a need exists for a more direct communication link between senders and receivers of information. Further, a need exists for a communication link that does not need to couple channel adapters, but still may act as a
channel. That is, a need exists for a communication protocol in which the advantages of a channel (e.g., having a direct memory adapter engine and offering protection of memory) may be realized. Additionally, a need exists for a communication protocol
that does not require large data buffers and does not significantly add to message latency.
SUMMARY OF THE INVENTION
The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a method of controlling the flow of information between senders and receivers of data. The method includes, for instance, initiating,
by a first system, a sending of a request, via a link, to a second system, in which the request has one or more data items associated therewith that are not sent with the request; indicating that a receiver of data is prepared to receive the one or more
data items from a sender of data, wherein the receiver is either the first system or the second system depending on the request, and the sender is the other of the first system or the second system; forwarding, in response to the indicating, the one or
more data items; and wherein the link is being used as a channel in that the link is not stopped while the receiver is preparing for the one or more data items.
In another aspect of the present invention, a method of controlling the flow of information across links between senders and receivers is provided. The method includes, for instance, including in a packet a sequence number usable in maintaining
delivery order of the packet. The packet, however, does not include a memory address and does not require an explicit individual response. The packet is sent from a sender to a receiver across a link, and the sequence number is used to determine if the
packet is in proper order for processing by the receiver.
In another aspect of the present invention, a method of controlling the flow of information across links between senders and receivers is provided. The method includes, for instance, including in a packet a continue indicator usable in
determining whether another packet is to follow; sending the packet from a sender to a receiver across a link; and using the continue indicator to determine if the another packet is to follow.
Advantageously, the flow control capabilities of the present invention provide a more direct communication path between senders and receivers. Further, the flow control protocol does not require large data buffers and offers low latency
messaging. Thus, the present invention is less expensive than other communication protocols.
Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention.
BRIEF
DESCRIPTION OF THE DRAWINGS
The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention will be
apparent from the following detailed description taken in conjunction with the accompanying drawings in which:
FIG. 1 depicts one example of a computer environment incorporating and using the flow control capabilities of the present invention;
FIG. 2 depicts one embodiment of a memory bus adapter of FIG. 1;
FIG. 3a depicts one example of a packet managed by a self-timed interface, in accordance with the principles of the present invention;
FIG. 3b depicts another example of a packet managed by a self-timed interface, in accordance with the principles of the present invention;
FIG. 4 depicts one example of a control word used in accordance with the principles of the present invention;
FIG. 5 depicts one example of a self-timed interface header for channel adapter memory requests, in accordance with the principles of the present invention;
FIG. 6 depicts one example of a self-timed interface header for channel adapter memory responses, in accordance with the principles of the present invention;
FIG. 7 depicts one embodiment of the flow of packets for an I/O input operation, in accordance with the principles of the present invention;
FIG. 8 depicts one embodiment of the flow of packets for an I/O output operation, in accordance with the principles of the present invention;
FIG. 9a depicts one embodiment of a frame sent across an intersystem channel;
FIG. 9b depicts one example of a header of the frame of FIG. 9a;
FIGS. 10a-10c illustrate examples of the flow of messages between an OS/390 processor and a coupling facility, in accordance with the principles of the present invention;
FIGS. 11a-11b illustrate additional examples of the flow of messages between an OS/390 processor and a coupling facility, in accordance with the principles of the present invention;
FIG. 12 depicts an example of the components of a memory bus adapter of the present invention;
FIG. 13 depicts one example of a header used for an integrated cluster bus (ICB) flow control, in accordance with the principles of the present invention;
FIG. 14 depicts an example of the components of the ICB control logic depicted in FIG. 12, in accordance with the principles of the present invention;
FIG. 15 depicts one embodiment of the logic associated with a write data example in which a message is sent from an OS/390 processor to a coupling facility using an integrated cluster bus, in accordance with the principles of the present
invention;
FIG. 16 depicts one embodiment of the logic associated with a read data example in which a message is sent from an OS/390 processor to a coupling facility using an integrated cluster bus, in accordance with the principles of the present
invention;
FIG. 17 depicts one embodiment of the logic associated with error checking performed when a command is received by a processor, in accordance with the principles of the present invention;
FIG. 18 depicts one embodiment of the logic associated with error checking performed when a packet is received over an ICB self-timed interface link, in accordance with the principles of the present invention;
FIG. 19 depicts one embodiment of the logic associated with sequence error checking performed when a packet is received over an ICB self-timed interface link, in accordance with the principles of the present invention;
FIG. 20 depicts one embodiment of the logic associated with checking the order of commands received from processors, in accordance with the principles of the present invention; and
FIG. 21 depicts one embodiment of the logic associated with checking the order of packets received over a link, in accordance with the principles of the present invention.
BEST MODE FOR CARRYING OUT THE INVENTION
In accordance with the principles of the present invention, the flow of information between senders and receivers is being controlled across links that are being used as channels.
One embodiment of a computing environment incorporating and using the flow control capabilities of the present invention is depicted in FIG. 1 and described in detail herein. Computing environment 100 is based, for instance, on the Enterprise
Systems Architecture (ESA)/390 offered by International Business Machines Corporation. ESA/390 is described in an IBM Publication entitled Enterprise Systems Architecture/390 Principles of Operation, IBM Publication No. SA22-7201-04, June 1997, which is
hereby incorporated herein by reference in its entirety.
Computing environment 100 includes, for instance, one or more central processing complexes (CPCs) 102 coupled to at least one coupling facility 104, each of which is described below.
Each central processing complex 102 includes one or more central processors 106 (central processing units) and main memory 108, which is accessed by the central processors. Each central processor may execute an operating system, such as the
OS/390 or Multiple Virtual Storage (MVS)/ESA operating system offered by International Business Machines Corporation.
Coupling facility 104 is a sharable facility, which includes one or more central processors 110 and storage 112, which is indirectly accessible by processors 106 and directly accessible by processors 110. Each central processor 110 may execute
coupling facility control code to perform operations requested by processors 106. In one embodiment, coupling facility 104 is a structured-external storage processor (SES).
Aspects of the operation of a coupling facility are described in detail in such references as Elko et al., U.S. Pat. No. 5,317,739, entitled "Method and Apparatus For Coupling Data Processing Systems", issued May 31, 1994; Elko et al., U.S.
Pat. No. 5,561,809, entitled "In A Multiprocessing System Having A Coupling Facility, Communicating Messages Between The Processors And The Coupling Facility In Either A Synchronous Operation Or An Asynchronous Operation", issued on Oct. 1, 1996; Elko
et al., U.S. Pat. No. 5,706,432, entitled "Mechanism For Receiving Messages At A Coupling Facility", issued Jan. 6, 1998; and the patents and applications referred to therein, all of which are hereby incorporated herein by reference in their entirety.
Multiple central processing complexes and coupling facilities may be interconnected. Further, each central processing complex and/or coupling facility processors may be logically partitioned, such that it can run multiple instances of the OS/390
and/or coupling facility control code.
Central processing complex 102 and coupling facility 104 are each coupled to a memory bus adapter 114. Memory bus adapter 114 provides assistance in fetching data from and storing data in memory. In particular, the memory bus adapter is used to
couple the memory to various peripheral components, such as input/output (I/O) devices 116. As examples, the input/output devices include disk drives, tape drives, local area network (LAN) attachments, and wide area network (WAN) attachments.
In one embodiment, in order to couple the memory bus adapters to the I/O devices, each memory bus adapter 114 is coupled to one or more channel adapters 118, 120 via memory interfaces, referred to as self-timed interfaces (STIs) 124. The channel
adapters then provide the I/O interface attachment points to the I/O devices. As examples, one of the channel adapters is a parallel channel adapter 118, which provides a parallel channel interface 126 to I/O devices 116; and another channel adapter is
an ESCON channel adapter 120, which provides an ESCON channel interface 128 to I/O devices 116. In addition to the above, an intersystem channel (ISC) adapter 122, which provides an intersystem channel interface 130 to, for example, another central
processing complex, is also coupled to the memory bus adapter via STI 124.
Channel adapters 118, 120, 122 are, for instance, direct memory adapter (DMA) engines, which move data to and from memory 108, 112 without any direct involvement of any of processors 106, 110. They also provide protection from the peripherals or
other components storing in any arbitrary location in memory. The peripherals or other components can only store into the memory locations known to the channel adapters, and those locations are determined by the programs running in processors 106, 110.
The channel adapters are considered a trusted, integral part of the central processing complex.
As described above, the channel adapters are coupled to memory bus adapter 114 via self-timed interface 124. In one example, self-timed interface 124 includes a link that operates at relatively short distances (about 10 meters). The link is
comprised of eight data signals, one combination parity and tag signal, and a clock signal in each direction. The maximum link propagation delay is about 40 nanoseconds, and thus, the time it takes to transmit a single 128 byte memory line is about 500
nanoseconds (including the packet overhead of headers and checking fields).
Coupled to one end of the STI link is memory bus adapter 114, which is described in further detail with reference to FIG. 2. Memory bus adapter 114 includes one or more STI macros 200 and switch logic 202. In one embodiment, each macro is
coupled to one end of a self-timed interface link 124.
Each STI macro is responsible for retiming the data and parity/tag signals of the link to the clock signal. The retiming compensates for propagation time variations of the various signals so that the data and parity/tag signals can be coherently
sampled by the clock signal. The propagation time variations are due to differences in the signal drivers, receivers and wire. (STI is further described in U.S. Pat. No. 5,757,297, entitled "Method and Apparatus for Recovering a Serial Data Stream
Using a Local Clock", Ferraiolo et al., issued on May 26, 1998; and U.S. patent application Ser. No. 08/660,648, now U.S. Pat. No. 5,859,881, entitled "Adaptive Filtering Method and Apparatus to Compensate For a Frequency Difference Between Two Clock
Sources", Ferraiolo et al., filed on Jun. 7, 1996, each of which is assigned to International Business Machines Corp. and each of which is hereby incorporated herein by reference in its entirety).
STI macros 200 are also responsible for transmitting, receiving, buffering, and regulating packets and control sequences over STI. Examples of packets managed by STI are depicted in FIGS. 3a-3b and described below.
Packet 300a (FIG. 3a) includes, for instance, a header 302a and a longitudinal redundancy check 304a. Header 302a has a variable length (e.g., 8 or 16 bytes) and includes various fields relevant for the particular packet being sent. For
instance, header 302a may include a buffer number indicating which buffer in the STI macro is to receive the packet, and a header length count indicating the length of the header.
Packet 300b also includes, for example, a header 302b and a longitudinal redundancy check 304b, as well as a payload 306 and a further longitudinal redundancy check 308. In this case, header 302b includes a payload length count indicating the
length of the payload (e.g., data), in addition to the buffer number and header length count. Payload 306 is, for example, variable in length in increments of 4 bytes, and has a maximum length of 128 bytes (the memory line size).
After a packet is transmitted or when there are no packets or control information to be transmitted, an idle sequence is transmitted.
To send and receive packets over STI, each STI macro 200 includes, for example, four transmit header buffers, four receive header buffers, two transmit payload buffers and two receive payload buffers. Each payload buffer is tied to one of the
header buffers. In one embodiment, each of the header buffers has space for a 16 byte header, and each of the payload buffers has space for 128 bytes (a memory line) of payload information. The payload buffers are sufficient to realize the full data
rate of the link.
The low level flow control provided by the STI macros regulate the flow of packets as they travel through the payload and header buffers. Some of the bits in the packet headers are under exclusive control of the STI macros and others may be used
by the STI user layer logic (such as channel adapters 118, 120, 122 and switch logic 202) for any purpose required.
Switch logic 202 (FIG. 2) is coupled to the STI macros. It is responsible for forwarding information received from the macros to memory and for passing information from memory to the STI macros.
When a user layer, such as the switch logic, wants to send a packet, it queries the STI macro to determine the type of buffers available. If the desired buffer (header and payload or only a header) is available, the user layer places its
information on the buffer(s), and tells the STI macro to proceed. The STI macro sends the packet to the other end of STI where it is placed in the receive buffer(s). The STI macro at the other end informs its user layer of the arrival of the packet,
and the user layer unloads it. After the user layer unloads the packet, the STI macro sends a control word (e.g., an acknowledgement) back to the side of the STI originating the packet. One example of a control word is depicted in FIG. 4.
As shown, a control word 400 can be inserted anywhere either in an idle sequence 402 or within a packet 404. At the most basic level, STI macros move packets between user layers, and for each packet moved, there is a control word
(acknowledgement) sent in the opposite direction after the packet has been successfully processed.
Negative acknowledgements in the form of control words, and additional control bits in the packet headers detect missing and/or damaged packets. In some cases, the STI macros can automatically retransmit a packet. This retransmission may cause
the packets to be presented to the receiving user layer in an order different from the order that the sending user layer transmitted the packets. If the user layer is sensitive to the ordering, it is up to the user layer to determine the order.
Returning to FIG. 1, channel adapters 118, 120, 122 use STI as an interface to memory. Their basic operation is sending memory requests. Each memory request receives a response. For a memory fetch, the memory address is in the request and the
response contains the data. For a memory store, the memory address and the data to be stored are in the request, and the response simply contains an indication of the successfulness of the store operation.
One example of a STI header for channel adapter memory requests is depicted in FIG. 5. Header 500 includes, for instance, one or more STI macro controls and routing information 502, an operation code (store or fetch) 504 and a memory address
506. The op code and memory address comprise a command.
The STI macro controls and routing information include, for example, a STI macro buffer number, a header length count, and a data length count for the memory operation.
One example of a STI header for channel adapter memory responses is depicted in FIG. 6. Header 600, which is similar to header 500, includes, for instance, one or more STI macro controls and routing information 602, and a memory response code
604. One response code is used to indicate a successful completion of a memory operation, while one or more other codes are used to indicate failures, such as an invalid memory address.
One example of the flow of packets for an I/O input (fetch or read) operation is depicted in FIG. 7. For clarity purposes, low level STI acknowledgements are not shown. The flow of information is between a memory bus adapter 700 and a channel
adapter 702. The channel adapter receives data from the peripherals (not shown in this figure) and typically, buffers this information in the adapter. Memory addresses and control information within the channel adapter is used to generate the STI
headers (e.g., header 500), and the buffered data are the payloads. A packet 704, including a header and a payload, is sent from adapter 702 to memory bus adapter 700. A further packet 706 is also sent from the channel adapter to the memory bus
adapter. In this particular example, two storage requests are outstanding at the same time. A response packet 708 (e.g., header 600) is then sent from memory bus adapter 700 to channel adapter 702. This response is paired to the request having the
same STI macro buffer number as the response. Thereafter, another request packet 710 is sent from the channel adapter to the memory bus adapter. Multiple exchanges are typically required, since the peripheral data is usually more than 128 bytes. Next,
two other response packets 712, 714 are sent from the memory bus adapter to the channel adapter.
One example of the flow of packets for an I/O output (store or write) operation is depicted in FIG. 8. For clarity purposes, low level STI acknowledgements are not shown. Again, the flow of information is between a memory bus adapter 800 and a
channel adapter 802. With a write operation, the channel adapter is instructed to send data to the peripherals. Memory addresses and control information within the channel adapters are, once again, used to generate the STI headers. Request packets
804, 806 (e.g., header 500) are sent from adapter 802 to memory bus adapter 800 requesting data that is to be written to the peripherals. Then, a response packet 808 (e.g., header 600) is sent from the memory bus adapter to the channel adapter, which
provides the data. Thereafter, another request packet 810 is sent from the channel adapter to the memory bus adapter. In response to the request packets, response packets 812, 814 are sent from the memory bus adapter to the channel adapter.
Described in detail above is one embodiment of how communication between memory and peripherals within a computing system occurs. Now, the discussion turns to communication between computing systems that are coupled to one another. In
particular, communication between central processing complex 102 and coupling facility 104 is described below.
In one example, communication between central processing complex 102 and coupling facility 104 is via an intersystem channel (ISC) 132 (FIG. 1). Intersystem channel 132 couples an intersystem channel adapter 122 on system 102 with an intersystem
channel adapter 122 on system 104. ISC supports connections up to approximately 20 kilometers, and using fiber optic conductors, the propagation time on the link (at 20 kilometers) is approximately 100 microseconds in each direction. To minimize
message latency, a minimum number of round trip acknowledgements is used. Thus, large data buffers are used for flow control.
Information is sent across an intersystem channel in frames, one example of which is depicted in FIG. 9a. A frame 900 includes a header 902, a cyclical redundancy check 904, a payload 906 and a payload cyclic redundancy check 908. Some frames
do not include the payload or payload cyclic redundancy check 908. These frames are used to control the link. Frames that include the payload are for the message commands, data and responses.
Header 902 includes, for instance, a "N" field 910 (FIG. 9b) specifying a buffer set number corresponding to a buffer set that can include, for example, a MRB, a MCB and/or data; a "R" field 912 differentiating requests (outbound) from responses;
and a "D" field 914 specifying a data frame, as opposed to a command or respon | | |
