Video and audio multiplex transmission system

Claims

We claim:

1. A video and audio multiplex transmission system having a transmission unit connectable to receive an audio input and a video input and to transmit a multiplexed signal, comprising:

an audio A/D converter connected to convert the audio input to a digital sound signal,

an audio coding unit for coding the digital sound signal, for outputting coded sound data having a format that enables selection of a transmission amount and for outputting audio content data,

a video A/D converter connected to convert the video input to a digital picture signal,

a video coding unit for coding the digital picture signal and for outputting coded picture data,

a coding control unit for determining and outputting control data including an allocation signal indicative of a transmission ratio of the coded sound signal and the coded video data in accordance with the transmission amount of at least one of the coded sound data and the coded picture data, and

a multiplexing unit for multiplexing the coded sound data and the coded picture data based on the allocation signal so as to obtain a certain transmission frame length, and

wherein said audio coding unit comprises an SB-ADPCM coding unit which divides the digital sound signal into a low frequency bit portion and a high frequency bit portion for coding and selects an amount of allocation of the high frequency bit portion based on the allocation signal.

2. A video and audio multiplex transmission system according to claim 1, wherein said audio coding unit is an adaptive type which outputs a plurality of coded sounds with different coding bit rates and wherein one of the plurality of coded sounds is selected based on an allocation bit rate output as the allocation signal.

3. A video and audio multiplex transmission system according to claim 2, wherein said audio coding unit outputs an optimal audio bit rate signal.

4. A video and audio multiplex transmission system according to claim 3, wherein said coding control unit outputs the optimal audio bit rate signal as the allocation signal.

5. A video and audio multiplex transmission system according to claim 1, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of the digital picture signal, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

6. A video and audio multiplex transmission system according to claim 2, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of the digital picture signal, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

7. A video and audio multiplex transmission system according to claim 3, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of the digital picture signal, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

8. A video and audio multiplex transmission system according to claim 1, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of said digital picture, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

9. A video and audio multiplex transmission system according to claim 1, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded video data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

10. A video and audio multiplex transmission system according to claim 2, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded video data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

11. A video and audio multiplex transmission system according to claim 3, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding invariable lengths of the coded picture data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

12. A video and audio multiplex transmission system according to claim 1, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded picture data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

13. A video and audio multiplex transmission system according to claim 7, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded picture data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video picture data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

14. A video and audio multiplex transmission system according to claim 8, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded picture data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video picture data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

15. A video and audio multiplex transmission system according to claim 6, wherein said coding control unit generates the allocation signal based on the interframe rate of change signal.

16. A video and audio multiplex transmission system according to claim 10, wherein said coding control unit generates the allocation signal based on the storage amount signal.

17. A video and audio multiplex transmission system according to claim 7, wherein said coding control unit generates the allocation signal in accordance with the interframe rate of change signal and the optimal audio bit rate signal.

18. A video and audio multiplex transmission system according to claim 11, wherein said coding control unit generates the allocation signal in accordance with the storage amount signal and the optimal audio bit rate signal.

19. A video and audio multiplex transmission system according to claim 13, wherein said coding control unit generates the allocation signal in accordance with the interframe rate of change signal, the storage amount signal, and the optimal audio bit rate signal.

20. A video and audio multiplex transmission system according to claim 14, wherein said coding control unit generates the allocation signal in accordance with the interframe rate of change signal and the storage amount signal.

21. A video and audio multiplex transmission system according to claim 1, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data nd the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

22. A video and audio multiplex transmission system according to claim 8, wherein said coding control unit generates the allocation signal based on the interframe rate of change signal.

23. A video and audio multiplex transmission system according to claim 12, wherein said coding control unit generates the allocation signal based on the storage amount signal.

24. A video and audio multiplex transmission system according to claim 13, wherein said coding control unit generates the allocation signal in accordance with the storage amount signal and the optimal audio bit rate signal.

25. A video and audio multiplex transmission system according to claim 2, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

26. A video and audio multiplex transmission system according to claim 3, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

27. A video and audio multiplex transmission system according to claim 1, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

28. A video and audio multiplex transmission system according to claim 6, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

29. A video and audio multiplex transmission system according to claim 13, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

30. A video and audio multiplex transmission system, wherein said system comprises:

a transmission unit connectable to receive an audio input and a video input, comprising:

an audio A/D converter connected to convert the audio input to a digital sound signal,

an audio coding unit for coding the digital sound signal, for outputting coded sound data having a format that enables selection of a transmission amount and for outputting audio content data,

a video A/D converter connected to convert the input to a digital picture signal,

a video coding unit for coding the digital picture signal and for outputting coded picture data,

a coding control unit for determining and outputting control data including an allocation signal indicative of a transmission ratio of the coded sound signal and the coded video data in accordance with the transmission amount of at least one of the coded sound data and the coded picture data, and

a multiplexing unit for multiplexing the coded sound data and the coded picture data based on the allocation signal to output a multiplexed signal so as to obtain a certain transmission frame length, and

a reception unit comprising:

a demultiplexing unit for receiving and outputting coded sound data, coded picture data and control data including an allocation signal,

an audio decoding unit for decoding the coded sound data to obtain a digital sound signal,

a video decoding unit for decoding the coded picture data to obtain a digital picture signal,

a decoding control unit for performing control of said audio decoding unit and said video decoding unit based on the allocation signal,

an audio D/A converter connected to convert the digital sound signal to an audio output, and

a video D/A converter for converting said picture signal to a video output, and

wherein said audio coding unit comprises an SB-ADPCM coding unit which divides the digital sound signal into a low frequency bit portion and a high frequency bit portion for coding and selects an amount of allocation of the high frequency bit portion based on the allocation signal, and

wherein said audio decoding unit performs SB-ADPCM decoding of said coded sound based on the allocation signal.

31. A video and audio multiplex transmission system according to claim 30,

wherein said audio coding unit is an adaptive type which outputs a plurality of coded sounds with different coding bit rates and wherein one of the plurality of coded sound is selected based on an allocation bit rate output as the allocation signal, and

wherein said decoding control unit performs control based on a demultiplexed allocation signal, i.e., said allocation bit rate.

32. A video and audio multiplex transmission system according to claim 31, wherein said audio coding unit outputs an optimal bit rate signal.

33. A video and audio multiplex transmission system according to claim 32, wherein said coding control unit includes the optimal bit rate signal in the allocation signal.

34. A video and audio multiplex transmission system according to claim 30, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of the digital picture signal, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

35. A video and audio multiplex transmission system according to claim 31, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of the digital picture signal, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

36. A video and audio multiplex transmission system according to claim 32, wherein said transmission unit further comprises an interframe rate of change judgment unit which finds a rate of change of interframe data of the digital picture signal, compares it with a threshold value, and outputs an interframe rate of change signal based on the comparison.

37. A video and audio multiplex transmission system according to claim 30, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded video data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

38. A video and audio multiplex transmission system according to claim 31, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths the coded video data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

39. A video and audio multiplex transmission system according to claim 32, wherein said video coding unit comprises:

a video coding unit for coding the digital picture signal to produce the coded picture data,

a variable length coding unit for coding in variable lengths of the coded video data to produce variable length coded picture data,

a buffer for temporarily storing the variable length coded video data, and

a buffer judgment unit for outputting a storage amount signal in accordance with an amount of the variable length coded picture data stored in said buffer.

40. A video and audio multiplex transmission system according to claim 30, wherein said reception unit further comprises a variable delay control unit for delaying the coded sound data in accordance with delay time data.

41. A video and audio multiplex transmission system according to claim 30, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

42. A video and audio multiplex transmission system according to claim 31, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

43. A video and audio multiplex transmission system according to claim 30, wherein said transmission unit further comprises a delay amount calculation unit which generates delay time data based on the coded video data and the digital sound signal and wherein said multiplexing unit multiplexes based on the delay time data.

44. A video and audio multiplex transmission system having a transmission unit to receive an audio input and a video input and to transmit a multiplexed signal, comprising:

an audio A/D converter connected to convert the audio input to a digital sound signal,

an audio coding unit for coding the digital sound signal to output coded sound data,

a video A/D converter connected to convert the video input to a digital picture signal,

a video coding unit for coding the digital picture signal to output coded video data,

a delay amount of calculation unit for calculating the variable time necessary to code the digital picture signal and output the coded video data in said video coding unit and for subtracting a fixed time necessary to code the digital sound signal and to decode the coded sound data from said variable time and output delay time data to be used for adaptively synchronizing a video reproduced signal and an audio reproduced signal in a reception unit connected to said transmission unit, and

a multiplexing unit for multiplexing the coded video data, the coded sound data and the delay time data.

45. A video and audio multiplex transmission system wherein said system comprises:

a transmission unit connectable to receive an audio input and a video input, comprising:

an audio A/D converter connected to convert the audio input to a digital sound signal,

an audio coding unit for coding the digital sound signal to output coded sound data,

a video A/D converter connected to convert the video input to a digital picture signal,

a video coding unit for coding the digital sound signal to output coded video data,

a delay amount calculation unit for calculating the variable time necessary to code the digital picture signal and output the coded video data in said video coding unit and for subtracting a fixed time necessary to code the digital sound signal and to decode the coded sound data from said variable time and output delay time data to be used for adaptively synchronizing a video reproduced signal and an audio reproduced signal in a reception unit connected to said transmission unit, and

a multiplexing unit for multiplexing the coded video data, the coded sound data and the delay time data, and

a reception unit comprising:

a demultiplexing unit connected to receive the multiplexed signal from the transmission line and to output audio data, coded picture data, and delay time data by subtraction of a fixed time necessary to code the digital sound signal and decode the coded sound data from a variable time necessary to code the digital picture signal and output the coded video data,

a variable delay control unit for delaying the audio data in accordance with the delay time data and for outputting delayed audio data,

an audio decoding unit for decoding the delayed audio data to output digital sound data,

an audio D/A converter connected to convert the digital sound data to an audio output,

a video coding unit for decoding the coded picture data, and

a video D/A converter for converting said decoded digital picture to a video output.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to a video and audio multiplex transmission system, more particularly relates to a system for coding audio data and video data and multiplexing and transmitting the same.

In recent years, there has been a boom in communications transmitting both audio data and video data, such as with video conferences and television-equipped telephones. In such a multiplex communications system, it is becoming important to performing the coding and decoding of the two at a high efficiency and to maintain a balance between the audio data and video data.

2. Description of the Related Art

In a video and audio coding system known in the past, which transmits coded audio data multiplexed with video coded data and other control data, in the case of a transmission speed of a particularly low bit rate such as in a 2.times.B system which uses two 64 kb/s lines, the 2B system which uses one 128 kb/s line, or the B system which uses one 64 kb/s line, the ratio of transmission between the audio and video signals is in general fixed to about 1:1 (for example, the case where the audio coding speed is 56 kb/s and the video coding speed is 64 kb/s), about 1:3 (for example, the case where the audio coding speed is 32 kb/s and the video coding speed is 96 kb/s), or about 1:7 (for example, the case where the audio coding speed is 16 kb/s and the video coding speed is 112 kb/s).

However, the ratio of the data density between the audio signal and video signal is inherently different by several hundred-fold. Despite this, the data is transmitted by the above ratios. Therefore, in the case of a transmission ratio of 1:1, the amount of video data transmitted per unit time becomes small, inviting deterioration of the quality of the reproduction of moving images. On the other hand, in the case of a transmission ratio of 1:7, the audio quality becomes poorer. In particular, in the latter case, when coding a picture with little movement, unnecessary bits (fill bits) are added for matching of the transmission bit rate so as to meet with the transmission capacity.

As mentioned above, in the conventional system, there has been the problem of a sacrifice of one of the video quality and audio quality.

In the video coding unit used in such a video and audio multiplexing system, the redundancy of the video data is reduced by interframe predictive coding, intraframe predictive coding, motion compensation predictive coding, variable word length coding, and other coding techniques, but the amount of data for video signals is far greater than for audio signals, so a delay occurs with respect to the coding and decoding processing times of audio signals.

Therefore, it was not possible to secure synchronization between the picture and sound at the receiving side (lip sync), and an unnatural reproduction with mismatched picture and sound resulted.

Therefore, the technique was used of finding in advance the average of the delay times of processing of video signals with respect to the sound and, based on this as a reference, causing a fixed delay time determined during the audio processing of the system. However, there was a problem in that the actual delay time changed depending on the content of the video data and therefore the unnaturalness could not be sufficiently improved.

Normally, also, the amount of the video data is far greater than that of the audio data, so not all of the video data is actually transmitted. That is, the portion which is not transmitted is ignored and the video data is sent with lapses. Due to this, the picture reproduced at the receiver side appears rough in movement. This is because picture quality is less important than real time transmission.

On the other hand, with respect to audio signals, there are silent periods in sound. Despite this, 16 kps/s worth or 56 kps/s worth of transmission capacity is secured. Therefore, it is desirable to use the silent periods in audio signals for the video signals.

SUMMARY OF THE INVENTION

The present invention has as its object to control the system as a whole, including video data processing and audio data processing, based on at least one of the transmitted video data and audio data, so as to maintain the optimal video quality and audio quality of the system as a whole.

First, this object is realized as follows according to one aspect of the present invention, i.e.,

a video and audio multiplex transmission system which is comprised of a transmission unit provided with an A/D converter for converting audio input to digital sound, an audio coding unit for coding said digital sound and outputting it as coded sound of a format enabling selection of the amount of transmission and outputting audio content data, an A/D converter for converting video input to a digital picture, a video coding unit for coding said digital picture and outputting the result as coded video data, a coding control unit for determining a transmission ratio of said coded sound and said coded picture in accordance with the amount of data of at least one of the said coded sound and said coded picture and outputting the result as an allocation signal, and a multiplexing unit for multiplexing said coded sound and said coded picture based on said allocation signal and further control data including said allocation signal so as to obtain a certain transmission frame length and a reception unit provided with a demultiplexing unit for receiving a multiplexed signal sent from the transmission unit to a transmission line and sent through the transmission line and for demultiplexing it to a coded sound, coded picture, and control data including an allocation signal, an audio decoding unit for decoding said coded sound to obtain a decoded digital sound, a video decoding unit for decoding said coded picture to obtain a decoded digital picture, a decoding control unit for performing control for said audio decoding unit and said video decoding unit based on said allocation signal, a D/A converter for converting said decoded digital sound to an audio signal, and a D/A converter for converting said decoded digital picture to a video signal, the transmission being performed by changing the transmission ratio of the sound and picture in accordance with the content of the transmission.

The audio data is smaller in amount than the video data, but decimations thereof etc. are not allowed. Therefore, the transmission ratio of the picture and sound was fixed and audio data of a certain quality was sent regardless of the content of the transmission.

In the present invention, the transmission ratio can be changed. Of course, for this, it is necessary to change the transmission ratio without interruption of the audio data. Therefore, adaptive type coding is performed for outputting the audio data as a plurality of coded sounds with different coding bit rates. The single most suitable coding bit rate is selected in accordance with the content of the transmission, so the audio quality does fluctuate, but the sound is never interrupted and transmission can be performed at the quality of the picture and sound most suited to the state at that time. Reproduction is performed at the receiving side in accordance with the coding bit rate.

Further, the plurality of coding bit rates are compared and the most suitable quality, that is, bit rate, is selected from the audio content.

Another adaptive type coder is an SB-ADPCM. This performs the coding dividing the data into a low region frequency part with a high data density and a high region frequency part with a low data density. The selection of the number of bits only causes a change in the sound quality.

In the case of this SB-ADPCM and a plurality of coding bit rates, the system as a whole can be made more balanced if one selects the number of bits giving consideration not only to the audio data, but also the video data and the state of the video buffer. As such video data, use is made of the rate of change of data among frames.

The video coding unit is designed to perform various types of processing for compressing the video data. Therefore, it is comprised of a video coding unit for coding and quantifying the digital picture, a variable length coding unit for coding in variable lengths the coded video data, a buffer in which this is stored temporarily before multiplexing, and a buffer judgment unit for showing the state of storage of this buffer. The video coding unit outputs a storage amount signal showing the state of buffer storage.

When the buffer becomes full, the video signal is decimated and time is taken until transmission.

To make the video and audio output match without deviation regardless of the content of transmission, there is provided a video and audio multiplex transmission system comprising a transmission unit provided with an A/D converter for converting audio input to digital sound, an audio coding unit for coding said digital sound, an A/D converter for converting video input to digital picture, a delay amount calculation unit for generating delay time data for synchronizing video reproduced output and audio reproduced output from input and output data of said video coding unit, and a multiplexing unit for multiplexing the coded video data and audio data and the delay time data, and a reception unit provided with a demultiplexing unit for demultiplexing a multiplexed signal from a transmission line into audio data, video data, and delay time data, a variable delay control unit for delaying the audio data in accordance with the delay time data, an audio decoding unit for decoding audio data from the variable delay control unit, a D/A converter for converting the decoded digital sound to an audio signal, a video decoding unit for decoding the coded picture, and a D/A converter for converting the decoded digital picture to a video signal.

To use the delay amount and eliminate the period of sound during the same less than a certain level, that is, the silent period, and transmit the audio portion all together, there is provided a video and audio multiplex transmission system which is provided, to deal with video signals, with a video coding unit for coding video signals and a variable length coding unit for variable length coding of the coded results, is provided, to deal with audio signals, with an audio signal coding unit, and is comprised so that the output from said variable length coding unit and the output corresponding to the said audio coding unit are transmitted multiplexed, the receiver side extracting the video signal and audio signal from what is transmitted, said video and audio multiplex transmission system being provided with a time division coding unit for extracting a signal during an effective period of sound and assembling it into a packet based on output from said audio coding unit, said time division coding unit is comprised so as to notify a system control unit of an audio transmission speed, said system control unit is comprised so as to receive said audio transmission speed and modify threshold data for controlling the coding amount of said video coding unit corresponding to the amount of data in a buffer memory in said variable length coding unit, and performing transmission in accordance with a frame format suited for said audio transmission speed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a conventional example of a video and audio multiplex transmission system;

FIGS. 2(a) and 2(b) show the transmission format based on CCITT draft recommendation Y.221;

FIG. 3 shows a conventional example provided with a delay unit;

FIGS. 4(a) and 4(b) are explanatory views of the feedback control of a buffer portion in the system shown in FIG. 3;

FIG. 5 shows the transmission frame format in the case of a different amount of transmission in the system shown in FIG. 3;

FIG. 6 is a view of the basic constitution of the present invention;

FIG. 7 is a block diagram of the transmitter side of an embodiment;

FIG. 8 shows the audio frame format;

FIG. 9 shows the audio frame format of different bit numbers;

FIG. 10 shows an example of the constitution of the multiplexing and demultiplexing unit;

FIGS. 11(1) and 11(2) show examples of allocation of BAS codes and a table of codes;

FIG. 12 is an explanatory view of the demultiplexing of a coding bit rate from BAS data;

FIG. 13 is an explanatory view showing the allocation of data to a transmission frame;

FIG. 14 to FIG. 17 show examples of the transmission frames when changing the number of bits of audio data;

FIG. 18 shows another example of a transmission frame format;

FIG. 19 shows a method of writing into the format of FIG. 18;

FIG. 20 to FIG. 23 show examples of the transmission frames when changing the number of bits of the audio data;

FIG. 24 shows an example of shifting of the selection unit of the bit rate to the multiplexing unit in the system shown in FIG. 7;

FIG. 25 shows an example of the constitution of the multiplexing and demultiplexing unit of the system shown in FIG. 24;

FIG. 26 shows the changes in the transmission ratio of sound and picture;

FIG. 27 shows the concept of the transmission of data in the case of a change in the transmission ratio of sound and picture;

FIG. 28 shows the basic constitution of another embodiment;

FIG. 29 is a block diagram of the transmitter side of the system shown in FIG. 28;

FIG. 30 is a flow chart showing the steps for determining the bit rate by the interframe rate of change of the picture and the optimal audio bit rate;

FIG. 31 shows the basic constitution of another embodiment;

FIG. 32 is an explanatory view of the buffer judgment portion of the system shown in FIG. 31;

FIG. 33 shows an example of a bit array of FAS data;

FIG. 34 shows an example of the allocation of BAS codes;

FIG. 35 is a view of the frame format showing an example of the allocation bits of sound;

FIG. 36 is a view for explaining the decoding operation based on the allocation bits of the sound;

FIG. 37 is a view showing the in-channel connection protocol;

FIG. 38 is a view showing an example of additional bits for inquiring about capacity in the AC data;

FIG. 39 shows the basic constitution of another embodiment;

FIG. 40 shows the concept of the frame data used in the system shown in FIG. 39;

FIG. 41 is a block diagram showing an embodiment of the delay amount calculation unit of the system shown in FIG. 39;

FIG. 42 is a block diagram showing an embodiment of the multiplexing and demultiplexing unit of the system shown in FIG. 39;

FIG. 43 is a view showing the control data in the frame format of the system shown in FIG. 39;

FIG. 44 shows the basic constitution of another embodiment;

FIG. 45 is an explanatory view of the transmission frame format of the system shown in FIG. 44;

FIG. 46 is an explanatory view of the processing in the time division coding unit; and

FIGS. 47(a) and 47(b) are explanatory view of the feedback control.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before explaining the embodiments of the present invention, an explanation will be made of the conventional audio and video multiplex transmission system using FIG. 1 to FIG. 5 so as to facilitate understanding of the present invention.

Note that throughout the figures, the same reference numerals are given to the same objects. FIG. 1 is a block diagram showing the constitution of a conventional video and audio multiplex transmission system and shows only the transmission unit and reception unit of the two terminal offices. As shown in the figure, the audio input is converted to digital sound by an A/D converter 1, then is coded by an audio coding unit 2 and input to a multiplexing unit 6. The video input is converted to a digital picture by an A/D converter 3, is quantified by the video coding unit 4, then is coded in variable lengths. The variable length coded video data is temporarily stored in a buffer and input to the multiplexing unit 6. The video coding unit 4 is comprised to have such a function. The coded sound and coded picture input to the multiplexing unit 6 are multiplexed at a certain ratio so as to give the transmission frame format shown in FIGS. 2(a) and 2(b), then is multiplexed with FAS, BAS, AC, and other control data and sent to a transmission line 10.

The multiplexed signal from the transmission line 10 is demultiplexed at a demultiplexing unit 11 into coded sound, coded picture, and control data. The demultiplexed coded sound is converted to digital sound by an audio decoding unit 12, then is further converted into an audio signal by a D/A converter 13 and then output. On the other hand, the demultiplexed coded picture is converted to a digital picture by a video decoding unit 14, then is converted into a video signal by a D/A converter 15 and output. The audio coding unit 2 and the audio decoding unit 12 are of corresponding types to enable a coded sound to be decoded to its original form. The same applies to the video coding unit 4 and video decoding unit 14.

The case where a delay is anticipated and a fixed delay is given is explained below:

FIG. 3 shows the conventional constitution of a terminal office in a video and audio transmission system. Reference numeral 20 in the figure is a terminal office, 3 is an A/D converter for the video signal, 41 is a video coding unit, 42 is a variable length coding unit, 1 is an A/D converter for an audio signal, 2 is an audio coding unit, 7 is a delay control unit, 6 and 11 are multiplexing and demultiplexing units, 68 is a transmission line interface unit, 14 is a video decoding unit, 15 is a D/A converter, 12 is an audio decoding unit, 13 is a D/A converter, and 19 is a system control unit.

The video signal is processed by the video coding unit 3 and then processed for coding in variable lengths, then is supplied to the multiplexing and demultiplexing units 6 and 11. On the other hand, the audio signal is coded at the audio coding unit 2 to the 4 kHz band 16 kbps or 7 kHz band 56 kbps. Provision is made of a delay control unit 7 for giving a delay corresponding to the delay for coding the video signal. The signal is delayed at the said delay control unit 7 and supplied to the multiplexing and demultiplexing units 6 and 11.

Next, the video signal and audio signal are multiplexed and sent from the transmission line interface unit 68 to the opposing terminal office.

The signal received from the opposing terminal office is demultiplexed by the multiplexing and demultiplexing units 6 and 11 into a video signal and audio signal. The video signal passes through the video signal decoding unit 14 and D/A converter 15 and is fetched as the video output. The audio signal passes through the audio decoding unit 12 and D/A converter 13 and is fetched as the audio output.

Note that the amount of delay control T in the above-mentioned delay control unit 7 is given by:

T=(t.sub.v1 +t.sub.v2)-(t.sub.A1 +t.sub.a2)

where,

t.sub.v1 : delay time at the processing units 3, 41, and 42

t.sub.v2 : delay time at the processing units 14 and 15

t.sub.A1 : delay time at the processing units 1, 2, and 7

t.sub.A2 : delay time at the processing units 12 and 13

Further, the transmission line has line speeds in multiples of 64 kbps from 64 kbps to 384 kbps and is fixed in capacity. Therefore, as shown in FIG. 4, feedback control is performed and the coding processing in the video coding unit 4 is controlled.

FIGS. 4(a) and 4(b) show the mode of the feedback control. Reference numerals 41, 4, and 19 in the figure correspond to FIG. 3, and 42 shows a variable length coding unit, 43 a buffer memory, 17 a first threshold value, and 18 a second threshold value. In the buffer memory 43 is stored the variable length coded video data. This is read out from the buffer memory 43 at a fixed speed and led to the multiplexing and demultiplexing units 6 and 11.

If, in accordance with the amount of video data stored in the buffer memory 43, the stage is reached where the storage amount exceeds the second threshold value 18 as shown in the illustration of FIG. 4(b), for example, the system control unit 19 commands the video coding unit 41 to stop the video coding. Alternatively, the quantification of the video coding unit is made rougher. Further, when the stage is reached where the storage amount falls to the first threshold value 17 or less, the system control unit 19 commands a restart of the video coding. Further, the quantification is made finer.

FIG. 5 shows the transmission frame format in the conventional case.

FIG. 5(A) shows the frame format in the case of 64 kbps, wherein an 8 kbps portion is allocated to the frame data portion, a 16 kbps portion is