 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
This invention relates to a data communication apparatus for communication
using a plurality of kinds of data, such as image data, audio data, and
code data (hereinafter referred to simply as data).
2. Related Background Art
With the recent development of communication technology, semiconductor
technology and optical technology, digital communication lines have been
provided to enable high-speed large-capacity data transmission.
Specifically, digital transmission is free from reduction in data quality
during transmission and ensures that data can be uniformly maintained at a
suitable quality level. Also, digital transmission enables integration of
media since it does not require selection of data transmission paths in
accordance with characteristics of media. Use of digital transmission has
therefore enabled transmission between composite media, and telephone
terminals capable of simultaneously transmitting images together with
speech have been developed from conventional telephone terminals
transmitting speech alone.
Under these circumstances, international standardization for enabling
communication between different kinds of composite terminals is being
forwarded by CCITT or other organizations, and service rules for
audio-visual (AV) services such as TV telephone or TV meeting system with
digital lines, protocol rules and multi-media multiplexed frame
construction rules have been made public in the CCITT recommendations
(drafts) H.320, H.242, H.221, etc.
In H.221, exchange between frame constructions and between terminal
capabilities, frame alignment signals (FAS) for communication modes, and
code assignment for bit allocation signal (BAS) with respect to AV service
from 64 to 1,920 kbps are defined. In H.242, protocols for capability
exchange and communication code changeover between AV terminals using BAS
are defined. In H.320, system aspects for overall AV service are defined.
These recommendations or drafts of recommendation provide a method for
performing, after setting an end-to-end physical connection and
establishing synchronization by FAS during in-channel, multi-media
communication between terminals using images, voice and data based on
procedures such as sequence of operations for terminal capability exchange
using BAS during in-channel and sequence of operations for mode changeover
by communication mode designation. However, changing the self terminal
capability at each terminal according to situations and selection of the
communication mode within the range of exchanged capability are not
included in the scope of the rules. The rate of information transfer on
each of media in multi-media communication is determined in such a manner
that the audio information transfer rate is determined by designating an
audio encoding method, the data information transfer rate is determined by
designating whether or not data is used and a transfer rate in the case of
use, and the image information transfer rate is determined as the
remainder of subtraction of the audio information transfer rate and the
data information transfer rate from the overall information transfer rate
of the set communication path.
In the system of conventional multi-media terminals defined as described
above, however, communication between terminals using image data is
performed as described below. Ordinarily, when a terminal is connected to
a communication partner's terminal, the display is changed over from a
picture showing the operation of the multi-media terminal to a received
image. Thereafter, displaying received images is continued unless the
communication is completed or the operator changes the display by
intention. However, the display cannot be changed also when the partner's
terminal stops the image data transmission or changes the transmitted
information for information through other media. In such a case, the
display continues showing meaningless images or shows a disturbed image.
According to the CCITT recommendation (or recommendation draft) H.261,
there are generally two modes of image signal encoding:
an intra-mode in which one-frame picture is divided into 12.times.33 blocks
or 3.times.33 blocks (macro-blocks), the difference between each of these
blocks and the original picture is calculated and image data on the
difference alone is transmitted; and
an inter-mode in which data on the true images of macro-blocks is
transmitted without using the above difference. This recommendation also
provides that transmission is performed in the inter-mode at least one
time every 132nd transmission with respect to macro-blocks to control
accumulation of errors.
The CCITT recommendation (or recommendation draft) H.221 provides that when
a picture renewal request BAS command is received, data for one-frame
picture is transmitted in the inter-mode.
In this conventional system, however, macro-blocks are renewed at
predetermined intervals and there is therefore a need to renewing the
picture based on user's judgment by transmitting a picture renewal command
to the partner's terminal each time an error occurs in a received image
signal during image transmission/reception.
On the other hand, processing at the time of occurrence of a frame
synchronization error is described in H.242. According to this
description, when a frame synchronization error occurs and it is detected
on the receiving side, a timer T3 (10 sec.) is set to recover frame
synchronization. If the frame synchronization is recovered before timer
termination, the ordinary operation is restarted. If the frame
synchronization is not recovered before timer termination, synchronization
is performed from the beginning.
In the above-described conventional system of multimedia terminals,
however, if a line disorder or a disorder of an image receiving unit of a
terminal takes place during image data communication between terminals, or
if image data is changed at the time of connection/disconnection with a
partner's terminal or at the time of changing the transmission capability,
omission or disorder of image data occurs and this condition directly
influences the displayed picture so that the screen is blacked out or an
image disturbance occurs. Moreover, during the period of time for
reestablishing the synchronized state, no video and audio signals are
supplied and there are no outputs on the screen, which condition gives the
operator a feeling of uneasiness, and the operator cannot discriminate
whether such a state follows occurrence of an error or completion of
communication. There is also the problem of a discontinuity between
pictures received before occurrence of a frame synchronization error and
after recovery of frame synchronization.
SUMMARY OF THE INVENTION
In view of the above-described problems, an object of the present invention
is to provide a data communication apparatus in which an omission or error
in an image data during communication does not appear in the resulting
picture.
Another object of the present invention is to provide a data communication
apparatus in which when an error such as synchronization error occurs in a
received image signal, an image for control is automatically displayed.
Still another object of the present invention is to provide a data
communication apparatus which is capable of communication using a
plurality of categories of data such as image data, voice data and code
data, and which determines whether or not there is image data in
communication data and displays an image for control if there is no image
data.
A further object of the present invention is to provide a data
communication apparatus arranged to prevent a discontinuity between
pictures received before occurrence of a frame synchronization error and
after recovery of frame synchronization when the synchronization error
occurs during reception of image data.
It is a still further object of the present invention to provide a data
communication apparatus capable of suitably informing the operator of the
condition of the apparatus by outputting error information speech or image
when an error in a received frame is detected.
It is a still further object of the present invention to provide a data
communication apparatus arranged to renew a picture by automatically
transmitting a picture renewal request command to a partner's terminal in
a case where an error occurs in a received image signal during image
transmission/reception.
These and other objects of the present invention will become apparent from
the following detailed description of the invention made with reference to
the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of a multi-media communication apparatus in
accordance with a first embodiment of the present invention;
FIGS. 2 and 3 are flow charts of a control operation in accordance with the
first embodiment;
FIG. 4 is a block diagram of a multi-media communication apparatus in
accordance with second and third embodiments of the present invention;
FIGS. 5 and 6 are flow charts of a control operation in accordance with the
second embodiment;
FIG. 7 is a sequential diagram of a channel D procedure;
FIG. 8 is a sequential diagram of an in-channel control procedure;
FIGS. 9A to 9C are diagrams of frame construction;
FIGS. 10A to 10C are diagrams of bit construction in FAS;
FIGS. 11A and 11B are diagrams of bit construction in BAS;
FIGS. 12 to 15 are flow charts of the operation of a multi-media terminal
in accordance with the third embodiment of the present invention;
FIG. 16 is a block diagram of a multi-media communication apparatus in
accordance with a fourth embodiment of the present invention;
FIG. 17 is a detailed block diagram of the construction of the video
interface unit and image data flows in accordance with the fourth
embodiment;
FIG. 18 is a flow chart of the operation procedure of the apparatus of the
fourth embodiment;
FIG. 19 is a block diagram of a multi-media communication apparatus in
accordance with a fifth embodiment of the present invention;
FIG. 20 is a sequential diagram of a communication start procedure; and
FIG. 21 is a flow chart of the operation procedure of the apparatus of the
fifth embodiment.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
Embodiments of the present invention will be described below with reference
to the accompanying drawings.
[First Embodiment]
FIG. 1 is a block diagram of a multi-media communication apparatus in
accordance with the first embodiment of the present invention. The
multi-media communication apparatus has a handset 1 provided as an audio
input/output means, a microphone 2 provided as an audio input means, and a
speaker 3 provided as an audio output means. The handset 1, the microphone
2 and the speaker 3 are connected to an audio interface unit 4. The audio
interface unit 4 is instructed by a later-described system control unit 14
to perform each of various kinds of processing including changeover
processing for changing the operations of the handset 1, the microphone 2
and the speaker 3 provided as audio input/output means, on/off-hook
detection processing for detecting whether the handset 1 is in an on-hook
state or an off-hook state, echo cancel processing for cancelling echoes
when the microphone 2 and the speaker 3 provided as audio input/output
means are operated, and processing for generating tones such as a dial
tone, a call tone, a busy tone and an incoming tone. The audio interface
unit 4 is connected to an audio encode/decode unit 5.
The audio encode/decode unit 5 is instructed by the system control unit 14
to encode a voice signal to be transmitted or to decode a received voice
signal in accordance with algorisms for encoding/decoding voice signals of
64 kbps PCM A-low, 64 kbps PCM .mu.-law, 64/56/48 kbps SB-ADPCM, 32 kbps
ADPCM, 16 kbps (e.g., APC-AB), 8 kbps, and the like.
A camera 6 is provided as an image input means to input a self-portrait
image and the like, and a picture image camera 7 is provided as another
image input means to input pictures, drawings and the like. A display unit
8 serves to display an image input through the camera 6 or the picture
image camera 7, an image received from the communication partner, an
operation picture and the like. The camera 6, the picture image camera 7
and the display unit 8 are connected to a video interface unit 9. The
video interface unit 9 is instructed by the system control unit 14 to
perform processing for changing the image input means, processing for
changing input and received images and operation pictures, image signal
composition processing for displaying these images by dividing the frame
on the display 8, and other kinds of processing. The video interface unit
9 is connected to a video encode/decode unit (image data error detection
means) 10 which encodes transmitted image signals and decodes received
image signals in accordance with the CCITT recommendation (or
recommendation draft) H.261.
A data terminal 11 for performing data communication is connected to a data
interface unit 12. The data interface unit 12 sends transmitted data
supplied from the data terminal 11 and the system control unit 14 to a
later-described multiplex/separation unit 15, and sends received data to
the data terminal 11 or the system control unit 14. An operation unit 13
including a keyboard, a touch panel or the like used to input control
information for overall control of the apparatus is connected the system
control unit 14 together with the video interface unit 9 and the data
interface unit 12. The system control unit 14 has a CPU, a ROM, a RAM, an
auxiliary memory and other components. The system control unit 14 monitors
the conditions of the other units of the apparatus to effect overall
control of the apparatus, forms operation/display pictures according to
the conditions, executes application programs and performs other
operations. The audio encode/decode unit 5, the video encode/decode unit
10 and the data interface unit 12 are connected to the
multiplex/separation unit (picture renewal request signal transmitting
means) 15. The multiplex/separation unit 15 multiplexes a voice signal
from the audio encode/decode unit 5, an image signal from the video
encode/decode unit 10, data from the data interface unit 12 and BAS from
the system control unit 14 with respect to transmitted frames in
accordance with the CCITT recommendation (or recommendation draft) H.221,
separates a received frame with respect to each of media constituting the
frame and sends separated data to the relating units. The
multiplex/separation unit 15 is connected to a line interface unit 16
which controls the line based on the ISDN user network interface.
The operation of the multi-media communication apparatus in accordance with
the present invention will be described below with reference to the flow
chart of FIG. 2. When the operation is started, data received through the
line is supplied to the multiplex/separation unit 15 through the line
interface 16 to be separated into a voice signal, an image signal and a
data communication signal, and the received image signal is decoded by the
video encode/decode unit 10 (step S1).
When the received image signal is decoded by the video encode/decode unit
10, BCH error checking in accordance with the CCITT recommendation (or
recommendation draft) H.261 is performed (step S2). If an error is
detected in the received image signal, that is, in the case of YES in step
S3, the video encode/decode unit 10 outputs an error detection signal to
the system control unit 14. If no error is detected in the received image
signal, that is, in the case of NO in step S3, this control is terminated
and the apparatus becomes ready for receiving the next image signal (step
S3).
When the system control unit 14 receives the error detection signal from
the video encode/decode unit 10, it outputs an instruction signal to the
multiplex/separation unit 15 to make this unit transmit a picture renewal
request BAS command to the partner's terminal. When the
multiplex/separation unit 15 receives this instruction signal, it sets a
picture renewal request in BAS and transmits a picture renewal request BAS
command to the partner's terminal through the line interface unit 16 (step
S4).
Next, the operation of the partner's terminal will be described below with
reference to the flow chart of FIG. 3. The partner's terminal has the same
construction as in the above. When a picture renewal request BAS command
is transmitted in step S4 of FIG. 2, the multiplex/separation unit 15 of
the partner's terminal receives this command through the line interface
unit 16 (step S5).
The multiplex/separation unit 15 separates the frame received in step S5
into four signals: voice signal, image signal, data communication signal
and BAS command, and transmits the voice signal to the audio encode/decode
unit 5, the image signal and the BAS command to the video encode/decode
unit 10, and the data communication signal to the data interface unit 12
(step S6).
The video encode/decode unit 10 analyzes the BAS command (step S7).
If there is a picture renewal request command, that is, in the case of YES
in step S8, the transmission encoding algorism is set to the intra-mode
(in which only image data on the difference from the original picture is
transmitted) (step S9). If there is no picture renewal request command,
that is, in the case of NO in step S8, the algorism is set to the
inter-mode (in which data on the true image is transmitted) (step S10),
thereby completing the control. Thus, if a picture updating request
command is transmitted from the partner's terminal, image data on the
difference from the original picture is transmitted to enable recovery
from an error by the partner's terminal.
At the time of detection of an error in the received image signal, if an
error occurs only one time, there is a possibility that the error is
regarded as a non-significant error such as temporary image disturbance.
The arrangement may therefore be such that an error detection signal is
supplied to the system control unit 14 if an error occurs a plurality of
times and is regarded as a significant error.
As described above, the multi-media communication apparatus in accordance
with the first embodiment of the present invention, which has an image
transmission/reception function, is characterized by being provided with a
detection means for detecting an error in a received image signal, and a
transmission means for transmitting a picture renewal request command to
the partner's terminal based on a detection signal from the detection
means.
It is therefore possible to automatically recover from image disturbance by
detecting the corresponding error in the received image signal and
automatically transmitting a picture renewal request command to the
partner's terminal.
[Second Embodiment]
The second embodiment of the present invention will be described below in
which if a data synchronization error occurs during communication using
image data, a control image is displayed instead of the disturbed received
image.
FIG. 4 is a block diagram of a multi-media terminal in accordance with the
second embodiment.
The multi-media terminal shown in FIG. 4 has a handset 101 provided as an
audio input/output means, a microphone 102 provided as an audio input
means, a speaker 103 provided as an audio output means, and an audio
interface unit 104. The audio interface unit 104 has functions started by
instructions from a later-described system control unit 114, i.e, a
function for changing the handset 101, the microphone 102 and the speaker
103 provided as audio input/output means, a gain control function for
controlling gain to control volume level, an on/off-hook detection
function for detecting whether the handset 101 is in an on-hook state or
an off-hook state, an echo cancel function for cancelling echoes when the
microphone 102 and the speaker 103 provided as audio input/output means
are used, and a tone generation function for generating tones such as a
dial tone, a call tone, a busy tone and an incoming tone. An audio
encode/decode unit 105 is instructed by the system control unit 114 to
encode, by A/D conversion, a voice signal to be transmitted or to decode a
received voice signal by D/A conversion in accordance with the algorism
for encoding/decoding voice signals of 64 kbps PCM (A-law), 64 kbps PCM
(.mu.-law), 64/56/48 kbps (SB-ADPCM), 32 kbps ADPCM, 16 kbps (e.g.,
APC-AB), 8 kbps, and the like.
A camera 106 is provided as an image input means to input a self-portrait
image and the like, and a picture image camera 107 is provided as another
image input means to input pictures, drawings and the like. A display unit
(display means) 108 displays an image input through the camera 106 or the
picture image camera 107, an image received from the communication
partner, and an image from the system control unit 114. A video interface
unit 109 has functions started by instructions from the system control
unit 114, i.e., a function for changing the image input means, processing
for changing input and received images and operation pictures, a function
for changing the display of the above-mentioned images and for dividing
the display frame, and a signal conversion function for establishing
electrical/physical matching with video signals from the image
input/output means. A video encode/decode unit 110 has a function for
encoding, by A/D conversion, an image to be transmitted and a function of
decoding a received image by D/A conversion. The video encode/decode unit
110 effects band compression of a large amount of raw image data by
various means such as movement compensation, frame thinning, inter-frame
compensation, in-frame compensation, DCT conversion, and vector
quantization conversion to reduce the amount of data so that the image
data can be transmitted through the digital line. Presently, the basic
interface of the ISDN line has a transmission speed of 65 kbps, and an
image encoding system enabling transmission at this transmission speed is,
for example, the one proposed in the CCITT recommendation draft H.261.
A data terminal 111 performs data communication. A data interface unit 112
sends transmitted data from the data terminal 111 and the system control
unit 114 to a multiplex/separation unit 115, and sends received data to
the data terminal 111 or the system control unit 114. An operation unit
113 includes a keyboard, a touch panel or the like used to input control
information for controlling the apparatus of FIG. 4. The system control
unit 114 has a CPU, a ROM, a RAM, an auxiliary memory, a character
generator, an image signal generation circuit and other components. The
system control unit 114 monitors the conditions of the other units to
effect overall control of the apparatus, forms operation/display pictures
according to the conditions, executes application programs and performs
other operations. The multiplex/separation unit 115 multiplexes voice data
from the audio encode/decode unit 105, image data from the video
encode/decode unit 110, data from the data interface unit 112 and BAS from
the system control unit 114 with respect to transmitted frames, separates
a received frame with respect to each of media constituting the frame and
sends separated data to the relating units. The multiplex/separation unit
115 operates in accordance with the CCITT recommendation H.221. A line
interface unit 116 controls the line based on the ISDN user network
interface.
A video input unit 109a and a video output unit 109b (video input/output
means) constitute the video interface unit 109. A video encode unit 110a
and a video decode unit 110b constitute the video encode/decode unit 110.
An image changeover circuit (image changeover means) 117 selects one of a
received image signal from the video decode unit 110b and a control image
signal from the system control unit 114, and transmits the selected signal
to the video output unit 109b. A sync detection unit 118 detects a sync
state of a received image signal from the video decode unit 110b and
informs the system control unit (control means) 114 of the result of this
detection.
The sync detection unit 118 monitors horizontal/vertical synchronization
signals in a video signal which is a received image signal, determines
that the signal is synchronized if a constant correct cyclic signal is
generated, or that the signal is not synchronized if no cyclic signal is
generated or if the cycle is not correct, and informs the system control
unit 14 of this state.
If a video encode error occurs in the video decode unit 110b, the sync
signal is disturbed to operate the sync detection unit 118. If a video
encode error occurs, it can be regarded as an synchronization error since
images are real-time processed as in the case of voice and since an error
which cannot be corrected by correction bits cannot be recovered by
re-transmission or the like. Therefore a detection method may
alternatively used in which the sync detection unit 118 is informed of
occurrence of this kind of error to recognize a sync error and to inform
the system control unit 114 of this state.
FIGS. 5 and 6 shows flows of the operation of the second embodiment.
Referring to FIG. 5, determination is first made as to whether
communication is performed by a multi-media terminal or a telephone
terminal (that is, by using voice alone) (step S101). In the case of
telephone communication, the video changeover unit 117 is operated to
establish connection between points C and B (steps S102, S103). A
telephone picture (control image signal) prepared by the system control
unit 114 is displayed by the display unit 108 (step S104). As the
telephone picture, a picture showing a menu necessary for telephone is
displayed. For example, at the time of calling, a telephone number list of
a telephone directory and a telephone number input by keys are displayed,
or at the time of call acceptance, the partner's telephone number, if the
partner can be identified, and other information are displayed. During
talking, the talking time and other information are displayed. Next, the
audio units, the components 101 to 105, are started (step S105) to
continue operating until disconnection (steps S106, S107). After
disconnection, an initial picture, e.g., a menu picture for selection of
multi-media terminals, telephone and other terminals is displayed.
In the case of communication by a multi-media terminal, the video
changeover unit 117 is operated to establish connection between the points
C and B (step S108). A multi-media terminal picture prepared by the system
control unit 114 is displayed by the display unit 108 (step S109). As this
picture, a picture showing a menu necessary for the multi-media terminal
is displayed. For example, at the time of calling, a picture for selection
of kinds of media used for communication, for setting capability with
respect to each medium, a list of partner's terminal numbers and a number
input by keys, used as in the case of telephone communication, is
displayed. At the time of call acceptance, the capability of the partner's
terminal or the capability determined by capability exchange is displayed
and, if the partner can be identified, the partner's telephone number is
displayed.
After connection to the partner's terminal and negotiation by capability
exchange (step S110), the audio units are started (steps S111, S112) if an
audio capability is required, and the data units, i.e., the components 111
and 112 are started (steps S113, S114) if a data capability is required.
If a vide | | |
