 |
Claims  |
|
|
What is claimed is:
1. A demultiplexer for a multiplexed audio video signal, the demultiplexer
comprising:
a clock producing a time reference;
a demultiplexing switch for separating a composite data stream into audio
sample batches, audio time stamps, video frames, and video time stamps,
wherein the audio time stamps are fewer or equal in number than the audio
sample batches and wherein the video time stamps are fewer or equal in
number than the video frames;
an audio phase-locked loop, receiving the audio time stamps and the time
reference, for comparing the time reference with the audio time stamps to
produce an audio timing signal and to associate one of the audio sample
batches with each of the audio time stamps;
an audio decoder receiving the audio sample batches and the audio timing
signal, and for decoding the audio sample batches in synchronization with
the audio timing signal;
a video phase-locked loop, receiving the video time stamps and the time
reference, for comparing the time reference with the video time stamps to
produce a video timing signal and to associate one of the video frames
with each of the video time stamps; and
a video decoder receiving the video frames and the video timing signal, and
for decoding the video frames in synchronization with the video timing
signal.
2. The demultiplexer of claim 1, further comprising a video time stamp
buffer receiving the video time stamps from the demultiplexing switch.
3. The demultiplexer of claim 1, further comprising an audio time stamp
buffer receiving the audio time stamps from the demultiplexing switch.
4. The demultiplexer of claim 1, further comprising a video buffer
receiving the video frames from the demultiplexing switch.
5. The demultiplexer of claim 1, further comprising an audio buffer
receiving the audio sample batches from the demultiplexing switch.
6. The demultiplexer of claim 1, wherein the time reference is indicative
of an absolute time elapsed since a reference time.
7. The demultiplexer of claim 1, wherein the clock operates at 90 kHz.
8. The demultiplexer of claim 1, wherein the multiplexed audio video signal
is an MPEG standard signal.
9. The demultiplexer of claim 6, wherein the one of the audio sample
batches associated by the audio phase-locked loop with each of the audio
time stamps is the one of the audio sample batches that, absent the audio
time stamp, would be decoded closest in time, as measured by the clock, to
the value of the audio time stamp.
10. The demultiplexer of claim 6, wherein the one of the video frames
associated by the video phase-locked loop each of the video time stamps is
the one of the video frames that, absent the video time stamp, would be
decoded closest in time, as measured by the clock, to the value of the
video time stamp.
11. A demultiplexer for an MPEG standard multiplexed audio video signal,
the demultiplexer comprising:
a 90 kHz clock producing a time reference indicative of an absolute time
elapsed since a reference time;
a demultiplexing switch for separating a composite data stream into audio
sample batches, audio time stamps, video frames, and video time stamps,
wherein the audio time stamps are fewer or equal in number than the audio
sample batches and the video time stamps are fewer or equal in number than
the video frames;
a video timestamp buffer receiving the video timestamps from the
demultiplexing switch;
an audio timestamp buffer receiving the audio time stamps from the
demultiplexing switch;
a video buffer receiving the video frames from the demultiplexing switch;
an audio buffer receiving the audio sample batches from the demultiplexing
switch;
an audio phase-locked loop, receiving the audio time stamps from the audio
timestamp buffer and receiving the time reference, for comparing the time
reference with the audio time stamps to produce an audio timing signal and
to associate one of the audio sample batches with each of the audio time
stamps, the one of the audio sample batches associated by the audio
phase-locked loop with each of the audio time stamps being the one of the
audio sample batches that, absent the audio time stamp, would be decoded
closest in time, as measured by the clock, to the value of the audio time
stamp;
an audio decoder receiving the audio sample batches and the audio timing
signal, and for decoding the audio sample batches in synchronization with
the audio timing signal;
a video phase-locked loop, receiving the video time stamps from the video
time stamp buffer and receiving the time reference, for comparing the time
reference with the video time stamps to produce a video timing signal and
to associate one of the video frames with each of the video time stamps,
the one of the video frames associated by the video phase-locked loop with
each of the video time stamps being the one of the video frames that,
absent the video time stamp would be decoded closest in time, as measured
by the clock, to the value of the video time stamp; and
a video decoder receiving the video frames and the video timing signal, and
for decoding the video frames in synchronization with the video timing
signal.
12. A decoder, comprising:
a clock producing a time reference;
means for separating a composite data stream into data units and data time
stamps, wherein the data time stamps are fewer or equal in number than the
data units;
a phase-locked loop, receiving the time stamps and the time reference, for
comparing the time reference with the time stamps to produce a timing
signal and to associate one of the data units with each of the data time
stamps, the one of the data units associated by the phase-locked loop with
each of the data time stamps being the one of the data units that, absent
the data time stamp, would be decoded closest in time, as measured by the
clock, to the value of the data time stamp; and
a data decoder receiving the data units and the timing signal, and for
decoding the data units in synchronization with the timing signal.
13. A demultiplexer for a multiplexed audio video signal, the demultiplexer
comprising:
a demultiplexing switch for separating a composite data stream into audio
sample batches, audio time stamps, video frames, and video time stamps;
an audio phase-locked loop, receiving the audio time stamps, for producing
an audio timing signal in synchronization with the audio time stamps;
an audio decoder receiving the audio sample batches and the audio timing
signal, and for decoding the audio sample batches in synchronization with
the audio timing signal;
the audio decoder including means for extracting an access unit count from
the audio sample batches and comparing the access unit count with a stored
access unit count to verify that the audio decoder is operating in proper
synchronization;
a video phase-locked loop, receiving the video time stamps, for producing a
video timing signal in synchronization with the video time stamps;
a video decoder receiving the video frames and the video timing signal, and
for decoding the video frames in synchronization with the video timing
signal; and
the video decoder including means for extracting an access unit count from
the video frames and comparing the access unit count with a stored access
unit count to verify that the video decoder is operating in proper
synchronization.
14. The demultiplexer of claim 13, further comprising a video timestamp
buffer receiving the video timestamps from the demultiplexing switch.
15. The demultiplexer of claim 13, further comprising an audio timestamp
buffer receiving the audio timestamps from the demultiplexing switch.
16. The demultiplexer of claim 13, further comprising a video buffer
receiving the video frames from the demultiplexing switch.
17. The demultiplexer of claim 13, further comprising an audio buffer
receiving the audio sample batches from the demultiplexing switch.
18. A decoder, comprising:
means for separating a composite data stream into data units and data time
stamps;
timing means receiving the data time stamps, for producing a timing signal
in synchronization with the data time stamps; and
a data decoder receiving the data units and the timing signal, for decoding
the data units in synchronization with the timing signal;
the data decoder including means for extracting an access unit count from
each of the data units and comparing the access unit count with a stored
access unit count to verify that the data decoder is operating in proper
synchronization.
19. The decoder of claim 18, wherein the timing means includes a
phase-locked loop.
20. A method for demultiplexing a multiplexed audio video signal, the
method comprising the steps of:
separating a composite data stream into audio sample batches, audio time
stamps, video frames, and video time stamps;
producing an audio timing signal in synchronization with the audio time
stamps;
decoding the audio sample batches in synchronization with the audio timing
signal;
extracting an audio access unit count from the audio sample batches and
comparing the audio access unit count with a stored audio access unit
count to verify audio synchronization;
producing a video timing signal in synchronization with the video time
stamps;
decoding the video frames in synchronization with the video timing signal;
extracting a video access unit count from the video frames and comparing
the video access unit count with a stored video access unit count to
verify video synchronization; and
incrementing the stored audio access unit count each time an audio sample
batch is decoded, and incrementing the stored video access unit each time
a video frame is decoded.
21. The method of claim 20, further comprising the step of duplicating a
video frame if the video access unit count for the video frame is larger
than the stored video access unit count.
22. The method of claim 20, further comprising the step of omitting a video
frame if the video access unit count for the video frame is smaller than
the stored video access unit count.
23. 3The method of claim 20, further comprising the step of omitting an
audio sample batch if the audio access unit count for the audio sample
batch is smaller than the stored audio access unit count.
24. The method of claim 20, further comprising the step of duplicating an
audio sample batch if the audio access unit count for the audio sample
batch is larger than the stored audio access unit count.
25. A method for processing a video and an associated audio signal, the
method comprising the steps of:
providing a signal including audio sample batches and video frames;
decoding the audio sample batches;
extracting an audio access unit count from the audio sample batches and
comparing the audio access unit count with a stored audio access unit
count to verify audio synchronization;
decoding the video frames;
extracting a video access unit count from the video frames and comparing
the video access unit count with a stored video access unit count to
verify video synchronization; and
incrementing the stored audio access unit count each time an audio sample
batch is decoded, and incrementing the stored video access unit count each
time a video frame is decoded.
26. The method of claim 25, further comprising the step of duplicating a
video frame if the video access unit count for the video frame is larger
than the stored video access unit count.
27. The method of claim 25, further comprising the step of omitting a video
frame if the video access unit count for the video frame is smaller than
the stored video access unit count.
28. The method of claim 25, further comprising the step of omitting an
audio sample batch if the audio access unit count for the audio sample
batch is smaller than the stored audio access unit count.
29. The method of claim 25, further comprising the step of duplicating an
audio sample batch if the audio access unit count for the audio sample
batch is larger than the stored audio access unit count.
30. The method of claim 25, wherein the audio video signal includes
additional audio and video time stamps respectively indicating
presentation times for ones of the audio sample batches and video frames.
31. A method for processing a stream of data bits divided into access
units, the method comprising the steps of:
receiving a first access unit;
initializing a register with an access unit count corresponding to the
first access unit;
receiving and decoding one or more subsequent access units following the
first access unit;
incrementing the access unit count stored in the register for each of the
subsequent access units and comparing the access unit count stored in the
register with an access unit count extracted from each of the subsequent
access units;
determining if the access unit count extracted from each of the subsequent
access units is the same as the access unit count stored in the register
to verify synchronization;
omitting one of the subsequent access units if the access unit count
extracted from said one of the subsequent access units is smaller than the
access unit count stored in the register;
detecting a discontinuity bit in the stream of data bits; and
reinitializing the register upon detection of the discontinuity bit.
32. The method of claim 31, wherein the register is reinitialized with an
access unit count extracted from an access unit following the
discontinuity bit. |
|
 |
|
 |
Claims |
|
|
|
Description |
|
|
BACKGROUND
1. Field of the Invention
This invention relates generally to the field of digital transmission or
storage of audio and video information. More particularly, this invention
relates to a technique for verifying the timing of multiplexed packetized
audio and/or video (A/V) information such as ISO's (International
Organization for Standardization) MPEG (Motion Pictures Expert Group).
2. Background of the Invention
The term "access unit" as used herein means either a frame of video data or
a batch of audio samples or a batch of other data samples. In general, a
decoder processes its "access units" and outputs decoded access units at
regular intervals. In the case of video, for example, this interval is the
picture rate, and for audio this is a constant integer (equal to the
number of audio samples in an audio access unit) times the audio sampling
rate.
In the case of an ISO standard MPEG system application, as defined for
example in ISO document number 1-11172 which is hereby incorporated by
reference, a "time stamp" indicating the decoding time for an access unit,
is included in the multiplex syntax for multiplexed video and audio
packets as shown in FIG. 1. These time stamps are included in the
multiplex syntax, i.e. in packet headers. A time stamp indicates the
decoding time of the first access unit header in that packet. Since each
packet can include multiple access units, each access unit is not
associated with a time stamp.
Although multiplex applications use multiple decoders, ISO's MPEG 1 system
standard can also be applied in applications that only have one decoder.
In a fixed bit rate (e.g. audio) application, perfect clocks do not exist,
therefore the digital storage media (or transmission) bit rate and
(depending on the sampling clock frequency error) the decoder input bit
rate vary. In such a system, the decoder generally reads one access unit
at a time. Furthermore, due to the differences in the clocks, the transfer
bit rate and the decoder input bit rate do not identically match. A buffer
can be used to compensate for these differences. In the case of video
data, consecutive access units (i.e. frames) are compressed to provide
variable length compressed (VLC) access units having a length that depends
on the picture content. Consequently the video decoder input bit rate has
large variations and a relatively large buffer is used.
However, even if a buffer is used, ideal bit rates generally do not exist
and therefore "buffer errors"(and even buffer overflow or underflow) can
occur. Two methods are generally used to prevent buffer overflow and
underflow. With one method (called Digital Storage Media slave) the
transfer rate (i.e. buffer input rate) is controlled. The other method
(called decoder slave) is realized by controlling the buffer output data
rate. In the case of video this can be done by adjusting the frame rate.
In the case of audio, this is done by adjusting the sampling rate. The
buffer output data rate is thus adjusted.. Another decoder slave method
skips or repeats access units in order to control the buffer output data
rate.
Adjustments of the decoder rate and adjustments of the transfer bit rate
are restricted by characteristics of the peripheral hardware. Therefore if
the buffer error (i.e. deviation from the ideal buffer fullness) is too
large, the appropriate control can become difficult or impossible. When
starting playback, a large buffer error can sometimes occur. Therefore
generally the decoder starts decoding after an appropriate start up delay
in order to reduce the initial buffer error.
In the MPEG 1 system standard, fields have been included in the multiplex
syntax which can be used to control the decoder or the transfer rate. In
the pack header a value called SCR (System Clock Reference) can be used to
control the transfer data rate. Time stamps in the video packet header can
be used to control the frame rate, and time stamps in the audio packet
header can be used to control the sampling rate. SCR indicates the time
when (the first part of the) packet data enters the decoder buffer and
time stamps indicate when a certain access unit in the packet data is to
be removed from the decoder buffer.
Both SCR and time stamps are absolute values of a clock that increments
continuously at a rate of 90 KHz. Therefore the difference between the
first read SCR and the first read time stamp can be used as a start up
delay.
Unfortunately, (except for the first audio and video time stamps for
initial start up) it is difficult to use consecutive time stamps. The
problem is that, after demultiplexing, the time stamps are separated from
their related access units. I.e., since the decoding system's demultiplex
switch separates time stamps from the packet data and then stores the
packet data (without their time stamps) in the respective buffers, it
becomes difficult to keep track of which time stamp belongs to which
access unit header. For example: a decoding system processes a certain
MPEG multiplex stream. The first SCR (System Clock Reference) is detected,
and the system uses this value to initialize a local (90 KHz) clock. From
now on this clock increments automatically at a 90 KHz rate. Then the
system detects the first video DTS. It indicates (with a 90 KHz clock
value) the decoding time for the first following picture header. However
generally there is a significant delay before this picture header should
be decoded (due to the buffer before the video decoder), and therefore the
time stamp must also be delayed or buffered before it can be used.
Furthermore, before the first picture is decoded, several more video
packets with time stamps can arrive at the demultiplexer switch and these
time stamps should also be delayed or buffered somewhere before they can
be used.
With the adopted definition for time stamps, it is a significant
shortcoming of the MPEG standard that the time stamps were not included in
the video and audio syntax specifications.
The problem is that the time stamps are in the "wrong" layer. This might
not be so bad if all access unit headers had time stamps. However, this is
not the case. Therefore, simply adding an additional time stamp buffer for
each decoder and assuming that each access unit has an accompanied time
stamp is not possible. Since a packet header contains only one time stamp,
and since the packet data can contain several access unit headers, not
every access unit header will have a time stamp. Even if the packet data
contains an access unit header, inserting a time stamp in the preceding
packet header is optional because the only MPEG requirement concerning
time stamps is that they occur at least once every 0.7 seconds.
If a particular application uses two or more decoders (e.g. one video and
one audio decoder) it is often necessary to synchronize these decoders.
According to the MPEG 1 system standard time stamps must be used to
perform this synchronization. The standard assumes that the audio and
video decoder processing delays are 0 ms. Consequently (in this model) the
time when an access unit is decoded is equal to the time when the decoded
access unit (i.e. "presentation unit") is output.
Besides the previously mentioned reason for the decoder start-up delay (to
minimize initial buffer error), in case of multi-decoder systems, for each
decoder an initial start-up delay is also necessary. This is because, for
example, the audio and video packets that occur together (in the same
pack) are often segments of the audio and video signal that, after
decoding, should be output at different times. This is possible, because
MPEG has agreed on a certain amount of buffering before each video and
audio decoder, which allows a flexible multiplex bitstream structure. A
different way to describe the MPEG multiplex standard is: any kind of pack
and packet structure is acceptable, as long as the buffers in the
reference decoder do not overflow or underflow.
Two kinds of decoding systems exist for synchronization. The first is a
locked system wherein the frame rate and sampling rate are locked to a
single common clock (e.g. 90 KHz in MPEG). The locked system has the
disadvantage that it can only play back bit streams that were generated by
an encoding system where the frame rate and sampling rate were also locked
to one common clock. Whether the video and audio encoder clocks are locked
or not will depend on the application. (In case of CDI-FMV, locking the
encoders is mandatory). In this system, (if the transmission error
characteristics are limited) after reading the first video and audio time
stamps and using them for the respective decoder start, all following time
stamps can be ignored. This kind of decoding system is relatively simple,
and does not need to keep track of which time stamps belong to which
access units. However, if a transmission error causes missing or false
access unit headers, a sync error results (and a corresponding buffer
error). Such problems can also occur in non-MPEG systems. The invention
provides a solution for this problem.
The second kind of decoding system (called non-locked decoding system) can
also play back non-locked encoded multiplex bit streams. Non-locked
encoded bit streams are generated by encoding systems that have
independent encoder frame and sampling rate clocks. In this case there is
no relation between the video encoder's frame rate error and the audio
encoder's sampling rate error; they vary independently.
Non-locked MPEG decoding systems are used if the multiplex bitstream was
generated by a non-locked encoding system, i.e. an encoding system where
the picture rate clock and audio sampling rate clock are independent.
Whether the non-locked encoded systems will be used or not will depend on
the application. For example, in case of CDI-FMV (i.e., Phillips' Compact
Disc Interactive with Full Motion Video extension, which has adopted the
MPEG 1 standard), independent video and audio encoder clocks are not
allowed. Instead, both these clocks must be locked to a single common
clock. However, in the future some applications may use non-locked MPEG
systems for certain applications.
When the MPEG standard is used, non-locked encoder (frame and sampling
rate) clock errors are recorded with time stamps and then included in the
bit stream. During playback, in order to prevent an AV sync error, at
least one decoder must have a PLL mechanism which uses time stamps
regularly and make the actual frame (or sampling) rate match the time
stamp values. The video decoder should thus read the video time stamps
(i.e. the DTS--the Decoding Time Stamps) or the video PTS (the
Presentation Time Stamps) and use these time stamps to control the picture
rate, or the audio decoder should read the audio DTSs and use these to
control the audio sampling rate, or both decoders should use time stamps
to control their clocks.
In FIG. 1, an MPEG or similar data stream 20 of packets is shown as a
mixture of video packets such as 22 and audio packets 24. Collections of
packets 22 and 24 are arranged in a larger pack preceded by a pack header
26. In each case, the actual video data 27 or audio data 28 are preceded
by a video packet header which contains (among other data items) time
stamp 30 or an audio packet header which contains (among other data items)
time stamps 32 respectively. The actual video data 27 are divided into
video frames, whereas the audio data 28 are divided into batches of
samples as illustrated.
According to one decoding method, the decoding system demultiplexes the
incoming packets into an audio bit stream and a video bit stream, takes
the time stamps from the packet header, and inserts them just before the
related access unit in each elementary stream. This generates the syntax
as shown in FIG. 2. In this syntax, for example, frame n+1 contains the
value of the video time stamp (VTS) (i.e. a time value from a 90 KHz
clock). Similarly, unit m+1 includes the audio time stamp (ATS) just prior
to the audio unit m+1.
In order to produce such elementary streams, a decoding system as shown in
FIG. 3 can be used. In this system, the multiplexed bit stream 20 is
provided to a demultiplexer 50 which separates the bit stream into video
data, video time stamps, audio data and audio time stamps. The video data
are passed through a video syntax modifier 54 while the audio data are
passed through an audio syntax modifier 58. The modified video and audio
data are buffered in buffers 60 and 62 respectively prior to decoding by
video and audio decoders 66 and 68 respectively under control of picture
rate control circuit 74 and sample rate control 76. Video and audio emerge
at the outputs 84 and 86 respectively.
This method has the drawback that extra modules are required before each
buffer in order to insert the time stamps in the right place of the
demultiplexed data streams. If the time stamps were in the correct layer,
this problem would not exist.
Also, this non-locked system has a further drawback that both the
elementary A/V bit streams are modified (i.e. the bit streams at the input
of the decoders do not comply with the respective audio and video
standards). Therefore, the decoders of this system cannot directly decode
non-multiplexed audio and video bit streams.
Finally, this non-locked system succeeds in maintaining the relation
between time stamps and access unit headers. Therefore, it has the ability
to detect whether access unit headers are lost or falsely generated (e.g.
due to a transmission error). Such an error would cause a large difference
between the intended access unit decoding time (i.e. the time stamp's
value) and the actual access unit's decoding time which would be detected
by the affected decoder's PLL. However, unfortunately, the PLLs are
designed to correct such differences by adjustments of the decoder's
clock, which in the worst case will cause a third drawback. The PLL will
try to repair such a large AV sync error by adjusting the decoder rate
very slowly (as usual). Such a slow correction procedure requires a long
time to repair the AV sync error and the corresponding buffer error.
Consequently, the chance of buffer underflow or overflow increases. Also,
since a large AV sync error could last for several seconds or longer, the
chance that the user will notice the AV sync error increases. If the PLL
attempts to repair the AV error by quickly adjusting the affected
decoder's clock, other audio or video artifacts would be generated (e.g.
vertical roll in the video or audio frequency shift in the audio).
Therefore in this system the adjustment of a decoder clock with time stamp
values is appropriate for replicating the small encoder clock errors that
exist in nonlocked encoded composite bit streams. This approach is not
always suitable for correcting large sync errors that were caused by a
certain number of lost or excess access unit headers.
The present invention was developed to alleviate these problems of
non-locked MPEG decoding systems and to overcome the disadvantage of
simple locked (MPEG and non-MPEG) decoding systems as described above.
Also, the system just described has the drawback that both elementary A/V
bit streams are modified (i.e. the bit streams at the input of the
decoders do not comply with the respective audio and video standard).
Therefore, the decoders in this system can not directly decode
non-multiplexed audio and video bit streams.
The present invention was designed to alleviate several problems as
described above. The simplest MPEG decoding system (as described above)
and the simplest non-MPEG decoding systems are locked decoding systems
(systems where the video decoder's picture rate and the audio decoder's
sampling rate are locked to a single common clock). These systems have the
drawback that they can not be used in applications that lose or falsely
generate access unit headers (for example, due to transmission or storage
errors).
The present invention overcomes these shortcomings by using an access unit
count value in each elementary stream. With this value each decoder can
detect missing or false access unit headers. The affected decoder can then
request a system reset or try to "repair" the synchronization error by
redecoding access units or by skipping access units. The decoder for such
a system is easily implemented. In addition to this, access count makes a
simpler non-locked decoding system possible without the disadvantages of
the non-locked decoding system above.
SUMMARY OF THE INVENTION
It is an object of the present invention to provide an A/V decoding system
which can detect missing or excess access unit headers.
It is another object to provide an A/V decoding system which can maintain
synchronization in an unlocked system.
It is another object to provide an A/V decoding system which can play back
edited multiplex and edited elementary bit streams.
It is an advantage that the present invention is simple to implement.
It is another advantage that the current invention solves the problem of
determining which time stamp belongs to which access unit, by looking at
the value of each time stamp and then associating it with a certain access
unit.
These and other objects, advantages and features of the invention will
become apparent to those skilled in the art upon consideration of the
following description of the invention.
In one embodiment of the invention, a demultiplexer for a multiplexed audio
video signal includes a clock which produces a time reference. A
demultiplexing switch separates a composite data stream into audio sample
batches, audio time stamps, video frames, and video time stamps, where the
audio time stamps are fewer than or equal in number to the audio sample
batches and where the video time stamps are fewer than or equal in number
to the video frames. An audio phase-locked loop receives the audio time
stamps and the time reference, and compares the time reference with the
audio time stamps to produce an audio timing signal which associates one
of the audio sample batches with each of the audio time stamps. An audio
decoder receives the audio sample batches and the audio timing signal, and
decodes the audio sample batches in synchronization with the audio timing
signal. A video phase-locked loop receives the video time stamps and the
time reference and compares the time reference with the video time stamps
to produce a video timing signal which associates one of the video frames
with each of the video time stamps. A video decoder receives the video
frames and the video timing signal, and decodes the video frames in
synchronization with the video timing signal.
A demultiplexer for an MPEG standard multiplexed audio video signal
includes a 90 KHz clock producing a time reference indicative of an
absolute time elapsed since a reference time. A demultiplexing switch
separates a composite data stream into audio sample batches, audio time
stamps, video frames, and video time stamps, where the audio time stamps
are fewer than or equal in number to the audio sample batches and the
video time stamps are fewer than or equal in number to the video frames. A
video timestamp buffer receives the video timestamps from the
demultiplexing switch and an audio timestamp buffer receives the audio
timestamps from the demultiplexing switch. A video buffer receives the
video frames from the demultiplexing switch, and an audio buffer receives
the audio sample batches from the demultiplexing switch. An audio
phase-locked loop receives the audio time stamps from the audio timestamp
buffer and receives the time reference. The audio PLL compares the time
reference with the audio time stamps to produce an audio timing signal
which associates one of the audio sample batches with each of the audio
time stamps. The audio phase-locked loop associates an audio time stamp
with the audio sample batch which is decoded closest in time, as measured
by the clock, to the value of the audio time stamp. An audio decoder
receives the audio sample batches and the audio timing signal, and decodes
the audio sample batches in synchronization with the audio timing signal.
A video phase-locked loop receives the video time stamps from the video
time stamp buffer and receives the time reference. The video PLL compares
the time reference with the video time stamps to produce a video timing
signal which associates one of the video frames with each of the video
time stamps. The video phase-locked loop associates a video time stamp
with the video frame which is decoded closest in time, as measured by the
clock, to the value of the video time stamp. A video decoder receives the
video frames and the video timing signal, and decodes the video frames in
synchronization with the video timing signal.
In another aspect of the invention, a decoder includes a clock producing a
time reference. The composite data stream is separated into data units and
data time stamps. The data time stamps are fewer than or equal in number
to the data units. A phase-locked loop receives the time stamps and the
time reference, and compares the time reference with the time stamps to
produce a timing signal which associates one of the data units with each
of the time stamps. The wherein the phase-locked loop associates a time
stamp with the data unit which is decoded closest in time, as measured by
the clock, to the value of the video time stamp. A decoder receives the
data units and the audio timing signal, and decodes the units in
synchronization with the timing signal.
In another aspect of the invention, a demultiplexer for a multiplexed audio
video signal includes a demultiplexing switch for separating a composite
data stream into audio sample batches, audio time stamps, video frames,
and video time stamps. An audio phase-locked loop receives the audio time
stamps, and produces an audio timing signal in synchronization with the
audio time stamps. An audio decoder receives the audio sample batches and
the audio timing signal, and decodes the audio sample batches in
synchronization with the audio timing signal. The audio decoder extracts
an access unit count from the audio sample batches and compares the access
unit count with a stored access unit count to verify that the audio
decoder is operating in proper synchronization. A video phase-locked loop
receives-the video time stamps, and produces a video timing signal in
synchronization with the video time stamps. A video decoder receives the
video frames and the video timing signal, and decodes the video frames in
synchronization with the video timing signal. The video decoder extracts
an access unit count from the video frames and comparing the access unit
count with a stored access unit count to verify that the video decoder is
operating in proper synchronization.
In another aspect of the invention, a decoder includes a demultiplexing
circuit for separating a composite data stream into data units and data
time stamps. A timing circuit receives the data time stamps, and produces
a timing signal in synchronization with the time stamps. A decoder
receives the data units and the timing signal, and decodes the data units
in synchronization with the timing signal. The decoder includes a circuit
for extracting an access unit count from the each data unit and compares
the access unit count with a stored access unit count to verify that the
decoder is operating in proper synchronization.
A method for demultiplexing a multiplexed audio video signal according to
the present invention includes the steps of: separating a composite data
stream into audio sample batches, audio time stamps, video frames, and
video time stamps; producing an audio timing signal in synchronization
with the audio time stamps; decoding the audio sample batches in
synchronization with the audio-timing signal; extracting an access unit
count from the audio sample batches and comparing the access unit count
with a stored access unit count to verify audio synchronization; producing
a video timing signal in synchronization with the video time stamps;
decoding the video frames in synchronization with the video timing signal;
extracting an access unit count from the video frames and comparing the
access unit count with a stored access unit count to verify video
synchronization; incrementing the stored access unit count each time a new
access unit is received.
Another method for processing a video and an associated audio signal
according to the invention includes the steps of: providing a signal
having audio sample batches and video frames; decoding the audio sample
batches; extracting an access unit count from the audio sample batches and
comparing the access unit count with a stored access unit count to verify
audio synchronization; decoding the video frames; extracting an access
unit count from the video frames and comparing the access unit count with
a stored access unit count to verify video synchronization; incrementing
the stored access unit count each time a new access unit is received.
Another method of the invention for processing a stream of data bits
includes the steps of: initializing a register with an access unit count
corresponding to a first frame header; receiving and decoding one or more
subsequent frames; incrementing the register for each the subsequent frame
and comparing the content of the register with an access unit count with
each the subsequent frame; determining if the access unit count from each
the subsequent frame is the same as that stored in the register to verify
synchronization; omitting a video frame if said access unit count for said
video frame is smaller than said stored access unit count; omitting an
audio sample batch if said access unit count for said audio sample batch
is smaller than said stored access unit count; detecting a discontinuity
bit; reinitializing the register upon detection of the discontinuity bit.
According to ISO's MPEG 1 system standard (which is intended for digital AV
storage media applications or digital AV transmission applications) in
order to detect and correct AV synchronization errors at the decoder,
"time stamps" are included in the packet headers of the multiplex syntax.
A time stamp indicates the presentation time of the first access unit
header (i.e. frame header or audio unit header) that occurs in that
packet. After demultiplexing time stamps are normally separated from their
related access units. This makes them difficult to use. In, for example,
non-locked decoding systems, some method must be used to find out which
time stamps belong to which access units. With one embodiment of the
current invention, a relatively simple decoding system makes use of the
redundancy of the time stamp values to determine this relationship. The
current invention solves the problem of determining which time stamps
belong to which access unit, by looking at the value of each time stamp
and then associating it with a certain access unit. This enables easy
non-locked decoding.
A compressed video signal and a compressed audio signal decoding apparatus
of an embodiment of the invention includes a local access unit counter
intended for digital AV applications where access unit headers (e.g.,
frame headers, audio unit headers or other kind of data unit headers) may
be lost or generated by storage media or transmission errors. In order to
prevent a resulting AV sync error, a value called "access unit count" is
included in each elementary stream. With this value each decoder can
detect lost or false access unit headers, and correct the AV
synchronization. In order to support editing, an extra bit can be added to
indicate discontinuity.
In embodiments where the access unit count fields are included in the
syntax of each elementary bit stream by each encoder, there is no need to
move this data from the multiplex layer to each elementary in the decoding
system. Therefore the decoding system's hardware or software complexity is
low. In this, decoding system time stamps are used for the initial start
up delay and synchronization, and also to follow the encoder's frame
and/or audio sampling rate clock error. Access Unit Count is used only to
detect missing or fake access unit | | |
