Digital audio workstations providing digital storage and display of video information

What is claimed is:

1. A digital audio workstation comprising:

a. means for storing digital audio information;

b. means for selecting a portion of digital audio information at audio sampling times spaced apart by less than one-thirtieth of a second;

c. a computer-readable randomly-accessible storage medium in which digital video information is stored and is associated in time with the digital audio information;

d. means for detecting selection by an operator of the portion of digital audio information; and

e. means for accessing the digital video information based on the selected portion of digital audio information selected by the operator.

2. The digital audio workstation of claim 1, wherein the accessed video information is from an identical time portion as the indicated audio information.

3. The workstation of claim 1, wherein the accessed video information is offset from the selected audio information by a particular time period indicated.

4. The workstation of claim 1, wherein the accessed video information is displayed essentially immediately upon the selection of the associated audio information.

5. The workstation of claim 1, further comprising means for displaying a visual timeline which represents waveforms of the audio information.

6. The workstation of claim 1, further comprising means for indicating segments of the digital audio information on a computer display and for allowing editing those segments by using computer user-interface techniques of cutting, pasting, and dragging.

7. The workstation of claim 1, further comprising a means for repeatedly playing a segment of the digital video information while allowing the operator to move the digital audio segment associated with that video segment forward or backward in time so as to synchronize events in the digital audio segment and the digital video information.

8. The workstation of claim 1, further comprising a means for repeatedly playing a segment of the digital video information while successively playing different versions of a segment of the digital audio information associated with the digital video information so as to enable the operator to choose one version of the digital audio segment.

9. The workstation of claim 1, further comprising a means for storing, in association with a particular point of the digital audio information, a digitized voice or textual message for later reference regarding the digital audio information.

10. The workstation of claim 1, further comprising a means for digitizing analog video information to produce the digital video information for storage.

11. The workstation of claim 10, wherein the accessed video information is from an identical time portion as the indicated audio information.

12. The workstation of claim 10, wherein the accessed video information is offset from the selected audio information by a particular time period indicated.

13. The workstation of claim 10, wherein the accessed video information is displayed essentially immediately upon the selection of the associated audio information.

14. The workstation of claim 10, further comprising means for displaying a visual timeline which represents waveforms of the audio information.

15. The workstation of claim 10, further comprising means for indicating segments of the digital audio information on a computer display and for allowing editing those segments by using computer user-interface techniques of cutting, pasting, and dragging.

16. The workstation of claim 10, further comprising a means for repeatedly playing a segment of the digital video information while allowing the operator to move the digital audio segment associated with that video segment forward or backward in time so as to synchronize events in the digital audio segment and the digital video information.

17. The workstation of claim 10, further comprising a means for repeatedly playing a segment of the digital video information while successively playing different versions of a segment of the digital audio information associated with the digital video information so as to enable the operator to choose one version of the digital audio segment.

18. The workstation of claim 10, further comprising a means for storing, in association with a particular point of the digital audio information, a digitized voice or textual message for later reference regarding the digital audio information.

19. A digital audio workstation comprising:

means for storing multiple tracks of digital audio information;

means for storing video information having frame rate and synchronized to said digital audio information in a storage medium that provides non-linear and random access to any selected portion of said video information;

means for selecting a portion of said multiple tracks of digital audio information at a resolution less than the frame rate of the video information; and

means for displaying portions of said video information associated with said digital audio information during editing of said digital audio information.

20. A digital audio workstation as defined in claim 19 wherein said means for selecting a portion of said multiple tracks of digital audio information includes means for editing said digital audio information at times corresponding to audio sampling boundaries.

21. A digital audio workstation as defined in claim 20 wherein said means for selecting a portion of said multiple tracks of digital audio information includes

means for generating a timeline display containing information representative of said multiple tracks of digital audio information in a selected clip;

means for displaying a list of available audio clips in a workreel; and

means for selecting audio clips in said workreel and transferring said audio clips from said workreel to said timeline to provide an edited clip in said timeline.

22. A digital audio workstation as defined in claim 21 further including means for controlling recording and playback of said digital audio information.

23. A digital audio workstation as defined in claim 21 further including means for associating locators representative of specific events with specific subframe locations in said digital audio information and means for accessing said specific subframe locations by selecting one of said locators.

24. A digital audio workstation as defined in claim 19 wherein said means for displaying portions of said video information includes means for repeatedly displaying a selected portion of said video information in a loop and wherein said means for selecting a portion of said multiple tracks of digital audio information includes means for recording digital audio information each time the selected portion of said video information is played.

25. A digital audio workstation as defined in claim 24 wherein said means for repeatedly displaying a selected portion of said video information in a loop includes means for automatically repeating said selected portion of video information after a predetermined delay.

26. A digital audio workstation as defined in claim 24 wherein said means for repeatedly displaying a selected portion of said video information in a loop includes means for displaying said selected portion of video information after manual initiation by a user.

27. A method for editing digital audio comprising the steps of:

storing multiple tracks of digital audio information;

storing video information having a frame rate and synchronized to said digital audio information in a storage medium that provides non-linear and random access to any selected portion of said video information;

selecting a portion of said multiple tracks of digital audio information at a resolution less than the frame rate of the video information; and

displaying portions of said video information associated with said digital audio information during editing of said digital audio information.

28. The method of claim 27 wherein said step of selecting a portion of said multiple tracks of digital audio information includes editing said digital audio information at times corresponding to audio sampling boundaries.

29. The method of claim 28 wherein the step of selecting a portion of said multiple tracks of digital audio information includes the steps of:

generating a timeline display containing information representative of said multiple tracks of digital audio information in a selected clip;

displaying a list of available audio clips in a workreel; and

selecting audio clips in said workreel and transferring said audio clips from said workreel to said timeline to provide an edited clip in said timeline.

30. The method of claim 29 further including the step of controlling recording and playback of said digital audio information.

31. The method of claim 29 further including the step of associating locators representative of specific events with specific subframe locations in said digital audio information and accessing said specific subframe locations by selecting one of said locators.

32. The method of claim 27 wherein said step of displaying portions of said video information includes repeatedly displaying a selected portion of said video information in a loop and wherein said step of selecting a portion of said multiple tracks of digital audio information includes recording digital audio information each time the selected portion of said video information is played.

33. The method of claim 32 wherein said repeatedly displaying a selected portion of said video information in a loop includes automatically repeating said selected portion of video information after a predetermined delay.

34. The method of claim 32 wherein said step of repeatedly displaying a selected portion of said video information in a loop includes displaying said selected portion of video information after manual initiation by a user.

Description Submit all comments and votes

FIELD OF THE INVENTION

The invention relates to the field of digital audio editing. More specifically, it is a system for editing the digital audio portion of video programs.

BACKGROUND OF THE INVENTION

Digital audio provides significant benefits over traditional analog audio recording, including more ease and precision in editing, easier addition of special audio effects, and the elimination of sound quality degradation in successive generations of recordings. Digital audio can also be stored in a computer memory and read directly to provide instant random access to any point on the program, For all of these reasons, digital audio workstations are common in the video and film industries.

The basic process of editing a video or film program with digital audio generally begins with the production of a video program with preliminary audio tracks. The program is then taken to a digital audio workstation, where more tracks are added, sound effects are built, and the existing tracks are cleaned up. At this stage, in contrast with the first, video-based stage, the audio editing is done in a more narrow and precise time frame than the one-thirtieth second period of an individual video frame or the one twenty-fourth second period of a film frame; therefore, the editing is known as "sub-frame" editing. Finally, in the last stage of the audio production, the program receives the final mixing and "sweetening" of the audio tracks.

It is the second and final stages of this process in which the invention is most useful. Those stages and the sub-frame precision they require , necessitate close interaction of the audio editing with the video portion of the program. The editing typically requires, among other things, synchronization of the audio effects with the action in the video program. As noted above, if the digital audio program is stored in a computer memory, it can be accessed immediately, greatly facilitating this editing process. (United Kingdom patent Application No. 2,245,745 discloses an application of this capability.) However, with current systems the video program is stored on a normal video tape recorder, which requires a great deal of time to rewind or fast-forward to the desired editing point and must be pre-rolled to its full speed for precise editing. Hence, the potential editing speed and convenience of the digital audio process is held back by the use of conventional video recording.

SUMMARY OF THE INVENTION

The invention disclosed herein takes full advantage of the potential of digital audio editing by combining it with integrated digital video capabilities. The invention is a digital audio workstation that provides the necessary facilities to store digital audio information and edit it with the required time precision. The basic editing precision for digital audio is generally the digital sampling period (usually approximately one-forty-thousandth of a second) , and is in any case much less than the one-thirtieth of a second for a full video frame. The invention adds to this digital audio editing capability the ability to digitize, store and display video information as well. It can detect an operator's indication of a point or segment of audio information and use it to retrieve and display the video images that correspond to the indicated audio programming.

Generally, the video information displayed will be from the same time point or segment as the indicated audio information. However, the invention also allows the operator to indicate a time offset, in which case the video information displayed is offset from the indicated audio information by the time period indicated by the operator.

In any event, the video information is displayed essentially immediately upon the indication of the associated audio information by the operator. This is the key advantage of the use of digital random access rather than linearly stored video signals.

The speed and random access of digital video retrieval allow several useful capabilities in preferred embodiments of the invention. For example, the invention can repeatedly play a segment of the video information while allowing the operator to move the audio segment associated with that video segment forward or backward in time so as to synchronize events in the audio and video segments. Another useful feature is the invention's ability to repeatedly play a segment of the video information while successively playing different versions of a segment of the audio information associated with that video segment so as to enable the operator to choose the best version of that audio segment.

Preferred embodiments of the invention also provide additional audio capabilities. The workstation includes a visual timeline that displays the waveforms of the audio information. This feature provides a visual representation of the overall audio program for the convenience of the operator. The preferred embodiment also provides fast and intuitive means of editing the audio segments by using computer user-interface techniques of cutting, pasting, and dragging.

Finally, another aspect of the invention is a labeling and notation system for recorded digitized audio or video information. The system provides a means for storing in association with a particular point of the audio or video information a digitized voice or textual message for later reference regarding that information. This aspect of the invention can be thought of as the equivalent in recording of adhesive-type notes in paper documents.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present invention, together with other and further objects, advantages and capabilities thereof, reference is made to the accompanying drawings, which are incorporated herein by reference and in which:

FIG. 1 is a block diagram of a digital audio workstation in accordance with the invention;

FIG. 2 shows the display screen organization used in the digital audio workstation;

FIG. 3 shows the timeline window used in the display screen of the digital audio workstation;

FIG. 4 shows the deck control window used in the display screen of the digital audio workstation;

FIG. 5 shows the workreel window used in the display screen of the digital audio workstation;

FIG. 6 shows the locator window used in the display screen of the digital audio workstation;

FIG. 7 shows the record settings window used in the display screen of the digital audio workstation;

FIG. 8 shows the external deck control window used in the display screen of the digital audio workstation;

FIG. 9 shows the clip editor window used in the display screen of the digital audio workstation; and

FIGS. 10A-10H illustrate edit operations performed on the display screen of the digital audio workstation.

DESCRIPTION OF THE PREFERRED EMBODIMENT

Digital recording of audio and video information has significant advantages over conventional analog recording. Although these advantages include greater precision and flexibility in the editing process, the primary benefit when video and audio are stored on digital random access media is probably the nearly instant access to any program point. Conventional recording generally stores material linearly, typically on a magnetic tape, as with a video tape recorder. Therefore, to access a particular point on the recording, the tape must be moved to that point. Digital recording, in contrast, allows the recorded information to be stored in a computer memory, where it can be accessed at random with negligible delay.

Although digital audio workstations are available which take advantage of the benefits of digital recording for audio programs, such systems remain tied to conventional linear recording for the video portions of their programs. The digital audio workstation disclosed herein uses the immediate random access possible with digital video storage to provide immediate display of the appropriate video portion of a recorded program when the audio portion is being edited. The invention eliminates the need for the editor to wait for a conventional video tape recorder to reach the correct position. This allows the editor to work more quickly and, in addition, allows the introduction of capabilities not available with current systems, as described in the following paragraphs.

The invention is a computer-based system similar in some respects to currently available editing systems. It has the typical structure of a general-purpose computer, with a central processing unit, memory, and various means for interacting with an operator. This general configuration is well-known in the art and is typified by commercially available systems such as the Avid Media Composer, manufactured by Avid Technology, Inc., Tewksbury, Massachusetts (which, however, is largely a video, rather than audio, editing system) .

A block diagram of a digital audio workstation in accordance with a preferred embodiment of the invention is shown in FIG. 1. Multiple audio inputs, which may be analog or digital, are supplied to an audio input converter and processor 10 from a multitrack audio recorder and playback unit (not shown) . The audio input converter and processor 10 can, for example, be supplied by Digidesign. The output of the audio input converter and processor 10 on line 12 is a stream of digital audio information supplied to a computer 14. The computer 14 can, for example, be an Apple Macintosh IIx, IIfx or Quadra 950. The computer 14 is connected to a monitor 16, typically having a 19 inch display screen.

Video input information is supplied from a video playback unit (not shown) in NTSC, PAL or other format to a video capture and frame store unit 20. The video capture and frame store unit 20 can be a Nuvista unit from Truevision. Digital video information is supplied through a pixel engine 22 and an expander/compressor 24 to a disk accelerator 26. The pixel engine 22 is described in application Ser. No. 07/807,433 (now U.S. Pat. No. 5,309,528) filed Dec. 13, 1991 and in U.S. Pat. No. 5,045,940, issued Sep. 3, 1991 to Peters et al, which are hereby incorporated by reference. The expander/compressor 24 compresses and decompresses video information as described in copending application Ser. Nos. 07/807,433, filed Dec. 13, 1991; (now U.S. Pat. No. 5,309,528) 07/807,117 filed Dec. 13, 1991; and 07/807,269 filed Dec. 13, 1991 and by G. K. Wallace in "The JPEG Still Picture Compression Standard", Comm. of the ACM, April 1991, Vol. No. 4, pages 31-44, which are hereby incorporated by reference. The expander/compressor 24 is preferably implemented using a CL-550 processor