 |
Description  |
|
|
FIELD OF THE INVENTION
This invention relates to the field of interactive digital media systems,
and particularly to a system for using media content in an interactive
digital media program.
BACKGROUND ART
Technological development is fostering an increasing convergence of
television, multimedia programming, and computers. The creation of a
world-wide information infrastructure will support the viewing of motion
pictures, multimedia programs, and newscast events on demand. It will
provide access to telecommunications networks, databases, and information
services over long distances, as well as facilitate the instantaneous
exchanging of governmental, business, research, institutional, medical,
and personal data, and teleconferencing and sharing of documents and
information among organizations, workgroups, and individuals spread out
over wide areas. The entry point for users to this information
infrastructure is principally the interactive use of a visual display
interface to the system.
Content is essential to the value users derive from use of the system.
While much of the content being offered is newly created to take advantage
of the latest developments in technology, there is a vast base of existing
content that is non-interactive which users may desire to have access to,
particularly media content in the form of movies, videos, video
advertising, television programming, etc. However, if existing media
content is merely offered as a digitized equivalent of its existing form,
then there is little or no value added over obtaining the same content
through the current media in which it is offered. The conversion of
existing media content to interactive digital media adds value by
rendering it capable of interactivity and linking to other forms of
digital media.
The conversion of media content to interactive digital media use has
heretofore been a laborious process as conversion tools have required
developers to perform conversion tasks essentially manually. Many types of
hyperlinking tools have been developed for rendering text and graphics
materials "live" for interactive use, e.g., as discussed in Multimedia and
Hypertext, edited by Jakob Nielson, published by Academic Press, Inc.,
1995. Typically, a link is created between a word, phrase, icon, image, or
other object appearing in the display to another text file (hypertext) or
to another program or media function (hypermedia) to deepen the user's
engagement in the system. Thus, when a user clicks with a pointing device
such as a mouse on an object appearing in the screen display, an
interactive media program will pull up another file or perform another
function so as to provide the user with further information, response, or
options. A series of hyperlinks may be followed to allow the user to
pursue a subject to any desired depth or relational complexity. Such
hyperlinking tools have found valuable use for online documentation, user
assistance, interactive manuals, graphical operating systems, information
retrieval, auditing and tracking systems, authoring systems, games,
audiovisual programs, edutainment programs, etc.
However, conventional hyperlinking tools require the developer to embed
linking codes or "anchors" manually in the content file which is to be
rendered interactive. For example, if the content is a voluminous
collection of "pages" to be displayed to the user, such as for an
electronic encyclopedia, then conversion would require a large amount of
time for the developer to .embed hyperlinking codes around each text
object for each page of content. A current candidate for a universal
language for marking documents and embedding hyperlinking codes is called
Standard Generalized Markup Language (SGML). A multimedia extension to
SGML known as HyTime has been accepted by the International Standards
Organization (ISO) for marking of documents which may incorporate audio
and video media. However, even when such hyperlinking tools are used for
media content, such as a digitized video sequence, the marking of the
sequence for "live" interactive use is currently accomplished by embedding
hyperlinking codes around the object in each frame of the sequence
(typically 30 frames per second for full motion sequence).
Digital video editing tools have also been developed for painting,
coloring, sizing, altering, or otherwise editing still and motion images,
compositing multiple images, text, and sound tracks together, animating
and morphing images, compressing multimedia files for storage or
transmission, etc. However, almost all such digital media editing tools
require alteration of the underlying raw content file in order to create a
new digital media content file. In most cases, conventional editing tools
embed proprietary codes or use proprietary file formats to modify or
re-specify an existing content file. As a result, the edited media file
can only be run on compatible systems or platforms that have complementary
display, playback, or decompression tools.
SUMMARY OF THE INVENTION
It is therefore a principal object of the present invention to provide a
system for allowing media content, particularly a broad base of existing
media content, to be used as interactive digital media programs. A
specific object is to render media content to interactive use without
locking it in to any particular system or platform, i.e., without
embedding proprietary codes in the original media content. It is a further
object to provide an authoring system for developing interactive digital
media programs from media content using automated tools which can reduce
the development time.
In accordance with the main object of the present invention, a system for
allowing media content to be used as an interactive digital media program
comprises: (a) media content in the form of digital data representing a
series of successive display frames having respective frame addresses
("Frame Data"); (b) object mapping data ("N Data") specifying display
location coordinates of objects intended to be interactive as they appear
in the display frames of the media content; (c) linkages provided through
an interactive digital media (IDM) program from the objects whose display
location coordinates are specified by the N Data to respective other
functions to be performed upon user selection of the objects in
conjunction with display of the media content; and (d) a user system for
operating the IDM program in conjunction with the display of the media
content by detecting when an object appearing in one or more display
frames is selected by a user and performing the function linked by the IDM
program linkage thereto.
In accordance with the specific object of the invention, the N Data
representing the display location coordinates and frame addresses of
mapped objects are maintained separately from the Frame Data for the media
content. The media content is thus kept intact and uncorrupted by any
embedded special codes, so that it can be run (played) on any media system
or platform. The N Data are preferably in a standard format so that they
can be widely used in the creation of many types and varieties of IDM
programs.
In accordance with a further object of the invention, an authoring system
comprises an editing subsystem for editing media content in the form of
digital data representing a series of successive display frames having
respective frame addresses ("Frame Data"); (b) an object mapping subsystem
for generating object mapping data ("N Data") specifying display location
coordinates of objects intended to be interactive as they appear in the
display frames of the media content; (c) interactive digital media (IDM)
program development tools including a hyperlinking tool for establishing
linkages from objects whose display location coordinates are specified by
the N Data to other functions to be performed upon user selection of the
objects in conjunction with display of the media content; and (d) said
object mapping subsystem having an object mapping tool for generating the
display location coordinates for an object appearing in a display frame
when an author marks the object as it appears in a display frame. The
object mapping subsystem further includes an object motion tracking tool
for generating the display location coordinates for an object in motion
based upon an author marking an object as it appears in one display frame
and detection of the marked object over subsequent frames of a series of
display frames.
In a preferred network system, media content in the form of movies, videos,
and the like, is used with an interactive digital media (IDM) program by
downloading the Frame Data for the movie and the N Data for designated
"hot spots" appearing therein from a network server to a subscriber
terminal upon request. An IDM program selected for the movie is also
downloaded from the server or, alternatively, is loaded by the subscriber
in the terminal. The subscriber terminal runs the IDM program in
conjunction with display of the movie and performs the hyperlinked
functions specified in the IDM program whenever the subscriber clicks on a
"hot spot" appearing in the movie, such as with a remote control pointer.
Thus, the previously non-interactive movie is rendered as interactive
entertainment to the subscriber.
A related aspect of the invention is a disk storage format for storing the
Frame Data and the N Data. The Frame Data for the media content is stored
physically or logically separate from the N Data for the designated
objects. The disk preferably has a main sector where the Frame Data are
stored, and a smaller, outermost sector where the N Data is stored. With
this format, movie or video disks having the N Data recorded in the
outermost sector can still be played in conventional player systems which
can only playback the movie and cannot use the N Data.
The present invention is described in greater detail below, together with
its further objectives, features and advantages, in conjunction with the
following drawings:
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic drawing showing the conversion of original media
content to digital frame data.
FIG. 2 is a schematic drawing showing the generation of object mapping data
designating "hot spots" in a display frame.
FIG. 3 is a schematic drawing showing the transmission of digital data for
the original media content and object mapping data for objects therein
from a network server to a subscriber terminal.
FIG. 4 is a schematic diagram of the components of a subscriber terminal
for use in conjunction with an interactive digital media program.
FIG. 5A is a procedural diagram for an object mapping tool for generating N
Data for objects in a display frame, FIG. 5B is a procedural diagram for
an object motion tracking tool for generating N Data for objects in motion
over a sequence of display frames, and FIG. 5c illustrates use of the
mapping and motion tracking tools for automatically generating N Data for
an object in motion.
FIG. 6 is a schematic illustration of a disk storage format for recording
media content data with object mapping data for an interactive digital
media program.
DETAILED DESCRIPTION OF THE INVENTION
Multimedia systems have evolved to sophisticated systems today that can
support photographic quality resolution (1280.times.1024 pixels), millions
of colors on a display screen, high-fidelity audio, large-scale storage
and retrieval of still and full-motion video, large-scale arrays of memory
storage, plug-and-play interfaces to multimedia devices, and high-capacity
network linkages that can support digital video and videoconferencing from
desktop systems. For an overview of hardware and software technologies
developed for multimedia systems, reference is made to Multimedia Systems,
edited by Jessica Keyes, published by Mc-Graw Hill, Inc., 1994.
The rapid technological advances of the last decade have made digital
full-motion video available on today's desktop systems. In the next
decade, advanced network technologies and integrated multimedia
distribution systems will permit full-motion video with high-fidelity
audio to be delivered on demand to offices and homes virtually anywhere in
the world. Such advanced systems and the possibilities for their use are
described in Interactive Television: A Comprehensive Guide for Multimedia
Technologies, by Winston W. Hodge, published by McGraw-Hill, Inc., 1995.
For such future, and even current, multimedia systems, a high demand will
be placed on being able to make interactive use of the huge base of
existing content, particularly media content such as movies, videos, and
television programming.
It is projected that a primary scenario for delivery of video-on-demand
(VOD) in the future will be through an office workstation or an
interactive television set at home connected via cable, fiber, or other
high-bandwidth link to network servers of a media services company for a
local area. The interactive television set is expected to have an advanced
set-top box for handling subscribers' requests and uses of interactive
media services. Principal services which customers are expected to ask for
include program and viewing time selection, order placing, menu
navigation, home shopping, interactive games, random scenes selection, TV
set controls, and subscriber billing review. For simplicity and ease of
use, the television and set-top box should be controlled by a simple
remote device which will include a light-beam pointer for pointing to menu
choices, icons, windows, photographs, and other objects of interest
appearing on the screen. A primary application of the present invention is
to facilitate the conversion of non-interactive media content to
interactive digital media use by establishing remote-controllable objects
or "hot spots" on the television screen display for user selection.
A basic concept of the invention is the mapping of objects in digital media
presentations as "hot spots" without embedding any special codes in the
original digital media content. This is accomplished by specifying the
display location coordinates of selected objects within a frame or series
of frames of a display and their frame addresses. The display location
coordinates and frame addresses of the "hot spots" are stored as data that
are physically of at least logically separate from the media content. This
allows the original media content to be accessed and run on any system
without having to handle proprietary or platform-dependent codes. The
coordinate/address data of the "hot spots" are preferably in a standard
format that can be accessed by any interactive digital media (IDM) program
written to run with that media presentation. When the media content is
played with the IDM program, a user can select "hot spots" appearing in
the display to trigger further developments. The IDM program responds to
user selection of "hot spots" by launching further layers of display
presentations and/or triggering other program functions, such as launching
another application, initiating the operation of another system, or
connecting to an external network such as a World Wide Web.TM. page or
service on the Internet.
The following description of the invention focuses primarily on the mapping
and use of "hot spots" appearing in the visual display of a digital media
presentation. However, it should be understood that a "hot spot" can be
any object identifiable in any type of digital presentation, including a
sound or music segment or even a bodily response in virtual reality
systems.
Interactive Digital Media System Overview
In a basic implementation of the invention, as illustrated in FIG. 1,
original media content 10, such as a movie, video program, or live
television program captured by a video camera, etc., is digitized via an
analog-to-digital (A/D) converter 12 into digital data representing a
series of display frames F.sub.i, F.sub.i+2, F.sub.i+3, . . . , in a time
sequence t for display on a display screen. Each frame F has a frame
address i, i+1, i+2, . . . corresponding to its unique time position in
the sequence, and is composed of an array of pixels p.sub.i uniquely
defined by location coordinates represented by j rows and k columns in the
display area of each frame. The pixels of the frame are also digitally
defined with chrominance and luminance values representing their color and
brightness levels on the display. For full motion video, a sequence of 30
frames is typically used per second of video. Each frame is composed of an
array of pixels forming the display at the screen's given resolution,
e.g., 640.times.480 pixels at a typical VGA resolution, or 1280.times.1024
at a higher SVGA resolution. Color resolution at a high 24-bit level may
also be used. Thus, for a desktop system using a 32-bit internal data bus,
and depending on whether and what data compression scheme is used, full
motion video of 30 frames per second at full color, SVGA resolution can
have a digital data stream from about 250 KBytes to 1.2 MBytes per second.
In FIG. 2, an individual frame is illustrated showing an image of an object
A such as a face next to an object B such as the sun. In interactive use,
the user can point at (click on) the face A or the sun B to connect to
further information or a further development in the story being presented.
In accordance with the invention, the original media content is converted
to interactive use without embedding special codes in the digital data for
the frames, by mapping the "hot spots" as separate data which are used in
an interactive digital media program associated with the media content.
Thus, for the frame F.sub.i, a "hot spot" area A'(F.sub.i) is mapped for
the object A, and a "hot spot" area B'(F.sub.i) is mapped for the object
B. The definition of a "hot spot" can be made by defining a set of pixels
in the display which comprise an outline around the designated area, e.g.,
p(a.sub.j,a.sub.k) . . . . Alternatively, the area may be defined by a
vector contour encompassing the designated area, or any other suitable
array definition method as is well known in the computer graphics field.
The display location coordinates of the defined pixels and the frame
addresses of the frames in which the area appears are stored separately as
object mapping data.
The original media content is thus rendered in the form of a stream of
digital data, referred to herein as "Frame Data", which represent the
series of display frames F constituting the movie or video sequence.
Concurrently, for each frame F.sub.i, the object mapping data, referred to
herein as "N Data", are generated to define the display location
coordinates of designated "hot spot" areas in the frames of the movie or
video sequence. In accordance with a basic principle of the invention, the
N Data mapping the "hot spots" are maintained as physically or at least
logically separate data from the Frame Data for the media content. For
example, the Frame Data and the N Data may be recorded as physically
separate sectors on a video laserdisk or CD, or may be stored as logically
separate data files in the memory storage of a video server. In this
manner, the objects which are rendered interactive in the original media
content are tagged for use in a compatible interactive digital media (IDM)
program without embedding any proprietary or platform-dependent codes in
the media content. Thus, the media content data can be run on any digital
media player and the N Data can be used by any IDM program.
The N Data defining the hot spots are preferably in a format that may
become established in the industry for handling the frame addresses and
display location coordinates for the designated objects, as explained
further herein. The standard-format N Data can thus be accessed by any
interactive digital media (IDM) program written in standard applications
programming languages. In accordance with the invention, the N Data define
the location of the designated "hot spots" or "anchors" to which
hyperlinks are established in the IDM program. This is represented in FIG.
2 by "IDM PROG." which references the "hot spot" N Data values as anchors
for hyperlinks to other files or executable functions ("GO TO . . . ").
Then when a user clicks on a designated "hot spot" by pointing to any
display position encompassed within the area defined by the object mapping
data, the IDM program recognizes that the object pointed to has been
selected, and consequently causes the other file or function linked to the
"hot spot" to be performed.
Running Media Content and IDM Program from Network Server
Interactive digital media programs in accordance with the invention can be
run on any of a wide range of platforms. In large media services networks,
the media content, N Data, and associated IDM programs are downloaded via
the network to user or subscriber terminals upon request. For individual
use, the programs are loaded via peripheral devices into personal
computers, game players, or other media playing consoles. The following
description focuses on the delivery of media content and IDM programs
through networks, such as cable TV, telephone networks, digital line and
fiber optic networks, and wide area digital networks. In the future, the
prevalence of network delivery of interactive media services is expected
to increase greatly toward a paradigm often referred to as the "multimedia
revolution".
An example of network delivery of interactive digital media programs in
accordance with the invention is shown schematically in FIG. 3. Typically,
a network server 30 provides media services from a node or hub in a
company's service area. The server 30 is coupled to subscriber terminals
through a suitable data transmission link DL, such as cable wiring, fiber
optic lines, telephone wiring, or digital data links. The subscriber's
terminal is typically in the form of a "set-top" box 32 connected to the
subscribers' TV or screen display 34, but it can also be a computer or
other type of terminal. An important concept for network media services is
"video-on-demand", wherein the server 30 can access large digital
libraries of movies, videos, and other types of media content and transmit
them to subscribers upon request. The server 30 transmits both the Frame
Data for the media content and the N Data and IDM program for rendering
the "hot spots" therein interactive to the subscriber's set-top box 32 via
the data transmission link DL. The subscriber uses a remote control device
36 to operate the set. For interactive use, the remote device 36 includes
an optical pointer which emits an infrared or other light beam. As known
conventionally, a sensor 33 in the set-top box is used to detect the
position and angle of the beam from the remote control pointer in order to
detect the area of the display 34 being pointed to.
The media content with N Data delivered to the subscriber is operated
interactively by the subscriber through the IDM program. The IDM program
can be a dedicated program indexed to N Data which are specific to a
single type of interactive use of the media content. Alternatively, a
production studio or studio library which owns the media content property
may find it more effective to publish a complete listing of N Data for an
owned property which includes a mapping of all "hot spots" likely to be of
interest for interactive programs. IDM program writers can then use the
published listing of N Data to create many and more diverse program
offerings for a particular media content property. For dedicated IDM
programs, the IDM program data can be stored together with the N Data in
association with the media content and transmitted together by the server
30 to the subscriber's terminal. For multi-use IDM programs, the N Data
can be stored in association with the media content and transmitted from
the server 30, while subscribers can choose any IDM program they wish to
play from a publishing or retail outlet and load it into their terminals
via a peripheral device provided with or connected to their set-top box
32, such as a CD-ROM drive or a ROM card insertion slot.
FIG. 4 illustrates schematically how an interactive digital media system
uses the media content Frame Data, N Data, and the IDM program together to
provide interactive entertainment. The system includes the aforementioned
set-top box 32, display 34, remote control pointer 36, and data link DL to
the external network server. An on-board CD-ROM player or other data
reading device 43 may be provided with the set-top box 32 for input of
data, indicated at 45, such as by loading from a selected CD or insertable
disk or card. Input from the remote control pointer 36 is detected by the
sensor 33 on the set-top box and processed to determine its target via a
pointer detection circuit 44.
In the principal mode of use, the subscriber inputs a request to the
service company for an interactive media program through the set-top box
32, using an on-board keypad 42 or through menu selection by using the
remote control pointer 36. For example, the subscriber can request the
interactive program "Movie Trivia Info" for the movie "The Maltese
Falcon". This interactive program will run the movie while displaying
pop-up movie trivia about the stars Humphrey Bogart, Sidney Greenstreet,
and Peter Lorre or objects such as the Maltese falcon whenever the user
clicks on these "hot spots" appearing in different scenes of the film. To
the user, movie viewing which had been a passive experience is rendered
interactive so that the user can play trivia games or spark conversations
in conjunction with the running of the movie.
A console processor 40 for the set-top box processes the subscriber request
and transmits it via the data link DL to the network server 30. In return,
the server 30 first transmits the IDM program data for "Movie Trivia Info"
and the N Data for the movie to the subscriber's set-top box where the
console processor 40 operates to store the data in a console RAM memory
46. The console processor 40 can load and run the IDM program as a
multi-tasking function concurrently with other console functions, as
indicated in FIG. 4 by the separate module 41. Alternatively, the IDM
program can run on a separate processor (41) in parallel with the console
processor.
The remote downloading and playing of games and other types of interactive
programs can be used even with conventional cable TV networks which do not
presently have a two-way data link DL between server and subscribers. In
an example for video games, the cable company broadcasts modulated signals
for the game data on a dedicated cable channel. In response to a
subscriber's telephone request, the cable company transmits a signal
enabling the subscriber's converter box to receive the data. The game data
is then demodulated through a modem connector and downloaded to the
subscriber's game player. For purposes of the present invention, this
would allow loading of the IDM program and N Data in the game player. The
game player can now operate the IDM program in conjunction with the media
content, as described next.
After the IDM program is loaded, the network server 30 begins to transmit
the movie as digital Frame Data to the subscriber's set-top box 32. The
Frame Data is routed by the console processor 40 to the video processor 48
and associated video RAM memory 50 which process the display of frames of
the movie via video display output 49 to the subscriber's television 34.
Audio processing is subsumed with the video processing and is not shown
separately. For typical video-on-demand servers, a requested movie can be
transmitted to the subscriber as a series of 30-second movie blocks within
a 6-minute start of a request. The video processor coordinates the receipt
of the blocks of transmitted data into a display of video output which the
user sees as a continuous movie.
As designed for interactive video systems, the remote control 36 includes
an optical pointer for digitally pointing to objects displayed on the
television screen. As the movie runs, the user can point the remote
control pointer 36 to a designated actor or object appearing on the
television display and click on the desired object. The N Data for the
movie defines the area encompassing the object as a "hot spot". Clicking
the pointer results in the target's display location coordinates being
detected by the pointer detector module 44. The target's coordinates are
input via the console processor 40 to the IDM program running concurrently
with the movie. As indicated at box 41a, the IDM program compares the
target's coordinates to the N Data mapping of "hot spots" stored in memory
to identify when a "hot spot" has been selected, and then executes the
response programmed by the hyperlink established for that "hot spot", as
indicated at box 41b.
For example, the hyperlinked response may be to display trivia information
about the actor or object clicked on. The IDM module retrieves the trivia
information stored with the IDM program in memory and sends it to the
console processor 40 to process a pop-up window, overlay display, audio
track, etc., in conjunction with the movie. To illustrate, upon the user
clicking on the Maltese falcon, the hyperlink established in the "Movie
Trivia Info" program can initiate a linked display of text or graphics
explaining the Maltese origins of the falcon in a pop-up window on the
television screen, or may execute another program function such as
initiating an Internet connection to a World Wide Web.TM. service which
offers a replica of the falcon for purchase. In this manner, unlimited
types and varieties of interactive actions can be activated for existing
movies, videos, and other media content.
As an option, upon selection by a user clicking on an object, the IDM
program can issue an instruction via the console processor 40 to the video
processor 48 to slow down or pause the running of the movie to allow time
for the user to absorb the IDM program response. Alternatively, the user
may wish to bypass the response and store it to be reviewed after the
movie is finished. By input from the remote control pointer 36 (e.g.,
clicking on a displayed "Save" button), the particular scene location and
clicked object and/or its linked response can be saved in the console RAM
46 for retrieval during a Review mode of the IDM program, as indicated at
box 41c in FIG. 4.
Authoring and Mapping of "Hot Spots" As N Data
The mapping of "hot spots" or objects appearing in original media content
to enable the operation of an interactive digital media (IDM) program is
an important aspect of the present invention. In the production of an IDM
program, the initial | | |
