 |
Claims  |
|
|
I claim:
1. A multimedia game playing system comprising:
a multimedia presentation comprised of a plurality of recorded media tracks
with at least a video track that is clocked in accordance with time
address points of a given time axis;
a game player unit having user input means for entering user input for
playing a game on said game player unit, game playing means including
programmed game sequence means for playing a programmed game sequence on
said game player unit incorporating video images therewith in conjunction
with the user input received through said user input means, and output
means including at least a video display screen for displaying video
images of the programmed game sequence in conjunction with the user input,
wherein said game player unit has associated therewith multimedia handling
means including at least video input means for receiving said video track
of said multimedia presentation and video output means for displaying said
video track on said video display screen,
wherein said video track of said multimedia presentation is divided into a
plurality of presentation segments having respective start and end time
address points with respect to the given time axis, and
wherein the programmed game sequence means of said game playing means
includes a plurality of game sequence subunits each of which is
operationally linked to the start and end time address points of a
respective one of said video track presentation segments,
wherein each of the game sequence subunits includes game choices which are
displayed in conjunction with display of the respective video track
presentation segment during a time period delimited by the start and end
time address points for the respective presentation segment, and
wherein said game playing means of said game player unit operates in a play
mode to play said presentation segments of said video track of said
multimedia presentation in a continuous sequence and to display the game
choices of the respective game sequence subunits linked thereto, and said
gate playing means receives a user input at any time during the time
periods of the multimedia presentation segments played in the continuous
sequence and enters the user input as a game choice for a respective game
sequence subunit linked to the corresponding presentation segment played
in the time period during which the user input is received,
whereby the programmed game sequence is played on the game player unit by
user input of selections for the game choices in conjunction with the
display of the game choices of the game sequence subunits together with
the display of the video track presentation segments of the multimedia
presentation.
2. A multimedia game playing system according to claim 1, wherein said game
player unit is a computer terminal, and said multimedia presentation is
input to said computer terminal through one or more devices of the group
comprising a CD-ROM unit, a magnetic disk storage unit, a laser disk
storage unit, and an audio/video input device providing the multimedia
presentation in digital format or through a digital converter.
3. A multimedia game playing system according to claim 1, wherein said
multimedia presentation includes a video motion track, an audio track, and
a text track synchronized together with respect to the given time axis.
4. A multimedia game playing system according to claim 1, wherein the
presentation segments of said video track are defined with contiguous
start and end time address points, and the time segments are ordered for
sequential presentation.
5. A multimedia game playing system according to claim 1, wherein said
multimedia presentation is a movie presentation having synchronized motion
video, sound audio, and dialog text tracks, the time segments are dialog
sequence units, and said game game sequence subunits include means for
inputting user choices for the dialog text and for providing feedback by
comparison of the user's choices for dialog text with the dialog text
track.
6. A multimedia game playing system according to claim 1, wherein said
multimedia presentation is a remote movie presentation, and said system
further comprises a receiver unit for receiving said movie presentation
remotely and downloading it to said game player unit.
7. A multimedia game playing system according to claim 1, wherein said game
sequence subunits of said programmed game sequence includes user feedback
associated with user selection of game choices for display on said video
display screen.
8. A multimedia game playing system according to claim 7, wherein said user
feedback includes means for evaluating a user input of selection of game
choices and displaying a value or response associated with the user
selection.
9. A multimedia game playing system comprising:
a multimedia presentation comprised of a plurality of recorded media tracks
with at least a video track that is clocked in accordance with time
address points of a given time axis;
a game player unit having user input means for entering user input for
playing a game on said game player unit, game playing means including
programmed game sequence means for playing a programmed game sequence on
said game player unit incorporating video images therewith in conjunction
with the user input received through said user input means, and output
means including at least a video display screen for displaying video
images of the programmed game sequence in conjunction with the user input,
wherein said game player unit has associated therewith multimedia handling
means including, at least video input means for receiving said video track
of said multimedia presentation and video output means for displaying said
video track on said video display screen,
wherein said video track of said multimedia presentation is divided into a
plurality of presentation segments having respective start and end time
address points with respect to the given time axis, and
wherein the programmed game sequence means of said game playing means
includes a plurality of game sequence subunits which are operationally
linked to the start and end time address points of respective ones of said
video track presentation segments, and wherein the game sequence subunits
includes game choices which are displayed in conjunction with display of
the respective video track presentation segments during a time period
delimited by the start and end time address points for the respective
presentation segments,
wherein the programmed game sequence is played on the game player unit by
user input of selections for the game choices in conjunction with the
display of the video track presentation segments of the multimedia
presentation, and
wherein said multimedia presentation is a music video presentation having
synchronized motion video, music audio, and lyric text tracks, the time
segments are measures of a song, and said game sequence subunits include
means for allowing the input of user choices for the lyrics for each
measure of the song in conjunction with displaying of the motion video
track and audio playing of the music audio track for the song and for
providing feedback by comparison of the user's choices for the lyrics with
the lyric text track.
10. A multimedia game playing system comprising:
a multimedia presentation comprised of a plurality of recorded media tracks
with at least a video track that is clocked in accordance with time
address points of a given time axis;
a game player unit having user input means for entering user input for
playing a game on said game player unit, game playing means including
programmed game sequence means for playing a programmed game sequence on
said game player unit incorporating video images therewith in conjunction
with the user input received through said user input means, and output
means including at least a video display screen for displaying video
images of the programmed game sequence in conjunction with the user input,
wherein said game player unit has associated therewith multimedia handling
means including at least video input means for receiving said video track
of said multimedia presentation and video output means for displaying said
video track on said video display screen,
wherein said video track of said multimedia presentation is divided into a
plurality of presentation segments having respective start and end time
address points with respect to the given time axis, and
wherein the programmed game sequence means of said game playing means
includes a plurality of game sequence subunits which are operationally
linked to the start and end time address points of respective ones of said
track presentation segments, and wherein the game sequence subunits
includes, game choices which are displayed in conjunction with display of
the respective video track presentation segments during a time period
delimited by the start and end time address points for the respective
presentation segments,
wherein the programmed game sequence is played on the game player unit by
user input of selections for the game choices in conjunction with the
display of the video track presentation segments of the multimedia
presentation, and
wherein said multimedia presentation is a karaoke presentation having
synchronized motion video, music audio, and song lyrics tracks, the time
segments are measures of a song, and said game game sequence subunits
include means for inputting a user voice audio track in conjunction with
displaying of the motion video track and audio playing of the music audio
track for the song and for providing feedback by comparison of the user's
voice audio track to the music audio track.
11. A multimedia game authoring system comprising:
a multimedia presentation comprised of a plurality of recorded media tracks
with at least a video track that is clocked in accordance with time
address points of a given time axis;
a game player unit having at least video input means for receiving said
video track of said multimedia presentation, user input means for entering
user input for playing a programmed game sequence on said game player
unit, game playing means for playing in a play mode said presentation
segments of said video track of said multimedia presentation in a
continuous sequence and a programmed game sequence incorporating video
images therewith in conjunction with the user input received through said
user input means, and output means including at least a video display
screen for displaying the video track of the multimedia presentation in
conjunction with the video images of the programmed game sequence and user
input; and
an authoring system having user interface means for dividing at least said
video track of said multimedia presentation into a plurality of
presentation segments having respective start and end time address points
on the time axis, for linking each of a plurality of game sequence
subunits of said programmed game sequence to start and end time address
points of a respective one of said video track presentation segments, for
defining game choices for each of the game sequence subunits which are to
be displayed in conjunction with display of the respective video track
presentation segment during a time period delimited by the start and end
time address points for the respective presentation segment, and for
assigning a value or feedback response to each of the defined game choices
which may be selected by a user,
said authoring system having time addressing means for defining the time
when a user input is received by the game playing means during the time
periods of the multimedia presentation segments played in the continuous
sequence and for addressing a respective game sequence subunit linked to
the corresponding presentation segment having the time period during which
the user input is received.
12. A multimedia game authoring system according to claim 11, wherein said
game player unit and said authoring tool are combined in a computer
terminal, and said multimedia presentation is input to said computer
terminal through one or more devices of the group comprising a CD-ROM
unit, a magnetic disk storage unit, a laser disk storage unit, and an
audio/video input device providing the multimedia presentation in digital
format or through a digital converter.
13. A multimedia game authoring system according to claim 11, wherein said
multimedia presentation includes a video motion track, an audio track, and
a text track synchronized together with respect to the given time axis.
14. A multimedia game authoring system according to claim 11, wherein the
presentation segments of said video track are defined with contiguous
start and end time address points, and the presentation segments are
ordered for sequential presentation.
15. A multimedia game authoring system according to claim 11, wherein said
authoring tool includes means for defining user feedback associated with
user selection of corresponding game choices for the game sequence
subunits of said programmed game sequence.
16. A multimedia game authoring system according to claim 15, wherein said
means for defining user feedback includes means for defining a value or
response associated with the user selection for display on said video
display screen. |
|
 |
|
 |
Claims |
|
|
|
Description |
|
|
1. BACKGROUND OF THE INVENTION
Movies, music videos and karaoke systems are some of the most successful
forms of modern entertainment. Movies and music videos are essentially
pre-recorded multimedia presentations with several synchronized tracks. A
track is a recorded sequence of specific media type, such as motion video,
audio, and text. (Examples of text tracks include movie subtitles and
closed captioning.)
Karaoke systems are also multimedia presentations with prerecorded video
and music audio tracks synchronized with a text display of lyrics,
allowing the user to create a new vocal track in real time by singing into
the microphone.
Video and electronic games are another popular entertainment form which
uses video images, sound, and sometimes text to challenge a player to
successfully execute a winning sequence of actions within a given time.
Most video games impose time limits on the user's input. Often, the
sequencing of video scenes is determined by the player's input. Some
computerized games have been tied to real time constraints, such as for
playing simultaneously with real-time events.
Other types of computerized games used for educational purposes include
narrative or task sequences based on multiple choice, composition,
fill-in-the-blanks or other related tasks. There are also instructional
games based on motion video presentation which are accompanied by a text
track, such as family of products from HyperGlot.TM. or subtitled films
used for language instruction.
The object of the present invention is to combine the high entertainment
value of movies, music videos, and other forms of multimedia presentations
with the functional value of video and computerized games for educational
and instructional tools.
2. SUMMARY OF THE INVENTION
The present invention provides an interactive system for playing a game,
educational or instructional sequence in conjunction with a prerecorded
multimedia presentation consisting of at least motion video, sound and
accompanying text. The multimedia presentation is partitioned into time
segments. The game sequence is also partitioned into task units that are
synchronized with the time segments. The time address markers for the
beginning and ending of each of the time segments initialize, execute and
close each task unit of a programmed game sequence.
This system can be used for motivating instructional games based on music
video, movies, animated cartoons and karaoke that provide the player with
the same excitement and challenge of the popular video games. These games
can contain educational contents and be used for teaching and training
reading comprehension and foreign language instruction. Similarly,
motivating tests based on this concept can accompany training in other
subjects. The invention can be the basis of many products in the emerging
"edutainment" industry that combines education and entertainment.
The invention also includes a corresponding authoring system which allows
an author to design and implement the time-segmentation of the multimedia
presentation and its functional interface with the game sequence.
2.1. DESCRIPTION OF THE DRAWINGS
The invention is described below in conjunction with the appended drawings,
as follows:
FIG. 1 shows the overall system architecture.
FIG. 2 shows the linking of multimedia presentation to the multimedia game.
FIG. 3a shows the overall control flow diagram of basic Multimedia Game.
FIG. 3b shows the control flow diagram of explore mode in basic Multimedia
Game.
FIG. 3c shows the control flow diagram of game mode in basic Multimedia
Game.
FIG. 4 shows the basic production process of the Multimedia Game.
FIG. 5 shows the screen layout of the Music Video Game.
FIG. 6 shows the screen snapshot from the Music Video Game.
FIG. 7a shows the control flow diagram of explore mode in Music Video Game.
FIG. 7b shows the control flow diagram of game mode in Music Video Game.
FIG. 8a shows the global control flow diagrams of game mode in Karaoke
Game.
FIG. 8b shows the control flow diagrams of game mode in Karaoke Game
(cont.).
FIG. 8c shows the control flow diagrams of game mode in Karaoke Game
(cont.).
FIG. 9a shows the overall control flow diagram of Remote Movie Game.
FIG. 9b shows the control flow diagram of game mode in Remote Movie Game.
FIG. 10 shows the screen layout of the Remote Movie Gain.
FIG. 11 shows the modules of Segmentation Authoring System.
FIG. 12 shows the screen snapshot from the Segmentation Authoring System.
FIG. 13a shows the graphical pane and its icons.
FIG. 13b shows the graphical pane when editing the end point.
FIG. 14 shows the lower panes of the segment specification area.
FIG. 15 shows the menu commands of Segmentation Authoring System.
3. DETAILED DESCRIPTION OF INVENTION
3.1. Overall System Architecture
Referring to FIG. 1, the overall system architecture preferably includes:
computer terminal or game player component
mass storage:
hard disk
CD-ROM or tape
multimedia input sources, including:
VCR+video frame digitizer
CD-ROM
audio player/recorder
laser disk player
keyboard or other means of text entry
microphone
sound or music input device with audio digitizer
input/game controls:
keyboard
mouse, joystick or other pointing device
voice command microphone, speech digitizer
output/display devices:
monitor
audio speakers
printer
recorder, VCR, audio
This system architecture can be used for production as well as for replay.
The computer can also be in the form of a game player (e.g. Sega.TM.)
connected to a TV monitor. The computer preferably incorporates hardware
that can either digitize audio and video (e.g. audio and frame digitizer
boards) or play it directly through the speakers and in a window on the
screen or a separate video monitor. If the multimedia game is used without
recording, the camera/video/audio recording equipment and microphone
components are not needed. Depending on the storage medium used for
distributing the game, a CD-ROM drive, hard disk, laser disk player or
another device is used. If broadcast services or network distribution is
used, then a modem, Ethernet, or connection to an electronic gateway is
provided.
3.1 Time Segmenting of Multimedia Presentation
In the basic process of this invention, a multimedia presentation is an
arbitrary network of prerecorded media sequences where at least one
sequence is dependent on time (i.e. the sequence is clocked while the
presentation is playing.) The time-dependent sequence is composed of
tracks that are synchronized with respect to a common time axis
(hereinafter "multimedia presentation"). The basic track consists of video
display images and is synchronized with at least one other track that
consists of audio or text display. The types of video tracks include, but
are not limited to, motion video, animation, or a sequence of still
images, such as a slide show. The types of audio tracks include, but are
not limited to, speech (such as dialog), song, or music. The types of
textual track include, but are not limited to, the transcript of a sound
track, such as transcript of movie dialog, song lyrics, a text narrative,
or other textual works.
The synchronization among the individual tracks can be either continuous or
discrete. In continuous synchronization, a track starts at a given point
of time and continues until a given point of time is reached (e.g. motion
video and audio). In discrete synchronization, a media asset is displayed
at a given point of time (e.g. still image appears, text section is
displayed or highlighted) and its display is altered at another given
point of time (e.g. still image disappears, text section is erased or
de-highlighted).
The multimedia presentation is segmented with respect to specific beginning
and ending points of segments on the time axis, i.e. there are one or more
points of time that partition the time axis into time segments. Typically,
the multimedia presentation can be played either as a whole or in a
segment-dependent fashion, for instance, when a pause or another media is
played in between two time segments, or when the user is allowed to select
and play an arbitrary time segment.
Other tracks of any other media format, such as motion video, audio,
sequence of still images, or text can be associated with a multimedia
presentation and be synchronized with respect to the presentation's time
or its segments or be independent of its time axis. Such tracks can be
provided or input by the user. An example of user's own input is recorded
speech, video, or sound.
3.2. Multimedia Game Playing System
For purposes of this invention, a game, teaching, or instructional tool
(hereinafter "multimedia game") is a software program that accepts a
user's input during the time a time-segmented multimedia presentation is
played, interprets the input as a selection among possible choices,
evaluates the choice and provides the user with feedback about the value
of the choice s/he has selected.
Referring to FIG. 2, the basic functional linking of a time-segmented
multimedia presentation and a game is accomplished in the following way.
The multimedia game has two layers, 1) the multimedia presentation, and 2)
the game program. The presentation has tracks that are synchronized and
partitioned into N time segments. Without loss of generality, one can
assume that the time segments are contiguous, i.e. each time segment I
starts at time T(I-1) and ends at T(I), and the presentation starts at
time T(0) and ends at T(n). (If there is a gap in between two time
segments or two segments overlap, one can introduce an additional time
segment that represents the gap or the overlap.)
By linking to the starting and ending times of each time segment of the
presentation played, the game layer presents to the user the choices
associated with each time segment. It then interprets the user's input as
a selection of a choice, evaluates the selected choice and gives the user
a feedback that reflects the resulting value.
Referring to FIG. 3a, control of the multimedia game proceeds in the
following steps. First, initial screen elements are displayed and the
multimedia game is retrieved and initialized. Then the user can choose to
enter the "explore" mode or the "game" mode (see below). Note that while
the "explore" mode can give the user a preview of the presentation, it
does not need to be included in the multimedia game. Finally, when the
user decides to quit, the multimedia game is terminated.
Referring to FIG. 3b, the explore mode of the multimedia game proceeds in
the following way. First, the section of the text linked to the time
segment is highlighted, and the time segment of the presentation is
retrieved and played. If there is no user input, the presentation
continues playing until its end is reached. Then the text is
de-highlighted and the presentation proceeds with the next time segment.
If the user does not interact with the game, all of the time segments of
the presentation will be played in sequence. The user can, however,
interrupt the play and choose an arbitrary time segment. This can be
realized, for instance, by clicking on any desired text section. Then an
associated media asset (e.g. native speaker's pronunciation of the
segment's text) can be played and the presentation resumes starting with
the selected segment. Other forms of interaction with the user are
possible, such as playing only one segment at a time or playing the time
segments in randomly shuffled order.
The control flow diagram in FIG. 3c illustrates the core multimedia
game-playing algorithm for an example of a game in which a user selects a
choice related to the text accompanying a multimedia presentation. It
depicts the case that time segments are played in sequence without
interruption. At the beginning of each time segment, the linked section of
the text track is highlighted, and the choices available to the user for
this segment are fetched. Then these choices can be presented to the user
and the time segment of the presentation starts playing. While the time
segment of the presentation is played, the multimedia game accepts user
input and interprets it as a selection of a choice. The game program then
fetches the value of the selected choice with respect to the current time
segment and gives the user a sensory feedback of this value. Once the time
segment reaches its ending point, the text is de-highlighted and the game
proceeds with the next time segment.
If choices are presented to the user, they either can be displayed on the
monitor or played as sound. For instance, the choices can be displayed as
a menu in the "game" area of the screen. While at present only visual and
audio output are available, any future sensory output is feasible. The
choices can be presented either in sequence or several at a time. They can
dynamically change according to time or the user's input. The set of
choices can vary depending on the segment currently played and can include
"distractor" choices that do not fit into the current context.
The user can input in numerous ways, e.g. click with a mouse on a choice
within the menu, type a key that identifies a choice, select a choice with
a joystick or pointer, push a software or hardware button while a choice
is presented, identify a choice using a microphone or any other input
device. The input can also be continuous, as in the recording of the
user's voice.
The evaluation of user's choices usually depends on the duration of the
current segment. (i.e. the value assigned to at least one user's choice
changes either at the start time or at the end time of at least one
segment.) Each choice is evaluated with respect to the current time and
segment index.
The evaluation can result in one or more numbers, or an identification of a
media element (e.g. image or sound), or simply in either a "correct" or an
"incorrect" feedback, or any combination there of. The value of the user's
choice can also depend on other factors, such as the history of the user's
performance. For instance, if several choices are related to the same
segment, selecting a choice might influence the evaluation of subsequent
choices, e.g. the user might be required to select the choices in a
certain order to receive the maximum score or an incorrect choice might
decrease the value of the next correct choice.
The feedback to the user can be visual, such as incrementing a score or
uncovering part of an image. It can be audio, such as a fanfare music, or
any other sensory input. Note that a value that might be identified by
omission of feedback, for instance the game program might offer feedback
for all values except for the "incorrect" value. The user then can
interpret the lack of feedback as an evaluation of his or her choice. The
feedback can be tied to the multimedia presentation and influence its
timing and sequencing. For instance, the current track can be repeated,
the text track can be modified to reflect the user's choice or another
media track can be played before the next segment starts playing.
There are several other variations on the schema depicted in FIG. 3c. For
instance, the sequence of time segments can be shuffled instead of being
played in its natural order. The choices can be displayed one at a time
instead of in a menu. Also, all choices can be displayed in the beginning
of the game. Choice of several variants can be offered to the user to
reflect the difficulty of the game. Moreover one or more tracks of the
presentation can be altered within the game mode.
3.3. Authoring System for Linking the Multimedia Game to Its Presentation
The construction of a time-segmented multimedia presentation and its
linking with the multimedia game can be greatly simplified using a
dedicated authoring system (Segmentation Authoring System or "SAS"
hereinafter) that facilitates the identification of the points in time
where a segment starts and the point where it ends. Such a system can also
simplify the specification of additional resources linked to each time
segment, the specification of user's choices linked to each segment, and
the value and feedback linked to each choice.
Conventional video editing tools such as Premiere.TM. produced by Adobe
Systems, Inc. of Mountain View, Calif., and products of Avid Technology,
Inc., of Tewksbury, Mass., allow the author to synchronize moving images
and sound tracks. Several of these tools allow the author to specify and
export "Edit Decision Lists" (EDL) that define start and end points ("in"
and "out" points) of a scene in time units (such as SMPTE.TM. time code).
These tools aim at composing video and audio segments into one linear
sequence rather than splitting one linear sequence into consecutive time
segments. Therefore, they do not facilitate the editing of a boundary
point in relation to its neighbor time segment. While these tools allow
the author to replay only one time segment and some limit the replay to
the "working space", they do not relate the replay time to the boundary
point being edited. These tools also lack provisions for synchronization
of EDLs with other resources, such as intervals of a separate text or
other media types or events. Multimedia authoring tools such as
Authorware.TM. or Director.TM., produced by MacroMedia of San Francisco,
Calif., allow the author to place events on a time axis (called "score"),
but they have no facilities to associate events directly with time
segments.
Therefore, a modified user interface is used here for the task of defining
the start and end points of segments. For instance, the task of splitting
human speech into segments is particularly sensitive to the exact
determination of those two boundary points.
Referring to FIG. 4, the authoring production of a multimedia game proceeds
in the following steps.
1) The author produces a synchronized multimedia presentation. The
recording and synchronization can be accomplished with well known hardware
and software systems:
audio and video can be recorded with microphone, tape recorder and
camcorder, such as these produced by Sony.TM. Corp. of Tokyo, Japan
audio and video can be digitized using a video grabber board (e.g. Media
100.TM. from Data Translations, Inc. of Marlboro, Mass.) or recorded in
analog form on a laser disk (as provided by laser disk service bureaus)
if necessary, audio and video tracks can be synchronized using well known
linear editing systems such as Premiere.TM. produced by Adobe Systems,
Inc. of Mountain View, Calif.
the digital components of multimedia can be recorded on a CD-ROM (e.g.
using CD-ROM burner available from Eastman Kodak Co. of Rochester, N.Y.)
on-line services provide downloading of files (e.g. Prodigy, Compuserve or
Internet).
2) Using the SAS, the author partitions the multimedia presentation into
time segments according to predominant time units, e.g. measures of song,
sound bites, or action sequences in a movie.
3) Sections of a text track are linked to the time segments.
4) Additional media resources are linked to each of the time segments.
5) Sets of choices available to the user are specified and linked to each
of the time segments.
6) Values are linked to the choices.
7) A feedback is linked to the value assigned to each choice.
8) The multimedia game is recorded onto a mass storage media, such as
CD-ROM | | |
