Multimedia interface and method for computer system

What is claimed is:

1. In a computer system, the combination comprising:

a first data processing unit coupled to a first display apparatus;

first memory means, coupled to said first data processing unit, for storing application software, wherein said application software displays messages on said first display apparatus and also transmits video display instructions;

a second data processing unit coupled to a second display apparatus, and coupled to said first data processing unit so as to receive said transmitted video display instructions;

random access video memory means, coupled to said second data processing unit, for storing a multiplicity of digitized video images; and

a digital video controller, coupled to said second data processing unit and to said video memory means, that displays on said second display apparatus selected ones and selected sequences of said digitized video images stored in said video memory means in accordance with said video display instructions transmitted by said application software;

said application software including means for coordinating said video images displayed on said second display apparatus with said messages displayed on said first display apparatus.

2. In a computer system, the combination comprising:

a first data processing unit coupled to a first display;

first memory means, coupled to said first data processing unit, for storing application software, wherein said application software displays messages on said first display apparatus and also transmits video display instructions; wherein said application software includes means for defining a predefined set of pieces of equipment and a predefined set of maintenance tasks in conjunction with corresponding ones of said pieces of equipment;

a second data processing unit coupled to a second display, and coupled to said first data processing unit so as to receive said transmitted video display instructions;

random access video memory means, coupled to said second data processing unit, for storing a multiplicity of digitized video images, a multiplicity of said digitized video images stored in said video memory means corresponding to one or more of said pieces of equipment;

a digital video controller, coupled to said second data processing unit and to said video memory means, that displays on said second display apparatus selected ones and selected sequences of said digitized video images stored in said video memory means in accordance with said video display instructions transmitted by said application software; and

user input means for specifying one of said equipment maintenance tasks to be performed in conjunction with corresponding ones of said pieces of equipment;

said application software including software that responds to specification of an equipment maintenance task, entered via said user input means, by generating video display instructions that instruct said digital video controller to display a single stored digitized video image for certain predefined equipment maintenance tasks and a sequence of said stored digitized video images for other predefined equipment maintenance tasks; said application software simultaneously displaying on said first display messages corresponding to said video images;

whereby said computer system can be used to provide assistance in the form of video sequences and message displays to a person performing a specified equipment maintenance task.

3. The combination set forth in claim 2, wherein said user input means includes voice input and recognition means for receiving spoken user commands.

4. The combination set forth in claim 2,

further including voice output means, coupled to said first data processing unit, for converting specified text strings into synthesized speech;

said application software generating and transmitting to said voice output means specified text strings.

5. The combination set forth in claim 2, including

portable housing that houses said computer system; said portable housing enabling said computer system to be taken to and used at the place where a selected piece of equipment is located.

6. In a computer system, the combination comprising:

a first data processing unit coupled to a first display;

first memory means, coupled to said first data processing unit, for storing application software, wherein said application software displays messages on said first display and also transmits video display instructions;

a second data processing unit coupled to a second display, and coupled to said first data processing unit so as to receive said transmitted video display instructions;

random access video memory means, coupled to said second data processing unit, for storing a multiplicity of digitized video images;

a digital video controller, coupled to said second data processing unit and to said video memory means, that displays on said second display apparatus selected ones and selected sequences of said digitized video images stored in said video memory means in accordance with said video display instructions transmitted by said application software; and

an audio speaker coupled to said digital video controller;

said random access video memory means storing both digitized video images and a multiplicity of distinct digitized audio sequences;

said application software transmitting audio play instructions to said second data processing unit; said second data processing unit forwarding corresponding audio play instructions to said digital video controller; and

said digital video controller sending selected digitized audio sequences stored in said video memory means in accordance with said audio play instructions transmitted by said application software;

whereby said computer system can play audio sequences in conjunction with video sequences and message displays.

7. The combination set forth in claim 6, wherein said application software includes means for defining a predefined set of pieces of equipment and a predefined set of maintenance tasks in conjunction with corresponding ones of said pieces of equipment;

a multiplicity of said digitized video images stored in said video memory means corresponding to one or more of said pieces of equipment,

said combination further including user input means for specifying one of said equipment maintenance tasks to be performed in conjunction with corresponding ones of said pieces of equipment;

said application software including software that responds to specification of an equipment maintenance task, entered via said user input means, by generating video display instructions that instruct said digital video controller to display a single stored digitized video image for certain predefined equipment maintenance tasks and a sequence of said stored digitized video images for other predefined equipment maintenance tasks; said application software simultaneously displaying on said first display messages corresponding to said video images;

whereby said computer system can be used to provide assistance in the form of video sequences and messages to a person performing a specified equipment maintenance task.

8. The combination set forth in claim 7, wherein said user input means includes voice input and recognition means for receiving spoken user commands.

9. The combination set forth in claim 7,

further including voice output means, coupled to said first data processing unit, for converting specified text strings into synthesized speech;

said application software generating and transmitting to said voice output means specified text strings.

10. In a computer system, the combination comprising:

a first data processing unit coupled to a first display;

first memory means, coupled to said first data processing unit, for storing application software, wherein said application software displays messages on said first display and also transmits video display instructions;

a second data processing unit coupled to a second display, and coupled to said first data processing unit so as to receive said transmitted video display instructions;

random access video memory means, coupled to said second data processing unit, for storing a multiplicity of digitized video images;

a digital video controller, coupled to said second data processing unit and to said video memory means, that displays on said second display selected ones and selected sequences of said digitized video images stored in said video memory means in accordance with said video display instructions transmitted by said application software; and

a portable, handheld satellite station, having

a third data processing unit;

a third display;

communication means for communicating with said first data processing unit; and

user input means for interacting with said application software;

said first and second data processing units including means for transmitting messages and video images to said satellite station;

whereby said satellite station allows use of said computer system by a peripatetic/ambulatory user without having to move said entire computer system.

11. The combination set forth in claim 10, wherein said application software includes means for defining a predefined set of pieces of equipment and a predefined set of maintenance tasks in conjunction with corresponding ones of said pieces of equipment;

a multiplicity of said digitized video images stored in said video memory means corresponding to one or more of said pieces of equipment,

said combination further including user input means for specifying one of said equipment maintenance tasks to be performed in conjunction with corresponding ones of said pieces of equipment;

said application software including software that responds to specification of an equipment maintenance task, entered via said user input means, by generating video display instructions that instruct said digital video controller to display a single stored digitized video image for certain predefined equipment maintenance tasks and a sequence of said stored digitized video images for other predefined equipment maintenance tasks; said application software simultaneously displaying on said first display messages corresponding to said video images;

whereby said computer system can be used to provide assistance in the form of video sequences and message to a person performing a specified equipment maintenance task.

12. The combination set forth in claim 11, wherein said user input means includes voice input and recognition means for receiving spoken user commands.

13. The combination set forth in claim 11,

further including voice output means, coupled to said first data processing unit, for converting specified text strings into synthesized speech;

said application software generating and transmitting to said voice output means specified text strings.

14. The combination set forth in claim 11, including

portable housing that houses said computer system; said portable housing enabling said computer system to be taken to and used at the place where a selected piece of equipment is located.

Description Submit all comments and votes

The present invention relates to the human interface aspects of computer systems, and particularly to systems and methods for adding video and voice capabilities to expert computer systems.

BACKGROUND OF THE INVENTION

The effectiveness of many computer systems is a function not only of the application software program which interacts with the user, but also a function of how well the computer system communicates with its human users.

In terms of expert systems, effectiveness is a function not only of the knowledge applied to its task domain, but also the delivery of that knowledge to the end user. In spite of the recent strides that have been made in expert system theory and development, no one has produced a truly effective delivery vehicle for communicating with the users of these systems.

Most expert systems are either implemented on a main frame computer and accessed through a terminal, or they are ported to smaller personal computers. In either case the human-computer dialogue passes through a standard text screen or monitor in one direction, and through a standard "QWERTY" keyboard in the other direction. This keyboard/monitor interface is not an interface that optimizes system performance.

In a large number of situations, the standard keyboard/monitor mode of communication is inadequate or not suitable. For instance, in situations where the user's hands are busy, keyboard input is clearly difficult. When the user must visually concentrate on a task, it may be dangerous for the user to avert his eyes so as to look at a computer screen. In other situations, computer monitors are inadequate visual media for conveying information needed by the user. For instance, computer monitors generally cannot be used to display photographic images or moving video pictures, such as a movie sequence showing what a particular piece of equipment looks like, or how to disassemble that piece of equipment to effect a particular repair.

All of the communication modules required for a multimedia interface, such as voice recognition modules, text-to-speech voice production modules, random access video memories and video displays are commercially available. However, these multimedia communication modules have generally not been incorporated into expert systems because of the difficulty of adding multimedia features to an existing expert system, or to a system for building expert systems. That is, there was an implicit assumption that one would have to totally redesign and rebuild an expert system's knowledge base and control software in order to incorporate multimedia features.

In other word words, the primary problem has been the difficulty of integrating the multimedia modules with traditional expert systems. It is therefore an object of the present invention to provide a multimedia software interface that can be added to an expert system with minimal changes to the expert system.

Another potential problem with a multimedia expert system is that increasing the number of communication modules increases the number of ways that the system can fail. It is therefore another object of the present invention to provide a modular multimedia software interface where the failure of any input or output channel does not impact the functionality of the remaining parts of the expert system.

Still another potential problem with building a multimedia expert system is that the multimedia features would be needed or useful mostly in situations requiring that the expert system be portable. While powerful portable computers are now available, these computers generally do not include voice input, voice output, video output and a printer because it has been assumed that the inclusion of all of these features would make the resulting system nonportable.

It is therefore an object of the present invention to provide a portable multimedia expert system delivery vehicle which weighs no more than 30 pounds, and is truly portable.

SUMMARY OF THE INVENTION

In summary, the present invention provides a multimedia interface, which presents information and receives user commands, for a computer system. The multimedia interface operates in parallel with another application software module, such as an expert system. To add multimedia features to the application software module, the module is modified so as to generate multimedia commands at the same time as it displays text on a text monitor. The multimedia commands, which are held in a queue, provide additional information in the form of video images and generated speech corresponding to the displayed text. In addition, the multimedia commands are split into at least two sets: one set which is dispatched to the user substantially immediately after displaying the corresponding text, and one set which is dispatched only upon request by the user.

In the preferred embodiment, information is presented through text, graphics, video, speech production, and printed output; control inputs are made through a special-function keypad and voice recognition. The preferred embodiment is a portable expert system which fits in a single portable suitcase sized package.

BRIEF DESCRIPTION OF THE DRAWINGS

Additional objects and features of the invention will be more readily apparent from the following detailed description and appended claims when taken in conjunction with the drawings, in which:

FIG. 1 is a perspective, partially cut away view of a prototype of a portable expert system incorporating the present invention.

FIG. 2 depicts a conceptual flow chart of the flow of information in the present information.

FIG. 3 is a diagram of the queue data structure used in a multimedia interface.

FIG. 4 is a flow chart of the system cycle of an expert system showing the multimedia interface tasks added to an expert system which previously did not have such tasks.

FIGS. 5A and 5B depict the keyboard layout for the portable expert system in FIG. 1.

FIG. 6 is a block diagram of the software routines used in the preferred embodiment.

FIG. 7 is a block diagram of the hardware for an expert system incorporating the present invention.

FIG. 8 shows a side by side presentation of a video image and corresponding text generated by a preferred embodiment of the present invention.

FIG. 9 is a block diagram of a dual processor embodiment of the present invention, plus an optional satellite station remotely located from the main system.

FIG. 10 depicts the system of FIG. 9 in a housing which has width and height comparable to a standard attache case.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The preferred embodiment of the present invention is a multimedia interface for an expert system. It should be understood, however, that the present invention can be used with many types of computer systems and many types of application software programs.

Overview of the Preferred Embodiment

The preferred embodiment of the present invention shown in FIG. 1 is a portable computer system incorporating a multimedia implementation of an expert system known as The Expert Adviser for Combustion Turbines, developed under a contract with the Electric Power Research Institute. This is a knowledge-based system designed to aid maintenance electricians when troubleshooting gas turbine power plant control systems. This knowledge base contains procedural knowledge for operating, maintaining, and troubleshooting electromechanical equipment, controls, and accessories.

In a prototype of the present invention the expert system was used for ground-fault detection. The ground-fault detection task is a good application for an expert system because it is characterized by a complex network of symptoms, complicated fault-isolation logic, and a considerable variance in the success rate and time-to-repair attributable to the technician's expertise. The work is performed in an environment that is cramped and has temperature extremes, excessive noise, and poor lighting. These environmental factors add to the desirability of providing the worker with a cognitive aiding device.

In order for an expert system to be usable and useful in the above described working conditions, it needs to be portable and able to communicate with the user when the user's hands are busy and also when the user's eyes need to be focused on the task at hand rather than on a computer or video display. As a result, an expert system for ground fault detection in gas turbine plants was determined to be a good test vehicle for the present invention.

Expert systems, as well as other computer systems, accept input from and present information to the end user in the form of a dialogue. A two-way exchange of information with an on-site, real-time system enhances the end user's perception of the system as an expert consultant. The system leads the user through a sequence of procedures culminating in the arrival at some desired end state, such as the answer to a question or a particular problem, an explanation of how a particular portion of a machine or system works, or an explanation of how to repair a particular piece of equipment.

The exact sequence of procedures performed and information provided by an expert system will depend on each user's needs and proficiency, but the end state reached should be user independent. From an expert user, the system may only request the minimum information needed for task completion. For less experienced users, the system can explain its requests by providing the user with additional information about the locations of various pieces of equipment and standard procedures, and can elaborate on acceptable responses.

The inventors have determined that the media options required for the most effective interaction between the user and an expert system are:

1. Input of user-commands via:

a. fixed function keys;

b. virtual (variable) function membrane keypads;

c. voice recognition for receiving voiced user-commands.

2. Output of text, video images and speech via:

a. text display;

b. video display for user-graphics and interactive video imagery;

c. speech generator;

d. printer for printing hard copy of text and/or images.

The system's voice recognition vocabulary is redundant with the keyboard, the audio speaker is redundant with the text display, and the printer is redundant with and an enhancement to the text display. Redundancy does not mean "identical with." Each input/output mode is customized to the manner in which humans best utilize information on that particular channel. Any redundant or primary feature may be disabled without greatly degrading the functionality of the system.

Referring to FIG. 2, there is shown a conceptual flow chart of an expert system 50 incorporating the present invention. The basic components of the expert system 50, excluding the multimedia features of the present invention, are a knowledge base 52 (i.e., a database of information), an expert system software module 54, a keyboard 55, a keyboard input module 56, and a text display module 58. The keyboard 55 in the preferred embodiment includes several special function keys to facilitate ease of use, and the text display module is a high quality flat screen computer monitor of the type commonly found in portable computers.

The user can enter commands manually using the keyboard 55, and orally by speaking into a microphone 60. The oral commands are decoded by a voice input module 62 coupled to the microphone 60. As explained below, the set of oral commands accepted by the system is a superset of the commands which can be entered using the keyboard 55.

Inputs from the user are routed by the expert system software 54 to an input filter 64, which stores multimedia commands from the user in a queue 74, and passes the remaining user inputs back to the expert system software 54. As will be explained in more detail below, the queue 74 stores multimedia commands generated by both the user and the expert system software 54.

The expert system software module 54, sometimes called an inference engine, interprets the user's keyboard and voice inputs and selects a new set of information from the knowledge base 52 based on the current "location" of the user in the knowledge base and the user's inputs.

The resulting output strings generated by the expert system are processed by an output parser 72, which stores multimedia commands in the output strings on the queue 74 and passes the remainder of the output strings back to the expert system software 54 for display on the text display 58.

A dispatch routine 66 controls the execution of the multimedia commands stored in the queue 74. As shown in FIG. 2, the multimedia dispatcher 66 calls subroutines for the system's video controller 80, voice output controller 82 and printer controller 84 when these multimedia features are used. The resulting presentations are transmitted to a video screen 86, speaker 88, and printer 90.

Simplified Expert System Model.

For the purposes of this explanation, a somewhat simplified structure of the knowledge base 52 and the expert system software module 54 is assumed. In particular, referring to FIG. 2, it is assumed that the knowledge base 52 is made up of a large number of discrete records 68, 70 and so on, each of which contains text to be displayed by a text display module 58. The records in the knowledge base also contain information regarding the "context" of the record, and also some "multimedia" information which will be discussed in more detail below. In addition, it is assumed that the expert system software module 54 reacts to user inputs simply by selecting a new record or set of records in the knowledge base 52, or by staying at the currently selected record or set of records.

As will be understood by those skilled in the art, this simplified model of an expert system permits explanation of the present invention without having to consider the many variations and complexities of expert system designs. While the preferred embodiment generally follows the above described model, it should be understood that the present invention can be used with virtually any expert system as well as many types of application software programs. For example, the present invention could be used to provide multimedia features in computer assisted training programs, and database access programs. Another use contemplated by the inventors is a calibration assistance program, in which the program assists the user calibrate a particular device by providing video and audio messages as needed by the user.

Multimedia Information in Knowledge Base.

When using the present invention, the text information in the knowledge base 52 is supplemented with additional text and information for the multimedia interface. More specifically, the author of the knowledge base can embed into the text in each knowledge base record 68 a number of special multimedia commands. Three types of multimedia commands specify three types of messages to be delivered by the multimedia interface: verbal messages, text to be printed, and selected video images or video sequences.

Thus each record 68 in the knowledge base 62 can contain text to be displayed by the text display module 58 as well as messages to be presented by each of the multimedia output modules. Other special multimedia commands embedded in the knowledge base's text can instruct the multimedia interface to repeat a verbal or video message, clear the video screen, and so on.

As will be explained in more detail below, the text and multimedia commands in each record are formatted so that the information for each output module can be easily identified and separated from the other information in the record. The format and function of each of the multimedia commands is also explained in more detail below.

Embedded Multimedia Commands.

Multimedia commands are embedded in the records of the knowledge database simply by inserting the commands into the portion of the record that specifies the text to be displayed on the system's text monitor. The beginning and end of each command is marked by a special delimiter, such as the at sign, "@", or any other character Which is generally not used for other purposes. The first non-blank character following the beginning delimiter identifies the multimedia command.

The following is an example of the text in a knowledge base record, with an embedded multimedia command:

The circuit breakers on the right hand panel are as follows . . . @sLooking at the circuit panel on the right side of the cabinet, trip the third breaker from the bottom to turn off power to the . . .@

The symbols @s . . . @ denote the beginning and end of an "s" command, which denotes "Question Text", and the remainder of the command is a parameter string - explanatory text that will be spoken when this record is selected for output.

An example of a record which contains a command to clear the video screen is as follows:

Repair sequence is now complete. Close . . . @m@

The symbol @m@ denotes a command to clear the video screen.

Table 1, discussed below, shows the full set of multimedia commands used in the preferred embodiment.

Output Parser

When the expert system software module 54 selects a new record 68 or set of records, it would normally send the corresponding output string directly to a text display module 58 for formatting and displaying the text. The present invention changes that by having the expert system module 54 send the output string in the selected record or records to an output parser routine 72.

The parser 72 identifies the multimedia commands in the output string, removes each multimedia command from the output string and stores it in the queue 74, and passes the remaining text, if any, back to the expert system software module 54 for processing by the text display module 58.

More specifically, in embodiments using the above described embedded commands, the parser works as follows. The expert system software 54 passes an array of output information to the output parser 72. The output parser 72 searches for "@" command markers, and stores the string between each set of command markers at the end of the queue 74, thereby creating a new queue entry. As a result, all of the multimedia commands in the output array are stored in the queue 74. The command markers and the multimedia commands are also removed from the output array so that when the output parser 72 returns control to the expert system software 54, only text to be displayed on the text display 58 remains in the array of output information.

As will be explained in more detail below, the use of an output parser 72 as shown in FIG. 2 enables a multimedia interface to be integrated with an already existing expert system with minimal changes to the structure of the expert system's software module 54.

Queue Data Structure and Use.

Referring to FIG. 3, the queue 74 is a linked list of variable size queue entries 76 which store the multimedia commands generated by the expert system software 54.

Each queue entry 76 temporarily stores a multimedia command, i.e , an output command specifier 76a and a variable size parameter or output string 76b. In terms of data variable size items. Thus each queue entry 76 has both forward and backward pointers 76c and 76d for creating forward and backward links.

In addition, the queue 74 has two pointers called the queue header 74a and the queue tail 74b, pointing to the beginning and end of the portion of the queue 74 which is currently occupied by queue entries. These pointers are updated by the input filter and dispatch software as items are added and deleted from the queue 74.

As shown in FIG. 3, the queue 74 is supplemented by three buffers: a Question Text Buffer 77, an Explain Text Buffer 78 and a Video Buffer 79. The function of these buffers will be explained shortly.

Every multimedia command that is generated by the system is added to the end of the queue 74. This is true regardless of whether the multimedia command is generated by the expert system software or by the user input (i.e., keyboard or voice input) modules.

Multimedia Dispatch and Output Control.

It should be noted that the multimedia commands stored in the queue 74 are not immediately and automatically executed. Execution and presentation to the user are controlled by a separate dispatch routine 66.

More particularly, all of the multimedia commands stored in the queue 74 are executed and processed whenever a "dispatch" command is executed. As will be explained in more detail below, a dispatch command causes all of the items in the queue 74 to be executed in the same order that they were added to the queue.

"Executing a multimedia command" means (1) generating the voice, video or print output denoted by the command, and/or (2) updating a corresponding one of the buffers 77-79. In addition, the dispatch command causes all of the queue entries 76 to be removed from the queue 74 as they are executed. Thus, after executing a dispatch command the queue is emptied.

TABLE 1 ______________________________________ MULTIMEDIA COMMANDS Com- mand Keybd Voice Driving Symbol Function Input Input Software ______________________________________ VOICE - QUESTION MESSAGES s Speak and add text to X question buffer. S Speak and add text to X question buffer, and close buffer. Next "s" or "S" command erases question buffer. a Speak specified text, but X do not add to question buffer. m Clear question buffer. X cntrl-R Repeat generation of text X X* in question buffer VOICE - EXPLAIN MESSAGES x Add text to explain X buffer. X Add text to explain X buffer and close buffer. Next "x" and "X" will erase explain buffer. z Speak text X X X* in explain buffer. K Clear explain buffer. X VIDEO IMAGES d Show specified video X image or video sequence and store in video buffer. r Replay last video. X X X* k Clear video buffer. X PRINTED TEXT p Print specified text. X OTHER COMMANDS v Toggle voice input mode X X X* between "mike on" and "mike off". M Clear the video screen. X Z Reset all: clear question X buffer, explain buffer, video buffer and video screen. cntrl-L List Choices: X X X* Verbally list the commands that the voice input system will accept. ______________________________________ *These commands can be generated by expert system software, but should normally be generated only by user.

This method of separating the definition of what to do (i.e., storing multimedia commands) and when to do it (by executing a dispatch command) prevents the multimedia interface from slowing down the user, because it enables the user to view and respond to text presented on the text display without having to wait for the slower video and voice outputs by the multimedia interface.

Table 1 lists the full set of multimedia commands which can be stored in the queue 74 in the preferred embodiment. As shown in the table, some of the multimedia commands are generated only by the driving software (i.e., the expert system software), while others are normally generated by the user's inputs via the keyboard or voice input system. The commands which a