Multimedia information add-on system

What is claimed is:

1. A multimedia information add-on system comprising:

screen monitoring means for detecting that an object character string is displayed on a display device by a character displaying system;

defined information means for storing a plurality of character strings and plural pieces of multimedia associated with each of the plurality of character strings; and

multimedia information outputting means for retrieving from said defined information means multimedia information which corresponds to the object character string detected by said screen monitoring means, and for displaying the retrieved multimedia information at a predetermined position on the display device, wherein said multimedia information add-on system is different from the character displaying system, and is added to the character displaying system.

2. The multimedia information add-on system according to claim 1, wherein said screen monitoring means detects that the specified information is displayed at a specified position on the display device, and wherein said multimedia information outputting means comprises:

media display information control means for receiving the specified information from said screen monitoring means and reading the multimedia information which corresponds to the specified information from said defined information means; and

media display control means for outputting for display to said display device the multimedia information read from said media display information control means.

3. The multimedia information add-on system according to claim 1, wherein the plurality of multimedia information stored in said defined information means includes layout information for use in a designation of a displaying format of multimedia information to be displayed, and actual multimedia information to be displayed.

4. The multimedia information add-on system according to claim 3, wherein

said media display control means displays actual information in a layout specified by the layout information.

5. The multimedia information add-on system according to claim 1, wherein

said screen monitoring means detects a character code outputted by the character displaying system.

6. The multimedia information add-on system according to claim 1, wherein said screen monitoring means monitors screen information for use in a determination of a necessity to refresh the screen, positions for character strings to be displayed with the multimedia information is extracted, and for monitoring the character strings.

7. The multimedia information add-on system according to claim 1, wherein

said character displaying system comprises a computer system; and

said screen monitoring means comprises a software driver for displaying and outputting data from said computer system to the display device.

8. The multimedia information add-on system according to claim 1, wherein an execution result of a first task performed by said character displaying system is detected by a second task performed by said character displaying system to output the multimedia information which corresponds to the specified information.

9. The multimedia information add-on system according to claim 8, wherein said character displaying system retrieves a data base by executing the first task, said second task detects that a result of the retrieval is displayed in a first window, and wherein the multimedia information which corresponds to the specified information is outputted to a second window.

10. A multimedia information add-on system for displaying multimedia information corresponding to a character string, comprising:

screen information generating means for generating, as defined screen information based on operator-inputted information at a setting of an environment, screen information for use in determining a necessity to refresh a screen of a display device, said screen being displayed by the character displaying system, and for generating information on a position at which the character string to be displayed with the multimedia information is extracted;

layout information generating means for determining, as defined layout information based on operator-inputted information at the setting of the environment, a position of a frame in which the multimedia information is displayed, a size of a frame, and a display format of the multimedia information;

multimedia information generating means for generating, as defined multimedia information based on operator-inputted information at the setting of the environment and based on the information on the position at which the character string to be displayed with the multimedia information is extracted, layout information and actual information on the multimedia information to be displayed;

screen monitoring means for monitoring the screen and extracting the character string to be displayed with the multimedia information according to preliminarily defined contents when an information change in at least one of the screen information and the information on the position at which the character string to be displayed with multimedia information is extracted is detected;

multimedia display information control means for retrieving, from the defined layout information and the defined multimedia information, information on the layout and multimedia information corresponding to the character string to be displayed with the multimedia information extracted from said screen monitoring means; and

multimedia display control means for analyzing information obtained from said multimedia display information control means and displaying the multimedia information on a display screen, wherein said multimedia information add-on system is configured to be attached to said character displaying system, without modifying said character displaying system.

11. The multimedia information add-on system according to claim 10, further comprising

means for defining the screen information for use in the determination of the necessity to refresh the screen of the display device and for defining the information on a position at which the character string to be displayed with the multimedia information is extracted, said means for defining setting a coordinate in a specified range of an area on a display screen, wherein a field specified with the coordinate is designated as a screen field position, said screen field position identifying the position at which the character string to be displayed with the multimedia information is extracted.

12. The multimedia information add-on system according to claim 10, wherein means for determining a necessity to refresh the screen monitors the character string in a specified range of an area indicating the position of the screen field on the display screen, and determines the necessity to refresh of the screen according to a change in the character string.

13. The multimedia information add-on system according to claim 10, wherein the character string to be displayed with the multimedia information is extracted by extracting a selected character string within a predetermined range of coordinates indicating a position at which the selected character string is extracted on the display screen.

14. The multimedia information add-on system according to claim 10, further comprising means for preliminarily setting, as defined layout information, a position and size of the frame and a display format of the multimedia information to be displayed in the frame,

wherein said means for preliminarily setting sets the position and size of the frame for each display format of a plurality of display formats, and sets the display format of the multimedia information for each type of information to be displayed in the frame.

15. The multimedia information add-on system according to claim 10, further comprising means for preliminarily setting, as the defined multimedia information the layout information and the actual information on the multimedia information to be displayed sets a layout to be used for each character string to be displayed with the multimedia information, and sets a name of a file in which actual data of each type of multimedia information to be displayed in the frame of the layout are stored.

16. The multimedia information add-on system according to claim 10, wherein said media display information control means receives character-string-to-be-displayed-with-media information from said screen monitoring means, searches the defined layout to be used corresponding to the character string information, retrieves the frame information depending on the display format, also retrieves the actual multimedia information for each type of multimedia information to be displayed in the frame, further retrieves the display position and size of the frame, and the information on the display format of the multimedia information displayed in the frame, and outputs the information to said multimedia display control means.

17. The multimedia information add-on system according to claim 10, wherein a window display system is operated by a second task, different from a first task operated by said character displaying system.

18. The multimedia information add-on system according to claim 17, wherein

said character displaying system monitors a first window, of said window display system and outputs the multimedia information corresponding to the predetermined information on a second window when the predetermined information is displayed on the first window.

19. A multimedia information add-on system for displaying multimedia information corresponding to predetermined information, said system comprising:

screen monitoring means for detecting that specified information is displayed by a character displaying system at a specified position on a screen of a display device;

defined information means for storing preliminarily defined screen monitoring information for use in a determination as to whether or not the specified information has been displayed at the specified position on the screen of the display device, and for storing preliminarily defined multimedia information to be added to the specified information; and

multimedia information outputting means for retrieving from said defined information means multimedia information corresponding to the specified information, wherein

said specified information is detected to be displayed at the specified position on the screen of the display device; and

said multimedia information corresponding to the specified information is retrieved for display from said defined information means.

20. A multimedia information add-on system, said system comprising:

a management information table containing multimedia information associated with each character string to be displayed on a screen by a character display system; and

a tool having a tool name and containing information regarding a media type, for detecting the character string displayed by the character display system, and retrieving corresponding multimedia information from said management information table using the detected character string as a key, without modifying the designated character string.

21. The multimedia information add-on system according to claim 20, wherein

selecting a character string as a key on the screen specifies multimedia information to be displayed;

in response to said specification of media information the tool name and the information regarding the media type corresponding to the selected multimedia information associated with the selected character string as a key are specified in said management information table, and the media specified as being associated with the media name are stored.

22. The multimedia information add-on system according to claim 20, wherein

said plurality of tools for outputting the multimedia information include image displaying tools, voice outputting tools, character outputting tools, and procedure tools.

23. The multimedia information add-on system according to claim 20, wherein

the character string as a key is specified in said management information table, as corresponding to a tool name and media type.

Description Submit all comments and votes

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a multimedia information add-on system capable of adding multimedia information such as static images, animation, voice, etc. to alphabetic and numeric information system when the information is displayed on a display unit.

The present invention generates defined information for each of screen information, layout information, and media information in response to an operator's entry when an environment is defined, monitors using a screen monitoring unit a change in predetermined position information during the actual operation, extracts a character string to be displayed with media information according to the contents of the defined information if a change in the information has been detected, obtains media information corresponding to the character string, that is, the name of layout to be used and actual media information to be displayed within a display frame, obtains the position and size of a frame and the display format of the media, each depending on the name of layout to be used, and displays media information corresponding to the character string to be displayed with the media information. Thus, character string information can be displayed with multimedia information added together only by mounting or installing the multimedia information add-on system without restructuring an existing system.

2. Description of the Related Art

Normally, alphabetic and numeric information (hereinafter referred to as a "character string") is used as common display information among the information displayed on the screen of a computer system. However, it has become possible to display a character string with multimedia information such as a static image, animation, voice, text, etc. with the increasing use of high-performance hardware at a lower cost. Accordingly, character strings have been displayed with the multimedia information in various fields of applications.

Conventionally, only a small precentage of personal computers and terminal units can process multimedia information. Actually, very few office computer systems having a host computer have the function of processing the multimedia information.

Actually, most office computer systems, etc. have not been provided with multimedia information add-on systems because they are designed to process character strings as their primary function. Therefore, a system can be modified to include a multimedia system only by greatly reorganizing a main system and the configuration of the entire system including a communications unit, etc. Furthermore, if conventional systems designed mainly to process alphabetic and numeric information are reconfigured to be multimedia information displaying systems, then a program for displaying each type of media information is required, and an operator's instruction is required to display each type of media information, thereby causing operational problems. Furthermore, if a plurality of types of media information are to be displayed, operational deterioration occurs due to the increase in number of operator's instructions. Additionally, no restrictions placed on display positions cause the problem that a display screen often becomes undesirably complicated.

SUMMARY OF THE INVENTION

The present invention has been developed based on the above described background, and enables multimedia information to be displayed without reconfiguring an existing system of displaying character strings. It aims at providing a multimedia information add-on system capable of detecting a predetermined character string displayed on an existing character string information display system, and automatically displaying media information and, furthermore, displaying plural types of media information simultaneously and in an optimally formatted layout without affecting operator's efficiency.

To attain the above described objects, the present invention generates defined information in response to operator's entries made at the definition of an environment, such as information for use in a determination of a change in the contents of data displayed on the display screen in an existing character displaying system, extraction position information for character strings to be displayed with media information, layout information specifying the position, size, etc. of a frame, actual media information to be displayed, etc.

The existing character displaying system carries out its task for displaying characters, and practically displays predetermined character strings on its display screen. Another task other than that executed by the existing system for displaying characters monitors a predetermined position on the display screen, reads the above described layout information and actual media information to be displayed in accordance with the character strings displayed on the predetermined position, and displays them on the display screen.

Thus, the present invention generates various types of defined information in response to an operator's entry made at the definition of an environment, and displays media information for the character strings to be displayed on the display screen of an existing character displaying system in a predetermined layout.

Thus, the present invention provides an existing character displaying system with the function of displaying multimedia information without reconfiguring the existing character displaying system or rewriting the existing program of the character displaying system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is the block diagram showing the principle of the first embodiment of the present invention;

FIG. 2 shows the configuration of the system according to the first embodiment of the present invention;

FIG. 3 shows the process performed by the screen information generating unit according to the first embodiment;

FIG. 4 shows the process performed by the layout information generating unit according to the first embodiment of the present invention;

FIG. 5 shows the process performed by the media information generating unit according to the first embodiment;

FIG. 6 shows the process performed by the control unit according to the first embodiment;

FIG. 7 shows the configuration of the office computer system operated according to the first embodiment;

FIG. 8 shows the environment defining process according to the first embodiment;

FIGS. 9A, 9B and 9C show examples of a display screen generated at the definition of an environment according to the first embodiment;

FIG. 10 shows the process performed during the actual operation according to the first embodiment;

FIGS. 11A, 11B and 11C show change of a display screen during the actual operation according to the first embodiment; and

FIG. 12 shows the configuration according to the second embodiment of the present invention.

FIG. 13 shows the contents of the management information table according to the second embodiment;

FIG. 14 is the flowchart indicating entries of media according to the second embodiment;

FIGS. 15A and 15B show practical examples (1) of the second embodiment;

FIGS. 16A and 16B show practical examples (2) of the second embodiment;

FIG. 17 shows a multimedia system according to the second embodiment;

FIG. 18 is the flowchart of the representation of multimedia (1 to 1) according to the second embodiment;

FIGS. 19A and 19B show examples of a displaying of multimedia information (1 to 1) according to the second embodiment;

FIG. 20 is the flowchart of the representation of multimedia (m to n) according to the second embodiment;

FIGS. 21A and 21B show examples of a displaying of multimedia information (m to n) according to the second embodiment; and

FIG. 22 is the flowchart of the representation of multimedia (1 to n) according to the second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention has been developed as a multimedia information add-on system capable of displaying multimedia information together with character strings by adding the add-on system to an existing character displaying system without redesigning the existing character displaying system for displaying character string information.

The first embodiment is explained below with reference to the attached drawings.

First, the fundamental principle of the present invention is explained before describing the first embodiment.

FIG. 1 is the block diagram showing the principle of the present invention (the first embodiment).

The defined information unit 3 stores defined screen information for use in determining whether or not predetermined character strings are being displayed at a predetermined position on the screen of the display device 1, and stores defined media information relating to media information added to the character strings. For example, the defined media information relating to the media information stored in the defined information unit 3 contains layout information for use in actually displaying the information and actual media information to be displayed.

The control unit 4 comprises a screen monitoring unit 5 for detecting that a predetermined character string is being displayed at a position on the screen specified with the defined information, a media display information control unit 6 for reading information on the defined screen information from the defined information unit 3 based on the detection result of the screen monitoring unit 5, and a media display control unit 7 for outputting to the display device 1 the information read by the media display information control unit 6.

Next, the operation of the multimedia information add-on system is described below.

An existing task is carried out first by selectively outputting a character string from the character displaying system 2 to display it on the display device 1, while another task is carried out such that the screen monitoring unit 5 in the control unit 4 determines whether or not a predetermined character string exists at a position specified by the defined screen information on the screen of the display device 1. Then, the media display information control unit 6 reads media information relating to the defined screen information from the defined information unit 3, and the media display control unit 7 displays the media information on the screen of the display device 1.

Thus, the multimedia information displayed by the multimedia information add-on system according to the present invention is automatically added to the character string displayed by the character displaying system, and displayed on the screen of the display device 1. Furthermore, since the present invention displays multimedia information without modifying an existing character displaying system, the existing character displaying system can be easily used as a multimedia information displaying device.

FIG. 2 shows the practical configuration of the entire system according to the first embodiment. The system according to the first embodiment mainly comprises an information generating unit 10, a defined information unit 20 generated in the information generating unit 10, and a control unit 30 for controlling a displaying of media information using the defined information in the defined information unit 20. It further comprises an existing character displaying system 40, a display device 50, etc.

The information generating unit 10 comprises three components, that is, a screen information generating unit 11, a layout information generating unit 12, and a media information generating unit 13 to appropriately determine an environment (preprocess) for displaying multimedia information. The three components generate various defined information based on the information specified by an operator (environment designer) at the setting of the environment. Processes performed by each of the components are described below.

The screen information generating unit 11 interactively defines and generates a position at which a character string is extracted for each screen in the existing character displaying system 40. The screen information generating unit 11 generates screen ID (identifier) field information and character string extraction field information based on operator-inputted information entered at the setting of an environment.

The screen ID field information indicates a field position at which a display is regenerated, and the character string contained in the field is processed as a screen ID. The character string extraction field information indicates a position at which a character string to be displayed with multimedia information is extracted. Since the information (character string extraction field information) is generated and managed for each screen ID, definition is performed for the necessary number of screens.

Next, the layout information generating unit 12 interactively defines and generates a format of a media display including the type and display format, etc. of media to be displayed. The layout information generating unit 12 defines a layout ID for each layout as shown in FIG. 4.

For each layout specified with a layout ID, the position and size of a frame (in which media information is displayed) in an existing character displaying window, and information on a media display format for the layout specified with the layout ID are generated in response to the operator-inputted information at the setting of an environment.

The information on the media display format describes the type, size, etc. of media. Each piece of information is managed with an assigned media ID. The media can be a static image, animation (digital animation and analog animation), voice, text, etc.

The layout information generating unit 12 generates a display format of media information, that is, a format in which media information is generated, and actual media information is generated by the media information generating unit 13.

As described above, the frame is provided to display media information in, and each piece of media information is displayed in the frame. The position and size of a display frame can be optionally determined relative to the position of an existing character displaying window.

The media information generating unit 13 interactively defines and generates applicable layout information indicating which layout among variations on layout generated by the layout information generating unit 12 for each character string to be displayed with media information, and actual media information to be displayed in the determined layout to be used as shown in FIG. 5.

The actual media information is displayed in the above described frame, and the data file name indicating the type of media displayed in the selected layout is defined as being assigned to a media ID defined in the frame to be displayed.

Thus, the information generating unit 10 and each piece of defined information are described by reference to FIG. 2. Next, the control unit 30 is explained.

The control unit 30 comprises three components; a screen monitoring unit 31, a media display information control unit 32, and a media display control unit 33, and extracts a character string in the existing character displaying system 40 to display multimedia information based on each piece of defined information generated by the information generating unit 10.

The above described screen monitoring unit 31 keeps checking the screen field information and the character string extraction field information on the display screen at predetermined intervals based on the defined screen information defined by the screen information generating unit 11 so as to monitor a regeneration of a display and a change in a character string to be displayed with media information. If no changes are detected in the value of the screen field information or the character string extraction field information, then it continues monitoring the information. If a change has been detected in the value of the screen field information and the character string extraction field information when the display is regenerated by the operator, etc., then a character string to be displayed with media information is extracted, and control is passed to the media display information control unit 32.

The media display information control unit 32 retrieves information required to display media from a character string to be displayed with media information received from the screen monitoring unit 31. First, it retrieves media information corresponding to the character string to be displayed with media information from defined media information 23, retrieves from defined layout information 22 applicable layout information defined in the retrieved media information, and passes the defined information to the media display control unit 33.

The media display control unit 33 analyzes the information received from the media display information control unit 32 to display media information on the display device 50. It also manages the present state of the screen. If media information is displayed corresponding to the character string being displayed on the screen, the media display control unit displays media information (new media information) corresponding to a character string to be displayed with new media information after clearing from the screen the media information corresponding to the character string being displayed on the screen.

In FIG. 2, display unit 51 in the display device 50 displays a character string to be displayed with media information, while display unit 52 displays media information corresponding to the character string to be displayed with media information. In FIG. 2, an arrow indicating data flow is written in dotted, bold, and fine lines. That is, a dotted line arrow indicates a flow of character information, a bold solid line arrow indicates a flow of media display information, and a fine solid line arrow indicates a flow of screen control information.

Thus, according to the first embodiment of the present invention, various kinds of defined information described above, that is, defined screen information, defined layout information, and defined media information, are generated by the information generating unit 10. Based on the defined information generated by the information generating unit 10, the control unit 30 controls the display of multimedia information corresponding to a character string to be displayed with media information on the display device 50. FIG. 6 shows the flow of the process mainly performed by the control unit 30. The process of converting media information after defined screen information, defined layout information, and defined media information have been generated by the information generating unit 10 is described below by referring to FIG. 6.

The screen monitoring unit 31 in the control unit 30 keeps monitoring a predetermined position on the display screen at predetermined intervals. At this time, it is assumed that the display device 50 in the existing character displaying system 40 displays screen A. Screen A displays character string w1, while media information XM corresponding to the character string XXXX to be displayed with media information is displayed.

Then, the screen monitoring unit 31 keeps monitoring at predetermined intervals the screen ID field information for use in determining a refreshing of display of information preliminarily defined by the screen information generating unit 11, and the character string extraction field information for a character string to be displayed with media information.

The position in the screen from which screen ID field information and the character string extraction field information are extracted are predetermined such that a refreshing of a display and a character string extraction field information can be appropriately determined. Practically, the coordinate of each position is predetermined, and a change in the information on the coordinate should be monitored.

If the screen monitoring unit 31 has detected a change in the screen ID field information (change in a screen ID), then it is assumed that the display screen has been regenerated to screen B. At this time, since the regeneration of a display indicates a change in the contents of media information to be displayed. If, after the regeneration of a display, a new character string to be displayed with media information is, for example, YYYY, then the position information of the new character string to be displayed with media information on the screen should be obtained first, and then the character string YYYY to be displayed with media information is extracted and control is passed to the media display information control unit 32. Even if the screen ID field information indicates no change and the display is not regenerated at all, a character string to be displayed with media information can be changed individually. That is, if the contents of data are changed although the screen ID remains the same, then "a change" is determined, a new character string to be displayed with new media information is extracted, and control is passed to the media display information control unit 32.

The media display information control unit 32 retrieves and extracts from the defined information generated by the media information generating unit 13 the media information corresponding to the character string (YYYY in the present case) to be displayed with media information received from the screen monitoring unit 31. That is, when a character string YYYY to be displayed with media information is extracted from the screen monitoring unit 31, the character string YYYY is compared with the contents of a table of character strings to be displayed with media information so as to extract from the table a pointer to an address of the media information corresponding to the character string YYYY.

Practically, an ID of layout to be used corresponding to a character string to be displayed with media information defined as media information is extracted, and media information (actual information) to be displayed according to the layout is extracted from a corresponding data file, etc. Data such as animation, etc. should be extracted taking the frame of a