 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to the field of data entry in computer systems.
2. Background Art
A typical computer system consists of a central processing unit (CPU), main memory such as random access memory (RAM), a data entry device, including a positioning device, a mass storage device such as one or more disk drives, a display and/or a
printer. In the prior art, the data entry device often consists of a keyboard, on which a user enters data by typing. The positioning device of a prior art computer system may consist of a "mouse" or other cursor positioning device.
Computer systems have also been developed that are directed to handwritten data entry rather than keyboard data entry. These systems are often characterized by the use of a pen, stylus, or other writing device, to enter handwritten data directly
on the display of the computer system. Alternatively, these systems may provide for a user to enter data on a digitizing tablet or other input device, with the image of the written input displayed on a separate computer display output device. The
writing device for entering handwritten or freestyle stroke input information is not limited to a pen or stylus, but may be any input device such as a mouse, trackball, pointer, or even a person's fingers. Such systems are not necessarily limited to
receiving data generated by human users; for example, machine generated data may also be inputted and accepted to such systems.
One class of this handwriting entry computer system that receives handwritten data input is referred to as a "pen based" computer system. In a pen based computer system, a writer can input information on a display by "writing" directly on the
display. A writing device, such as a pen or stylus, is used to enter information on the display. In a typical pen-based computer system, a user touches the stylus to the display and writes as he would on a piece of paper, by making a series of pen
strokes to form letters and words. A line appears on the display that follows the path of travel of the pen point, so that the pen strokes appear on the display as ink would appear on a handwritten page. Thus, the user can enter information into the
computer by writing on the display. These strokes are referred to herein as "ink". A related series of strokes or characters are referred to as a "block of ink".
In addition to handwriting, special functions based on input strokes are supported on pen based computer systems. For example, a writer may strike certain previously entered handwritten input by making the strokes of an "X" over the entry.
These special functional strokes or actions are referred to herein as "gestures". Pen based computers typically have a display surface that serves as both an input receiving device and as an output display device.
The display of a pen-based entry or other handwriting entry computer system may have bordered regions called "fields" that define locations or areas where data is to be entered, or that are associated with particular actions. For example, if the
software application provides a data entry display that is an information form with fields for "First Name", "Last Name", "Age", "Occupation", etc., certain information is generally associated with each field. These locations are referred to here as
"objects" or "fields". Each object has a boundary associated with it. When data is entered, the computer must determine which object is to be associated with the writing or gesture. The object intended to be associated with the data input is called
the target object. The process of identifying the target object is referred to here as targeting. The method and apparatus that performs the function of targeting is referred to here as a targeting system.
One characteristic of handwriting entry computer systems is the ability to translate original handwritten strokes of ink or blocks of ink into machine readable words or characters for display. This translation is accomplished via a "character
recognition" algorithm. The strokes of ink that form handwritten words are translated into, for example, ASCII characters. After the translation, the appearance of the displayed characters is as if they had been typed in via a keyboard.
To translate a handwritten character into a machine readable character, the handwritten character is compared to a library of characters to determine if there is a match. A description, or "template" for each character is defined and stored in
memory. Handwritten characters are compared to the stored templates. Match coefficients, reflecting how closely a handwritten character matches the template of a stored character, are calculated for each template character. The template character with
the highest match coefficient is identified. The character represented by this template provides the "best fit" for the handwritten character. If the match coefficient for the "best fit" character exceeds a predetermined minimum threshold., the "best
fit" character is adopted. If the match coefficient for the "best fit" character is less than the minimum threshold value, no translation is done. If the handwritten character cannot be translated, the character must be re-entered.
A disadvantage of current character recognition algorithms is limited accuracy. Often, handwritten characters are not translated at all or are mistranslated as an ASCII character other than the handwritten character. The mistranslated character
must then be rewritten by the user, sometimes repeatedly, until a correct translation is made.
A number of prior art handwriting recognition and translation schemes have been described in United States patents. These prior art schemes are described below.
Fukunaga, U.S. Pat. No. 4,641,354, is directed to an apparatus for recognizing and displaying handwritten characters and figures. The apparatus includes a stylus, a liquid crystal display, and handwritten character and figure recognition
means, based upon input stroke information. The Fukunaga system displays the result of recognition as it is completed, erasing from the display screen the stroke information used for recognition, and retaining on the display the stroke information which
has not yet been used for recognition. Fukunaga discusses the use of a character recognition algorithm that converts handwritten text as soon as the algorithm determines that the stroke information for a character is completed. To conserve memory, all
handwritten information is erased from the display as soon as it is recognized.
Conoval, U.S. Pat. No. 4,703,511, is directed to a handwriting input apparatus and method of determining handwriting dynamics information. The Conoval system uses a portable unit and a code generating stylus that modulates handwriting
markings, thereby embedding time code information within the writing path. The coded markings are delivered to the writing surface via electromechanical, electrothermal, or electrostatic printhead devices. After the writing is produced, an image
detection means may convert the written image to an electrical signal representation thereof for further dynamic analysis. The writing device of Conoval emits a modulated signal which is memorialized on the writing surface as the user hand writes text.
Temporal information is stored in the strokes comprising the handwritten characters. The handwritten characters are displayed as they are written. However, the characters are not line-segmented. Conoval suggests that the writing surface, on which the
characters are written, can later be provided to a character recognition algorithm for conversion to typewritten fonts.
The system described in More, U.S. Pat. No. 4,839,634, is a portable electro-optic data input/output, storage, processing, and display device responsive to hand printed text and hand drawn graphics. The More system incorporates a character
recognition method optimized for automatic and computationally efficient on-line recognition of hand printed text characters. Hand printed text characters are stored in a compact and standard format, like ASCII, for later display, processing or output
to external equipment. In one preferred embodiment, the user places hand printed characters into a text conversion area, where they are recognized, converted to typewritten fonts on a line-by-line basis, and displayed in a document viewing/processing
area. With the device of More, a user hand writes a line of characters in a text conversion area. The characters are displayed in this area as they are written. When the user activates a "CONVERT" button on the device, the entire line of handwritten
text is operated on by a character recognition algorithm. Any handwritten characters in the text conversion area are erased. The recognition output is displayed as typewritten fonts in a larger, separate document viewing/word processing area. All
editing functions are performed on the typewritten fonts in a display area which is separate from the area where data is inputted. Even though character recognition does not take place until the user initiates the recognition algorithm, the user can
enter only a single line of unconverted text at a time. More does not allow the user to work with and perform functions on the untranslated handwritten characters. Further, the user is prompted to correct any translating errors as each line is
converted. More requires untranslated, handwritten characters to be resident in a specific and limited region.
A portable keyboardless entry computer system is described in Sklarew, U.S. Pat. No. 4,972,496. The invention in Sklarew temporarily displays the hand printed characters and then translates them into pre-selected fonts by way of a character
recognition algorithm. The user is prompted to make any corrections to handwritten data as it is being inputted. The system in Sklarew also includes a word processing editor. The portable unit described in Sklarew also suffers from some of the
disadvantages of Fukunaga. The device of Sklarew concurrently displays handwritten text and the result of its recognition algorithm on the screen. The user is prompted to correct any errors before the translated characters are permanently transferred
to the screen.
Shojima U.S. Pat. No. 4,718,103, and Jeanty, U.S. Pat. No. 5,001,765, are both specifically related to character recognition algorithms.
A disadvantage of the prior art translation systems is that the character recognition and translation functions are invoked in real time, immediately after entry of a handwritten word. When the characters are not translated correctly, the user
is prompted to correct the translation or reenter the handwritten data. This interrupts the flow of data entry, increasing data entry time, frustrating the user, and decreasing the usability of handwriting entry computer systems.
SUMMARY OF THE INVENTION
The present invention provides for improved data entry for handwriting entry computer systems by disabling the automatic display of a translation of the entered data. Instead, the original ink entered as data is displayed. This is referred to
here as "deferred translation". The invention allows a user to select modal deferred translation, automatic deferred translation, and/or manual deferred translation.
Modal deferred translation provides for the automatic display of original ink input in all cases. A translated version of the input data is generated and retained. The user may elect to display either the original ink or the translated text at
a later time. In operation, a user enters data into a handwriting entry computer system, and the writing is associated with a target field or object. A translation of the entered data is performed and the untranslated "ink" is displayed. At a later
time, if desired, the user can elect to display the translated text. If there are errors or mistranslations, the user can make corrections or revert back to the ink.
Automatic deferred translation provides for the display of the original ink whenever there are errors in the translated text. If there are no errors in the translated text, it is displayed instead of the original ink. In operation, a user
enters data into the computer system, and a translation of the data is performed. If any of the characters cannot be translated, the original ink is displayed. In an optional step, if all of the characters are translated, the resultant word is compared
to a dictionary of words, numbers and symbols. If the resultant translated word does not match a dictionary entry, the original ink is displayed. If there is a match, the translated text is displayed.
In manual deferred translation, a user enters data, and a translated version of the data is displayed. The user may elect to correct the translated text, if necessary, display the original ink instead, or do nothing. This decision can be made
on a field by field basis.
The present invention retains both the original ink and translated text so that a user can elect to display either version as desired.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 illustrates a handwriting entry computer that may be used with the present invention.
FIG. 2A-2C illustrates the entry of data and the correction of data in the preferred embodiment of the present invention.
FIG. 3 is a flow diagram of the operation of the present invention.
FIG. 4 is a flow diagram of automatic deferred translation in the present invention.
FIG. 5 is a flow diagram of manual deferred translation in the present invention.
FIG. 6 is a flow diagram of the steps used to generate an ink object in the present invention.
FIG. 7 is a flow diagram of the operation of the present invention for branching of fields.
FIG. 8 is a block diagram of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
A method and apparatus for providing deferred translation of data entered into a computer system is described. In the following description, numerous specific details, such as processor type, processor speed, operating system type, etc. are set
forth in detail to provide a more thorough description of the present invention. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without these specific details. In other instances, well known
features have not been described in detail so as not to unnecessarily obscure the present invention.
The preferred embodiment of the invention is incorporated in computer software for use, for example, with the PenPoint.TM. operating system for pen-based computers, such as those manufactured by GO.TM. Corporation of Foster City, Calif.
However, the invention can be used with a wide variety of other operating systems and other computers and computer systems.
FIG. 1 illustrates an example of a handwriting entry computer of a type that may be used with the present invention. It consists of a pen sensitive display 102 disposed in a housing 101. The front of-the housing features an on/off button 110.
Input/output ports for a disk drive, communications, and a printer, a power supply socket, and a contrast control (not visible) are located along the top 111 of housing 101.
The display 102 senses the position of a special stylus 103 when the tip of the stylus is adjacent to the display. The display 102, for example, shows as display output, the outline of a form for data entry. This form can be generated, for
example, by an application program running on the pen based computer system. The form may include a number of objects or fields. For example, in the example of FIG. 1, the form includes the "First Name" field 104, the "Last Name" field 105, the
"Company" field 106 and the "Date" field 107. A user may write data directly in one or more of the fields using the stylus 103. For example, the name "Tom" is entered in the "First Name" field 104, the name "Jones" is entered in the "Last Name" field
105, and the name "International" is entered in the "Company" field 106.
Internally, the computer of FIG. 1 uses an Intel 80286 microprocessor chip running at 12 Mhz or more. Any suitable processor may be used with the present invention. Preferably, at least four megabytes of random access memory (RAM) is included..
"Gestures" are simple types of one or two stroke pen movements that invoke certain specified commands. The present invention utilizes approximately a dozen standard "gestures" that form part of its user interface. Examples of these "gestures"
are left and right brackets ("[" and "]"), often used for selecting words or a section of text, an up-caret (" "), for accessing an editing pad that can be used for data entry, single strokes or "flicks" right, left, up or down, which are used for
scrolling, and "taps" or "presses" which are used to invoke commands.
The operating system of the preferred embodiment of the present invention also provides data to an application about any writing that is done on the display with the stylus. The operating system records events such as "pen down", "pen up", pen
position, and pen movement, and makes the record of such events available to applications running on the computer system.
An example of the entering and translation of data in a handwriting entry computer system is illustrated in FIGS. 2A-2C. Referring first to FIG. 2A, an array of fields 201-209 is illustrated. These fields are areas of a display or data entry
means that are to be associated with particular data. In the example of FIG. 2A, the boundaries between the fields are illustrated. In actual implementation, however, it is not required that the boundaries be displayed. In the example, a user has
written the words "Acme Sales" into field 205. After a user has written a block of ink, such as the words "Acme Sales", the block of ink is targeted to a target field. One method of targeting is described in U.S. patent application copending,
commonly-assigned Ser. No. 07/819,450, filed Jan. 10, 1992 entitled "Targeting.".
In this scheme, the bounds of the smallest rectangle that contains all of the strokes in the block is determined. This bound is shown as bound 210 surrounding the strokes that comprise the word "Acme Sales" in FIG. 2A. The area overlap between
the bounded rectangle and every object or field touched by the rectangle is calculated. If a preselected threshold percentage of the bounded rectangle overlaps a single field or object, that field or object is identified as the target. If no one field
or object is overlapped by at least the preselected percentage area of the bounded rectangle, bounding rectangles are defined for each character or stroke in the block of data. The amount of overlap of the bounded characters is determined, and the field
having the greatest area of overlap is identified as the target field.
Any suitable targeting scheme may be utilized without departing from the scope of the present invention. For example, in an alternate embodiment, a bounded rectangle is generated for the data block and the center of the rectangle is determined.
The object overlapped by the center of the writing rectangle is identified as the target object.
After the block of ink has been targeted, a translation is performed to translate the strokes and characters of ink into translated text. In the preferred embodiment, the strokes are translated into characters (e.g., Unicode or ASCII). This
translation can be performed using any of several well known translation algorithms. In one embodiment of the present invention, a multi-tiered translation scheme is employed. This scheme is described in U.S. patent application Ser. No. copending,
commonly-assigned Ser. No. 07,795,009, filed Nov. 18, 1991, entitled "Method for Entering and Manipulating Spreadsheet Cell Data" assigned to the assignee of the present invention.
Fields in the present invention may be one of three types of data formats: text, numbers, or "ink". A field may be designated as having a particular format prior to data being entered in the field. That is, a field may be designated to be a
"text" field, a "value" field, or an "ink" field. In the preferred embodiment of the present invention, designating a field a particular format does not preclude data having a different format from being entered into the field. It does, however,
indicate what kind of data is expected to be entered into the field, thereby aiding the character-recognition process.
Regardless of the translation scheme used, there can sometimes be errors in the translation. These errors can take the form of untranslatable characters or mistranslated characters. After a translation is attempted, the translation results are
displayed to the user. Referring to FIG. 2B, the attempted translation of "Acme Sales" is displayed as "Ac?e Sole?". The translation system correctly recognizes the "A", "c" and "e" of "Acme" and the "S", "l" and "e" of "Sales". It is unable to
recognize the "m" of "Acme" and the "s" of "Sales", and incorrectly translates the "a" in "Sales" as an "o". A question mark "?" is used to represent an untranslatable character.
After the system has displayed the translation, the user may seek to correct the translation, if necessary. This is accomplished by invoking an "editing pad" for the translated text.
FIG. 2C illustrates an embodiment of an editing pad that can be invoked to correct the entry in field 205 of FIG. 2B. The editing pad 220 shown in FIG. 2C consists of a rectangular box containing a format display 221 that indicates the format of
the field (in this case, Text), an entry display 222 that displays the content of the field in a format based on the designated format of the field, and navigation buttons 223A to 223F that are used to, for example, accept an edited entry.
To edit the field entry, corrections are made with the stylus by writing on the entry display 222. Entry display 222 illustrates the existing field content in a format that depends on the format of the field. The format of field 205 is Text.
For a field that has a text format, the field content is displayed on a character by character basis in translated form, to the extent translation is possible. The same multi-tiered recognition process described above for translating handwritten field
entries is used, with the following two exceptions: (1) an entry is treated on a character by character basis, rather than as a whole, such that each recognized character is displayed regardless of whether other characters are recognized and regardless
whether all characters of an entry have the same format (or, in the case of a text entry, whether the entry is found in a dictionary) and (2) if a character is not recognized, a question mark ("?") is displayed in lieu of that character.
To correct the entry, the correct characters are handwritten over the incorrect or unrecognized characters. To replace the "?" in "Ac?e", an "m" is written over the "?" with the stylus. Occasionally, a character must be corrected more than once
before the proper translation is made. The same process used to correct the "?" in "Ac?e" can be used to correct the "o" and the "?" in "Sole?".
The need to correct translation errors interrupts the flow of data entry in prior art handwriting entry computer systems. The present invention eliminates the need to correct translation errors by enabling the user to elect to display only the
original ink as data is entered. This is referred to here as "deferred translation". The translation of entered data may take place in a background process, but only the original ink is displayed. At a later, more convenient time, the user can elect
to display translated text and perform corrections, if necessary, at that time.
The present invention provides a number of methods for disabling the automatic display of translated text. These methods are referred to as "modal deferred translation", "automatic deferred translation", and "manual deferred translation". These
methods are described below.
Modal Deferred Translation
A user may wish to prevent or disable the automatic display of translated text that is characteristic of prior art handwriting entry computer systems. For example, a user may wish to disable the automatic display of the translation of ink when
entering large volumes of data, or sequentially entering data as part of completing a sales order form, inspection sheet, medical history form, when entering data quickly, or in any other situation where the automatic display of translated text is a
barrier to data entry.
One method provided by the present invention to disable automatic translation of ink to translated text is referred to here as "modal deferred translation". In modal deferred translation, the data entry system is configured so that no translated
text is displayed unless specifically requested by the user. When modal deferred translation is enabled, only ink strokes and characters are displayed for all fields into which data may be entered.
FIG. 3 is a flow diagram illustrating the operation of modal deferred translation using the present invention. At step 301, a block of ink is collected. At step 302, the block of ink is targeted and the appropriate target field is identified.
At decision block 303, the argument "signature field?" is made. If the argument is true, the system proceeds to step 304. A signature field is never translated. A signature field is reserved for written signatures and other items that are to remain in
the written form.
If the argument at decision block 303 is false, the system proceeds to step 305 and the field is identified as a data field. At decision block 306, the argument "defer translation?" is made. If the argument is true, the system proceeds to step
307 and the block of ink is displayed as ink only. At step 308, the block of ink is translated so that if the user subsequently wishes to display the translated characters, they are available.
If the argument at decision block 306 is false, the block of ink is translated at step 309 and the result is displayed at step 310. After either of steps 310 or 308, the user may, at step 311, initiate a gesture at the target field associated
with the block of ink. This gesture may be a "tap" gesture, where the stylus or data entering device is tapped once over the target field. The gesture may be translated by a multi-tiered translation process. Alternatively, another gesture or
activating means may be used.
At decision block 312 the argument "Deferred ink here?" is made. This argument determines if the contents of the target field are currently displaying ink or translated text. If the argument is true, meaning the target field is displaying ink,
the system proceeds to step 313 and the ink is replaced with the translated text. The translated text may or may not have errors that require correction at this time. The user may elect to make corrections if necessary, or may elect to redisplay the
ink. This can be accomplished by initiating the tap gesture (or its equivalent), or by some other activating means. If the argument at decision block 312 is false, the system proceeds to step 314 and some other action is taken. This action may be the
provision of an editing pad to aid in the correction of the translated text. Alternatively, another action or actions may be taken.
Modal deferred translation may be enabled by selecting and activating a menu command or its equivalent. The user may also use a menu command to disable manual deferred translation. In one embodiment of this invention, when modal deferred
translation is disabled, subsequent entry of strokes and characters are automatically translated. Ink that had been entered while the computer system was in modal deferred translation remains displayed as ink until the user selectively indicates, on a
field by field basis, that the translated text for that target field is to be displayed.
In another embodiment of this invention, when modal deferred translation is disabled, all strokes and characters that had been entered while the computer system was in modal deferred translation are displayed as translated text. The user may, on
a field by field basis, elect to display the original ink instead.
Automatic Deferred Translation
In some circumstances, the user may wish for the translated text to be automatically displayed, but only if it is a correct translation. Otherwise, the user prefers that the original ink be displayed instead. The present invention provides a
method, referred to here as "automatic deferred translation", in which translated text is reviewed for untranslatable characters or for mistranslated words or characters. Only if there are no translation errors is the translated text displayed.
Otherwise, the original ink is displayed.
The present invention provides a two part test to implement automatic deferred translation. After the translation process has been performed, the translation is reviewed. If there are any untranslatable characters, the translated text is not
displayed. In an optional step, if each character has been translated, the word is compared to entries in a dictionary. If the translated block is not found in the dictionary, it is assumed that a mistranslation has occurred and the original ink is
displayed. The dictionary may be a system dictionary or a user-selected dictionary.
FIG. 4 is a flow diagram illustrating automatic deferred translation in the present invention. At step 401, a block of ink is collected. At step 402, the target field of the block of ink is identified and the block of ink is associated with the
target field. At decision block 403, the argument "signature field?" is made. If the argument is true, the system proceeds to step 404 and no translation takes place. If the argument at decision block 403 is false, the system proceeds to step 405 and
the block of ink is translated, (translation may be performed in the background).
At decision block 406, the argument "is there a translation error?" is made. That is, whether one or more strokes or characters are untranslatable. If the argument at decision block 406 is true, the system proceeds to step 407. At step 407,
the ink is displayed and at step 408 the attempted translation is saved for future use.
If the argument at decision block 406 is false, the system displays the text. In an optional step, the system proceeds to decision block 409. At decision block 409, the argument "Is word in dictionary?" is made. If the translated word,
although having no untranslatable characters, is not found in the dictionary, i.e., the argument is false, the system returns to step 407 where the ink is displayed and the translation is saved. If the argument at step 409 is true, an acceptable
translation has been made and the text is displayed at step 410.
Automatic deferred translation can be enabled and disabled by a menu command or some other suitable activation means. In one embodiment, after automatic deferred translation is disabled, target fields displaying ink continue to display ink. The
user can select the attempted translations on a field by field basis. In another embodime | | |
