 |
Description  |
|
|
FIELD OF THE INVENTION
The present invention relates to apparatus and methods for typing a handwritten document and for performing similar tasks. This application includes a listing in the form of microfiche appendix A comprising 3 microfiche which contain a total of
236 frames.
BACKGROUND OF THE INVENTION
Operating systems, such as Microsoft's Windows '95, enable several applications to run at the same time, as well as to display a separate user interface window for each one.
Several software applications, such as Delrina's WinFax, enable displaying hand-written and other faxed documents. As described in the following excerpt from the Delrina's WinFax Pro user guide:
"Delrina Fax PRO provides sophisticated yet easy-to-use faxing capabilities for the Macintosh. With Delrina's Fax, you can send faxes to and receive faxes from any fax machine or fax/modem. You can quickly and easily send faxes from within
Macintosh applications using the FaxWriter DA (Desk Accessory), or with FastFax. You can also automatically receive faxes, store them on your hard disk, view them at your convenience, convert them to editable text files, or forward them to another
location, without ever having to print them."
Several OCR software applications, such as ExperVision's TypeReader, enable scanning a printed paper document and identifying alphanumerical characters therein in a process known as "optical character recognition". These applications also
support proofreading tools which display the scanned and identified text, along with suspected erroneous characters next to the relevant scanned image, as described in the following excerpt from the TypeReader Pro user guide:
"The On-Screen Verifier is only available in the text view when the current page has been recognized and its page image is available.
The On-Screen Verifier is a pop-up window that displays the part of the page image corresponding to selected text.
This saves you the trouble of having to refer back to the hard copy of your document to see what was actually on the page.
When you use Proof to find and select a character or a word, the On-Screen Verifier is automatically displayed above the line containing the character or word. You can hide it by choosing Hide On-Screen Verifier from the Edit menu. When you've
selected text manually, the On-Screen verifier is not automatically displayed. You can show it by choosing Show On-Screen verifier from the Edit menu."
The disclosures of all publications mentioned in the specification and of the publications cited therein are hereby incorporated by reference.
SUMMARY OF THE INVENTION
The present invention seeks to provide improved apparatus and methods for typing a hand-written document and for performing similar tasks.
There is thus provided, in accordance with a preferred embodiment of the present invention, a method for displaying an image of an original document based on a digital representation of the original document, the method including displaying a
first image of at least a portion of an original document to a typing operator, and displaying a second image of a typed-in interpretation of at least a portion of the original document, wherein the first and second images are simultaneously visible to
the typing operator.
Further in accordance with a preferred embodiment of the present invention, the method also includes generating a digital representation of the original document.
Still further in accordance with a preferred embodiment of the present invention, the step of generating a digital representation includes the step of scanning the original document.
Further in accordance with a preferred embodiment of the present invention, the step of generating a digital representation includes the step of digitally photographing the original document.
Still further in accordance with a preferred embodiment of the present invention, the step of displaying a second image includes simultaneously displaying a second image of a typed-in interpretation of at least a portion of the original document
on the same screen on which the first image is being displayed.
Also provided, in accordance with another preferred embodiment of the present invention, is apparatus for displaying an image of an original document based on a digital representation of the original document, the apparatus including an original
document display generator operative to display a first image of at least a portion of an original document to a typing operator, and an interpretation display generator operative to display a second image of a typed-in interpretation of at least a
portion of the original document, wherein the first and second images are simultaneously visible to the typing operator.
Further in accordance with a preferred embodiment of the present invention, the apparatus also includes a digitizing tablet for generating the document wherein a digital output representing the document is provided by the digitizing tablet to the
original document display generator.
Still further in accordance with a preferred embodiment of the present invention, the document includes a handwritten document.
Further in accordance with a preferred embodiment of the present invention, the apparatus also includes a typing process manager operative to store a representation of at least a portion of the original document in association with a
representation of at least a portion of the typed-in interpretation of at least a portion of the original document.
Also provided, in accordance with a preferred embodiment of the present invention, is apparatus for monitoring a user's progress in processing a first sequence of information elements relative to a matching second sequence of information
elements, the apparatus including a tracker operative to automatically track a user's progress in inputting a first sequence of information elements thereby to monitor the user's current location within the first sequence, and a matching location display
generator operative to receive information regarding the user's current location in the first sequence from the tracker, to identify a matching location within the second sequence which matches the current location within the first sequence, and to
display the matching location.
Further in accordance with a preferred embodiment of the present invention, the apparatus also includes a scroller operative to scroll the first image in response to input other than a keyed-in scroll command
Still further in accordance with a preferred embodiment of the present invention, the scroller is operative in response to a voice command.
Additionally in accordance with a preferred embodiment of the present invention, the scroller is operative in response to a non-manual scroll command.
Still further in accordance with a preferred embodiment of the present invention, the non-manual scroll command includes a foot-activated command.
Also provided, in accordance with another preferred embodiment of the present invention, is apparatus for monitoring a user's progress in processing a first sequence of information elements relative to a matching second sequence of information
elements, the apparatus including a tracker operative to automatically track a user's progress in inputting a first sequence of information elements thereby to monitor the user's current location within the first sequence, and a matching location display
generator operative to receive information regarding the user's current location in the first sequence from the tracker, to identify, without
relying on recognition of information elements, a matching location within the second sequence which matches the current location within the first sequence, and to display the matching location.
Further in accordance with a preferred embodiment of the present invention, the matching location display generator is operative to receive corrective input from a user and to identify the matching location by adjusting the information regarding
the user's current location to take into account the corrective input.
Still further in accordance with a preferred embodiment of the present invention, the matching location display generator is operative, in response to a user's command, to display at least a portion of the second sequence of information including
the matching location.
Additionally in accordance with a preferred embodiment of the present invention, the matching location display generator is operative to constantly display at least a portion of the second sequence of information including the matching location.
Still further in accordance with a preferred embodiment of the present invention, the original document display generator includes a tracker operative to track the typing operator's progress in typing in the original document, and a matching
location display generator operative to receive information regarding the operator's progress from the tracker, to find a current location within the original document which the operator is currently typing in and to display a portion of the original
document which includes the current location.
Further in accordance with a preferred embodiment of the present invention, the tracker tracks a user's progress in processing the first sequence of information elements at least partly by counting the information elements without relying on
recognition of the information elements which may include words and/or characters and/or sets of characters such as lines or paragraphs.
Still farther in accordance with a preferred embodiment of the present invention, each set of characters includes a field and each sequence of information elements includes a spreadsheet.
Further in accordance with a preferred embodiment of the present invention, the first sequence of information elements includes a text document and the second sequence of information elements includes a graphic image.
Still further in accordance with a preferred embodiment of the present invention, the matching location display generator includes apparatus for moving a cursor through the second sequence.
Additionally in accordance with a preferred embodiment of the present invention, the user employs a software application to process the first sequence and the tracker includes an application interface operative to elicit, from the application,
information regarding the user's progress within the first sequence of information elements.
Further in accordance with a preferred embodiment of the present invention, the user employs an input device to express commands which define a desired process to be performed on the first sequence and wherein the tracker includes an input device
interface for interfacing with the input device and receiving information therefrom regarding the commands.
Also provided, in accordance with another preferred embodiment of the present invention, is a method for facilitating document generation, the method including displaying original contents of a source document, adding contents to the source
document by inputting a sequence of user-generated input elements into the source document, and generating a new document including only the sequence of user-generated input elements without the original contents.
Further in accordance with a preferred embodiment of the present invention, the adding step includes adding a subsequence of user-generated input elements to the source document which corresponds to a portion of the original contents, at a
location adjacent to the location of the portion of the original contents.
Also provided in accordance with another preferred embodiment of the present invention, is a document input system operative in conjunction with a document input application, the system including a document display device operative to display at
least a portion of a graphically represented source document, and a user input responsive display controller operative to control the document display device responsive to inputting of a user-generated document.
Further in accordance with a preferred embodiment of the present invention, the system also includes a document scanner operative to generate the graphically represented source document from a hard copy document.
Still further in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a scroller selecting the portion of the graphically represented source document displayed by the document
display device wherein the scroller is operative to scroll the source document responsive to inputting of the user-generated document.
Additionally in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a cursor mover operative to move a cursor through the source document wherein the document display device is
operative to superimpose the cursor in its current position onto the portion of the graphically represented source document which is being displayed.
Further in accordance with a preferred embodiment of the present invention, the graphically represented source document contains the same information as the user-generated document.
Still further in accordance with a preferred embodiment of the present invention, the user-generated document does not contain the same information as the graphically represented source document but is derived therefrom.
Additionally in accordance with a preferred embodiment of the present invention, the user-generated document includes a translation of the graphically represented source document into another language.
Additionally in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a cursor mover operative to move a cursor through the graphically represented document wherein the document
display device is operative to superimpose the cursor in its current position onto the portion of the graphically represented document which is being displayed.
Further in accordance with a preferred embodiment of the present invention, the source document includes a handwritten text and wherein the application includes a word-processor.
Still further in accordance with a preferred embodiment of the present invention, the document display device is operative to simultaneously display at least a portion of the user-generated document and at least a corresponding portion of the
graphically represented source document.
Also provided, in accordance with another preferred embodiment of the present invention, is a document input system operative in conjunction with an application used to input a user-generated document and a document display device operative to
display at least a portion of a graphically represented source document, the system including a user input responsive display controller operative to control the document display device responsive to inputting of the user-generated document, wherein the
user input responsive display controller includes a scroller determining the portion of the graphically represented document displayed by the document display device -wherein the scroller is operative to scroll the graphically represented document
responsive to inputting of the user-generated document, and a cursor mover operative to move a cursor through the graphically represented document wherein the document display device is operative to superimpose the cursor in its current position onto the
portion of the graphically represented document which is being displayed.
Still further in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes an optical character recognizer operative to optically recognize characters in the user-generated document.
Additionally in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a counter operative to identify and count objects inputted into the user-generated document. The objects may
include words, spaces, rows, cells within a table, cells within a spreadsheet and the like.
Further in accordance with a preferred embodiment of the present invention, the step of adding a subsequence includes the step of displaying each subsequence of user-generated input elements corresponding to a line of original contents adjacent
to the line.
Still further in accordance with a preferred embodiment of the present invention, the subsequence of user-generated input elements corresponding to a line of original contents is displayed in a space between the line and an adjacent line.
Additionally in accordance with a preferred embodiment of the present invention, the display controller is operative to select the portion of the graphically represented source document to be displayed by the document display device responsive to
inputting of the user-generated document and to a between-language factor indicating the average ratio between the number of words in a typical document in the language of the graphically represented source document and the number of words in a document
which is a translation of the typical document into the language of the user-generated document.
Further in accordance with a preferred embodiment of the present invention, the matching location display generator includes a learning capability operative to use corrective input from a user, pertaining to an incorrect match between the user's
current location in the first sequence and an identified matching location in the second sequence, in order to improve future incorrect matches between subsequent locations of the user in the first sequence and subsequent identified matching locations in
the second sequence.
Still further in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a document input application interface operative to receive, from the document input application, information
pertaining to a user's progress in inputting the user-generated document and to select the portion of the graphically represented source document to be displayed by the document display device, at least partly based on the information.
Further in accordance with a preferred embodiment of the present invention, the user-generated document is inputted by a user using an input device and wherein the user input responsive display controller includes an input device interface
operative to capture, from the input device, information pertaining to a user's progress in inputting the user-generated document and to select the portion of the graphically represented source document to be displayed by the document display device, at
least partly based on the information.
Still further in accordance with a preferred embodiment of the present invention, the source document is in a first language and the user-generated document is in a second language and wherein user input responsive display controller includes an
automatic translator operative to generate a translation of the user-generated document into the first language, and a document synchronizer operative to synchronize the documents based at least partly on the translation.
Further in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a scrolling monitor operative to monitor a user's manual scrolling operations, and a scrolling operation learner
operative to compare the manual scrolling operations with information pertaining to the user's progress in inputting the user-generated document and to learn a relationship between the rate at which the graphically represented source document should be
scrolled and the rate at which the user inputs the user-generated document.
The user generated document may include a database.
Further in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a document location analyzer operative to compare a relative location of an object which is now being inputted in
the user-generated document to the location of the same object in the source document.
Still further in accordance with a preferred embodiment of the present invention, the tracker is operative to automatically track the user's progress without employing user-supplied information other than the first sequence of information
elements.
Additionally in accordance with a preferred embodiment of the present invention, the first sequence of information elements includes a user-generated sequence of information elements.
Still further in accordance with a preferred embodiment of the present invention, the user input responsive display controller includes a scrolling monitor operative to monitor a user's manual scrolling operation which causes a current portion of
the source document to be displayed and to associate the current portion, in memory, with a current location in the user-generated document, and a manual scrolling responsive display controller operative to cause the document display device to display
the current portion of the source document when the current location in the user-generated document is returned to.
The term "document" is intended to include text documents as well as non-text documents such as databases, spreadsheets, and representations of graphic images.
The term "scrolling", as used herein, includes displaying a first screen/display window's worth of a document which is more than one screen/display window long and subsequently displaying a second screen/display window's worth of the document
which is not identical to the first screen/display window although the two may overlap. Scrolling may or may not include adjustment of the cursor position.
Scrolling may be based on a counting process which matches a counting process performed during tracking. For example, the number of typed letters, words or lines may be counted by the tracker and the same quantity may be counted out by the
scroller. For example, if the tracker finds that 215 letters, or 17 words, or 4 lines have been processed, the source document image would then be scrolled to a vicinity of the 216th letter, or 18th word, or 5th line, respectively, after performing the
object identification method of FIG. 5 to identify all letters, words or lines, respectively, in the source document image. The cursor would typically be set on the 216th letter, or 18th word, or 5th line, respectively.
If the document is a spreadsheet or has a tabular structure, rather than being a text document or other document which is sequential in character, then instead of counting, the relative location of the object currently being processed may be
employed. For example, if the tracker indicates that the user is now keying within the third field of the fourth row, then the scroller similarly proceeds to the third field of the fourth row within the source document.
BRIEF DESCRIPTION OF THE
DRAWINGS AND APPENDICES
The present invention will be understood and appreciated from the following detailed description, taken in conjunction with the drawings and appendices in which:
FIG. 1A is a prior art figure illustrating a secretary typing a handwritten hard copy document clipped onto a stand;
FIG. 1B is a pictorial illustration of apparatus constructed and operative in accordance with a preferred embodiment of the present invention in which a display of a handwritten text is provided, e.g. by scanning in the handwritten text, and in
which the handwritten text is scrolled such that the portion thereof which is included in the display window corresponds to the secretary's progress in typing;
FIG. 1C is a simplified block diagram of an information sequence synchronization and display system constructed and operative in accordance with a preferred embodiment of the present invention;
FIG. 2 is a simplified flowchart illustration of a preferred method of operation, including object identification, for the source document image processing unit of FIG. 1C;
FIG. 3 is a simplified flowchart illustration of a preferred method for performing the resolution reduction step of FIG. 2;
FIG. 4 is a simplified flowchart illustration of a preferred method for performing the dilation step of FIG. 2;
FIG. 5 is a simplified flowchart illustration of a preferred method for performing the object identification step of FIG. 2;
FIG. 6A is a simplified flowchart illustration of a preferred method for performing the page layout model superimposition step of FIG. 2;
FIG. 6B is a simplified flowchart illustration of the source document attribute learning step of FIG. 2;
FIG. 7 is a simplified flowchart illustration of a preferred method for performing the object serial number definition step of FIG. 2;
FIG. 8 is a simplified flowchart illustration of a preferred method of operation for the tracking/scrolling unit of FIG. 1C;
FIG. 9 is a simplified flowchart illustration of a preferred method for performing the pre-setting step of FIG. 8;
FIG. 10A is a simplified flowchart illustration of a preferred method for performing the tracking step of FIG. 8;
FIG. 10B is a simplified flowchart illustration of a preferred method for performing the input application checking step of FIG. 10A;
FIG. 11A is a simplified flowchart illustration of a preferred method for implementing the user command processing step of FIG. 10A;
FIG. 11B is a simplified flowchart illustration of a preferred method for performing the "store and learn from scrolling correction" of FIG. 11A;
FIG. 12 is a simplified flowchart illustration of a preferred method for performing the scrolling step of FIG. 8;
FIG. 13 is a simplified flowchart illustration of a preferred method for performing the image cursor position and scrolling offset computation step of FIG. 12 for the source document;
FIG. 14A is a simplified flowchart illustration of a preferred method for computing the serial number of the current object in the user-generated document for each of eight different synchronization sub-modes;
FIG. 14B is a simplified flowchart illustration of a preferred method for performing step 890 in FIG. 13 in which an object in the source document is found which has a particular serial number, i.e. which matches a particular object in the user
generated document;
FIG. 15A is an example of a screen display generated by a information sequence synchronization and display system constructed and operative in accordance with a preferred embodiment of the present invention which system is particularly suited to
facilitating the process of typing hand-written documents;
FIG. 15B is an example of a subsequent display generated by the same system after the user has typed the word "scanning" and has begun typing the next word, "device";
FIG. 15C is an example of a subsequent display generated by the system of FIGS. 15A and 15B for an example in which the insertion point has been moved back, by the user, to a earlier location within the user-generated document in order to correct
existing text;
FIG. 16A is an example of a screen display generated by a system constructed and operative in accordance with another embodiment of the present invention;
FIG. 16B is an example of a subsequent display generated by the same system after the user has typed the word "publishers" and has begun typing the next word, "and";
FIG. 17A is an example of a screen display generated by a information sequence synchronization and display system constructed and operative in accordance with a preferred embodiment of the present invention which system is particularly suited to
facilitating the process of generating spreadsheets;
FIG. 17B is an example of a subsequent display generated by the same system after the user has typed "3065" and has begun typing "Dan" in the next cell;
FIG. 18 is a pictorial illustration of the contents of an source document image database for a handwriting sample illustrated in FIG. 19;
FIG. 19 is a pictorial illustration of a handwriting sample;
FIG. 20 is a graph of the centers of gravity of the objects (words) in the handwriting sample of FIG. 19 and of a set of best-fit lines for the centers of gravities;
FIG. 21A is a diagram of an adjustment database;
FIG. 21B is a diagram of the adjustment database of FIG. 21A after a user has requested that the source document cursor position be corrected by advancing one object forward, from Object No. 5 to the user's present location at Object No. 6;
FIG. 21C is a diagram of the adjustment database of FIG. 21B after a user has made an additional source document cursor position correction request, namely that the cursor advance one more object forward, from Object No. 10 to the user's present
location at Object No. 11;
FIG. 22A is a screen display generated by a system constructed and operative in accordance with a preferred embodiment of the present invention, when operating in the translating or between-language factor sub-modes;
FIG. 22B is an example of a subsequent screen display generated by the system of FIG. 22A after the user has finished typing the cursor row of the document to be translated and has begun typing the next row of the document to be translated;
FIG. 23A is an example of a screen display generated by an information sequence synchronization and display system constructed and operative in accordance with a preferred embodiment of the present invention which system is operating in Between
Line sub-mode and is operative to display a portion of a source document and, between the lines thereof, portions of a user-generated document;
FIG. 23B is an example of a first subsequent screen display generated by the system of FIG. 23A after the user has finished processing one row of the document to be processed and has begun inputting the next row of the document to be processed
which row is displayed below the processed previous row;
FIG. 23C is an example of a second subsequent screen display generated by the system of FIG. 23A after the user has finished processing the document to be processed, wherein the document which has been generated has been separated from the
document to be processed;
FIG. 24 is an illustration of a landmark database useful particularly when the system shown and described herein is operating in Landmark sub-mode; and
FIG. 25 is an example of an input hand-written source document which can be processed by the computer program of Appendix A.
Attached herewith are the following appendices which aid in the understanding and appreciation of one preferred
embodiment of the invention shown and described herein:
Microfiche Appendix A is a computer listing of a software implementation of an information sequence synchronization and display system constructed and operative in accordance with a preferred embodiment of the present invention.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it
appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
FIG. 1A is a prior art figure illustrating a secretary typing a handwritten hard copy document 2, clipped onto a stand 4.
FIG. 1B is a pictorial illustration of apparatus constructed and operative in accordance with a preferred embodiment of the present invention in which a display 6 of the handwritten text is provided, e.g. by scanning in the handwritten text. The
handwritten text is scrolled such that the portion thereof which is included in the display window 6 corresponds to the secretary's progress in typing. For example, in the illustration, the secretary has typed "Faxscanner is a powerful tool for" and is
in the course of typing the next word, "today's". The handwritten text has been scrolled to the vicinity of the word "today" and the cursor is pointing to the word "today's".
FIG. 1C is a simplified block diagram of an information sequence synchronization and display system constructed and operative in accordance with a preferred embodiment of the present invention. The system of FIG. 1C includes a scanning device
10, which may be a flatbed scanner such as a ScanJet 4s by Hewlett Packard, or a digitizing device 20, such as an ArtPad II by Wacom Computer Systems. Device 10 or device 20 are both operative to generate a graphics representation 30, such as a bitmap
representation, of an original or source document 40 which is typically a hard copy and may for example be a handwritten paper. Although the source document need not necessarily comprise a handwritten document, the present specification refers to the
source document on occasion as "the handwritten document", for clarity.
A source document image processing unit 50 is operative to analyze the graphics representation 30 and to identify objects therein, such as words, lines, columns, cells and/or paragraphs. The source document image processing unit 50 is described
in detail below with reference to FIG. 2. The output of the source document image processing unit 50 preferably includes a source document image database 60 storing at least one characteristic of objects identified in the source document. An example of
contents of source document image database 60 is illustrated in FIG. 18. The output of unit 50 also preferably includes a image 70 to be displayed of at least a portion of the source document 30 as illustrated in FIGS. 15A-15C.
The output of source document image processing unit 50 is fed to a tracking and scrolling unit 80 which is operative to track a user's progress through a first sequence of information and scroll the second sequence of information accordingly.
The tracking and scrolling unit 80 typically performs the following operations:
a. tracking the user's location within a user-generated document, typically by identifying that the user's location has changed and if so, quantifying the extent of the change.
b. identifies a corresponding location within a source document by synchronizing the source document to the user-generated document using any suitable synchronization method and
c. scrolls the source document to display that corresponding location.
For example, the user-generated document may comprise a typed version of a source document which is hand-written. A suitable input device 82 such as a keyboard and/or mouse may be used to key in the user-generated document. The
tracking/scrolling unit 80 may track the user's progress in generating the user-generated document 110, e.g. by obtaining user progress information 84 directly from an input application 86 in which the user is working, or alternatively by analyzing the
operation of the input device 82. The tracking/scrolling unit 80, typically after "mapping" or learning the relevant attributes of the entire source document 40 as described below with reference to FIG. 2, scrolls to an appropriate location within the
source document 40 and adds suitable cursor and scrolling information 87 to a display 88 of the source document image 70, which appears on a display 89 such as the display screen of a computer.
Once tracking has ceased, the source document image database 60 and source document image-to-display 70 are preferably stored in association with t | | |
