 |
Description  |
|
|
BACKGROUND OF THE INVENTION
The present invention relates generally to computer systems, and more
particularly to computer-implemented assistance methods and apparatus.
Computerized personal organizers are becoming increasingly popular with a
large segment of the population. Computerized personal organizers tend to
be small, lightweight, and relatively inexpensive, and can perform such
functions as keeping a calendar, an address book, a to-do list, etc. While
many of these functions can also be provided in conventional computer
systems, personal organizers are very well suited to the personal
organization task due to their small size and portability. Personal
organizers are available from such companies as Sharp and Casio of Japan.
A relatively new form of computer, the pen-based computer system, holds
forth the promise of a marriage of the power of a general purpose computer
with the functionality and small size of a personal organizer. A pen-based
computer system is typically a small, hand-held computer where the primary
method for inputting data includes a "pen" or stylus. A pen-based computer
system is commonly housed in a generally rectangular enclosure, and has a
dual-function display assembly providing a viewing screen along one of the
planar sides of the enclosure. The dual-function display assembly serves
as both an input device and an output device. When operating as an input
device, the display assembly senses the position of the tip of a stylus on
the viewing screen and provides this positional information to the
computer's central processing unit (CPU). Some display assemblies can also
sense the pressure of the stylus on the screen to provide further
information to the CPU. When operating as an output device, the display
assembly presents computer-generated images on the screen.
The dual-function display assemblies of pen-based computer systems permit
users to operate the computer as a computerized notepad. For example,
graphical images can be input into the pen-based computer by merely moving
the stylus on the surface of the screen. As the CPU senses the position
and movement of the stylus, it generates a corresponding image on the
screen to create the illusion that the stylus is drawing the image
directly upon the screen, i.e. that the stylus is "inking" an image on the
screen. With suitable recognition software, text and numeric information
can also be entered into the pen-based computer system in a similar
fashion.
One approach to computerized assist operations is to provide assistance
automatically when a situation in which assistance could be provided is
recognized. However, such an approach may provide unsatisfactory results
when the user is provided with assistance that is unwanted or
disproportionate.
Simply stated, concerns have arisen about assist functions being undertaken
by the computer without adequate user control and interaction. When the
assist function has been undertaken without adequate user control,
assistance would be provided awkwardly and at times when the assistance
was not necessarily desired.
It is essential that the control, the timing, and the application of the
assistance is considered appropriate and well-tuned. The confidence of the
user is undermined when the assistance provided only obliquely addresses
particularized user needs without precisely providing the particular
results objectively and subjectively required.
SUMMARY OF THE INVENTION
According to the invention, a method and apparatus has been developed for
providing computer-assisted implicit and explicit assistance for a variety
of user-supportive information functions. If no implicit assist actions
are desired or indicated, then a logical process is initiated to determine
whether explicit assistance should be undertaken. If implicit assistance
is indicated, a list of action alternatives is displayed for the user.
Alternatively, a process can be undertaken to automatically select a best
action alternative of several identified alternatives.
If explicit assistance is desired by the user, particular object(s) from
which the assistance may be inferred are entered into an assistance
operation. An attempt is then made to recognize possible intents expressed
by the objects entered into the assistance process. If no user intent is,
in fact, recognized, the assist operation is usually terminated. If a
possible intent is recognized, the actual intent is hypothesized. A check
is further undertaken, to determine whether a hypothesis is in fact
available. If no hypothesis is found, the process permits the user to
supply a proposed action. If no hypothesis is found and no user action is
proposed, assistance efforts terminate. However, if a hypothesis is
available, preparations for execution are undertaken. A final inquiry is
made as to whether to undertake the hypothesized assist. If the response
to an inquiry whether to assist as hypothesized is affirmative, execution
of the hypothesized action is undertaken.
Accordingly, it is an intent of the invention to provide user explicit
assist functions which are provided under oversight and with the
interaction of the user, and implicit assist functions in certain
structured instances. The involvement of the user in the implementation of
implicit assist operations is a key aspect of the invention. User intent
is not merely deduced, but specific user queries are made to ensure
controlled application of assist operations.
These and other advantages of the present invention will become apparent
upon reading the following detailed descriptions and studying the various
figures of the drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of a computer system in accordance with the
present invention.
FIG. 2 is a top plan view of the screen, case, and keypad of the computer
system of FIG. 1.
FIG. 3 is a flow diagram of a process according to the invention for
providing controlled computer-assisted user assistance.
FIG. 4a shows a process for determining whether or not implicit assistance
is desired by the user, in connection with providing computer assisted
support to the user.
FIG. 4b shows an example of an implicit assist operation with a phone slip
window having a smart name field evoked, for example, by either
highlighting the verb "call" or by writing it on the note field before
evoking the window.
FIG. 4c shows the phone slip window of FIG. 4b with the name ISAAC having
been recognized and established in the name field in formal font form.
FIG. 5 shows an example of a generic <PERSON> type frame along with a
particular set of specific frames of the <PERSON> type.
FIGS. 6a-6c show respective assist windows in successive stages of an
assist process, including first a window containing a first informational
level directed at the name ISAAC alone, a second window with a pop-up menu
offering a user choice among several known ISAACs, and a third window
showing the selection of a particular ISAAC, that is ISAAC ASIMOV, having
been accomplished.
FIG. 7 illustrates a brief flow diagram illustrating a heuristic process
for the selection of a particular choice among alternatives when the user
is not queried for selection of alternatives.
FIGS. 7a-7c show selected examples of heuristic rules of thumb which are
effective in making automatic choices between alternative ISAACs,
respectively directed toward selection schemes such as selecting the last
used ISAAC, selecting the last in order of ISAACs according to particular
position within a selected table, and finally selecting a particular ISAAC
based upon prior frequency of choice of that particular ISAAC.
FIG. 8a is a flow diagram illustrating the updating process for data base
information in linked smart fields.
FIG. 8b is a call slip illustration of an updated smart field window in
which the phone number field information has been updated.
FIG. 9a is a flow diagram of the process according to the invention in
which a query is made as to whether a specific selection has been made as
to a particular object.
FIGS. 9b-9c indicate graphically the performance of the selection query
operation as expressed in FIG. 9A.
FIG. 9d illustrates the transferal of the highlighted objects of FIG. 9c
transferred to a selected window.
FIG. 10a illustrates the input of a handwritten object into a smart field
in a window.
FIG. 10b illustrates the recognition of the handwritten object of FIG. 10a
and its conversion into formal font form.
FIG. 11a is a flow diagram illustrating the recognition of objects process.
FIG. 11b illustrates an object combination under operation.
FIG. 11c shows a template for organizing in preset form a variety of object
combinations.
FIG. 12a illustrates a process for hypothesizing user intent as to
particular activities.
FIG. 12b is a flow diagram setting forth a process for determining whether
the user wishes to provide or supply a particular action.
FIG. 12c shows a window with a menu partially overlapping its topside in
order to provide the user with an array of activity choices.
FIG. 12d shows the window of FIG. 12c with the activity of calling having
been selected, establishing the combined objects CALL and ISAAC in the
activity field of the window.
FIG. 13 illustrates the process for preparing for execution of a particular
activity.
FIG. 14a is a short flow diagram of a process for determining whether to
proceed with execution of a particular selected assist activity.
FIG. 14b illustrates an example of a confirmation of action slip that could
be produced upon completion of a particular activity, in this case
completion of the process of faxing information to another party.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The present invention is well suited for pointer based computer systems
such as the pen-based, pen-aware and mouse controlled systems that are
currently popular. For the purposes of illustration, the invention will be
described in connection with a pen-based system.
As shown in FIG. 1, a pen-based computer system 10 in accordance with the
present invention includes a central processing unit (CPU) 12, read only
memory (ROM) 14, random access memory (RAM) 16, input/output (I/O)
circuitry 18, and a display assembly 20. The pen-based computer system 10
may also optionally include a mass storage unit 22, a keypad (or keyboard)
24, a serial port 26, an infrared (I/R) port 28, and a clock 30.
The CPU 12 is preferably a commercially available, single chip
microprocessor. While CPU 12 can be a complex instruction set computer
(CISC) chip, it is preferable that CPU 12 be one of the commercially
available, reduced instruction set computer (RISC) chips which are known
to be of generally higher performance than CISC chips. CPU 12 is coupled
to ROM 14 by a unidirectional data bus 32. ROM 14 contains the basic
operating system for the pen-based computer system 10. CPU 12 is connected
to RAM 16 by a bi-directional data bus 34 to permit the use of RAM 16 as
scratch pad memory. ROM 14 and RAM 16 are also coupled to CPU 12 by
appropriate control and address busses, as is well known to those skilled
in the art. CPU 12 is also coupled to the I/O circuitry 18 by
bi-directional data bus 36 to permit data transfers with peripheral
devices.
I/O circuitry 18 typically includes a number of latches, registers and
direct memory access (DMA) controllers. The purpose of I/O circuitry 18 is
to provide an interface between CPU 12 and such peripheral devices as
display assembly 20, mass storage 22, keypad 24, serial port 26, and FR
port 28.
Display assembly 20 of pen-based computer system 10 is both an input and an
output device. Accordingly, it is coupled to I/O from a variety of
vendors. The input device of display assembly 20 is preferably a thin,
clear membrane which covers the LCD display and which is sensitive to the
position of a stylus 38 on its surface. With such a structure, the display
assembly 20 can serve as an input "tablet." These position sensitive
membranes are also readily available on the commercial market.
Alternatively, other types of tablets can be used, such as inductively
coupled tablets. Combination display assemblies such as display assembly
20 which include both the LCD and the input membrane are commercially
available from such vendors as Scriptel Corporation of Columbus, Ohio.
Some type of mass storage 22 is generally considered desirable. Mass
storage 22 can be coupled to I/O circuitry 18 by a bi-directional data bus
40. However, the mass storage 22 can be eliminated by providing a
sufficient amount of RAM 16 to store user application programs and data.
In that case, the RAM 16 can be provided with a backup battery to prevent
the loss of data even when the pen-based computer system 10 is turned off.
However, it is generally desirable to have some type of long term mass
storage 22 such as a commercially available miniature hard disk drive,
nonvolatile memory such as flash memory, battery backed RAM, a (Personal
Computers Memory Card International Association PCMCIA) card, or the like.
The keypad 24 can comprise an array of mechanical buttons or switches
coupled to I/O circuitry 18 by a data bus 42. Alternatively, keypad 24 can
comprise an entire, standard QWERTY keyboard. In the present embodiment, a
separate keypad 24 is not used in favor of a "pseudo" keypad 24'. This
"pseudo" keypad 24' comprises "button" areas which are associated with a
bottom edge of the tablet membrane that extends beyond the lower edge of
the LCD display. These button areas are defined by a printed or
silk-screened icons which can be seen through the transparent membrane of
the input tablet. When the "buttons" are selected by engaging the stylus
38 with the membrane over these printed icons, the membrane senses the
pressure and communicates that fact to the CPU 12 via data bus 38 and I/O
18. An example of pseudo keypad 24' is shown in FIG. 2.
Other types of pointing devices can also be used in conjunction with the
present invention, for example, an interrupt port of the CPU 12 which can
count the clock pulses to provide the time function. However, this
alternative clock embodiment tends to be wasteful of CPU processing power.
Clock 30 is coupled to CPU 12 by a data bus 48.
In operation, information is input into the pen-based computer system 10 by
"writing" on the screen of display assembly 20 with the stylus 38.
Information concerning the location of the stylus 38 on the screen of the
display assembly 20 is input into the CPU 12 via data bus 38 and I/O
circuitry 18. Typically, this information comprises the Cartesian (i.e. x
& y) coordinates of a pixel of the screen of display assembly 20 over
which the tip of the stylus is positioned. Commercially available
combination display assemblies such as the aforementioned assemblies
available from Scriptel Corporation include appropriate circuitry to
provide the stylus location information as digitally encoded data to the
I/O circuitry of the present invention. The CPU 12 then processes the data
under control of an operating system. While the method of the present
invention is described in the context of a pen-based system, other
pointing devices such as a computer mouse, a track ball, or a tablet can
be used to manipulate a pointer on a screen of a general purpose computer.
Therefore, as used herein, the terms "pointer", "pointing device",
"pointing means", and the like will refer to any mechanism or device for
pointing to a particular location on a screen of a computer display.
Serial port 26 is coupled to I/O circuitry by a bi-directional bus 44. The
serial port 26 can be used to couple the CPU to external devices and
networks.
Infrared (I/R) port 28 is coupled to I/O circuitry by a bi-directional bus
46. The I/R port can be used for outgoing information (e.g. to control a
printer or some other external device, or to communicate with other
computer systems) or for incoming information from other computers or
devices.
Clock 30 preferably comprises a real-time clock to provide real-time
information to the system 10. Alternatively, clock 30 can simply provide
regular clock pulses td possibly an application program stored in ROM 14,
RAM 16, or mass storage 22. The CPU 12 next produces data which is
transferred to the display assembly 20 via I/O circuitry 18 and data bus
38 to produce appropriate images on the screen portion of the display
assembly 20.
In FIG. 2, the pen-based computer system 10 of FIG. 1 is shown housed
within a generally rectangular enclosure 50. The CPU 12, ROM 14, RAM 16,
I/O circuitry 18, and clock 26 are preferably fully enclosed within the
enclosure 50. The display assembly 20 (FIG. 1) is mostly enclosed within
the enclosure 50, but a viewing screen 52 of the display assembly is
exposed to the user. As used herein, the term "screen" will refer to the
portion of the display assembly 20 which can display an image that can be
viewed by a user. Also accessible to the user is the pseudo keypad 24'
that was described with reference to FIG. 1.
Upon power-up, pen based computer system 10 displays on screen 52 an
initial "note" area 54a including a header bar 56a and a number of
guidelines 58. The header bar 56a preferably includes the date of creation
of the note area 54a and a number of icons and "soft" buttons, not
particularly germane to the discussion of the present invention. For this
reason, the header bar 56a will not be discussed in detail herein. The
optional guidelines 58 aid a user in entering text, graphics, and data
into the pen-based computer system 10. A graphic object G in the form of a
triangle is shown entered within note area 54a.
Additional note areas, such as a note area 54b, can be formed by the user
by drawing a substantially horizontal line across the screen 52 with the
stylus 38. The substantially horizontal line is recognized by the system
10 and is converted into a second header bar 56b. Additional text,
graphical, and other data can then be entered into this second note area
54b. For example, the text object T comprising "ISAAC" has been entered
into second note area 54b.
In this preferred embodiment, the keypad 24', as explained previously,
comprises a printed or silk-screened member 60 provided beneath a lower
edge of a thin, clear, stylus-sensitive membrane 62 of the input "tablet."
Alternatively, a keypad could comprise a mechanical keypad (or keyboard)
24, or a keypad could comprise "soft buttons" i.e. images generated at
convenient locations on the screen 52, in which case a "button" would be
activated by touching the stylus to the screen over the image of the
button. The keypad 24' preferably includes a number of dedicated function
buttons 64 and a pair of scroll buttons 66a and 66b. The operation of the
note areas 54a, 54b, etc., scroll buttons 66a and 66b, and other aspects
of computer system 10 are discussed in greater detail in co-pending U.S.
patent application Ser. No. 07/868,013, filed 04/13/92 on behalf of Tchao
et al., assigned to the assignee of the present invention and incorporated
herein by reference in its entirety.
The screen illustrated in FIG. 2 is referred to as the "notepad", and is
preferably an application program running under the operating system of
the pen based computer system 10. In this preferred embodiment, the
notepad is a special or "base" application which is always available
beneath higher level applications. The notepad application, like other
applications, runs within a window, which in this instance comprises the
entire viewing screen 52. Therefore, as used herein, a "window" is the
entire screen or any portion of an entire screen which is dedicated to a
particular application program.
A status bar 68 is provided at the bottom of the notepad application. The
status bar 68 is provided with a number of active and display areas, which
again are not particularly germane to the present invention and will
therefore not be discussed in detail herein. U.S. patent application Ser.
No. 07/976,970 filed 11/16/92 on behalf of Foster et. al, entitled "Status
Bar for Application Windows" and assigned to the assignee of the present
invention describes how to make and use the status bar, and is
incorporated herein by reference in its entirety.
The enclosure 50 is preferably provided with apertures 70 which permit the
free transmission of sound from a speaker (not shown) which is housed
within enclosure 50. The speaker can be driven by the CPU 12, by I/O
circuitry 18, or by specialized sound chips, as is well known to those
skilled in the art. The speaker can be used to provide user feedback, or
to transmit audible information to a user.
The term "object" will be used extensively in the following discussions. As
is well known to software developers, an "object" is a logical software
unit comprising data and processes which give it capabilities and
attributes. For example, an object can be queried as to its type and can
return such data as the number of words that it contains, what its
bounding box (BBOX) is, etc. Objects can contain other objects of the same
or of a different type. Objects can also be used to project images on a
screen according to their object type. Example of object types used in the
following description include paragraph, line, and word objects. There are
many well known texts which describe object oriented programming. See, for
example, Object Oriented Programming for the Macintosh, by Kurt J.
Schmucher, Hayden Book Company, 1986.
In the present invention, objects are preferably implemented as part of a
frame system that comprises frame objects related by a semantic network. A
description of semantic networks can be found in "A Fundamental Tradeoff
in Knowledge Representation and Reasoning", Readings in Knowledge
Representation, by Brachman and Leveseque, Morgan Kaufman, San Mateo,
1985.
It will be noted there is a liberal use of graphic elements in the present
invention. For example, the header bars 56a and 56b include lines and
other graphical elements. Processes for drawing lines on a computer screen
are well known to those skilled in the art. For example, graphics software
such as QUICKDRAW from Apple Computer, Inc. of Cupertino, Calif. can be
used to draw lines, simple geometrical shapes, etc. A description of the
QUICKDRAW graphics software is found in the book Inside Macintosh, Volumes
I, II, and III, by C. Rose et al., Addison-Wesley Publishing Company,
Inc., July 1988. With such graphics software, a line can be drawn by
simply specifying the coordinates of the beginning and the end of the
line, and by specifying the width of the line.
Another preferred tool for implementing the system of the present invention
is a view system. Various types of view systems are well known to those
skilled in the art. In the present system, the notepad application on the
screen 52 forms a first or "root" layer, with the status bar 68, for
example, positioned in a second layer "over" the root layer. The various
buttons of the status bar 68 are positioned in a third layer "over" the
second and root layers. The view system automatically handles "taps" and
other gestures of the stylus 38 on the screen 52 by returning information
concerning the tap or gesture and any object to which it may be related.
Again, the status bar 68 and the view system is described in greater
detail in co-pending U.S. patent application Ser. No. 07/976,970, which
has been incorporated herein by reference. It is therefore clear that the
object oriented programming and view system software makes the
implementation of the processes of the present invention less cumbersome
than traditional programming techniques. However, the processes of the
present invention can also be implemented in alternative fashions, as will
be well appreciated by those skilled in the art.
A method or process 100 for providing implicit or explicit assistance in
the provision of computer implemented services in accordance with the
present invention is shown in FIG. 3. The process begins at step 102 on
power-up of the computer system 10 and runs concurrently with other system
functions.
At step 104, the process recognizes whether or not an implicit assistance
function is to be provided by computer system 10. As will be seen,
implicit assistance may, for example, arise from an entry into a smart
field by a user. If a user does enter information into a "smart field,"
the computer database will be queried at step 106 to determine whether
assistance is possible given the user input.
A smart field is considered to be a predefined region on screen 52 of
computer system 10 shown in FIG. 2, or a predefined region within a window
which appears on screen 52, as suggested below with reference to FIG. 4b
and which will be discussed in greater detail in the text description
below associated with that Figure. For convenience, the smart fields are
typically rectangular in shape. The particular geographic bounds of a
smart field can conveniently be stored in computer memory by simply saving
four numbers defining the corners of the rectangular shape of the field. A
particular field is considered smart, because of the specialized
capabilities of the smart field to respond with particularized
effectiveness and intelligence to user needs, indications, or events
registered, for example, by pen 38, within the bounds of the particular
smart field.
However, implicit assist may be indicated not just by entry of an
indication in a smart field, but by the happening of any of a number of
predefined allowable events which lead to a query of the database at
process step 106. A user entry made into a smart field is not the only way
computer system 10 is caused to undertake an implicit assist operation.
Certain kinds of events on screen 52, for example, such as the writing of
a particular indication or word on screen 52 outside of a particular smart
field may trigger an implicit assist. In general, implicit assist can be
triggered by the happening of any of a number of predefined allowable
events.
If, however, a decision is made at decision process 104 not to perform the
implicit assist function or approach suggested in FIG. 3, a check is made
at decision process 110 whether an explicit assist function should be
undertaken. If neither implicit or explicit assist is indicated at
decision processes 104 and 110, operation continues past point A and along
line 112 repeatedly checking for an indication whether implicit or
explicit assistance is required, at the two decision processes 104 and
110.
An example of an indication of user desire to have explicit assistance
undertaken is the act of using pen 38 in FIG. 2 to tap or click on the
assist icon or button 64 shown on the surface of stylus-sensitive membrane
62 or a keypad 24 including a range of dedicated function buttons 64. If
the query at process step 108 produces a negative, i.e., null, response to
the question of whether any implicit assist actions are available in the
database of computer 10, indicating that no assistance actions are
identified for performance, then process control returns to point A.
If a non-null result to the query for implicit assistance has been
established by step 108, a determination is made at process step 115 as to
whether the user should be queried. This determination can be made by the
system, or can be set by the user in a preference field. If it is decided
that the user should be queried, step 117 displays a list of action
alternatives for user selection. The user can make the selection of a
particular assistance action according to step 119, by highlighting the
particular course of action selected, for example. Alternatively, if it is
desired that no user query is desired for selection of a particular mode
of assistance, then a process is undertaken at step 121 to select the best
alternative.
In either the case of an automatic selection of a best alternative
according to process step 121 or in the case of steps 117 and 119
involving a presentation of alternatives to the user followed by user
selection of a particular alternative, i.e., whether a user query has been
undertaken, or whether an automatic selection of a best alternative has
been made, upon accomplishment of the selected assistance action, the
database information and any linked information are updated at step 123.
Once implicit assistance has been completed, process control returns to
point A. If step 104 determines that there is no implicit assist, then
step 110 determines if there is an explicit assist to be undertaken. For
example, if the name ISAAC had been entered on screen 52 of FIG. 2 and no
time-out had occurred, and additionally the assist button (the rightmost
of buttons 64) had been clicked, then an explicit assist will have been
requested. If an explicit assist has been indicated at step 110, then a
step 130 determines, if a particular selection as to the explicit
assistance has been made. Selection is typically made by clicking on a
particular word with pen 38, for example, and thereby highlighting the
item selected. If a user selection has been made, particular selected
objects are entered into the assistance operation in step 132. If no user
selection has been made, objects entered since a delimiter are entered
into the assistant in a step 133. Since no objects have specifically been
selected, the objects to be entered into the assistant are selected
automatically by a delimiter process.
An example of how the delimiter process can be accomplished, for example,
involves the entry of only those objects on the screen 52 which are
delimited in some fashion from the other objects which may have been
entered on the screen. For example, if several paragraphs have been
entered on the screen, only the last paragraph's objects will be
considered for entry as objects into the assistant. Time may also be used
as a delimiter. For example, if a considerable period of time separates a
given object on the screen from another, only the most recent object will
be entered into the assistant. The time threshold separating the
particular objects may for example be a pre-set time-out.
Next, an attempt is made at step 135 to recognize the possible intent
expressed by the objects entered into the assistance process. If no intent
is recognized, the explicit assistance operation is considered for local
termination by return of process control to point A, according to decision
process 136, or for continuation by return to the top of step 133 to wait
for new objects which might permit intent to be recognized or to await
user or process intervention.
If a possible intent is recognized at step 135, the intent is hypothesized
at step 137. Next, step 139 checks whether there is a hypothesis
available. If no hypothesis is found, process step 141 permits the user to
supply a proposed action. If no hypothesis is found and no user action is
proposed, process control returns to point A. However, with the
availability of a user supplied action or if a hypothesis is available,
preparations for execution of actions consistent with the action requested
or hypothesized are undertaken at ste | | |
