WikiPatents - Community Patent Review
Create Free Account  |  License or Sell Your Patent  |  WikiPatents Marketplace  |  WikiPatents Blog
Username:  Password:  
    
Advanced Search
Method and apparatus for structuring and managing the participatory evaluation of documents by a plurality of reviewers    
United States Patent5706452   
Link to this pagehttp://www.wikipatents.com/5706452.html
Inventor(s)Ivanov; Vladimir I. (White Plains, NY)
AbstractA method for structuring the process of participatory document evaluation by a plurality of reviewers utilizing a feed-forward synchronization schema defined as a workflow graph (52) construct. The workflow graph construct allows one to define efficient document evaluation strategies that take advantage of the inherent parallelism within a business process, and take into account the data dependencies exhibited by the business process. Independent aspects of each document are evaluated in parallel, and a stage evaluation is started when all required data (including data provided by other reviewers) is available. The workflow graph construct assembles together a set of workflow objects (review (46), roles (48), stages (50), document (44)), and associates a set of functional properties to each of them. The topology of the workflow graph describes the synchronization schema, while the functional properties provide additional flexibility and further refine the evaluation strategy. A method for managing the process of participatory document evaluation by a plurality of reviewers utilizing a programmed computer system. Documents to be reviewed are prepared by a set of preparers, and are stored into a database (38). Reviewers participate in the review process by evaluating aspects of the document within their area of expertise. Depending on the information in each individual document, a set of reviewers is notified that the document is ready for their review. Each reviewer is notified at the time appropriate for his/her role via electronic mail by a workflow manager (30).
   














 Title Information Submit all comments and votes
 
Patent Text Patent PDF Print Page Summary File History
Plain text PDF images Print Summary File History
Drawing from US Patent 5706452
Method and apparatus for structuring and managing the participatory

     evaluation of documents by a plurality of reviewers - US Patent 5706452 Drawing
Method and apparatus for structuring and managing the participatory evaluation of documents by a plurality of reviewers
Inventor     Ivanov; Vladimir I. (White Plains, NY)
Owner/Assignee    
Patent assignment
All assignments
Publication Date     January 6, 1998
Application Number     08/568,378
PAIR File History     Application Data   Transaction History
Image File Wrapper   Patent Term   Fees
Litigation
Filing Date     December 6, 1995
US Classification    
Int'l Classification    
Examiner     McElheny Jr.; Donald E.
Assistant Examiner    
Attorney/Law Firm    
Address
Parent Case    
Priority Data    
USPTO Field of Search    
Patent Tags     structuring managing participatory evaluation documents plurality reviewers
   
Enter a comma (,) or semicolon (;) between multiple tag words/phrases.
Describe this patent:
 Amusing   
 Clever   
 Complex   
 Efficient   
 Historic   
 Important   
 Innovative   
 Interesting   
 Practical   
 Simple   
[no votes]
Patent WIKI

Share information and news about this patent, including information and news about the technology, inventors, company, ligation and licensing.

 References Submit all comments and votes
 
*references marked with an asterisk below are user-added references
 U.S. References
 
Add a new US reference:  
ReferenceRelevancyCommentsReferenceRelevancyComments
5537526
Anderson
715/515
Jul,1996

[0 after 0 votes]
5535322
Hecht
705/1
Jul,1996

[0 after 0 votes]
5438659
Notess
715/505
Aug,1995

[0 after 0 votes]
5434995
Oberlin

Jul,1995

[0 after 0 votes]
5398336
Tantry
707/103R
Mar,1995

[0 after 0 votes]
5388203
Kaneko
715/837
Feb,1995

[0 after 0 votes]
5337407
Bates
715/751
Aug,1994

[0 after 0 votes]
5319543
Wilhelm
705/3
Jun,1994

[0 after 0 votes]
5312478
Reed
715/503
May,1994

[0 after 0 votes]
5283856
Gross
706/47
Feb,1994

[0 after 0 votes]
5216603
Flores
704/1
Jun,1993

[0 after 0 votes]
5208748
Flores
704/1
May,1993

[0 after 0 votes]
5132900
Gilchrist
707/9
Jul,1992

[0 after 0 votes]
5008853
Bly

Apr,1991

[0 after 0 votes]
 Foreign References
 Other References
 Market Review Submit all comments and votes
   
Market Size
Estimate the gross annual revenues of the relevant market sector:
> $10B
$5B - $10B
$2B - $5B
$500M - $2B
$100M - $500M
$10M - $100M
$1M - $10M
$500K - $1M
$100K - $500K
< $100K
[No votes]
$0
 
$0   $2.5B   $5B   $7.5B   $10B
Market Share
Estimate the percentage of the relevant market sector this invention will capture:
75% - 100%
50% - 74.99%
25% - 49.99%
10 - 24.99%
5 - 9.99%
2 - 4.99%
1 - 1.99%
< 1%
[No votes]
0.0%
 
0%   25%   50%   75%   100%
Reasonable Royalty
What percentage of gross sales should the inventor or assignee be paid?
75% - 100%
50% - 74.99%
25% - 49.99%
10 - 24.99%
5 - 9.99%
2 - 4.99%
1 - 1.99%
< 1%
[No votes]
0.0%
 
0%   25%   50%   75%   100%
Public's "Guesstimation" of Royalty Value
Market SizeN/A[No votes]
xMarket ShareN/A[No votes]
xReasonable RoyaltyN/A[No votes]

N/A

License Availablity
If you are NOT the owner or assignee, answer here:
Yes, license is available for purchase

No, license is not currently available



[No votes]
License Availablity
If you ARE the owner or assignee, answer here:
Yes, license is available for purchase

No, license is not currently available



[No votes]
Competitive Advantage
Does this invention have a significant competitive advantage over similar technologies?
Yes

No



[No votes]
Most helpful competitive advantage comment
[No comments]

Commercial Alternatives
Are there viable commercial alternatives for this invention?
Yes

No



[No votes]
Most helpful commercial alternative comment
[No comments]

 Technical Review Submit all comments and votes
 Claims Submit all comments and votes
 


I claim:

1. A computer implemented method for structuring the evaluation of at least one document by a plurality of reviewers who participate in and contribute to said evaluation, said at least one document prepared by at least one preparer, said method comprising the steps off

(a) defining a workflow graph structure which specifies said evaluation, and storing it into a memory, said workflow graph structure comprising a generic review, a plurality of roles, a plurality of stages, a generic document, a criterion to finalize the role for each of said plurality of roles, a criterion to finalize the stage for each of said plurality of stages, and a criterion to finalize the document for said generic document;

(b) creating said at least one document having a plurality of aspects to be reviewed independently, each of said plurality of aspects corresponding to one of said plurality of roles;

(c) associating zero or more authorized reviewers to said at least one document and to each of said plurality of roles;

(d) creating at least one review for said at least one document associated to at least one of said plurality of roles;

(e) analyzing said at least one document and said at least one review relative to said workflow graph structure, processing a subset of said plurality of roles and a subset of said plurality of stages responsive to the actions of said plurality of reviewers, consistent with said criteria to finalize the role associated to said subset of said plurality of roles, said criteria to finalize the stage associated to said subset of said plurality of stages, and said criterion to finalize the document;

(f) notifying said authorized reviewers associated to said plurality of roles outgoing from each of the processed stages when its associated said criterion to finalize the stage is satisfied.

2. The method of claim 1, wherein each of said criterion to finalize the role, each of said criterion to finalize the stage, and said criterion to finalize the document is a number to finalize criterion.

3. The method of claim 2, wherein each of said number to finalize criterion is selected from the group consisting of: ONE, ALL, and percentage.

4. The method of claim 1, wherein each of said criterion to finalize the role, each of said criterion to finalize the stage, and said criterion to finalize the document is a condition to finalize criterion.

5. The method of claim 1, further comprising the steps of:

(a) further refining said workflow graph structure by including a role transfer on finalize action for each of said plurality of roles, and a stage transfer on finalize action for each of said plurality of stages;

(b) from said at least one review for one of said plurality of roles to said at least one document, transferring information according to corresponding said role transfer on finalize action upon satisfying said criterion to finalize the role for said one of said plurality of roles;

(c) from said at least one review for one of said plurality of roles incoming into one of said plurality of stages to said at least one document, transferring information according to corresponding said stage transfer on finalize action upon satisfying said criterion to finalize the stage for said one of said plurality of stages.

6. The method of claim 1, further refining said workflow graph structure by:

(a) for each of said at least one review associated to one of said plurality of roles, ranking the aspect corresponding to said one of said plurality of roles using a grade and a weight;

(b) for each of said plurality of roles, defining a role grade formula to be applied to said grade and said weight of said at least one review associated to said each of said plurality of roles;

(c) for each of said plurality of stages, defining a stage grade formula to be applied to said grade and said weight of said plurality of roles incoming into said each of said plurality of stages;

(d) for said generic document, defining a document grade formula to be applied to said grade and said weight of all said plurality of stages.

7. The method of claim 6, wherein each of said role grade formula, each of said stage grade formula, and said document grade formula is selected from the group consisting of: MAX, MIN, SUM, AVG, CNT, WSUM, WAVG, EARLIEST, and LATEST.

8. The method of claim 1, further comprising the steps of:

(a) further refining said workflow graph structure by defining a plurality of stop conditions, comprising:

(i) for said generic review, a review stop condition to be evaluated upon processing of the associated said at least one document;

(ii) for each of said plurality of roles, a role stop condition to be evaluated upon processing of said each of said plurality of roles;

(iii) for each of said plurality of stages, a stage stop condition to be evaluated upon processing of said each of said plurality of stages;

(iv) for said generic document, a document stop condition to be evaluated upon processing of said at least one document;

(b) stopping said document evaluation upon satisfying any of said plurality of stop conditions.

9. The method of claim 8, further comprising the steps of:

(a) further refining said work flow graph structure by including a review transfer on stop action for said generic review, a role transfer on stop action for each of said plurality of roles, and a stage transfer on stop action for each of said plurality of stages;

(b) from said at least one review to said at least one document, transferring information according to said review transfer on stop action upon satisfying said review stop condition for said at least one review;

(c) from said at least one review for one of said plurality of roles to said at least one document, transferring information according to corresponding said role transfer on stop action upon satisfying said role stop condition for said one of said plurality of roles;

(d) from said at least one review for one of said plurality of roles incoming into one of said plurality of stages to said at least one document, transferring information according to corresponding said role transfer on stop action upon satisfying said stage stop condition for said one of said plurality of stages.

10. The method of claim 1, wherein said notifying step comprises the steps of:

(a) specifying at least one notification template;

(b) identifying one of said at least one notification template to be used in notifying step via a template formula.

11. An apparatus for managing the participatory evaluation of at least one document by a plurality of reviewers, said participatory evaluation described by a workflow graph structure, the apparatus comprising:

(a) a graph description interface module for specifying said workflow graph structure using an external representation, said workflow graph structure comprising a generic review, a plurality of roles, a plurality of stages, a generic document, a criterion to finalize the role for each of said plurality of roles, a criterion to finalize the stage for each of said plurality of stages, and a criterion to finalize the document for said generic document;

(b) a preparer interface module for allowing a plurality of preparers to create, store, retrieve, and modify said at least one document, having a plurality of aspects to be reviewed, each of said plurality of roles corresponding to one of said plurality of aspects;

(c) an identifying procedure used to associate zero or more authorized reviewers to said at least one document and to each of said plurality of roles, thereby allowing said authorized reviewers to review the aspects corresponding to said each of said plurality of roles;

(d) a reviewer interface module used by said authorized reviewers to create, store, retrieve, and modify a plurality of reviews for said at least one document and said plurality of roles;

(e) a translation module for generating an internal representation of said workflow graph structure from said external representation, and storing it in a memory;

(f) a workflow manager accessing said at least one document, said plurality of reviews, said internal representation; analyzing and updating said at least one document based on its associated said plurality of reviews;

(g) a notification module for informing said authorized reviewers of the appropriate time for creating said plurality of reviews for said plurality of roles.

12. The apparatus of claim 11, wherein each of said criterion to finalize the role, each of said criterion to finalize the stage, and said criterion to finalize the document is a number to finalize criterion.

13. The apparatus of claim 12, wherein each of said number to finalize criterion is selected from the group consisting of: ONE, ALL, and percentage.

14. The apparatus of claim 11, wherein each of said criterion to finalize the role, each of said criterion to finalize the stage, and said criterion to finalize the document is a condition to finalize criterion.

15. The apparatus of claim 11, further comprising:

(a) said workflow graph structure further comprising a role transfer on finalize action for each of said plurality of roles, and a stage transfer on finalize action for each of said plurality of stages;

(b) a role transfer submodule in said workflow manager for transferring information from said plurality of reviews for one of said plurality of roles to said at least one document, according to corresponding said role transfer on finalize action upon satisfying said criterion to finalize the role for said one of said plurality of roles;

(c) a stage transfer submodule in said workflow manager for transferring information from said plurality of reviews for one of said plurality of roles incoming in one of said plurality of stages to said at least one document, according to corresponding said stage transfer on finalize action upon satisfying said criterion to finalize the stage for said one of said plurality of stages.

16. The apparatus of claim 11, further comprising:

(a) a grading interface submodule in said reviewer interface module for ranking said plurality of aspects, using a grade and a weight;

(b) said work flow graph structure further comprising:

(i) for each of said plurality of roles, a role grade formula to be applied to said grade and said weight of said plurality of reviews associated to said each of said plurality of roles;

(ii) for each of said plurality of stages, a stage grade formula to be applied to said grade and said weight of said plurality of roles incoming into said each of said plurality of stages;

(iii) for said generic document, a document grade formula to be applied to said grade and said weight of said plurality of stages.

17. The apparatus of claim 16, wherein each of said role grade formula, each of said stage grade formula, and each of said document grade formula is selected from the group consisting of: MAX, MIN, SUM, AVG, CNT, WSUM, WAVG, EARLIEST, and LATEST.

18. The apparatus of claim 11, further comprising:

(a) said workflow graph structure further comprising a plurality of stop conditions,

(i) for said genetic review, a review stop condition to be evaluated upon processing of said at least one document;

(ii) for each of said plurality of roles, a role stop condition to be evaluated upon processing of said each of said plurality of roles;

(iii) for each of said plurality of stages, a stage stop condition to be evaluated upon processing of said each of said plurality of stages;

(iv) for said generic document, a document stop condition to be evaluated upon processing of said at least one document;

(b) a stop submodule in said workflow manager for stopping said participatory evaluation upon satisfying any of said plurality of stop conditions.

19. The apparatus of claim 18, further comprising:

(a) said workflow graph structure further comprising a review transfer on stop action for said generic review, a role transfer on stop action for each of said plurality of roles, and a stage transfer on stop action for each of said plurality of stages;

(b) a review transfer submodule in said workflow manager for transferring information from each of said plurality of reviews to said at least one document, according to said review transfer on stop action upon satisfying said review stop condition for said each of said plurality of reviews;

(c) a role transfer submodule in said work flow manager for transferring information from said plurality of reviews for one of said plurality of roles to said at least one document, according to corresponding said role transfer on stop action upon satisfying said role stop condition for said one of said plurality of roles;

(d) a stage transfer submodule in said workflow manager for transferring information from said plurality of reviews for one of said plurality of roles incoming into one of said plurality of stages to said at least one document, according to corresponding said role transfer on stop action upon satisfying said stage stop condition for said one of said plurality of stages.

20. The apparatus of claim 11, further comprising:

(a) a specification module for specifying at least one notification template; (b) said workflow graph structure further comprising a role template formula for each of said plurality of roles, and a document template formula for said generic document.
 Description Submit all comments and votes
 


1. FIELD OF THE INVENTION

This invention relates generally to computerized methods of doing business, and in particular to methods for defining and managing the process of document review by a plurality of individuals who assess and express their opinions about portions of a document, and/or contribute new information to the document.

2. BACKGROUND

In one sense, aspects of this invention relate to such fields as database management, electronic mail, communication management, and synchronization of parallel processes. In another, more important sense, this invention provides solutions for a newly identified class of problems by integrating state of the art computer and communication tools within a new methodological framework. The general characteristics of a problem in this class are,

a preparer collects information within a discourse domain and assembles it into a document,

the document is reviewed by a set of reviewers who rank it collectively; each reviewer evaluates specific aspects of the document, eventually renders a grade which contributes to the overall rank, and possibly adds new information to the document,

the document is reviewed in stages; a stage acts like a synchronization barrier for the review of certain aspects of the document,

the system manages the evaluation process by monitoring the status of each document and notifying reviewers of the proper time to review particular aspects.

2.1. DESCRIPTION OF THE PRIOR ART

2.1.1. Databases

Database systems have been used traditionally to organize and store data for efficient retrieval. A database management system defines the structures used to store data, and the operations for assembling disparate data. A database system can thus be used to store documents in different stages of completion, reviews in different stages of preparation, and relate the documents and the associated reviews when needed.

In addition, a database management system allows a certain degree of concurrent access to data, and can hide sensitive information based on user identity. For example, two reviewers residing at different locations can examine a document at the same time and can work on their reviews independently. Also, a user can be denied access to a document if she is not a reviewer or is not explicitly granted this privilege.

Database systems lack the functionality to play an active role in managing the evaluation of documents. Within the strict framework of database systems, reviewers can not be notified that documents are ready for their review but have to constantly monitor the system for "work to do." In addition, the database system alone can not enforce a predefined review sequence; instead, the review discipline must be voluntarily observed by each reviewer.

2.1.2. Electronic Mail

Like the traditional communication systems (regular mail, phone, radio, etc.) electronic mail systems have been used to transfer unstructured information between participants in conversations residing at different locations. Unlike the traditional systems, electronic mail can be used to transfer structured data as well, data which is readily available for other computational purposes.

Each electronic mail user communicates via a computer, has an electronic identity, and an electronic mail box where the incoming messages are delivered. The underlying assumption in electronic mail is that the user monitors the personal mail box for new messages periodically. In the current prevalent practice in electronic mail, the structures relevant for our purpose consist of the sender and receiver(s) identities, along with an unstructured text body which stores the message itself. The user of such a system is provided with general choices of action. For example, the "Send" action allows one to send a message to one or more recipients, "Reply" allows a response to be sent to the original sender, and "Forward" sends a copy of the message along with receiver's comments to a third party.

Electronic mail systems do not provide assistance in structuring the flow of communication toward accomplishment of collective goals and results. The management of the communication is essentially left to the discretion and ability of the users of the system.

Documents to be evaluated can be sent to reviewers via electronic mail, but the preparer has to explicitly know and specify the first reviewer(s), and reviewers have to know and specify the next reviewer(s) as well. In many cases this is unacceptable, in part because the evaluation strategy is decided at a different level, and because rules change without reviewers necessarily being aware of it. U.S. Pat. No. 5,283,856 to Gross et al. (1994) discloses an event-driven rule-based mail messaging system in which a set of rules can be defined to automatically specify the next reviewer(s). The rules are stored in each person's private rule repository, which makes it difficult to maintain all repositories consistent when rules change. Furthermore, since rules take into account the contents of one message only, events associated to a multitude of messages (and requiring synchronization) are difficult (if not impossible) to handle.

2.1.3. Conversation Management Systems

The existence of "forms" in the standard business sense derives from the existence of certain recurring communications in which the collection and transmission of relevant details has the same structure each time. In addition to standardized forms, there can also be standardized "procedures" in which a sequence of actions follows a regular pattern.

Throughout this presentation a business process is a collection of steps, not necessarily performed in sequence, that accomplish a specific business goal. Examples of business processes are the evaluation of competing bids or the approval of purchase requisitions. Processes remove elements of randomness from work, making it more efficient and predictable.

Paper based forms and their associated routing procedures which implement particular business processes are still being used in offices today. Computer technology makes it possible to automate forms generation, processing, and routing by embedding them as programs in data processing systems. For example, U.S. Pat. No. 5,319,543 to Wilhelm (1994) discusses a particular parameterized work procedure for automatic assignment of work to resources in a hospital. A person using such a system acts within the strict framework of the system and the limited options presented in accordance with the procedures embodied in the computer system.

Acknowledging the fact that systems like those described in the Wilhelm patent implement limited communication protocols, U.S. Pat. Nos. 5,208,748 and 5,216,603, both to Flores et al (1993) present a general method for managing conversations between individuals within a community of participants on the basis of a methodology which is structured and adaptable to different types and categories of conversations.

While in the Flores patents parallelism is handled at the communication level ("a plurality of conversations"), the synchronization of conversations due to data dependencies in the business process is not handled. Similarly, while the Wilhelm patent suggests the parallel routing of cases to a multitude of work queues, it does not address the issue of synchronization once a case exits the queues.

Action Workflow.RTM., a product from Action Technologies, Inc. of Alameda, Calif., is another embodiment of the Flores patents. While providing solutions to the problem of parallel work and synchronization, Action Workflow is still limited in the kinds of parallelism and synchronization allowed (for example, "all child workflows triggered in parallel must return to the same point in their parent work flow," Action Workflow Application Builder for Notes User's Guide, pp. 20-21, Action Technologies, Inc.).

2.1.4. Synchronization of Tasks in Parallel Processing

When dealing with complex documents that have to be evaluated by a multitude of reviewers, a sequential evaluation might result in unacceptable performance. Further, not only that each review becomes a critical step in the process, but each delay increases the overall document evaluation time.

Parallel evaluation is a possible solution to the performance problem. Complications arise when there are data dependencies in the evaluation process. For example, if reviewer A produces result a, reviewer B produces result b, and reviewer C cannot review unless results a and b are available, then C cannot review in parallel with A and B. C's review must be synchronized with A and B's results.

Solutions for similar problems in parallel processing have been suggested both in hardware and in software. For example, U.S. Pat. No. 5,434,995 to Oberlin et al (1995) discloses a barrier mechanism used to synchronize processing elements in a massively parallel processing system.

The evaluation speed of a document ultimately depends on how quickly each reviewer completes his/her evaluation. However, delays in non critical reviews can be masked by the exposed parallelism and could affect the overall evaluation speed little or at all (see Arvind, D. E. Culler, and G. K. Maa, "Assessing the Benefits of Fine-grain Parallelism in Dataflow Programs," in Proc. 3rd Intl. Conf. Supercomputing, 1988, pp. 60-69; V. I. Ivanov, "Performance Evaluation of Task Grain Programs," in Proc. 8th Intl. Parallel Proc. Symp., pp. 644-648, 1994).

2.2. Related Work

The Investment Proposal (IP) system developed by the author in 1995, and not believed to be prior art to the present invention, is an approval/rejection system used to manage expenses within an organization by requiring an approval for each major project.

Preparers compose investment proposal documents which are then reviewed by a set of reviewers depending on specific information in each document. The IP system implements roughly a three tier hierarchical approval/rejection business process. First, each document is reviewed in parallel from a detailed perspective by experts in each category with non zero expenses. Second, final reviewers for each of the technology and real estate areas review a document when all the associated detail reviews are available. Finally, the chief administrative officer reviews a document after all the detail and final reviewers have approved.

Reviewers can either approve or reject a proposal; all reviewers (within an equivalence class) must approve in order for the document to be approved, and a single rejection from any reviewer suffices to reject the document.

Documents and their associated reviews are stored into a database which is permanently monitored by a workflow manager. Reviewers are notified by the workflow manager that a document is ready for their review via electronic mail, and they can review a document in parallel. A rigid OR/AND synchronization schema is hard coded into the system. Although roles and barriers can be added, eliminated, or moved around, the methodology suggested by the IP system lacks flexibility as exemplified below.

The IP system is a particular approval/rejection system. It does not allow reviewers to grade documents or to express a degree of confidence in their grading. Such feature is important in systems that rank documents for comparative purposes, like bid evaluation, paper review, project proposal, etc. It also does not allow reviewers to contribute any information to the reviewed document, which is important in cases when earlier reviewers conduct research and publish the results for the use of later reviewers.

The IP system does allow equivalent reviews. For example, if there are five reviewers for a role, any one review is sufficient for the role to be considered reviewed. Quorums (voting systems) are not supported; such feature is useful, for example, when a project proposal system has to handle exceptions (unavailable reviewers); and to eliminate the bias introduced by a single person's opinion. A critical role could require, for example, that at least three of the five reviewers grade a specific aspect of a proposal.

The IP system statically computes the reviewer lists for each document when the document is submitted. It does not allow earlier reviewers to dynamically change (add, modify, remove) next reviewers neither directly nor indirectly through their actions. Such feature is needed, for example, when a reviewer encounters a controversial aspect in a document and wants to involve a higher ranking person in the decision process, person who otherwise would not be involved.

The IP system allows a fixed number of templates to be used for notifications. It also provides limited substitution of tokens in notifications with actual values from the document. It does not allow the notifications to be customized depending on recipients or other characteristics of the document. Such feature is important in order to provide coaching to reviewers and pinpoint the exact elements that they need to review.

The IP system allows only a uniform OR/AND synchronization schema which can be used in collaborative review systems. Competitive review systems based on an AND/OR schema, fuzzy, quorum based, or hybrid systems cannot be implemented with the methodology suggested by the IP system.

The only exception that the IP system handles is a reviewer's rejection. More complex criteria, such as bad reviews by a "majority" of reviewers in a specific role (voting systems) cannot be implemented within the system or with the methodology suggested by the system.

The IP system implements a limited collective approval/rejection schema and does not suggest a general method for managing a general document evaluation process, which is structured and adaptable to different types and categories of business processes.

3. OBJECTS AND ADVANTAGES

It is the principal object of this invention to provide a flexible methodology for defining efficient strategies for collective document evaluation.

It is another object of this invention to provide a computerized system for managing the process of document evaluation for strategies defined using the aforementioned methodology.

Some of the features of a workflow application built according to the present invention are:

(a) provides different access (security) levels based on user's electronic identity, to protect sensitive information from unauthorized users,

(b) allows concurrent access to shared data from a multitude of locations,

(c) maintains a history of actions related to each document; this includes, who did what and when, and what was the impact of each action,

(d) participants in the evaluation process, including the document author, are notified of events related to the document via electronic mail messages; the personal mail box is the single place that a user has to check regularly, regardless of how many evaluation applications require his/her reviews,

(e) the system manages the review process based on the logic specified in the application; participants do not have to know or select who has to review a document next; the routing logic is stored in a central place where it can be maintained consistent for all the people involved in the evaluation process,

(f) by performing independent reviews in parallel, the efficiency of the review process is increased; furthermore, the impact on performance of delayed rendering of non critical reviews could be minimized,

(g) allows the definition of a feed-forward review schema, to take advantage of the intrinsic parallelism of the business process while observing all its data dependencies.

More specifically, features and advantages unique to this invention include:

(a) each reviewer is specialized in some aspects, and evaluates the corresponding portions, of a document; the result of the evaluation is an inherently subjective grade or level of confidence; grades from individual reviewers for aspects of the document are combined to yield a more objective grade for the entire document,

(b) the method of this invention allows a designer to specify how to handle multiple reviews from one reviewer evaluating one aspect of a document; it also allows flexible handling of reviews from different reviewers evaluating one aspect of a document (different reviewers can have different confidence levels); finally, some aspects of the document could weigh more than others (for example, marketability could weigh twice as staff availability),

(c) the method allows the grouping of reviewers in roles or equivalency classes; reviews could be required from one, a quorum, or all members of a role,

(d) reviewers can contribute new information to a document; this can be used by the preparer as feedback, or by subsequent reviewers as argumentation of predecessor's position,

(e) reviewers can also modify the review process itself by specifying which additional aspects need to be reviewed on a case by case basis,

(f) the methodology allows a designer to specify an escalation schema (how to handle cases when reviewers do not review a document within the allowed time frame),

(g) the method of this invention makes use of a programming language to express complex logic and manipulate data in the document; part of the method flexibility stems from the use of the programming language,

(h) modifications in the review authorizations between the moment the document is submitted and the moment reviews are requested can be reflected in the review process; in this respect, the reviewer list can be computed using the programming language just before notifications are sent out,

(i) exceptionally, the review process can be stopped if conditions specified in the programming language are satisfied; or the normal review flow based on quorum logic can be altered when other conditions are satisfied,

(j) actual notifications sent to inform users of events related to a document are based on notification templates; the notification template to be used is identified for each role by evaluating a programming language expression; the text of the notification is further customized substituting programming language expressions in the template with the result of their evaluation.

Further objects, features, and advantages will become apparent from the following detailed description of the method of this invention, the current embodiment of a system which incorporates the method of this invention, and the sample applications built using this system.

4. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the structure of a networked computer system.

FIG. 2 is a schematic diagram illustrating the basic components in a system incorporating the method of this invention.

FIG. 3 illustrates the workflow object hierarchy used in the general method of this invention.

FIG. 4 presents a sample workflow graph for a particular application implemented using the method of this invention.

FIGS. 5A-5D present the four basic configurations used to assemble a workflow graph.

FIGS. 6A-6D illustrate the essential data components and functional properties of workflow objects used in the general method of this invention.

FIGS. 7A-7F illustrate the external representation of the basic constructs used to define a workflow graph in the current embodiment of the method of this invention.

FIG. 7G presents an actual notification obtained from the template in FIG. 7F, and received by a reviewer in an application developed using the current embodiment of the method of this invention.

FIG. 8 presents the data structures used internally by the current embodiment of the workflow manager.

FIGS. 9A-9I present the pseudo code implementing the current embodiment of the work flow manager.

FIGS. 10A and 10B are schematic diagrams of alternative system architectures which may be employed in implementing the method of this invention.

5. DETAILED DESCRIPTION

This invention comprises a method for managing the process of participatory document evaluation by a plurality of reviewers utilizing a programmed computer system. The method of this invention comprises a powerful tool which can be configured in a variety of ways and with different degrees of complexity to achieve productivity increases for a wide class of business processes.

The current embodiment of the method of this invention, the Rank Workflow system, integrates existing database and electronic mail technologies with a unique workflow manager. The database and electronic mail layers in this embodiment are provided by Lotus Notes.RTM., a product from Lotus Development Corporation of Cambridge, Mass. The workflow manager is produced by the author specifically for this platform, and thus the resulting system benefits from the capabilities, and is restricted by the limitations, in Lotus Notes. Those skilled in the art will understand that the particular implementation is only an illustration of the method of this invention, and that the same features can be implemented for other platforms and in a variety of ways.

5.1. Overview

FIG. 1 is a high level representation of a networked computer system 20, in particular, a Lotus Notes system. Such system connects a number of personal computers 22 and a number of server computers 24, allowing them to communicate. A user seated in front of personal computer 22 can create, access, modify, and delete data which is shared with other users and resides on server computer 24. Each user has an electronic identity and can communicate with other users via an electronic mail system.

Users of a system implementing the method of this invention can be categorized by the type of actions they take into designers, preparers, and reviewers. An application designer, or designer, builds an application by implementing the method of this invention for a particular business process, for example, a project proposal approval/rejection process, or a bid evaluation process. A preparer gathers together the information that is subject to the review process, for example, the specification of a project proposal, or the details of a bid. Finally, a reviewer participates in the evaluation process by reviewing parts of the document, and eventually, influences the evaluation process itself.

All components presented in FIG. 2 reside on server computer 24. A designer specifies the application to be implemented by entering a workflow graph description 40 into the system. In the current embodiment, workflow graph description 40 is stored as a set of records in a database using a database management system. The particular procedures for accessing a database, programmatically or via a user interface, or for accessing electronic mail components are well known to those skilled in the art.

A preparer accesses a documents/reviews database 38 and creates a document 44. A workflow manager 30 examines document 44, examines workflow graph description 40, and deposits one or more notifications 42 into a system mail box 34. A postman program 32 permanently monitors system mail box 34 and dispatches notifications 42 to appropriate private mail boxes 36 based on the electronic identity specified in each notification 42. A reviewer retrieves not