Repetitive event analysis system - Patent Review 5414644

What is claimed is:

1. A method for observing and comparing repetitive behavioral activities during the life span of a life form, said life span being recorded in a series of visual records of the behavioral activities, each visual record of the behavioral activity constituting a segment of the life span, comprising:

a) converting each visual record in the series of visual records into a series of digitized images each digitized image having a location with a specific address and storing a time based identity for each digitized image,

b) establishing a textual database and storing in the textual data base written descriptions of the digitized images and a glossary of key words describing each behavioral activity exhibited in the digitized image, each key word being associated with at least one exemplary digitized image representative of said key word, each written description being associated with at least one digitized image the digital images, the location of each digitized image the address of each digitized image and the associated written descriptions and key words forming an information library which is stored in a microprocessor system, the microprocessor being programmed so that a user can search and retrieve from the information library,

1) multiple digitized images of a selected behavioral activity,

2) precursor digitized images showing precursor behavior occurring at a time prior to each digitized image of the selected behavioral activity,

3) digitized images showing responsive behavior occurring at a time following each digitized image of the selected behavioral activity, and

4) digitized images of contemporary events occurring at the same time as the selected behavioral activity,

such that each retrieved digitized image of selected, precursor, responsive and contemporary behavioral activity can be compared and contrasted.

2. A computer based system to analyze a time based visual record of defined events comprising:

a) means for storing, in a numerical order, audio/visual information regarding defined events and related textual information,

b) a plurality of software defined entry paths for searching and retrieving the stored audio/visual information and related textual information including:

1) a first software defined entry path means for searching and retrieving selected audio/visual information, precursor, audio/visual information, responsive audio/visual information, concurrent audio/visual information, and related textual information regarding a selected event,

2) a second software defined entry path for searching a glossary of key words and for retrieving, audio/visual information and textual information related thereto,

3) a third software defined entry path for searching a relationship chart and for retrieving audio/visual images and textual information related thereto, and

4) a fourth entry path means for searching sequentially through audio/visual records, and

c) output means for generating a set of displays of the retrieved audio/visual and textual information, whereby the display generated using the entry path allows the analysis, comparison and contrasting of the displayed audio/visual and textual information.

3. A method of analyzing behavior of one or more subjects of a living species comprising the steps of:

a) collecting audio/visual records illustrating portions of the life of a subject, said portion of the life of the subject including an activity, and optionally, events preceding the activity and events following the activity;

b) creating text describing the audio/visual record of each said portions, activity or events;

c) storing said audio/visual records;

d) indexing said audio/visual records using a sequential coding;

e) associating said text with said audio/visual records;

f) associating said keywords with said audio/visual records;

g) recording said association between said audio/visual records and said text;

h) recording the association between said audio/visual records and keywords;

i) searching said audio/visual records, keywords and text to locate desired visual records;

j) retrieving the audio/visual record, associated keywords and associated text; and

k) comparing and contrasting the audio/visual records having the same keywords associated therewith,

so that the same event recorded for different subjects can be contrasted and the preceding activity and following activity related thereto can be contrasted.

4. The computer based system of claim 2 wherein the defined activity is a repeated event of a selected subject wherein the subject is chosen from the group consisting of one or more species, living subject, organism, lifeform, living or non-living systems, organic or inorganic reaction and physical phenomena.

5. The computer based system of claim 2 wherein said numerical order can be alternatively chosen from a time based numbering system and a straight sequential numbering system.

6. The computer based system of claim 4 wherein the defined activities are different events related to each other, the relationship being shown on a relationship chart.

7. The computer based system of claim 6 wherein the relationship chart can be changed, amended or deleted.

8. The computer based system of claim 6 wherein the relationship chart is based on genealogical information.

9. The computer based system of claim 2 further having means to access additional records of activities and to add new information relating to the previously collected audio/visual records.

10. The method of claim 3 wherein said audio/visual records, keywords and text are displayed in an observable manner such that one or more audio/visual records, keywords and textual description can be concurrently viewed for one or more subjects and for one or more behaviors.

11. The method of claim 3 wherein said audio/visual records, keywords and text can be modified and supplemented and the associations between the audio/visual records, keywords and text can be changed.

12. The method of claim 10 wherein said displayed audio/visual records, keywords and textual descriptions can be compared, analyzed and contrasted.

13. The method of claim 3 wherein the recorded audio/visual records and associated text include more than one subject and records of similar behaviors of each subject is observable concurrently by selecting a behavior which is common to each subject.

14. The method of claim 13 wherein a subject is chosen from the list consisting of one or more species, living subject, organism, lifeform, living or nonliving systems, organic or inorganic reaction and physical phenomena.

15. The method of claim 1 wherein said audio/visual record is selected from the group consisting of still and motion images, series of images, videotaped records, film, notes, maps, photographs, videoclips, individual video frames and video frame sequences.

16. The method of claim 15 wherein said audio/visual record is delivered from the group of media consisting of CD-ROM, laser disc, video file server, video ethernet and data delivery network.

17. The method of claim 1 wherein the microprocessor is further programmed for creation of a second textual database containing written descriptions of the digitized images and a second glossary of keywords describing selected behavioral activities; whereby each keyword in the second glossary can be different than previously established keywords such that a customized user specific database is established.

18. The method of claim 1 wherein the textual database and written descriptions can be modified and supplemented by a user so that said user can rearrange the visual records and classify the rearranged records with a unique coding selected by the user.

19. The method of claim 1 wherein a user can add new textual input to said textual database.

20. The method of claim 1 wherein a source tag can be assigned to said visual records along with the time based identity to allow a user to view visual records organized alternatively by information source and by said time based identity number.

21. The method of claim 1 further incorporating means to compare behavior activities at different times, different activities at the same time, different views of the same activity, activities concerning different subjects at the same or different activities concerning different subjects at the same time.-

22. The method of claim 1 further including the step of displaying a series of visual records relating to a selected behavioral activity beginning at a selected period of time prior to the activity and displaying each successive record related thereto, whereby the visual records are viewed in the same manner as a motion picture.

23. The method of claim 1 wherein textual information regarding an observed incident is viewed.

24. The method of claim 1 wherein information regarding an observed lifeform, the parentage, siblings and offspring of the observed lifeform and descriptive text related thereto can be viewed along with the visual record.

25. The method of claim 1 wherein different visual records showing the same behavior can be viewed.

26. The method of claim 25 wherein each different visual record can be identified by discrete addresses and time based identity within the data base and textual information relating thereto can be displayed.

27. The method of claim 25 wherein each different visual record can be viewed as a collection of screen frames displaying the behavioral activity along with textual descriptions for each sequence.

28. The method of claim 27 wherein the textual descriptions regarding a visual sequence is accessible separate from the visual record.

Description Submit all comments and votes

BACKGROUND

The present invention relates to techniques to evaluate behavior so that causes and/or intentionality (purpose) of time based events or acts can be identified; learned behavior can be analyzed; and responsive conduct can be predicted.

In the past, investigators desiring to study learning, behavior or stimulus/response interplay had to spend years observing activities of the subject and related subjects. Even when timebased video tapes, movies, audio tapes and notes were extensively compiled, it was virtually impossible to identify and analyze similar behavior in an efficient, scientific, and methodical manner. Additionally, since different researchers may perceive the same act in the same species or organism in a different light, their different methodologies made it difficult to establish a consistent basis for comparison. Thus, the majority of behavioral studies were anecdotal and were not amenable to statistical analysis or even side by side comparison.

Systems have been developed for indexing single sets of information and retrieving information, whether textual or visual, for viewing. However, these systems have been designed simply to read and observe the information and possibly append related information. For example, material presented in books, encyclopedias, magazines, catalogs, etc. and related visual information (i.e., still or moving pictures) can be stored on digital recording media or in a computer memory. This information can then be accessed interactively, i.e. simultaneously presented on both audio and visual equipment. Prior art systems, such as the Grolier CD-ROM based encyclopedia, had limited access to various types of information. The Grolier CD-ROM allowed access to textual information but did not have the capability of storing and accessing pictures or sound. Other systems, such as the World GeoGrahics software, a graphics based atlas, allowed access to visual images but only provided access to textual material directly connected to the graphic image selected. U.S. Pat. No. 5,241,671, issued Aug. 31, 1993 to Reed et al, which is incorporated herein by reference, discloses a multimedia search system using a plurality of entry path means to retrieve similarly coded textual and graphical information stored in a data base so that the information located in different data bases can be simultaneously presented by different media modes. However, neither Reed et al. nor the other prior disclosed systems disclose or suggest that they can be used for comparing, analyzing, or predicting repetitive behavior in a life form.

Thus there is a need for a system which will allow a researcher to readily access information regarding repetitive behavior of life forms contained in voluminous multimedia information databases for comparison. There is also a need for researchers to simultaneously observe precursor and responsive behavior in the same subject performing the same or similar activity over a period of time. There is still a further need to be able to readily observe, compare and analyze different subjects or different species showing similar behavior in the same or disparate environments.

SUMMARY

These needs are met by the present invention which comprises a method of storing and retrieving textual and time based graphic information regarding repetitive behavioral activities and displaying that information so that similar acts or developmental stages can be visualized and compared and precursor events and responsive acts can be analyzed. This allows behavior or cognitive processes to be understood and quantified and performance and responses to be predicted.

DRAWINGS

These and other features, aspects and advantages of the present invention will become better understood with reference to the following description, appended claims, and accompanying drawings, where:

FIG. 1 shows the Activity/Behavioral access screen.

FIG. 2 shows the Subject/Genealogical access screen.

FIG. 3 shows the Library screen.

DESCRIPTION

A system which incorporates features of the present invention includes graphic information (still and motion images) stored in an accessible medium, such as a CD-ROM, laser disc, video file server, or information accessible via a connection, such as a video ethernet or other data delivery network, or any other recording medium that can be computer controlled.

There was no suggestion in the prior art systems that labeled and indexed visual information regarding life forms or living species could be used for comparing acts of the form or species recorded at different times, with different subjects and by different observers to determine cause/effect relationships and/or identify learned behavior or cognitive processes. This capability is enhanced by the ability of the system user to access textual information related to the images, textual information describing similar behavior, or textual information appended by the user or other observers of the displayed image to reach a hypothesis regarding the observed behavior. Further, the system gives the media observer the ability to access old information in a new manner and to arrive at new conclusions. Because of the manner of storage and ease of complete retrieval, hypotheses arrived at can then be supported by a statistical analysis or directly observed representations rather than just anecdotally supported conclusions.

The technique of the present invention is distinguishable from prior art indexing systems in that the prior systems were used merely to locate and display discrete instances of behavior or information and not behavioral events in the context where they occurred. As such there was no need or suggestion that these prior systems were applicable to observed behavior or that they could be used to analyze or compare cause and effect, behavior or cognitive processes in related situations by contextualized information from related time based events.

Systems incorporating the invention can utilize any technique of storage of audio/visual information whereby specific events can be cataloged, and all audio/visual information similarly cataloged can be readily accessed and retrieved. It is preferred to convert audio/visual information originally recorded in an analog form into a digital format such as CD-ROM. However, other readily accessible storage media for audio/visual information, such as hard drives, video tape, magneto-optical drives, laser disks, magnetic tape or any other means of storing digitized audio/visual media can be utilized in conjunction with computerized controls. Also, any means of accessing such information, such as a cable, network or a video ethernet can be utilized.

The system is designed to operate on any computer system, such as an Apple Macintosh, Amiga, IBM PC, Sun Workstation, or any other computerized system having an adequate memory, an audio/visual display, and a two way interface with the stored audio/visual information. This system can also work on a TV top system or any system with a means for storing, displaying and searching video, sound and text or any other sensory information. A particular preferred system comprises an Apple Macintosh Model II having at least 4 Megs of memory, a 20 Meg hard drive, a running System 7, Hypercard software utilizing the Quicktime driver, a mouse and keyboard and a color monitor with a resolution of 640 by 480 able to display text, play sound and display video clips, and a CD-ROM drive (or a laser disk) able to deliver text and video clips. A more preferred system can include a higher resolution monitor, a faster CD-ROM drive, a larger hard drive and more memory.

While prior art systems are primarily adapted for use as a multimedia encyclopedia database and search system, the present system is specifically designed for identifying, labeling, and comparing events recorded with a time based code as they occur and in their original context, particularly events which involve life forms or living subjects, so that the recorded event can be compared or contrasted with similar events involving the same living subject recorded at a different time or similar events involving different live subjects in the same or different locales or environments, whether recorded by the same or different observers.

Also, non-time dependent information as well as visual and textual records can be given a sequence, rerecorded with a time code and then appended to the time based records. Thus, these non-time dependent events and visual records can be merged into a time-based system and retrieved, searched and viewed in the same manner that a time based record can be retrieved, searched and viewed. Such a system allows for one consistent access mechanism and thus simplifies the retrieval of dissimilar and separate information. This logic also can be extended to other sensory input such as sound, speech and smell, which would allow access to any stored information as a type of multi-sensory movie or a time-slice from any sequence. While technology to store and retrieve smells, taste and tactile sensations are presently not available, the systems incorporating the invention would be applicable to identifying, labeling and comparing the sensory outputs once the technology is developed.

Moreover, the technique permits the automatic splicing of time-slices so that an appropriately scientific presentation can be made from the provided information. This presentation could show the clips in the requested sequence, including voice and text annotations and background sounds of the subject presented.

For descriptive purposes, the life form under observation is referred to as "the subject." To utilize the system a glossary is established by assigning a key word or phrase to each behavioral archetype. The key word is then elaborated both verbally and by a representative series of images showing the behavior. Each living subject is identified by name and image; and if the subject is part of a community under observation, then a genealogical chart and relationships within the community or other network of association is defined. In addition, a chronological time base is established for each subject as well as the community at large which can be linked to world or solar time. A detailed log is then made of the recorded field observations, identifying each recorded event using the glossary with the event being referenced to the particular subject, the subject's chronological history, and the chronology of the community or network under observation. Additionally, each image or time segment of video within the activity, as well as each activity having an assigned key word or phrase, is identified by a location in the time based data archive. A second textual data base is established which includes any written records of events transcribed contemporaneously with the audio/visual record, as well as any later recorded analysis or comparison, these transcribed records being likewise keyed to the subject, chronological time and the time based image location in the graphic database. In this manner, any specific activity and the context of that activity, including prior (a precursor event) or following (responsive) events or contemporaneous event or activities as well as any textual or graphical information related thereto, no matter how voluminous the data base, can be observed and compared with any other similar activity (i.g., EKG output, weather patterns, time of day, etc.) Likewise, where a particular precursor event is believed to cause the activity under study, all instances of the same type of precursor event can be readily accessed and represented in some media form to confirm or disprove the incidence of the cause/effect relationship. The recording, labelling and categorization of particular repetitive behaviors can then be analyzed in a methodical and scientific manner and statistically analyzed. Until now, it was often necessary to wade through hours of video records to locate one telling moment. As a result, a researcher trying to analyze and compare video records of repetitive behavior usually found the task to be so time consuming as to prohibit a thorough and comprehensive scientific analysis.

The utility of the system is best illustrated by specific examples. Jane Goodall spent 32 years observing chimpanzees in their natural habitat. Over the course of those years she collected one of the most comprehensive records of animal behavior ever compiled, consisting of over five hundred hours of film and video taped records of the chimpanzee community at Gombe, Tanzania along with voluminous field notes, maps and photographs. A portion of this information has been transferred to a CD-ROM, laser disk or other digitized storage medium and the remainder is in the process of being transferred. A text-and-video glossary of 333 basic activities or behaviors has been established. Added to this was an identification of 150 chimps in the Gombe community along with the relationship of each chimp in the community (parentage, position in the hierarchy) and typical behaviors. It was then possible, for example, to view the life of a particular chimp as well as its family. By entering in the name of a particular chimp, Fifi for example, and moving the cursor along Fifi's life line, the whole recorded life of the chimp from infancy to old age can be viewed without confusion with other subjects. This reveals the developmental trajectory of Fifi over her life and allows the researcher to see how much Fifi learned about a certain activity and what the learning apparati were. The operator also learns that Flo is Fifi's mother and Frodo is an offspring of Fifi. A characteristic activity of the chosen chimp can then be observed, such as "stone throwing", and that activity can be observed from its earliest incidence to a later modified stage or a still further evolution of the activity.

In this manner, the use of some tools may be observed from its earliest discovery by a single animal, along with any instances of use by the animal being observed by others or learning the act. Evolution of a learned activity can thus be observed using this technique. In a specific example, chimps have learned to use sticks to pick up insects crawling on land. By observing this behavior over a period of time it was discovered that the chimp transferred this learning by attempting to adapt the behavior to other environments, such as attempting to use a stick to pick up water bugs in a stream bed.

While researchers in the past observing video recordings of similar acts occurring at different times have reached the conclusion that a dominant chimp actually communicated by soft vocalization and eye contact to formulate a course of action against strangers to the community, this conclusion would be easier to reach and more statistically based or confidently framed by using a system incorporating the invention.

Using the system, activities of separate communities of similar species, such as other chimpanzee communities, can also be compared for similarities and differences. In this way, the chimps living in a natural environment can be compared with chimps in captivity, allowing the enrichment of the environment of the captive chimps by providing a more natural living experience. Further, by comparing the behavior of primates in captivity to that of humans in an enclosed setting, such as in an institution, prison or nursing home, similar psychopathic behaviors can be discovered.

A preferred embodiment of the system has three methods of accessing the information with each method having a screen view associated therewith. The Activity/Behavior screen is shown in FIG. 2, the Biographical screen is shown in FIG. 3 and the Library screen is shown in FIG. 4. The discussion below uses a standard mouse to activate (click on) a screen toggle or move (drag) an image to a new location. Every screen has a return button 18 which allows the user to switch to the previously viewed screen type.

Activity/Behavior Screen

The Activity/Behavior screen (FIG. 1) allows access to similar behaviors and precursors and responsive behaviors, related behaviors of a chosen subject. The behavior keyword 20 identifies what is currently being viewed in the video/list panel 22 and set forth in the descriptive text 24. If at any time the user wishes to switch to a different behavior, clicking on the visible behavior keyword 20 will present in the video/list panel 22 a list of behavior keywords 26 which can be selected. Clicking on this selected keyword 26 will display an appropriate video clip.

Positioned below the video/list panel is a button bar 28 which includes a library button 30, a biography button 32 and a next behavior button 34. Selecting the next behavior button 34 will present a subsequent instance of the current behavior. The keyword 20 identifying the type of behavior graphically displayed is indicated at the top of the screen. For example if `Play` is the next behavior keyword 20 then the system, organized chronologically, will search for the next time a `Play` classified behavior occurred and the associated section of the visual image will be displayed. Selecting the biography button 32 will jump the user to the biography screen FIG. 3 and show or indicate the identity of the subject being displayed.

Below the button bar 28 is the descriptive text 24. The descriptive text 24 is a brief textual description of the displayed behavior, related behaviors, the subjects involved and related subjects. Certain words in the descriptive text 24 are highlighted keywords 36 and can be clicked on to access other related behavior or subject biography information. When the highlighted keywords 36 are clicked on the related information and visual displays are made available for viewing.

If the library button 30 is pressed the highlighting on the words disappears and all of the words in the text can act as an index into the library. Therefore, a user can click on any word and this will cause a search serially in the table of contents, index, full text and any other text stored in the data base until a hit or near hit is made (i.e., fuzzy logic). The result of the search will appear on the library screen (FIG.