System for recognizing image - Patent Review 5008946

Description Submit all comments and votes

FIELD OF THE INVENTION

The present invention relates to a system which detects an image and also the elements of the image and, more particularly, to a system which recognizes an image, such as an object or person to turn on or off desired electrical devices, to increase or decrease the output power of the devices, or to otherwise control the devices by responding to the motion or operation of the image in a noncontact manner, it being noted that the present invention is not limited to these applications.

BACKGROUND OF THE INVENTION

The prior art techniques of this kind are used in automatic doors employing photosensors, footboards, etc., warning devices for informing a person of entry or intrusion, and metal sensors. Any of these devices makes use of a noncontact sensor, such as a photosensor, microswitch, electrostatic field-type proximity switch, or electromagnetic sensor, or a mechanical switch, and detects opening or closure of an electrical contact, making or breaking of an electromagnetic wave path, a change in an electric field, or a change in a magnetic field which is caused when an object or a person makes contact with, approaches, or passes through, the device to turn on or off a desired electrical device, such as a buzzer, meter, automatic door, relay, monitor television , or an electrically controlled machine.

This electrical device cannot be controlled, e.g., turned on and off, unless an object or person is close to the device and makes relatively large movement. Since a change in the state of a minute portion of an object or human body cannot be detected by a sensor, an input device consisting principally of keyswitches has been heretofore most frequently used to energize various electrical devices. As an example, various electrical devices are installed on an automobile, and various keyswitches, volumes, etc. are disposed corresponding to those electrical devices. However, if the driver stretches his or her arm or twists around to operate a switch or volume control, then the driving is endangered. Also, it is easy to meticulously operate a switch or volume control, because the driver cannot keep his or her eyes off the front view for a relatively long time to watch a device. Accordingly, it may be contemplated to install a speech recognition apparatus which recognizes the driver's speech and controls various electrical devices. Unfortunately, a large amount of noise takes place inside the automobile and so the recognition involves noticeable error.

In order to automatically control or energize various electrical devices according to the change in the state of a small portion within a broad region and to permit the driver to control various electrical devices relatively precisely in a noncontact manner without requiring great care or large motion, the present inventor has developed an apparatus that turns on and off devices installed on a vehicle in response to the motion of driver's eyes and mouth, as disclosed in Japanese Patent application No. 272793/1985.

This apparatus makes use of image pattern recognition techniques. In particular, this apparatus uses a camera means for converting an image, or information in the form of light, into an electrical signal and a position-detecting means that detects the position of certain portions of the image. In operation, the apparatus takes a picture of an object or person, such as an automobile driver, and detects the positions of the certain portions of the picture, such as the driver's eyes and mouth.

Since the brightness inside the automobile varies, an illuminating means for illuminating the driver, a brightness-setting means for setting the brightness of the illuminating means, and a brightness control means are provided. The brightness control means detects the brightness on the driver's face and adjusts the setting of the brightness-setting means to change the brightness. Thus, the brightness on the driver's face is maintained constant to prevent the image processing from producing error due to variations in the brightness.

The position of the driver's face may be changed by vibration of the automobile body or may vary because of his or her unintentional minute motion or a change in the posture. Also, the eyes and mouth may be intentionally moved to control electrical devices in a noncontact manner as described later. To precisely extract information about the eyes and mouth from image information in response to the changes in the positions of the face, eyes, and mouth, the apparatus further includes a storage means for storing the detected positions, a window setting means for setting a region narrower than the image produced by the camera means according to the stored positions, a means for setting the region covered by a position-detecting means to the narrower region after a certain period of time elapses since the detected positions are stored in the storage means, and an updating means for updating the positions of the aforementioned certain portions within the narrower region which are stored in the storage means. Once the positions of the certain portions, i.e., the eyes and mouth, are detected, the scan made to detect the eyes and mouth is limited to the narrower region and so they can be detected quickly. Further, the accuracy with which the detection is made is enhanced. Consequently, the apparatus follows the eyes and mouth quickly and precisely.

This apparatus is further equipped with a state change-detecting means for detecting the states of the eyes and mouth at successive instants of time to detect the changes in the states. Also, the apparatus includes an output-setting means which supplies a control signal or electric power to an electrical device according to the changes in the states. Specifically, when the states of the monitored eyes and mouth are found to change in a predetermined manner, i.e, it is ready to activate the electrical device, electric power is supplied to the device according to the change.

The apparatus enables the driver to control the electrical device by moving his or her eyes or mouth while assuming a posture adequate to drive the automobile. Therefore, the electrical device installed on the automobile can be quite easily operated. This contributes to a comfortable and safe drive. As an example, when the driver utters a word to indicate something, the electrical device is controlled according to the shape of the mouth. If the driver utters no word but moves the mouth intentionally as if to utter a word, then the electrical device is controlled according to the shape of the mouth. Since the operation of the device is not affected by utterance, the detection involves no error in spite of noise produced inside the passenger's compartment. Also, if the radio set is played, or if a passenger is speaking loudly, it is unlikely that the electrical device is caused to malfunction.

The concept of the aforementioned apparatus can be similarly applied to the case where a person other than an automobile driver is monitored. For example, a similar apparatus allows a patient with an advanced disease to operate, stop, or control the surrounding medical instruments or assisting instruments with his or her eyes and mouth.

The apparatus can also monitor a machine to detect abnormality and protect the machine. A certain part or portion of the machine is checked for trouble. If this part or portion operates abnormally, the operation of the machine is stopped, or a warning device is operated. In this way, the above described apparatus can be also employed with similar utility to monitor an object other than a person.

Further, the invention can be utilized to monitor a broad region such as a natural sight, especially to monitor animals or vehicles moving in the region. For instance, a gate in a safari park can be opened and closed according to the movement of a vehicle or fierce animals. For a manufacturing plant a belt conveyor line can be monitored to check the parts or products on the conveyor. When they move in a given direction, a safety device is operated, or equipment for the next manufacturing step is run. In this way, the aforementioned apparatus can be used with similar utility in the same manner as the foregoing.

The apparatus described above can detect the driver's head, face, and pupils with high accuracy and yield the foregoing advantages when the monitored object, such as the face of the automobile driver, has a relatively uniform brightness, typically encountered when no car is running in the opposite direction at night and substantially only the interior light illuminates the face, thus permitting the monitoring. However, when the driver's face or head is illuminated with intense light emanating from the headlamps either on a car running in the opposite direction or on a succeeding car even at night, or when the sunlight is intense in the daytime, the external light stronger than the light emitted from the interior light is reflected or intercepted by the driver's face or head. In this situation the brightness on the face frequently becomes nonuniform. That is, intense light is reflected from only a portion of the face; the remaining portion is in shadow and darker. As an example, when the automobile is running in fine weather under the sun located to the right of the automobile, the surroundings of the right eye are very bright, while the surroundings of the left eye are quite dark. In this nonuniform illumination, the accuracy with which the driver's pupils are detected deteriorates, because the apparatus uses only one threshold value in digitizing the whole obtained image. Also, the shape of the driver's mouth is detected with decreased accuracy.

Accordingly, the present inventor has developed an improvement over the aforementioned known apparatus to detect elements, such as the pupils or the mouth or both, of a monitored object, such as the driver's face, with increased accuracy, as disclosed in Japanese Patent application No. 169325/1987. The improved apparatus arithmetically obtains a first gradation histogram for each of small neighboring regions, for example the right half and the left half, within a desired portion such as a human face included in the monitored image. Then, a threshold value for each region is determined, based on the histogram. Information about the gradation of the image is digitized, and a characteristic index (HTY) which indicates the boundary between the hair and the forehead, for example, is determined. This boundary extends through the neighboring regions on the monitored face. Opposite sides of the boundary differ in gray level. A second gradation histogram is created from information about the gradation of an image of a set region S.sub.d based on the determined characteristic index (HTY). The set region S.sub.d contains the eyes. Then, a threshold value (TH.sub.e) is determined according to this histogram to digitize the gradation of the image of the region (S.sub.d). Thus, the positions of certain small portion or portions, such as pupils, within the region (S.sub.d) are detected. The certain small portion can be a mouth instead of pupils.

Determination of a threshold value from a gradation histogram and digitization an analog signal are known in the field of object recognition image processing. These techniques are adequate to separate an object located in front of the background from the background of the image when the concentration of the image varies. Accordingly, this improved apparatus can precisely detect the characteristic index which indicates the upper end of the forehead. This digitization is adequate to detect a characteristic index (HTY) indicating the boundary between the background, or hair, and the main portion, or forehead, in each divided region even if the monitored object is not uniformly illuminated or the brightness of the light source itself varies. Hence, the index (HTY) can be detected with accuracy. The index (HTY) represents a reference position on the detected object, or face.

The region (S.sub.d) surrounding the eyes is set according to the characteristic index (HTY). A threshold value is set according to a gradation histogram obtained from this region (S.sub.d). Then, an analog signal is transformed into binary codes, using the threshold value. These techniques are adequate to define the given region (S.sub.d) containing the certain small regions, or pupils, of the detected object, and to separate the pupils whose gray levels suddenly change in the region (S.sub.d), from the background, or the surroundings of the pupils if the object is illuminated asymmetrically or the brightness of the light source itself varies. Consequently, the certain small portions, or the pupils, can be detected accurately. Also, the small portions can be mouth or lips.

In this manner, the improved apparatus is capable of detecting given portions of an object accurately if the object is illuminated asymmetrically or the brightness of the light source itself varies.

If the driver sitting on the driver's seat of an automobile shifts the seat forward or rearward to adjust the posture for driving, the distance between the camera means and the subject, or face, changes. At this time, an automatic focusing device prevents the obtained image from getting blurred. However, the possibility that elements of the image are incorrectly detected, e.g., the nostrils are regarded as the mouth, increases.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a system capable of detecting elements of an image with increased accuracy.

The above object is achieved in accordance with the invention by a system comprising: a camera which converts optical information obtained from the image into an electrical signal; a position-detecting circuit for detecting three or more elements of the image and their positions according to the electrical signal; a distance detecting circuit for detecting the distances between the detected elements; a normalizing circuit for normalizing data about the detected distances with the distance between given two of the detected elements; a storage circuit which hold reference values previously assigned to the elements of the image; a similarity degree calculating circuit which compares the normalized data about the distances with the reference values and produces data about the degrees of similarity to the elements of a reference image; and a determining circuit which determines whether the image has been successfully detected, from the data about the degrees of similarity of the detected elements.

The position detecting circuit detects the positions of three or more elements, such as the right pupil, the left pupil, the nostrils, and the mouth, of an image such as a human face. The distance-detecting circuit detects the distances between the elements. The distances are normalized with the distance between certain elements. Therefore, the normalized data indicating the distances between the elements are substantially independent of the distance between the camera means and the image. The similarity degree-detecting circuit compares the normalized data with reference values which are stored in the storage circuit and have been previously assigned to the elements of the image to produce data about the degrees of similarity of the detected elements to the elements of the reference image. The degrees of similarity indicate the degrees to which the positions of the elements of the optical image formed by the camera bear resemblance to the positions of the elements of the reference image, or the normalized data about the distances between the elements. As the degrees of similarity of the elements increase, the optical image formed by the camera means approaches the reference image. The determining circuit determines whether the elements have been detected successfully, based on the data about the degrees of similarity. That is, if a high degree of similarity is obtained, then it is found that the image formed by the camera approximates the reference image. Conversely, if a low degree of similarity is obtained, then the image formed by the camera is judged to be different from the reference image.

Accordingly, where the elements such as the pupils and the mouth of an automobile driver's face, for example, are detected to turn on and off or otherwise control electrical devices installed on an automobile according to the shapes of the elements and the pattern of change in the shapes, the decision to determine whether the image formed by the camera is the face or not can be made precisely. Therefore, the electrical devices can be controlled with reduced error. Especially, where the distance between the camera and the driver's face change as encountered when the driver moves the seat forward or rearward, the pupils and the mouth can be detected correctly. The electrical devices can be controlled precisely according to the positions of the elements and the changes in the shapes.

Other objects and features of the invention will appear in the course of description thereof which follows.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1a is a block diagram of a system according to the invention;

FIG. 1b is a perspective view of the dashboard of an automobile, for showing the arrangement of the camera 3 and the light 4 shown in FIG. 1a;

FIG. 2 is a flowchart schematically illustrating a sequence of operations performed by the microprocessor 6 shown in FIG. 1a;

FIGS. 3, 4, 5a, 5b, 5c, 6, 7a, 7b, 7c, 8a, 8b, 8c, 8d, 9a, 9b, 9c, 9d, 10a, 10b, 11a, 11b, 12a, 12b, and 12c are flowcharts particularly illustrating operations performed by the microprocessors 6 and 8 shown in FIG. 1a;

FIGS. 13a, 13b, 13c, 13d, 13e, 13f, 13g, and 13h, 13i, 13j are plan views of all or some of images taken by the camera 3 shown in FIGS. 1a and 1b;

FIG. 14 is a diagram showing the relations between the degrees of similarity F.sub.1 -F.sub.14 and the degrees of certitude F.sub.21 -F.sub.32 calculated in the routine (FAD) illustrated in FIG. 9a for checking detection of a face;

FIGS. 15a, 15b, 15c, 15d, 15e, 15f, and 15g are plan views of window regions W.sub.e formed for searching for pupils and detected pupils;

FIG. 16a is a plan view of a window region W.sub.m formed for searching for a mouth and a detected mouth; and

FIG. 16b is a plan view of a mouth, for showing various shapes of the mouth taken to pronounce vowels.

DETAILED DESCRIPTION OF THE INVENTION

Referring to FIG. 1a, there is shown a system embodying the concept of the present invention. This system is installed on an automobile and acts to turn on or off, increase or decrease the power, or otherwise control electrical devices installed on the automobile, according to intentional movement of the pupils and the mouth of the driver's face.

The system includes a TV camera 3 and a light 4 that illuminates at least the driver's face. The camera 3 and the light 4 are combined into a unit and mounted on the instrumental panel 2 so as to be movable vertically and horizontally, as shown in FIG. 1b. Indicated by numeral 1 in FIG. 1b is the steering wheel turned by the driver to steer the vehicle.

Referring again to FIG. 1a, the light 4 is turned on and powered by a light controller 19, which also controls the brightness of the light. The light 4 consists of an incandescent lamp using a filament as a light source. The controller 19 rapidly switches on and off a direct voltage, using a thyristor chopper and applies it to the light 4. When an OFF signal is applied to the controller 19 to turn off the light, the controller turns off the thyristor chopper. When an ON signal is applied to the controller 19, it switches on and off the chopper at a normal duty cycle. When a signal indicating an increase in the brightness arrives at the controller, it increases the duty cycle by one step. When a signal indicating a decrease in the brightness is fed to the controller, it reduces the duty cycle by one step. If the brightness is increased to its maximum level or reduced to its minimum level, the brightness is no longer changed. A microprocessor 6 supplies the aforementioned signals to the light controller 19 via an interface 14 to turn on or off the light or to increase or decrease the brightness.

The TV camera 3 uses a two dimensional CCD array constituting 256 by 256 picture elements per frame. The camera 3 produces an analog video signal. The microprocessor 6 supplies a signal via an interface 17 to the camera 3 to turn on or off the camera. This camera 3 delivers successive video signals indicating 256.times.256 picture elements per frame to an A/D converter 18, and applies pixel sync pulses to the converter 18 for synchronization with A/D conversion. Further, the camera 3 furnishes frame sync pulses, line sync pulses, and pixel sync pulses to the microprocessor 6 via the interface 17.

The microprocessor 6 writes data about one frame of an image to a frame memory 13 in synchronism with the frame sync pulses. The memory 13 consists of a RAM. The address at which data is written to the memory 13 is advanced in synchronism with the line sync pulses and the pixel sync pulses.

In this example, the A/D converter 18 converts its input signal to 8-bit, digital data indicating 256 different gray levels. That is, 8 bits of image data are written to the frame memory 13 per pixel.

Mode switches 5E and 5M are connected to the microprocessor 6. The switch 5E is used to designate a mode in which the system responds to the pupils. When the switch 5E is closed to establish this mode, the microprocessor 6 controls the operation of electrical devices 20, 21, 29, 30 (described later) according to the positions of the pupils and the pattern of a blink.

The switch 5M is used to specify a mode in which the system responds to the mouth. When this switch 5M is closed to establish this mode, the microprocessor 6 controls the operation of electrical devices 20-30 (described later) according to the pattern of change in the shape of the mouth.

Another microprocessor 8 which performs arithmetic operations for making various decisions is connected with the microprocessor 6. A frame memory 11, a ROM 9 for normal computer system control, a RAM 10, a bus controller 16, and the aforementioned frame memory 13 are connected to the microprocessor 6. The frame memory 11 consists of a RAM and temporarily stores binary-coded data about one frame of image. Every bit of the data indicates the presence or absence of a black spot on a pixel. Also, the electrical devices 20-30 and the light controller 19 which are controlled by this microprocessor 6 are connected to the microprocessor 6.

The electrical device 20 is a radio controller that electrically turns on or off the power of a radio set and electrically controls the volume. When an ON signal is applied to the controller 20, it turns on the power of the radio set. When an OFF signal is applied to the controller 20, it turns off the power. When a signal indicating an increase in the volume is fed to the controller 20, it steps up the volume by one step. When a signal indicating a decrease in the volume is applied to the controller 20, it decreases the volume by one step. When the volume reaches its maximum or minimum level, the volume is no longer varied.

The electrical device 21 is an air conditioner controller which simply turns on or off an air conditioner. A controller incorporated in the body (not shown) of the air conditioner makes a switch between cooling and warning, increases or decreases the power, and turns on or off a cleaner according to the room temperature and contamination of the air.

The electrical device 22 is a cruise controller which acts only to operate or stop the cruise control system. As is known in the art, when cruise control is selected, one type of cruise control unit stores the present road speed. Then, the throttle opening is controlled to maintain the actual speed of the automobile at the speed stored in a memory. When the cruise control is ended, the throttle opening is no longer controlled. In another type of cruise control system, when cruise control is selected, the throttle opening is controlled until the actual speed of the vehicle reaches the preset or stored speed which may or may not be altered, taking the present speed into account When the cruise control is stopped, the control over the throttle opening is stopped. In either type, when it is instructed to increase the speed, it increases the set speed by one step. When it is instructed to decrease the speed, it decreases the set speed by one step. These operations are performed under the control of the cruise controller 22. When an ON signal is applied to the controller, it initiates cruise control. When an OFF signal is applied to the controller, it stops cruise control. When a signal indicating an increase in the speed, it increases the set speed by one step. When a signal indicating a decrease in the speed is fed to it, it decreases the set speed by one step.

The electrical device 23 is a door lock controller 23 which locks or unlocks the doors according to the conditions of a driver's scat door lock/unlock switch mounted in the driver's seat door, an all door lock/unlock switch, and lock/unlock switch mounted in the other doors, and also according to the vehicle speed and other conditions of the vehicle. When an ON signal is applied to the controller 23, it causes the doors to be locked unless there is any precluding condition. When an OFF signal is applied to the controller 23, it causes the doors to be unlocked unless there is any precluding condition.

The electrical device 24 is a sunroof controller which, in this example, drives a sliding sunroof. When a normally installed switch is operated, the controller opens or closes the sunroof. When an OFF signal is applied to the controller, it opens the sunroof fully. When an OFF signal is applied to the controller, it closes the sunroof fully. When an UP signal is applied to it, it opens the sunroof by one step. When a DOWN signal is applied to it, it closes the sunroof by one step.

The electrical device 25 is a window controller which vertically moves the sliding windowpanes at the doors to open or close them. When a normally installed switch is operated, the controller 25 opens or closes the sliding windowpanes. When an ON signal is applied to the controller 25, it fully closes the sliding windowpane at the driver's seat door. When an OFF signal is applied to the controller 25, it fully opens the windowpane. When an UP signal is fed to the controller 25, it closes the windowpane by one step. When a DOWN signal is supplied to the controller 25, it opens the windowpane by one step.

The electrical device 26 is a wiper controller that drives the wipers when a normally installed switch is operated. When an ON signal is applied to the controller 26, it drives the wipers at a standard speed. When an OFF signal is applied to the controller 26, it stops the wipers and places them in their standby positions. When an UP signal is applied to the controller 26, it increases the speed of the wipers by one step. When a DOWN signal is applied to the controller 26, it decreases the speed by one step.

The electrical device 27 is a cigar lighter controller. When an ON signal is applied to the controller 27, it recesses a cigar lighter mechanism. When an OFF signal is applied to the controller 27, it extrudes the lighter mechanism even if a predetermined temperature is not reached. The controller operates in a conventional manner in other respects. When the mechanism is recessed, it makes a mechanical contact with the electric circuit, so that the cigar lighter is energized. When the predetermined temperature is reached, the lighter is automatically extruded and gets deenergized.

The electrical device 28 is a headlamp controller which responds to a headlamp switch having three positions--OFF, PARK, and HEADLAMP. In the HEADLAMP position, the controller 28 lights the parking lamps, raises the headlamps from their recessed positions, lights them, selects a low-beam position or a high-beam position according to the setting of a dimmer switch. In the PARK position, the controller 28 puts out the headlamps and lights only the parking lamps. In the OFF position, the controller puts out the parking lamps and recesses the headlamps. When an ON signal is applied to the controller, it lights the parking lamps, raises the headlamps, and lights them. When an UP signal is applied to the controller, it sets the headlamps in the next higher position. When a DOWN signal is supplied to the controller, it sets the headlamps in the next lower position. When an OFF signal is applied to the controller, it puts out the headlamps and lights only the parking lamps.

The electrical device 29 is a defroster controller which energizes or deenergizes the heater embedded in the rear window when a normally installed switch is closed or opened. When an OFF signal is applied to the controller 29, it energizes the embedded heater. When an OFF signal is applied to the controller, it deenergizes the heater.

The electrical device 30 is a buzzer controller. When an ON signal is applied to the controller 30, it operates a buzzer (not shown). When an OFF signal is applied to the controller 30, it ceases to operate the buzzer. As described later, when the driver keeps closing his or her eyes for a given period of time, the controller regards the driver as taking a nap and activates the buzzer. When the mode switch 5E or 5M is closed to carry out the routine responding to pupils or mouth, if the image pattern processing is disabled, then the controller 30 activates the buzzer.

FIG. 2 illustrates the main routines performed by the microprocessors 6 and 8. Particulars, or subroutines, of the main routines are illustrated in FIGS. 3-9d. The manner in which the microprocessors 6 and 8 control the electrical devices in accordance with a set program is described next by referring to these figures.

Referring to FIG. 2, when the electric power is turned on, the microprocessor 6 initializes the input/output ports to turn off the electrical devices 19-30 and the TV camera 3 and to clear or reset internal registers, counters, flags, and the RAMs 10, 11, 13 (step 1).

Then, control proceeds to a subroutine for judging the input mode (IPD), where a decision is made to see whether the switches 5E and 5M are open or closed. This subroutine is particularly illustrated in FIG. 3, where if either the switch 5E or 5M is closed, then control goes to a subroutine for adjusting light (ILC), in which the light controller 19 and the TV camera 3 are turned on. When at least one of the switches 5E and 5M is closed, if both switches are opened, then the light controller 19 and the TV camera 3 are turned off. Also, the buzzer controller 30 is not operated (step 4). If both switches 5E and 5M are open, the microprocessor waits until at least one of them is closed, and the image pattern processing is not carried out.

When at least one of the mode-indicating switches 5E and 5M is closed, control goes to the subroutine for adjusting light (ILC). This subroutine is particularly illustrated in FIG. 4, where the microprocessor 6 supplies ON signals to the light controller 19 and to the TV camera 3. Then, the microprocessor waits until the brightness of the light stabilizes and the driver's face becomes motionless at the driving position. Thereafter, the microprocessor starts to write data about one frame of image to the frame memory 13 in synchronism with the frame sync pulses delivered from the TV camera 3. The data representing gray levels comprises 8 bits per pixel. Thus, the data derived from one frame, or 256 by 256 pixels, is written to the memory 13 (step 5).

The microprocessor 8 then totals the number of pixels belonging to each of discrete gray levels of the image data stored in the memory 13. The lowest gray level, or level 0, indicates the maximum brightness. The highest level, or level 255, represents the darkest condition. In this way, a histogram showing the brightness distribution is created. In this histogram, the gray levels i (i=0-255) are plotted on the horizontal axis and the number of pixels belonging to each level is plotted on the vertical axis. The histogram is stored in the ROM 10. Then, the microprocessor 8 calculates the deviation of the histogram from a reference histogram stored in the ROM 9 for each gray level. The microprocessor 8 totals the squares of every deviation (step 6), and compares the calculated sum with a reference value (step 7). If the sum lies outside a given range, then the microprocessor supplies an UP or DOWN signal to the light controller 19 according to the sum of all the deviations (step 10). Then, after the brightness of the light 4 is regulated at a modified brightness, the microprocessor writes data about another frame of image to the memory 13 (step 5). In this manner, the various arithmetic operations are carried out by the microprocessor 8. Thus, the brightness of the light 4 is controlled in such a way that the histogram originating from the image data lies within a certain range (steps 5, 6, 7, 10).

During the execution of the subroutine for adjusting the light (ILC), when the brightness is set to its highest or lowest level, the light controller 19 informs the microprocessor 6 of this fact. If the microprocessor 6 is informed that the brightness is set to its lowest level (step 8), then it turns off the light controller 19 (step 11), thus ending the adjustment of the brightness. After the light 4 goes out, the microprocessor reads data about one frame of image and writes it to the memory 13 (step 12). If the microprocessor 6 is informed that the brightness is set to its highest level, then it ends the adjustment of the brightness (step 9).

After the execution of the subroutine for adjusting light (ILC) is complete, control proceeds to a head detection subroutine (HDD). This subroutine is particularly illustrated in FIG. 5a. A subroutine 13 for determining a threshold value TH.sub.h used for detection of the head is effected to determine the threshold value. Then, a flag register which acts to store data indicating whether the head is successfully detected or not is first cleared (step 14). Then, a subroutine 15 for detecting the head is carried out, using the threshold value TH.sub.h to detect the position ATY of the front end of the head (FIG. 13c). The detection might end in failure. When it is successfully detected, 1 is written to the flag register in the subroutine 15. In case of failure the contents of the register remain zero.

Upon completion of the execution of the subroutine 15, the contents of the flag register are checked (step 16). If the value held in the register is 0, control goes to a subroutine for error processing 1 (ERP1). This subroutine is particularly illustrated in FIG. 6. A register IPDF which counts the number of errors or unsuccessful detections is incremented (step 17). A decision is made to ascertain whether the number is equal to or in excess of 16 (step 18). If so, the buzzer controller 30 is instructed to operate the buzzer (step 19). Subsequently, control returns to step 5, where an image is read. If the number is less than 16, control goes back to step 5 without operating the buzzer.

If the result of the check made at step 16 (FIG. 5a) is that the value held in the flag register is larger than 16, then control goes to a subroutine for detecting the forehead (BRD). This subroutine is particularly illustrated in FIG. 7a, where a subroutine 20 is first carried out to set the threshold value TH.sub.f used for detection of the forehead. Then, the flag register is cleared (step 21), followed by execution of a subroutine 22 for detecting the forehead. If the forehead is successfully detected in this subroutine 22, 1 is written to the flag register. In case of failure, the contents of the register remain zero.

After completing the subroutine 22 for detecting the forehead, the contents of the register are checked (step 23). If the value held in it is zero, control proceeds to the subroutine for error processing 1 (ERP1) (FIG. 6), where the register IPDF which counts errors is incremented (step 17). A check is made to see whether the total count is equal to or larger than 16 (step 18). If so, the microprocessor instructs the buzzer controller 30 to operate the buzzer (step 19), after which control returns to the subroutine 5 for reading an image. If the total count is less than 16, control goes back to the subroutine 5 without operating the buzzer.

If the check made at step 23 (FIG. 7a) reveals that the value held in the flag register exceeds 0, then control proceeds to a subroutine for detecting the right eye (RED). This subroutine is particularly illustrated in FIG. 8a, where 0.05 and 0.4 are held in registers K.sub.1 and K.sub.2, respectively, which store constants used for detection of pupils (step 24). Then, a subroutine 25 for determining the threshold value TH.sub.e used for detection of pupils is effected.

After completion of the execution of the subroutine 25, the flag register is cleared (step 26). Then, a subroutine 27 for detecting pupils is carried out. In this subroutine 27, if pupils are detected while satisfying a second, stricter condition, then 2 is held in the flag register. If pupils are detected in such a way that the second condition is not met but that a first, milder condition is fulfilled, then 1 is held in the flag register. If pupils are detected in such a manner that neither the first condition nor the second condition is satisfied, then the contents of the flag register are retained at zero.

After the end of the execution of the subroutine 27, the contents of the flag register are checked (steps 28, 29). If the contents are 2, step 31 and the following steps are carried out to calculate the central positions of the pupils. If the contents of the register are 1, a subroutine 30 for modifying the threshold value is executed. Also, the subroutine 25 for determining the threshold value TH.sub.e used for detection of pupils is carried out. If the number held in the flag register is 0, then control goes to step 17.

If the number held in the flag register is 2, the buzzer controller 19 is instructed not to operate the buzzer (step 31). Subsequently, the error count register IPDF is cleared (step 32). A subroutine 33 is then carried out to calculate the central positions of the eyes. Next, a subroutine 33A is executed to calculate a feature. In particular, the number of successive black pixels on the right pupil is counted. Also, the length of the boundary is measured. The obtained values are held in a given register. Then, control proceeds to step 34 (FIG. 2) for executing a subroutine for setting a window W.sub.e. In this subroutine 34, a small region of a given area is defined around the center of the pupil.

Then, a subroutine (LED) for detecting the left eye is carried out in the same manner as the subroutine (RED) for detecting the right eye. That is, the central position of the left pupil, the number of successive black pixels, and the length of the boundary are calculated. The resulting data is held in given registers.

Subsequently, a subroutine 35 is executed to detect opposite ends BRX and BLX (FIG. 13i) of the eyebrows. Control then goes to a subroutine 36, where the mouth is arithmetically found. Control proceeds to a subroutine (MOD) for detecting the mouth. In this subroutine, the central position (MCX, MCY) (FIG. 13j) of the mouth is detected. Then, a window W.sub.m is set (step 39) to define the small region W.sub.m of a given area around the central position of the mouth.

After setting the windows W.sub.e and W.sub.m in this way, a subroutine (NOD) for detecting nose is carried out to detect the center of a group of black pixels in the nose, including the two nostrils. Then, a subroutine (FAD) for checking detection of face is executed to determine whether the image including the detected elements, i.e., both pupils, the mouth, and the nostrils, is a face.

If the result of the decision is that the image is a human face, then control proceeds to step 40 and the subsequent steps for automatic detection and control. Specifically, data about a new image is accepted (step 40). The analog data is digitized (step 41). The image is searched for the pupils (step 44). The image is searched for the mouth (step 51). Decision 1 is made to detect the opening and closing pattern of the eyes and to detect the positions of the pupils (step 49). Decision 2 is made to detect the shape of the mouth and to detect the pattern of change in the shape of the mouth (step 52). An output signal is produced (step 54). If the result of the decision is that the image is a human face, then control goes to the subroutine for error processing 1, whence control returns to the subroutine (IPD) for judging input mode. Then, the aforementioned series of subroutines beginning with the adjustment of light (LCL) and ending with check on detection of face (FAD) is repeated.

The automatic detection and control beginning with the subroutine 40 are carried out in the manner described below. First, data about another frame of the image is written to the memory 13 (step 40). Then, a subroutine 41 is effected to digitize the analog data within the window region W.sub.e, using the threshold value TH.sub.e determined in the subroutine 25. Inside the window region W.sub.m, the analog data is digitized, using a threshold value TH.sub.m determined in the subroutine MOD6 for detection of the mouth. In the other regions, the analog data is digitized, using threshold values TH.sub.fr, TH.sub.fm, and TH.sub.fL determined in the subroutine 20 for detection of the forehead. The resulting digital data is written to the memory 11. Subsequently, the condition of the mode switch 5E is examined (step 42). If it is closed, the subroutine 44 for searching for the pupils is carried out. If the search results in failure, a subroutine (ERP2) for error processing 2 is executed. This error processing 2 is similar to the error processing 1 illustrated in FIG. 6 except that step 18 is replaced by "IPDF.gtoreq.8?". Again, image data is accepted (step 40). The data is digitized (step 41). The image is searched for the pupils (step 44). If the pupils cannot yet be detected after this series of operations is repeated eight times, the present position of the face is regarded as inappropriate or the detection of the pupils made in the subroutine 27 is regarded as inappropriate. In this case, control returns to step 2. If the search for the pupils is successfully made, then decision 1 is made (step 49) to detect the pattern of opening and closing of the eyes and to detect the movement of the positions of the pupils. Data corresponding to the result of the decision is stored in the output register. The IPDF counter which counts the number of unsuccessful detections is cleared (step 50).

Then, a check is performed to ascertain whether the mode-specifying switch 5M is closed (step 43). If it is closed, the image is searched for the mouth (step 51). If the mouth is not yet detected after this series of operations is repeated eight times, then either the present position of the face or the detection of the mouth position made at steps 35-39 is regarded as inappropriate. Control then returns to step 2. If the search for the mouth is successfully made, decision 2 is made (step 52) to detect the shape of the mouth and the pattern of change in the shape. Data corresponding to the result of the decision is held in the output register. The IPDF counter that counts the number of errors is cleared (step 53). In an output subroutine 54, the contents of the output register are transferred to the interface 14.

While the search for the pupils and the search for the mouth are successful, the series of operations including the reading of image (step 40), digitization (step 41), the search for the pupils and the mouth (steps 44, 51), the decisions 1, 2 (steps 49, 52), and the output subroutine (step 54) are repeated. During this process, the aforementioned various threshold values are not modified to detect the pupils and the mouth at a high speed. If the search eventuates in failure, the series of operations beginning with the "decision on input mode (IPD)" and ending with the "check on detection of face (FAD)" is now carried out to update the various threshold values.

In this example, in order to precisely detect various portions of the face even if the face is not illuminated uniformly, the following subroutines, steps, and means are taken into account: subroutine 13 for determining threshold value TH.sub.h used for detection of head; subroutine 15 for detection of head; subroutine 20 for determining threshold value TH.sub.f for detection of forehead; subroutine 22 for detection of forehead; subroutine 24 for determining threshold value TH.sub.e for detection of pupils; subroutine 27 for detection of pupils; the addition and the contents of subroutine 30 for modifying the threshold values; subroutine 24 and step 25 in subroutine 36 for arithmetically finding the mouth region; that the threshold value TH.sub.m used for detection of mouth is determined in the same manner as in subroutine 27 and 30 for detection of the pupils and for modifying the threshold values; that analog image data is digitized, using the threshold value TH.sub.e determined in subroutine 24 for determining threshold value TH.sub.e used for detection of pupils and the threshold value TH.sub.m determined in subroutine 36 for arithmetically finding the mouth region within the set windows W.sub.e and W.sub.m in digitization subroutine 41; calculating the threshold value used for detection of nose (NOD) in the same way as the detection of right eye (RED), digitizing the image information derived from the nose region, using the threshold value to detect the nostrils; and digitizing the image information about the other portions, using the threshold values TH.sub.fr (region 1), TH.sub.fm (region 2), and TH.sub.fL (region 3) (FIG. 4c) determined in subroutine 20 for determining the threshold value TH.sub.f used for detection of forehead. The pupils, the mouth, the nose, etc. are detected with accuracy if the brightness of the illuminating light changes or if the face is not uniformly illuminated, owing to the above-described processings. Therefore, during the execution of the light adjustment subroutine (LCL), if the brightness of the light 4 is required to be adjusted outside the intended range of brightness, the microprocessor does not produce an error signal, but rather detection and control operations subsequent to the head detection subroutine (BRD) are per