Image processing apparatus for vehicles

We claim:

1. An image processing apparatus, comprising:

image pickup means mounted on a vehicle for shooting an image in front of the vehicle to attain image information;

edge pixel extraction means for scanning for each line the image information of an image in front of the vehicle obtained through the shooting operation by the image pickup means, checking boundaries between a particular color and other colors, and extracting pixels having the particular color as edge pixels;

candidate line extraction means for referencing position coordinates of the extracted edge pixels, assuming the edge pixels to be classified into straight lines, obtaining straight lines by coupling the edge pixels corresponding to respective results of the classification, and setting the obtained straight lines as candidate lines wherein multiple candidate lines are obtained for at least one of a left and a right line;

line selection means for selecting the left line and the right line in front of the vehicle from the candidate lines; and

wherein the line selection means includes means for selecting two of the candidate lines which have the largest numbers of edge pixels on the left and right sides of the image information of the vehicle as the left and right lines of the vehicle, respectively.

2. An image processing apparatus according to claim 1, wherein the line selection means includes means operative in an operation to select the left and right lines for selecting as the left and right lines two candidate lines satisfying a particular positional relationship.

3. An image processing apparatus according to claim 2, wherein the line selection means includes means operative in an operation to select the left and right lines by assuming a particular horizontal line which is parallel to a horizontal direction of the front image information and which has a particular length for selecting as the left and right lines two candidate lines arranged to be brought into contact with both ends of the horizontal line.

4. An image processing apparatus, comprising:

image pickup means mounted on a vehicle for shooting an image in front of the vehicle to attain image information;

edge pixel extraction means for scanning for each line the image information of an image in front of the vehicle obtained through the shooting operation by the image pickup means, checking boundaries between a particular color and other colors, and extracting pixels having the particular color as edge pixels;

candidate line extraction means for referencing position coordinates of the extracted edge pixels, assuming the edge pixels to be classified into straight lines, obtaining straight lines by coupling the edge pixels corresponding to respective results of the classification, and setting the obtained straight lines as candidate lines wherein multiple candidate lines are obtained for at least one of a left and a right line;

line selection means for selecting the left line and the right line in front of the vehicle from the candidate lines; and

wherein the line selection means includes means for selecting as one of the lines of the vehicle, namely, a selection line, one of the candidate lines having the largest number of edge pixels on the left or right side of the image information and selecting, in an operation to select the other line by assuming a particular horizontal line which is parallel to a horizontal direction of the front image information and which has a particular length, as the other line a candidate line arranged to be brought into contact with an end of the horizontal line when the selection line is most close to the other end thereof.

5. An image processing apparatus for identifying a line of vehicle disposed on a running road surface according to color image data of the road surface, comprising:

color image pickup means for shooting an image in front of a vehicle along a running direction thereof;

edge pixel extraction means for scanning for each line the image data of an image in front of the vehicle obtained through the shooting operation by the color image pickup means, checking boundaries between a particular color and other colors, obtaining pixels having the particular color as edge pixels, and extracting pixels satisfying a particular extraction color condition from the obtained edge pixels;

color sampling means for sampling color data of the decided pixels from a single screen of the image data;

extraction condition decision means for determining the extraction color condition according to the sampled color data; and

line image extraction means for attaining as a line of the vehicle a line image configured by coupling the extracted edge pixels into a straight line.

6. An image processing apparatus according to claim 5, wherein the extraction color condition means includes means for deciding the sample pixel positions according to road surface data representing the recognized line.

7. An image processing apparatus according to claim 5, further including:

means for deciding an illumination environment of the running road surface according to the sampled color data; and

means responsive to a result of the decision for controlling a state of an iris of an optical system disposed in the color image pickup means and a color conversion gain of an image data processing system thereof.

8. An image processing apparatus according to claim 5, further including means for updating the extraction color condition according to a state of the sampled color data and a number of pixels satisfying the extraction color condition.

9. An image processing apparatus according to claim 5, wherein the line selection means includes means for selecting, in an operation to select either one of the left and right lines by assuming a central perpendicular line of front image information, as a line of the vehicle, one of the candidate lines which has the largest number of edge pixels on the left or right side of the central perpendicular line.

10. An image processing apparatus according to claim 9, wherein the line selection means includes means operative in an operation to select the left and right lines for selecting as the left and right lines two candidate lines satisfying a particular positional relationship.

11. An image processing apparatus according to claim 10, wherein the line selection means includes means operative in an operation to select the left and right lines by assuming a particular horizontal line which is parallel to a horizontal direction of the front image information and which has a particular length for selecting as the left and right lines two candidate lines arranged to be brought into contact with both ends of the horizontal line.

12. An image processing apparatus according to claim 9, wherein the line selection means includes means for selecting as one of the lines of the vehicle, namely, a selection line by assuming a central perpendicular line of front image information, one of the candidate lines having the largest number of edge pixels on the left or right side of the central perpendicular line and selecting, in an operation to select the other line by assuming a particular horizontal line which is parallel to a horizontal direction of the front image information and which has a particular length, as the other line a candidate line arranged to be brought into contact with an end of the horizontal line when the selection line is brought into contact with the other end thereof.

13. An image processing apparatus, comprising:

image pickup means for shooting an image to attain image information;

image magnification process means for magnifying a size of the obtained image information;

edge pixel coordinate sense means for scanning for each line the magnified image information, checking boundaries between a particular color and other colors, and extracting pixels on the boundaries, and obtaining position coordinates of the extracted edge pixels;

edge pixel coordinate conversion means for converting the position coordinates of the extracted edge pixels into coordinates before the magnification process;

line image extraction means for referencing the converted coordinates of the edge pixels and coupling the edge pixels into a straight line, thereby attaining a line image; and

magnifying condition storage means for storing therein a zoom rate as a magnitude of magnification of the image information and a magnification center coordinate indicating a reference position for the execution of the magnification process of the image information, the magnification factor and the magnification center coordinate being defined for each horizontal line, wherein

the image magnification process means includes means for referencing storage contents in the magnifying condition storage means and magnifying for each horizontal line the size of the image information with the magnification center coordinate set to a center of magnification according to a magnification factor indicated by the zoom rate.

14. An image processing apparatus according to claim 13, wherein the image magnification process means includes means for magnifying the obtained image information only in the horizontal direction.

15. An image processing apparatus according to claim 13, wherein the magnifying condition storage means stores therein values of the zoom rate, the values continuously varying for each horizontal line.

16. An image processing apparatus according to claim 13, wherein the line extraction means includes means for obtaining, as the magnification center coordinate of each horizontal line, a center position of a vehicle running lane enclosed with the left and right lines in each horizontal line and storing the coordinate in the magnifying condition storage means.

17. An image processing apparatus according to claim 13, wherein the magnifying condition storage means beforehand stores therein values of the zoom rate, the value for horizontal lines existing above a particular horizontal line in the perpendicular direction being different from that for horizontal lines including the particular horizontal line and existing below the particular horizontal line in the perpendicular direction.

18. An image processing apparatus according to claim 14, wherein the magnifying condition storage means stores therein values of the zoom rate, the values continuously varying for each horizontal line.

19. An image processing apparatus according to claim 14, wherein the line extraction means includes means responsive to the coordinate data of extracted lines obtained by the process in a previous line extraction cycle for attaining a zoom rate and a magnification center coordinate for each horizontal line and storing the zoom rate and the magnification center coordinate in the magnifying condition storage means.

20. An image processing apparatus according to claim 14, wherein the line extraction means includes means for obtaining, according to a value obtained by dividing a predetermined screen width by a width of a vehicle running lane enclosed with the left and right lines in each horizontal line, the zoom rate for each horizontal line and storing the zoom rate in the magnifying condition storage means.

21. An image processing apparatus according to claim 15, wherein the line extraction means includes means for obtaining, as the magnification center coordinate of each horizontal line, a center position of a vehicle running lane enclosed with the left and right lines in each horizontal line and storing the coordinate in the magnifying condition storage means.

22. An image processing apparatus according to claim 14, wherein the magnifying condition storage means beforehand stores therein values of the zoom rate, the value for horizontal lines existing above a particular horizontal line in the perpendicular direction being different from that for horizontal lines including the particular horizontal line and existing below the particular horizontal line in the perpendicular direction.

23. An image processing apparatus, comprising:

image pickup means mounted on a vehicle for shooting an image in front thereof to attain image information;

image magnification process means for magnifying a size of the obtained image information;

edge pixel coordinate sense means for scanning for each line the magnified image information, checking boundaries between a particular color and other colors, and extracting pixels on the boundaries, and obtaining position coordinates of the extracted edge pixels;

edge coordinate conversion means for converting the position coordinates of the extracted edge pixels into coordinates before the magnification process; and

line extraction means for referencing the converted coordinates of the edge pixels and coupling the edge pixels into a straight line, thereby attaining the left and right lines and magnifying condition storage means for storing therein a zoom rate as a magnitude of magnification of the image information and a magnification center coordinate indicating a reference position for the execution of the magnification process of the image information, the zoom rate and the magnification center coordinate being defined for each horizontal line, wherein

the image magnification process means includes means for referencing storage contents in the magnifying condition storage means and magnifying for each horizontal line the size of the image information with the magnification center coordinate set to a center of magnification according to a magnification factor indicated by the zoom rate.

24. An image processing apparatus according to claim 23, wherein the line extraction means includes means responsive to the coordinate data of extracted lines obtained by the process in a previous line extraction cycle for attaining a zoom rate and a magnification center coordinate for each horizontal line and storing the zoom rate and the magnification center coordinate in the magnifying condition storage means.

25. An image processing apparatus according to claim 23, wherein the line extraction means includes means for obtaining, according to a value obtained by dividing a predetermined screen width by a width of a vehicle running lane enclosed with the left and right lines in each horizontal line, the zoom rate for each horizontal line and storing the zoom rate in the magnifying condition storage means.

26. An image processing apparatus according to claim 23, wherein the image magnification process means includes means for magnifying the obtained image information only in the horizontal direction.

27. An image processing apparatus according to claim 23, wherein the magnifying condition storage means stores therein values of the zoom rate, the values continuously varying for each horizontal line.

28. An image processing apparatus according to claim 23, wherein the line extraction means includes means for obtaining, as the magnification center coordinate of each horizontal line, a center position of a vehicle running lane enclosed with the left and right lines in each horizontal line and storing the coordinate in the magnifying condition storage means.

29. An image processing apparatus according to claim 23, wherein the magnifying condition storage means beforehand stores therein values of the zoom rate, the value for horizontal lines existing above a particular horizontal line in the perpendicular direction being different from that for horizontal lines including the particular horizontal line and existing below the particular horizontal line in the perpendicular direction.

Description Submit all comments and votes

TECHNICAL FIELD

The present invention relates to an image processing apparatus for selecting an image indicating a particular straight or curved line according to image information obtained through an image pickup operation, and in particular, to an apparatus operative according to image information obtained through an image pickup operation by an onboard camera for correctly recognizing a line for a vehicle with respect to a lane on which the vehicle is running.

BACKGROUND ART

An apparatus for recognizing a running route or lane area according to image information obtained through an image pickup operation by an onboard camera has been described, for example, in the JP-A-4-134503 (article (1)).

The apparatus above includes scan means for extracting a running route edge from image information inputted via image input means and scanning upward an image associated with the extracted running route edge beginning at a particular reference point in a horizontal direction and means for setting a center of the scanned area to a particular reference point in the horizontal direction for the subsequent scanning operation such that the running route zone according to the area scanned by the scan means.

In other words, a color corresponding to pixels which constitute the image information and which are at the lower portion of the screen is assumed as a particular color (color of the road surface) to examine an area including a set of pixels having color information equivalent to the particular color, thereby determining a zone or an area enclosed with the particular color as a road area in front of the vehicle.

However, in accordance with the prior art described in article (1), although the zone enclosed with the particular color is assumed as a road area in front of the vehicle, there still remains a problem, for example, occurrence of an event in which the road zone enclosed with road edges (corresponding to lines of the vehicle) cannot be exactly extracted only by the technology.

For example, in a case in which the contour of the road area in front of the vehicle is complicated due to presence of wheel tracks, it is determined that there exist edges of the running route in addition to the line in the road. Resultantly, the edge cannot be appropriately recognized. In addition, there have frequently occurred a case in which the road zone cannot be correctly recognized.

On the other hand, for such a vehicle which runs on a road, it is necessary for the driver to appropriately drive the vehicle so that the vehicle runs along a predetermined running lane.

Consequently, the driver is required to continuously monitor the running road surface to pay attention to the state of the road surface and presence of other vehicles and obstacles. Therefore, a heavy psychological and physical load is imposed on the driver.

As a result, if the driver loses concentration or the driver's attention is distracted in a driving state in which the vehicle exceeds its own running lane at random, there may possibly occur a grave accident.

In particular, on a general road including opposing traffic lanes in which vehicles run mutually in the opposite directions, if a vehicle exceeds a central line of the road into the opposing traffic lane, there will possibly occur quite a serious accident.

To solve the problem, the JP-A-62-221800 (article (2)) has disclosed an apparatus in which an image of a road surface in front of the pertinent vehicle is obtained by a color image pickup device. According to the obtained color image signal, a line of the vehicle is decided according to the difference in color between the line and the road surface to thereby issue an alarm message to the driver of the vehicle.

In the conventional technology described in article (2), the existence of the line is determined according to a color representing the line, namely, yellow or white.

Since the road surface is grey and hence it is guaranteed to discriminate the road surface from the yellow line. However, the discrimination is impossible by the color difference between the white line and the grey road surface and it is therefore necessary to achieve the discrimination according to luminance.

Consequently, in a case in which the state around the road, namely, the road environment is altered, for example, luminance varies in a tunnel (an obtained image is yellow due to the yellow illumination) or due to illumination at night, it is difficult to recognize the line of vehicle according to the change in the state.

On the other hand, in the apparatus in which the line of vehicle is recognized according to image data obtained by the onboard camera, there has been generally used a method in which the state in front of the vehicle is shot by one camera. However, such an apparatus is attended with the following problems. To recognize the state in a wide range, when there is disposed a wide-angle lens, it is difficult to recognize the line in the faraway place. On the other hand, to recognize the line in the faraway place, when there is adopted a zoom mechanism of the optical system, the recognizable field of vision is restricted.

To improve the apparatus, there has been disclosed an apparatus described, for example, in the JP-A-06-229760 (article (3)).

The apparatus includes a solid-state image pickup block including an optical system having a low zoom rate and a solid-state image pickup block including an optical system having a high zoom rate. Comparing image data attained by the block of low zoom rate with that obtained by the block of high zoom rate, it is recognized that the running route in a relatively faraway place in front of the vehicle has a contour of a curved route or that the vehicle is in a state to enter a curved route.

However, according to the prior art of article (3) described above, the configuration includes a solid-state image pickup block including an optical system having a high zoom rate to recognize the contour of the line in a relatively faraway place. Therefore, there are required two optical systems for the high and low zoom rates and two solid-state image pickup blocks for the respective optical systems. This leads to a problem, namely, the size of the apparatus is increased and hence the apparatus is not necessarily suitable as a device to be mounted on a vehicle; moreover, the production cost thereof is inevitably soared.

Additionally, in consideration of the environment related to the mounting of the apparatus, namely, in the environment of a high temperature and a large amplitude of vibration, it is strongly desired that the number of optical systems is to be possibly reduced in the configuration of the apparatus.

It is therefore a first object of the present invention in relation to the problem of the prior art of article (1) to provide an apparatus in which when recognizing a line of vehicle for a running lane of the vehicle according to image information obtained by a camera mounted thereon, the line of vehicle can be correctly recognized even there exists such an edge other than the line as a noise in the road area in front of the vehicle.

A second object of the present invention in relation to the problem of the prior art of article (2) is to provide a road surface monitor apparatus employing an image process in which changes in luminance and color of the road surface just in front of the vehicle are determined to discriminate the road surface from the line of vehicle according to the luminance and color, thereby appropriately recognizing the line.

A third object of the present invention in relation to the problem of the prior art of article (3) is to provide a road surface monitor apparatus employing an image process in which when recognizing a line of vehicle for a running lane of the vehicle according to image information obtained by a camera mounted thereon, the configuration of the apparatus is not required to include a plurality of optical systems and the line can be correctly recognized in a running lane in a relatively faraway place while retaining a wide field of vision.

DISCLOSURE OF INVENTION

To achieve the first object, there is provided according to the present invention an image processing apparatus including an image pickup unit for shooting an image to attain image information, an edge pixel extraction unit for scanning for each line the image information obtained through the shooting operation by the image pickup unit, checking boundaries between a particular color and other colors, and extracting pixels having the particular color as edge pixels, a line image extraction unit for coupling the extracted edge pixels into a straight line, thereby attaining a line image, and a line image selection unit for selecting two line images according to a predetermined rule from the extracted plural line images.

The image processing apparatus favorably includes an image pickup unit mounted on a vehicle for shooting an image to attain image information, an edge pixel extraction unit for scanning for each line the image information obtained through the shooting operation by the image pickup unit, checking boundaries between a particular color and other colors, and extracting pixels having the particular color as edge pixels, a candidate line extraction unit for referencing position coordinates of the extracted edge pixels, assuming the edge pixels to be classified into straight lines, obtaining straight lines by coupling the edge pixels corresponding to respective results of the classification, and setting the obtained straight lines as candidate lines, and a line selection unit for selecting a left line and a right line in front of the vehicle from the candidate lines.

Incidentally, in the image processing apparatus, the line selection unit favorably selects, in an operation to select either one of the left and right lines by assuming a central perpendicular line of front image information, as a line of vehicle one of the candidate lines which has the largest number of edge pixels on the left or right side of the central perpendicular line.

In addition, favorably, an image in front of the vehicle is shot by an image pickup unit mounted on the vehicle, thereby obtaining image information.

Desirably, the edge pixel extraction unit scans the obtained image information for each line, checks boundaries between a particular color and other colors, and extract pixels having the particular color as edge pixels. Moreover, the candidate line extraction unit references position coordinates of the extracted edge pixels, assumes the edge pixels to be classified into straight lines, obtains straight lines by coupling the edge pixels corresponding to respective results of the classification, and sets the obtained straight lines as candidate lines

Favorably, the line selection unit selects the left and right lines in front of the vehicle from the plural candidate lines, thereby obtaining the left and right lines.

Favorably, the line selection unit may select, in an operation to select lines by assuming a central perpendicular line of front image information, as a line of vehicle one of the candidate lines which has the largest number of edge pixels on the left or right side of the central perpendicular line.

As above, according to the present invention, the left and right lines can be detected from the plural candidate lines according to information such as an arrangement relationship between lines. Resultantly, the road surface area on which the vehicle can run can be correctly recognized.

To achieve the second object, there is provided according to another aspect of the present invention an image processing apparatus for identifying a line of vehicle disposed on a running road surface according to color image data of the road surface, including a color image pickup unit for shooting an image in front of a vehicle along a running direction thereof, an edge pixel extraction unit for scanning for each line the image data obtained through the shooting operation by the color image pickup unit, checking boundaries between a particular color and other colors, obtaining pixels having the particular color as edge pixels, and extracting pixels satisfying a particular extraction color condition from the obtained edge pixels, a color sampling unit for sampling color data of the decided pixels from a screen of the image data, extraction condition decision means for determining the extraction color condition according to the sampled color data, and a line image extraction unit for attaining as a line of vehicle a line image configured by coupling the extracted edge pixels into a straight line.

Favorably, there is disposed a unit to decide an illumination condition of the running road surface according to the sampled color data such that the opening of the iris of the optical system included in the color image pickup unit and the color conversion gain of an image data processing system thereof are controlled according to a result of the decision.

Desirably, the image extraction unit extracts pixels satisfying the determined extraction color condition from the screen configured with the image data, the color sampling unit samples color data from the pixels extracted by the pixel extraction unit, and the extraction condition deciding unit decides the extraction condition according to the sampled color data.

In this way, since the line recognition is carried out according to the extracted pixels, the road surface can be discriminated from the line according to the luminance and colors in association with the change in the state of the periphery of the road, thereby appropriately recognizing the line.

To achieve the third object, there is provided according to further another aspect of the present invention an image processing apparatus including an image processing apparatus including an image pickup unit for shooting an image to attain image information, an image magnification process unit for magnifying a size of the obtained image information, an edge pixel coordinate sense unit for scanning for each line the magnified image information, checking boundaries between a particular color and other colors, and extracting pixels on the boundaries, and obtaining position coordinates of the extracted edge pixels, an edge pixel coordinate conversion means for converting the position coordinates of the extracted edge pixels into coordinates before the magnification process, a line image extraction unit for referencing the converted coordinates of the edge pixels and coupling the edge pixels into a straight line, thereby attaining a line image, and a magnifying condition storage unit for storing therein a zoom rate as a magnitude of magnification of the image information and a magnification center coordinate indicating a reference position for the execution of the magnification process of the image information, the magnification factor and the magnification center coordinate being defined for each horizontal line.

Favorably, the image magnification process unit conducts a process to reference storage contents in the magnifying condition storage means so as to magnify for each horizontal line the size of the image information with the magnification center coordinate set to a center of magnification according to a magnification factor indicated by the zoom rate.

Desirably, the image processing apparatus includes an image pickup means mounted on a vehicle for shooting an image in front thereof to attain image information, an image magnification process unit for magnifying a size of the obtained image information, an edge pixel coordinate sense unit for scanning for each line the magnified image information, checking boundaries between a particular color and other colors, and extracting pixels on the boundaries, and obtaining position coordinates of the extracted edge pixels, an edge pixel coordinate conversion unit for converting the position coordinates of the extracted edge pixels into coordinates before the magnification process, a line extraction unit for referencing the converted coordinates of the edge pixels and coupling the edge pixels into a straight line so as to attaining a left line and a right line, and a magnifying condition storage unit for storing therein a zoom rate as a magnitude of magnification of the image information and a magnification center coordinate indicating a reference position for the execution of the magnification process of the image information, the magnification factor and the magnification center coordinate being defined for each horizontal line.

Favorably, the image magnification process unit conducts a process to reference storage contents in the magnifying condition storage unit so as to magnify for each horizontal line the size of the image information with the magnification center coordinate set to a center of magnification according to a magnification factor indicated by the zoom rate.

Favorably, the image magnification process unit conducts a process to magnify the size of the obtained image information only in the horizontal direction.

Favorably, an image in front of the vehicle is shot by an image pickup unit mounted on the vehicle to obtain image information.

Desirably, stored in the magnification condition storage unit are a zoom rate as a magnitude of magnification of the image information and a magnification center coordinate indicating a reference position for the execution of the magnification process of the image information, the magnification factor and the magnification center coordinate being defined for each horizontal line.

Favorably, the size of the obtained image information is magnified by the image magnification process unit.

Specifically, the image magnification process unit executes a process to reference the storage contents in the magnifying condition storage means so as to magnify for each horizontal line the size of the image information with the magnification center coordinate set to a center of magnification according to a magnification factor indicated by the zoom rate. In this operation, the image magnification process unit conducts a process to magnify the size of the obtained image information only in the horizontal direction.

Additionally, the edge pixel coordinate sense unit scans the magnified image information for each horizontal line, checks boundaries between a particular color and the other colors, and extracts pixels on the boundaries as edge pixels to obtain position coordinates of the extracted edge pixels. Moreover, the edge pixel coordinate conversion unit converts the position coordinates of the extracted edge pixels into coordinates before the magnification process.

In addition, the line extraction unit references the converted coordinates of the edge pixels and couples the edge pixels into straight lines to obtain the left and right lines.

As above, according to the present invention, the image information obtained by the image pickup unit is magnified (simply referred to as "zooming an image" for convenience herebelow) such that the running traffic lines can be correctly recognized according to the image data undergone the magnification process.

In other words, adopting the image zooming operation, it is possible to extract pixels (edge pixels) indicating the boundaries between the road surface and the lines in a state in which the line width is magnified and hence the lines in the faraway place can be appropriately recognized.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing the overall configuration of a first embodiment of the image processing apparatus according to the present invention.

FIG. 2 is a flowchart for explaining the entire processing of the embodiment of FIG. 1.

FIGS. 3A and 3B are diagrams for explaining an edge detecting operation in the embodiment of FIG. 1.

FIG. 4 is a flowchart for explaining operations of line calculation and estimation processes in the embodiment of FIG. 1.

FIGS. 5A and 5B are diagrams showing an edge data extraction example for explaining the line estimation process.

FIG. 5C is a flowchart for explaining the line recognition process in FIG. 5B.

FIG. 5D is a diagram for explaining the line recognition process in FIG. 5B.

FIG. 6 is a diagram showing an edge data extraction example for explaining another example of the line estimation process.

FIG. 7 is a flowchart for explaining the line estimation process in FIG. 6.

FIGS. 8A and 8B are diagrams for explaining an ideal lane width.

FIG. 9 is a diagram showing an edge data extraction example for explaining still another example of the line estimation process.

FIG. 10 is a flowchart for explaining the line estimation process in FIG. 9.

FIG. 11 is a flowchart for explaining a variation of the line estimation process in the embodiment of FIG. 1.

FIG. 12 is a diagram showing an edge data extraction example for explaining the line estimation process in the variation of FIG. 11.

FIG. 13 is a block diagram showing the overall construction of a second embodiment of the image processing apparatus according to the present invention.

FIG. 14 is a flowchart for explaining the entire processing of the embodiment of FIG. 13.

FIG. 15 is a flowchart for explaining a condition setting process in the embodiment of FIG. 13.

FIG. 16 is a diagram showing sampling pixel positions in the initialization of the embodiment of FIG. 13.

FIGS. 17A to 17G are diagrams for explaining the setting and update of extraction color conditions in the initialization of the embodiment of FIG. 13.

FIG. 18 is a diagram for explaining sampling pixel positions in the setting of extraction color conditions in the embodiment of FIG. 13.

FIG. 19 is a diagram for explaining a screen area employed to calculate a distribution ratio in the embodiment of FIG. 13.

FIG. 20 is a diagram for explaining a method of updating the extraction color conditions in the embodiment of FIG. 13.

FIG. 21 is a flowchart for explaining a process of updating the extraction color conditions in the embodiment of FIG. 13.

FIGS. 22A and 22B are diagrams for explaining a process of minimizing a setting color area of the extraction color conditions in the embodiment of FIG. 13.

FIGS. 23A and 23B are diagrams for explaining a process of minimizing a setting luminance area of the extraction color conditions in the embodiment of FIG. 13.

FIGS. 24A and 24B are diagrams for explaining a process of magnifying a setting color area of the extraction color conditions in the embodiment of FIG. 13.

FIGS. 25A and 25B are diagrams for explaining a process of magnifying a setting luminance area of the extraction color conditions in the embodiment of FIG. 13.

FIG. 26 is a diagram showing a setting characteristic of the color emphasis gain in the embodiment of FIG. 13.

FIG. 27 is a block diagram showing the overall construction of a third embodiment of the image processing apparatus according to the present invention.

FIG. 28 is a flowchart for explaining the entire processing of the embodiment of FIG. 27.

FIGS. 29A to 29E are diagrams for explaining a line recognition process in the embodiment of FIG. 27.

FIG. 30 is a flowchart showing an edge extraction process in the embodiment of FIG. 27.

FIGS. 31A and 31B are diagrams for explaining the principle of a edge coordinate conversion process.

FIG. 32 is a flowchart for explaining an electronic zoom process.

FIGS. 33A an 33B are diagrams for explaining the coordinates of a zoom center in an image.

FIG. 34 is a diagram for explaining a method of attaining the zoom center coordinates in an image.

FIG. 35 is a diagram for explaining a method of attaining the zoom center coordinates in an image.

FIGS. 36A and 36B are diagrams for explaining the principle of a zoom process.

FIGS. 37A to 37C are diagrams for explaining images before and after the zoom process and a zoom rate.

FIGS. 38A to 38D are diagrams for explaining an edge decision process.

FIGS. 39A to 39D are diagrams for explaining an edge decision process.

FIGS. 40A to 40C are diagrams for explaining images before and after zoom processes and zoom rates when the zoom rate varies in a discontinuous fashion.

FIGS. 41A to 41C are diagrams for explaining the zoom center coordinates when the zoom rate varies in a discontinuous manner.

FIGS. 42A to 42D are diagrams for explaining process results of a replacement circuit.

BEST MODE FOR CARRYING OUT THE INVENTION

Referring now to the drawings, description will be given of an embodiment of the image processing apparatus according to the present invention.

FIG. 1 shows the configuration diagram of a first embodiment of the image processing apparatus according to the present invention.

The apparatus includes an image pickup block 2, a processing block 4, and a memory 6. Moreover, the processing block 4 includes an image processing section 402 and a decision section 404. In this configuration, signals are communicated between the decision section 404 and a vehicle state sensor 8, a vehicle motion controller 10, and an alarm device 12. Incidentally, like in the embodiment shown in FIG. 11F, a display section may be connected to an output side of the imaging block (image pickup block) also in this embodiment.

The respective constituent components will be next described.

First, the constituent elements of the image pickup block 2 will be described.

A CCD camera 202 is image pickup means to shoot an image in front thereof so as to obtain image information and has a function to attain image information of an object in front thereof and to convert the information into a color analog signal. In the onboard apparatus, the CCD camera 202 need only be arranged, for example, at a position in a favorable environment with little dirt in the vicinity of the grille in the front side of the vehicle.

Subsequently, an A/D converter 204 accomplishes a process to convert an analog image signal outputted from the CCD camera 202 into a digital signal.

Next, a color difference converter circuit 206 is a circuit having a function to multiply the digital signal produced from the A/D converter 204, i.e., a digital signal of the object including three primary colors R(red), G (green), and B (blue) components by a color emphasis gain including .gamma.R, .gamma.G, and .gamma.B components to thereafter convert the signal into a luminance signal Y and color difference (R-Y) and (B-Y)) image signal.

In this connection, although the process of the multiplication of the color emphasis gain is not necessarily required, it is favorable to conduct this process.

The image signals Y, R-Y, and B-Y are defined according to the R, G, and B signals as follows. First, R', G', and B' are defined using the R, G, and B signals and color emphasis gain components .gamma.R, .gamma.G, and .gamma.B as shown below.

Employing R', G', and B' thus defined, the luminance and color difference signals are defined as follows.

As above, the color difference converter circuit 206 multiplies the digital signals of object including three primary colors R(red), G (green), and B (blue) components by the color emphasis gain (.gamma.R, .gamma.G, and .gamma.B) and then converts the resultant signals into the image signals including the luminance signal Y and color difference signals R-Y and B-Y to send the signals to the image processing section 402.

Next, the image processing section 402 will be described. However, since the contents of operation thereof will be described in detail later, the functions of the respective elements will be briefly described in the following paragraphs.

As shown in FIG. 1, the processing block includes the image processing section 402 and the decision section 404.

First, description will be given of the constituent components of the image processing section 402.

An extraction process circuit 40202 is a circuit having two primary functions.

According to one of the functions, the image signal sent from the color difference converter circuit 206 is processed for each horizontal scan line so as to extract therefrom pixels corresponding to extraction color condition data (which will be described later) transferred from an extraction color condition determining section 40204. Thereafter, position coordinate data (edge coordinate data) corresponding to the extracted pixels on one horizontal line is transferred to an edge decision section 40206. Incidentally, the image signal may be supplied to the extraction process circuit 40202 via a buffer, not shown.

In this regard, in the investigation of one horizontal line, assuming a change point from a pixel not satisfying the extraction color condition data to a pixel satisfying the extraction color condition data, the pixel satisfying the extraction color condition data is called a rising edge and its coordinate data is referred to as edge coordinate data of rising edge. Similarly, in the investigation of one horizontal line, assuming a change point from a pixel satisfying the extraction color condition data to a pixel not satisfying the extraction color condition data, the pixel satisfying the extraction color condition data is called a falling edge and its coordinate data is referred to as edge coordinate data of falling edge. In this regard, the edge coordinate data includes, in addition to coordinate information, an information item indicating that the pertinent edge coordinates are related to a rising or falling edge.

According to the remaining one of the functions of the circuit, color data including Y, R-Y, and B-Y components of a particular pixel (pixel at a particular position of one frame) is sampled from the image information obtained by the pickup operation and is then transferred to the extraction color condition determining section 40204.

Subsequently, according to data (Y, R-Y, B-Y) corresponding to the road color beforehand stored in a control data section 604 of the memory 6 and the color data (Y, R-Y, B-Y) sampled by the extraction process circuit 40202, the extraction color condition determining section 40204 decides an extraction color condition and then transfers the condition to the extraction process circuit 40202.

The extraction color condition data is color data of pixels. As described above, according to the data, the circuit 40202 extracts pixels satisfying the extraction color condition data to obtain edge coordinate data. Incidentally, there may be considered, for example, a method of determining the extraction color condition data in which the beforehand stored data corresponding to the road color is used immediately after the apparatus is initiated. At lapse of a predetermined period of time thereafter, the color data sampled by the extraction process circuit 402020 is utilized. A more specific method of deciding the data will be described later.

Next, an edge decision section 40206 is a circuit to decide according to a predetermined algorithm that each of the edge coordinate data items (edge coordinate data items respectively of rising and falling edges) corresponds to either one of the edge coordinate data assumed to constitute a left line of the running lane of the vehicle, the edge coordinate data assumed to constitute a right line of the running lane of the vehicle, and other edge coordinate data. The algorithm will be described later. In this connection, the other edge coordinate data includes, for example, edge coordinate data extracted for image information of a vehicle in front of the pertinent vehicle.

The edge decision section 40206 then transmits edge coordinate data considered to form the right and left lines to a line calculation section 40208 and any other edge coordinate data to a hazard decision section 40404.

Next, according to the edge coordinate data set from the edge decision section 40206 as data considered to form the right and left lines, the line calculation section 40208 decides linearity thereof and calculates as a candidate line a straight line constituted with edge coordinate of each line. Only one candidate line may be obtained or two or more such lines may be generated depending on cases. In this connection, the decision of linearity will be described later.

An ideal lane width data acquiring section 40210 possesses a function to acquire ideal lane width data and the like from an ideal lane width data section 602. In this context, the ideal lane width data is data which indicates the width of a running lane defined by the right and left lines.

A line estimation section 40212 checks, according to the ideal lane width data and the like acquired by the data acquiring section 40210, the candidate lines received from the line calculation section 40208 to determine right and left lines to be recognized (recognition lines) and then sends edge coordinate data and the like corresponding to the right and left lines to the hazard decision section 40404.

Subsequently, the constituent elements of the decision block 404 will be described.

In this connection, the decision block 404 is an example in which the embodiment is applied to a facility to stimulate attention of the driver. The decision block itself is not included in the primary portion of the embodiment.

A vehicle state decision section 40402 determines the running state of the pertinent vehicle according to a signal sent from the vehicle state sensor 8 and then transfer the result of decision to the hazard decision section 40404.

In this construction, the sensor 8 is means to sense the momentum of the vehicle, driving intention of the driver, and the like and includes, for example, a vehicle speed sensor to measure the speed of the vehicle, a direction indicator, a steering angle sensor, or the like.

In the hazard decision section 40404, the running lane of the pertinent vehicle is recognized according to the data of right and left recognition lines sent from the line estimation section 40212, presence of a car, an obstacle, or the like in front of the vehicle according to the edge coordinate data transmitted