 |
Description  |
|
|
FIELD OF THE INVENTION
The present invention relates to an object recognition system which is mounted on a vehicle and recognizes an object ahead of the vehicle, and more particularly to an object recognition system which is mounted on a vehicle and recognizes the
outline of an object ahead of the vehicle.
BACKGROUND OF THE INVENTION
In recent years, devices which determine the distance and size of objects in front of a vehicle, and which appropriately control the vehicle in accordance with this determination, have been proposed for improving the safety of vehicle operation.
A typical method for obtaining information about an object ahead includes the steps of: extracting horizontal edges and vertical edges from a captured image; determining whether each of the extracted edges belongs to the object such as a vehicle ahead;
and recognizing the object based on the edges determined to belong to the object.
Japanese Patent Application Kokai No. Hei 10-97699 describes a device for determining whether or not an object exists. The device recognizes left and right guidelines such as white lines in a captured image, and extracts horizontal edges from an
area between the left and right guidelines. A probable area in which an obstacle may exist is estimated based on the extracted horizontal edges. Vertical edges are then extracted in the probable area. The device judges that no object exists if the
number of vertical edges whose lengths exceed a predetermined value is less than two.
Japanese Patent Application Kokai No. Hei 9-16782 describes a device for determining whether or not an object exists. According to the device, a captured image is divided into a plurality of small areas. An area containing detected vertical
edge points is defined as a vertical edge area, and an area containing detected horizontal edge points is defined as a horizontal edge area. Intersections of a group of continuous vertical edge areas and a group of continuous horizontal edge areas are
determined. If four or more intersections are detected and they form a rectangle, then the rectangle is recognized as the object.
However, according to Japanese Patent Application Kokai No. Hei 10-97699, if the color of a part of an object is similar to, and therefore has not a sufficient contrast to the color of the background, and two or more vertical edges cannot be
obtained in the probable area, then the recognition error may take place that `no object` exists even if there is actually an object. According to Japanese Patent Application Kokai No. Hei 9-16782, if four or more intersections of vertical edges and
horizontal edges cannot be obtained because of an insufficient contract, then the recognition error may take place that `no object` exists even if there is actually an object.
Accordingly, one object of the invention is to provide a system capable of recognizing the outline of an object from horizontal edges and/or vertical edges even if desired horizontal edges or vertical edges cannot be obtained.
SUMMARY OF THE INVENTION
According to one aspect of the invention, an object recognition system including a position sensor, an image sensor, and a controller is provided. The position sensor determines the position of an object, and the image sensor captures an image
of the object. The position sensor can be implemented with radar or a pair of image sensors. The controller sets a processing area within the image captured by the image sensor based on the position of the object determined by the position sensor and a
predetermined size for the object to be recognized. It is preferable that the predetermined size is set to surround the object to be recognized.
The controller extracts horizontal edges from the processing area. Extraction of edges can be carried out by detecting a variation of intensity values of pixels contained in the processing area. The controller then identifies horizontal edges
indicating or belonging to the outline of the object from the extracted horizontal edges. Thus, the object can be recognized based on the horizontal edges only. Similarly, the controller can extract vertical edges from the processing area, and identify
vertical edges belonging to the outline of the object.
According to one aspect of the invention, the outline of the object is recognized by determining upper, lower, left, and right ends of the object. Thus, the vehicle ahead traveling in front of the vehicle mounting the system of the invention is
recognized by the positions of the upper, lower, left, and right ends of the outline of the vehicle ahead.
According to one aspect of the invention, the controller selects upper, lower, left, and right candidate ends from the identified horizontal edges, and selects upper, lower, left, and right candidate ends from the identified vertical edges. The
controller then determines the upper, lower, left, and right ends of the object based on the upper, lower, left, and right candidate ends selected from the horizontal edges and the upper, lower, left, and right candidate ends selected from the vertical
edges.
Preferably, the left and right candidate ends selected from the vertical edges are chosen as the left and right ends of the object respectively. For the upper end of the object, the higher one of the upper candidate end selected from the
horizontal edges and the upper candidate end selected from the vertical edges is chosen as the upper end of the object. For the lower end of the object, the lower one of the lower candidate end selected from the horizontal edges and the lower candidate
end selected from the vertical edges is chosen as the lower end of the object.
According to one aspect of the invention, if either one or both of the left and right candidate ends cannot be selected from the vertical edges, then the left or right end of the object is determined by the left or right candidate end selected
from the horizontal edges in lieu of the left or right candidate end selected from the vertical edges.
According to another aspect of the invention, if either one or both of the left and right candidate ends cannot be selected from the horizontal edges, then the left and right ends of the object are determined by estimated left and right candidate
ends in lieu of the left and right candidate ends selected from the horizontal edges. The estimated left and right candidate ends can be determined based on the position of the object recognized in a previous recognition cycle.
The controller can comprise a micro-controller which typically includes a central unit (CPU), or a micro-processor, a read-only memory (ROM) containing control programs that when executed by the processor performs respective functions which are
to be described hereafter. The controller also includes a random-access memory (RAM) that provides a working area for the CPU and temporary storage for various data and programs.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram illustrating the overall structure, and functional blocks of the controller of one embodiment of the present invention.
FIG. 2 is a diagram illustrating the principle of measurement by the triangulation method.
FIG. 3 is a diagram showing the processing area in accordance with one embodiment of the present invention.
FIG. 4 is a diagram showing the way for setting a processing area in accordance with one embodiment of the present invention.
FIG. 5 is a diagram showing another processing area with allowance for pitching in accordance with one embodiment of the present invention.
FIG. 6 is a flowchart of the method for extracting edges.
FIG. 7 (A) is a diagram showing the filter for extracting horizontal edges, and FIG. 7(B) is a diagram showing the coordinates of filter elements.
FIG. 8 is a histogram showing intensity values of the captured image.
FIGS. 9 is a diagram illustrating the template and method of determining labels used in accordance with one embodiment of the present invention.
FIG. 10 is a diagram illustrating labeling scheme in accordance with one embodiment of the present invention.
FIG. 11 is a diagram showing the filter for vertical edges.
FIG. 12 is a diagram showing the scheme for determining horizontal edges in accordance with one embodiment of the present invention.
FIG. 13 is a block diagram,illustrating in detail the object outline recognition part in accordance with one embodiment of the present invention.
FIG. 14 is a diagram showing the scheme for selecting candidate ends of the object from horizontal edges in accordance with one embodiment of the present invention.
FIG. 15 is a diagram showing the scheme for selecting candidate ends of the object from vertical edges in accordance with one embodiment of the present invention.
FIG. 16 is a flowchart of the method for recognizing the outline of the object in accordance with one embodiment of the present invention.
FIG. 17 is a diagram showing the case in which the left vertical candidate end cannot be selected.
FIG. 18 is a diagram showing the case in which the left and right vertical candidate ends cannot be selected.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
The embodiments of the present invention will be described below with reference to the attached drawings. FIG. 1 is an overall block diagram of an object recognition system in accordance with one embodiment of the present invention. Other than
the image sensor 1 and the object position sensor 3, all the blocks in FIG. 1 may be incorporated in a controller which comprises a single chip or multiple chip semiconductor integrated circuit. Thus, FIG. 1 shows functional blocks of the controller.
Respective functions of the blocks are performed by executing respective programs stored in the ROM of the controller.
In the present embodiment discussed below, it is assumed that an object to be recognized by the object recognition system is a vehicle ahead that traveling ahead of the vehicle mounting the system of the invention.
The image sensor 1 shown in FIG. 1 captures a view ahead of the vehicle mounting the system of the invention. The image sensor 1 is typically two-dimensional CCDs, and can be two-dimensional photo-sensor arrays. When usage in the night is
considered, an image sensor using infrared light is advisable. In this case, it is preferable to install infrared-transparent filters in front of a lens, and to design the system such that the object is illuminated at a predetermined period from an
infrared light source. The image sensor senses the infrared light reflected from the object. The image captured by the image sensor 1 is converted into digital data by an analog-to-digital converter (not shown), and is stored in an image memory 2.
An object position sensor 3 is typically implemented by laser radar or millimeter-wave radar. The position sensor 3 radiates laser or millimeter-wave to the object and receives the signal reflected by the object to measure the distance from the
vehicle mounting the system of the invention to the object as well as the relative direction of the object to the vehicle. The position sensor 3 may be a scan-type radar apparatus with a single beam for scanning over a certain angle range ahead of the
vehicle mounting the system of the invention. Alternatively, the scan-type radar with a plurality of beams may also be used. The angle range covered by beams of the radar is set based on the range in which the object to be recognized can be captured by
the image sensor 1. In this embodiment, since the object to be recognized is the vehicle ahead, the radar is set to cover at least the lane of the vehicle mounting the system of the invention. Alternatively, the direction of the beams of the radar may
be changed as appropriate according to the position of the vehicle ahead obtained in a previous recognition cycle such that the beams are correctly radiated to the vehicle ahead. Thus, the distance D and the direction .theta. of the vehicle ahead are
determined and then stored in an object position memory 4 shown in FIG. 1.
According to another embodiment, the position sensor 3 is implemented by using a pair of optical image sensors. By way of example, FIG. 2 shows the principle of measuring the distance by the triangulation method. For the sake of simplicity, the
pair of image sensors shown in FIG. 2 are described as one-dimensional line sensors. A line sensor 28 and lens 26 constituting one of the pair of image sensors are arranged at a specified distance, i.e., at a distance equal to the base line length B in
the horizontal direction from the line sensor 29 and lens 27 constituting the other of the pair.
The line sensors 28 and 29 are respectively positioned at the focal length f of the lenses 26 and 27. Assume that an image of an object 25 located at distance "D" from the plane of the lenses 26 and 27 is formed at a position shifted by a
distance d1 from the optical axis of the lens 26 in the case of the line sensor 28, and is formed at a position shifted by a distance d2 from the optical axis of the lens 27 in the case of the line sensor 29. Then, according to the principle of
triangulation, the distance "D" to the object 25 from the plane of the lenses 26 and 27 is determined by the equation:
In the embodiment, the images are digitized. Accordingly, the distance (d1+d2) is digitally calculated. The sum of the absolute values of the differences between the digital values indicating the intensities of the corresponding pixels of both
images obtained from the line sensors 28 and 29 is determined while one or both of said images are shifted, and this sum is taken as a correlation value. The amount of shift of the images when this correlation value is at a minimum indicates the
positional deviation between the two images, i.e., (d1+d2). In idealized terms, the distance by which the two images obtained from the line sensors 28 and 29 must be moved in order to cause said images to overlap as shown in FIG. 2 is (d1+d2). The
direction .theta. of the object 25 to the vehicle mounting the system of the invention can be determined by the equation, tan .theta.=d2/f with respect to the optical axis of the lens 27.
In one embodiment, the captured image is divided into a plurality of windows and the above determination of the distance is carried out for each of windows. The determined distance is compared with a distance to the road that is calculated and
stored beforehand for each window. If the determined distance is shorter than the distance to the road, the determined distance is judged to be the distance to the object. Thus, by determining the distances of the windows that correspond to the lane
area of the vehicle mounting the system of the invention with a steering angle of the vehicle taken into account, the distance to the vehicle ahead can be calculated.
In another embodiment, the position sensor 3 is implemented by using the radar and the pair of image sensors in combination. By the way of example, in the situation in which it is difficult to determine the distance and the direction using the
image captured by the pair of image sensors (for example, when the vehicle ahead is faraway from the vehicle mounting the system of the invention, or when the vehicle ahead is running through a tunnel and a view ahead of the vehicle cannot be stably
captured), the radar is used to determine the distance and direction. On the other hand, using the radar to determine the distance and the direction limits the angle range of the vehicle mounting the system of the invention to a predetermined range. In
other words, with the radar, it is difficult to cover a wide range that can be covered by the image sensors. Therefore, preferably, the direction of the beams of the radar is changed according to the direction of the vehicle ahead determined by the
image sensors.
An object recognition part 21 shown in FIG. 1 sets a processing area within the image captured by the image sensor 1, and recognizes the vehicle ahead using edges extracted from the processing area. The process of recognizing the object is
repeatedly executed at predetermined time intervals (for example, 100 milliseconds). Described below in detail is the method for recognizing the object implemented by the object recognition part 21.
Setting Processing Area
A processing area setting part 5 shown in FIG. 1 sets a processing area within the image captured and stored in the image memory 2 based on the position of the object stored in the object position memory 4 and a predetermined size for the object
to be recognized. The predetermined size for the object to be recognized is set beforehand to surround the object to be recognized.
The process of setting the processing area is described below by referring to FIGS. 3 and 4. FIG. 3 shows an example of the captured image in which the vehicle ahead 40 running forward is included. As shown in FIG. 3, an x-axis and a y-axis are
fixed in the image, and a processing area 30 is defined by the coordinates (Xa1, Ya1) and (Xa2, Ya2).
FIG. 4 shows the way for setting the processing area 30. FIG. 4(A) shows the way for determining the x coordinates, that is, Xa1 and Xa2, and FIG. 4(B) shows the way for determining the y coordinates, that is, Ya1 and Ya2.
In FIG. 4, the image sensor 1 is mounted on the vehicle mounting the system of the invention. The image sensor 1 captures the vehicle ahead 40 that is traveling in front of the vehicle mounting the system of the invention. Reference character f
denotes the focal length of the lens 45 mounted on the image sensor 1, which is specified depending on the characteristic of the lens. Reference characters W and H denote predetermined width and height of the object to be recognized, that is, the
vehicle ahead in the present embodiment, respectively. The width and height are preset to surround the object to be recognized. For example, for the vehicle ahead, W may be set to 2 m, and H may be set to 2.5 m. Reference characters D and .theta.
denote the distance to the vehicle ahead and the relative direction of the vehicle ahead stored in the object position memory 4, respectively. Reference character h denotes the height from the road to the center of the lens 45, which is predefined
depending on the position of the image sensor 1 in the vehicle mounting the system of the invention.
The processing area setting part 5 extracts D and .theta. stored in the object position memory 4 to compute the coordinates (Xa1, Ya1) and (Xa2, Ya2) using the predetermined parameters W, H, and h as follows.
In one embodiment, for Ya1 and Ya2, a pitching allowance value ".alpha." is used in consideration of pitching of the vehicle mounting the system of the invention, as follows.
Thus, the processing area 30 is defined within the captured image by the coordinates (Xa1, Ya1) and (Xa2, Ya2) as shown in FIG. 3, or is defined as shown in FIG. 5 with the pitching taken into account.
Extracting Horizontal and Vertical Edges
A horizontal edge extraction part 7 and a vertical edge extraction part 9 shown in FIG. 1 extract horizontal edges and vertical edges respectively from the processing area 30. Since both horizontal and vertical edges are extracted in the same
way, only the process of extracting horizontal edges is described below. The extracted edges show a portion in which the variation of intensity is large in the image. FIG. 6 is a flowchart of extracting edges, which is carried out by the horizontal
edge extraction part 7.
First, the horizontal edge extraction part 7 performs a filtering process on each pixel within the processing area 30 in the horizontal direction to enhance edge portions indicating a large difference in intensity in the horizontal direction
(step 61). FIG. 7(A) shows an example of a horizontal edge filter. For convenience in the following computation, coordinates are assigned to each element of the filter as shown in FIG. 7(B).
A computation shown in the equation (7) is executed for the intensity value of each pixel within the processing area 30 while the processing area 30 is scanned by the horizontal edge filter. ##EQU1##
In equation (7), x and y are coordinates identifying the position of each of pixels in the processing area 30. G(x, y) indicates the intensity value of the pixel at (x, y), and F(i, j) indicates the value of the filter element at (i, j) of the
horizontal edge filter. P(x, y) indicates the intensity value of the pixel at (x, y) after the filtering process has been performed. Since values of elements of the horizontal edge filter are predetermined such that horizontal edges are enhanced,
horizontal edges can be detected by performing the above computation on each pixel.
In another embodiment, instead of the equation (7), the filtering process is carried out by differentiation. In this case, the difference in intensity between vertically adjacent pixels is calculated as shown in equation (8), where n is an
integer, for example, may be set to 1 (n=1).
Then, an intensity histogram is created based on the intensity value P of each pixel (step 63). The intensity value used in this embodiment is represented as digital data having 256 gradations (ranging from pure black "0" to pure white "255")
FIG. 8 shows an example of the intensity histogram. The horizontal axis indicates the intensity values obtained by the filtering process while the vertical axis indicates the number of pixels corresponding to each of the intensity values.
On the basis of the histogram, the intensity value at which the ratio between the number of pixels in lower intensities and the number of pixels in higher intensities matches a predetermined value is determined as a threshold for binarization
(step 65). Alternatively, the ratio between the number of the pixels forming edges of the vehicle ahead and the number of pixels of the processing area 30 may be estimated beforehand, and the intensity value that corresponds to the estimated ratio and
that is greater than a predetermined intensity value may be determined as the threshold for binarization.
With respect to the threshold, for example, the higher intensities are assigned 1, the lower intensities are assigned 0, there by producing a binary image of the processing area 30 (step 67). The pixel having the value of 1 is referred to as an
edge point.
There are a number of conventional methods for setting the threshold for binarization. The above method for setting the threshold is merely an example and is not meant to exclude another method.
Then, pixels having the value of 1, that is, edge points, are extracted. If two or more edge points continue, they are grouped into a horizontal edge. In the present embodiment, a labeling process using a template is used as a method for
grouping edge points. The labeling process is described in detail in U.S. patent application Ser. No. 09/567,734 which is incorporated herein by reference.
With reference to FIG. 9, the labeling process is described below. FIG. 9 shows a template for the labeling process. T1 through T3 in FIG. 9(A) indicate positions in the template. V1 through V3 in FIG. 9(B) indicate the values (1 or 0) of
pixels corresponding to the positions T1 through T3 respectively when the template is positioned such that T2 assumes the place of an edge point to be processed. L1 through L3 in FIG. 9(C) indicate labels assigned to pixels corresponding to the
positions T1 through T3 respectively.
The table in FIG. 9(D) shows the type of label L2 that is assigned to the edge point at position T2 based on the value of the pixels at positions Ti through T3 when T2 is placed at the edge point to be processed. For example, if the values V1
through V3 at positions T1 through T3 satisfy condition 4 in FIG. 9(D), then a label L1 is assigned to the edge point at T2. The label L is assigned when the condition 1 is satisfied requiring a new label. A horizontal edge extraction part 7
successively scans the edge points placing T2 of the template at respective edge points, thus assigning label L2 to respective edge points, as shown in FIG. 9(E).
The process of assigning labels to respective edge points is described more specifically with reference to FIG. 10. FIG. 10(A) is an example of a part of the image after binarization, wherein the value of 0 is represented by a dot. The template
is placed such that position T2 of the template is at respective edge points, which have the value of 1, to be processed. FIG. 10(B) shows the image after assigning labels to edge points. As seen in FIG. 10(B), the same labels are assigned to the
continuous edge points.
Here, referring to FIG. 9(D) again, when condition 5 is satisfied, labels corresponding to positions T1 and T3 are connected or joined together, and the label corresponding to T3 is replaced with the label corresponding to T1. In the example
shown in FIG. 10(B), edge points 91 and 92, and edge points 92 and 93 satisfy condition 5. Therefore, all edge points having the labels D and E are re-assigned label C (see FIG. 10(C)). By connecting labels, all continuous edge points are integrated
into an edge group assigned the same labels. FIG. 10(C) shows three edge groups with labels A, B and C. Thus, three horizontal edges are extracted.
In another embodiment, the process of connecting labels is performed after scanning all edge points in the processing area 30 and assigning labels to them.
The vertical edge extraction part 9 extracts vertical edges from the processing area 30 in the same way as the horizontal edge extraction part 7 except that a vertical edge filter shown in FIG. 11 is used in step 61 (FIG. 6).
Judging Horizontal and Vertical Edges
Referring to FIG. 1, a horizontal edge judgment part 11 judges whether or not each of horizontal edges extracted by the horizontal edge extraction part 7 indicates or belongs to the object based on characteristics of the object to be recognized.
In other words, each of the extracted edges is judged based on characteristics of the object when it is displayed in the image. In the present embodiment, since the object to be recognized is the vehicle ahead, the judgment is carried out based on
characteristics, such as having a box-shape, having relatively a large number of horizontal edges, and having linear edges.
By way of example, the following judgment conditions are used to judge whether or not each of the extracted horizontal edges belongs to the vehicle ahead.
1) It is judged that an edge containing a pixel located on a boundary of the processing area belongs to an object other than the vehicle ahead. This is because the processing area is set to surround the vehicle ahead as described above.
2) The linearity of each of the extracted horizontal edges is examined, and it is judged that an edge having a poor linearity belongs to an object other than the vehicle ahead. This is because a horizontal edge belonging to the vehicle ahead
does not draw a curve in the horizontal direction.
3) The slope of each of the extracted horizontal edges is examined, and it is judged that an edge having a large slope belongs to an object other than the vehicle ahead. This is because a horizontal edge belonging to the vehicle ahead does not
have a large slope in the horizontal direction.
With reference to FIG. 12, the above condition 1) through 3) will be described in detail below. FIG. 12(A) shows an example of the captured image. A marking on the road 120, a mountain ridge 130, and white lines 140 are included in the image.
The marking 120 extends across the processing area 30. FIG. 12(B) shows a binary image of horizontal edges extracted for the image in FIG. 12(A). In this binary image, not only the edges belonging to the vehicle ahead 40, but also edges 125 and 126
belonging to the marking 120, edges 135 through 137 belonging to a part of the mountain ridge 130, and edges 145 and 146 belonging to the white lines 140 are included.
For the above condition 1), the horizontal edge judgment part 11 examines x coordinates of the pixels forming each of the horizontal edges to judge whether the x coordinates include the edge point having Xa1 or Xa2. The processing area 30 is
specified by the coordinates (Xa1, Ya1) and (Xa2, Ya2) as described above. Therefore, if the x coordinates include the edge point having Xa1 or Xa2, then it is judged that the horizontal edge extends across a boundary of the processing area 30, and that
the horizontal edge belongs to an object other than the vehicle ahead. In the example shown in 12(B), since the edge 125 and 126 contain pixels having the x coordinates Xa1 and Xa2, each of the edges 125 and 126 is judged to belong to an object other
than the vehicle ahead.
For the above condition 2), the horizontal edge judgment part 11 calculates a variance of y coordinates of the edge points forming each of the extracted horizontal edges. If the variance is larger than a predetermined value, then the horizontal
edge is judged to belong to an object other than the vehicle ahead. In the example shown in FIG. 12(B), each of the edges 135 through 137 having poor linearity is judged to belong to an object other than the vehicle ahead.
For the above condition 3), the horizontal edge judgment part 11 approximates each of the extracted horizontal edges by straight lines in accordance with a conventional manner such as the least-squares method, to examine the slope of the edge
approximated by the straight lines. If the slope is larger than a predetermined value, the horizontal edge is judged to belong to an object other than the vehicle ahead. In the example in FIG. 12(B), each of the edges 145 and 146 having a large slope
is judged to belong to an object other than the vehicle ahead.
Horizontal edges judged to belong to the vehicle ahead and horizontal edges judged to belong to an object other than the vehicle ahead are stored separately in a horizontal edge memory 12. For example, the edge points forming the horizontal
edges judged to belong to the vehicle ahead are stored with value 1, and the edge points forming the horizontal edge judged to belong to an object other than the vehicle ahead are stored with value zero. Alternatively, a flag may be set only for the
edge points forming the horizontal edges judged to belong to an object other than the vehicle ahead, and these edge points are stored with the flag so that the edges cannot be used in subsequent processes. FIG. 12(C) shows a binary image of horizontal
edges after removing the edges 125, 126, 135, 136, 137, 145, and 146.
Horizontal edges containing a pixel on a boundary of the processing area may originate from a building in the background and a marking on the road such as a stop line. Horizontal edges having a poor linearity may originate from natural objects
such as a tree, and a flag used as a signboard of a shop. Horizontal edges having a large slope may originate from a guardrail, a sign of no-passing lane, and a roadside structure. According to the above process, erroneous recognition | | |
