 |
Description  |
|
|
BACKGROUND OF THE INVENTION
The present invention relates to an image segmentation method for segmenting the image region to be extracted from an input image using a reference image, an image identification method for identifying the image region to be extracted from an
input image using a reference image, an image segmentation apparatus, an image processing apparatus and method for extracting a specific image region from an image, and a storage medium storing the image processing method.
As general techniques for implementing image extraction, a chromakey method using a specific color background, a videomatte method for generating a key signal by predetermined image processing (histogram processing, difference processing,
differential processing, contour enhancement, contour tracing, and the like) (The Television Society Technical Report, Vol. 12, pp. 29-34, 1988), and the like are known.
A technique for performing image extraction based on the difference from the background image is a state-of-the-art one, and for example, Japanese Patent Laid-Open No. 4-216181 discloses a technique for detecting or extracting a target object in
a plurality of specific regions in an image by setting a mask image (specific processing region) in difference data between the background image and the image to be processed. Furthermore, Japanese Patent Publication No. 7-16250 discloses a technique
that uses a color model of the object to be extracted to implement image extraction by obtaining the existence probability distribution of the object to be extracted from color-converted data of an original image including background and lightness
difference data between the background image and the original image.
In the difference method from the background image, the luminance level or color component difference between the pixels of the background image and the subject image is normally expressed by a predetermined evaluation function, and the
evaluation function is subjected to thresholding to extract a region having a difference level equal to or higher than a predetermined value. As the evaluation function, the correlation between blocks having individual points as centers and a
predetermined size (Rosenfeld, A. and Kak, A. C., Digital Picture Processing (2nd ed.), Academic Press, 1982), a normalized principal component features (Journal of the Institute of Electronics, Information and Communication Engineers, Vol. J74-D-II pp.
1731-1740), a weighted sum value of a standard deviation and a difference value (Journal of the Television Society, Vol. 45, pp. 1270-1276, 1991), a local histogram distance associated with hue and luminance level (Journal of the Television Society,
Vol. 49, pp. 673-680, 1995), and the like are used.
As a technique for identifying or recognizing a specific object, the following method is popularly used. That is, a model image or template associated with that object is prepared in advance. An image region of the object to be identified is
separated from other regions, the size of the image region to be identified is normalized or its position is fixed, or a plurality of model images having different sizes are prepared. Scanning a target image, the similarities between the model image or
template and the object to be identified are determined using a measure such as correlation or the like.
The background difference method poses a problem when a partial region which has a similar luminance, color, or pattern to the background image is included in the object to be extracted. In this case, since no difference in that region is
estimated between the background image and the input image including the object to be extracted, extraction or detection errors take place. As a step against such problem, Japanese Patent Laid-Open No. 8-44844 adopts a method of calculating the
gradients of both the background image and the input image, and taking logical OR of the difference absolute value between the gradients and that of image signals. On the other hand, Japanese Patent Laid-Open No. 8-212350 adopts a method of performing
thresholding by calculating a feature, the rate of its change with respect to changes in pixel density of the input image of which decreases when the background image has a middle pixel density, and increases when the background image has a high or low
pixel density.
However, in image extraction, the chromakey method is hard to use outdoors due to serious background limitations such as a requirement for a specific color background, and causes color omissions in the subject region having the same color as the
background. On the other hand, in the videomatte method, since contour designation must be manually and accurately performed in units of pixels, such operation requires much labor and skill.
Furthermore, in the background difference method, the background is hard to distinguish from the subject in a partial region of the subject similar to the background, and this method does not normally allow image sensing condition differences
(e.g., the exposure condition, magnification, illumination condition, focusing condition, view point position, and the like) between the background image and input image. Especially, when background obtained by removing the subject from the input image
is different from the background image, the tolerance to their difference is considerably low even if they are similar to each other. In addition, it is very hard to extract contour and three-dimensional shape details of the subject while removing the
noise influence.
Also, the background difference method requires distinct image characteristic differences (e.g., pixel values and the like) between the background image and the subject region everywhere, and it is hard to apply the method to a general
background. Even in the methods that take a step against the partial region of the subject similar to the background (Japanese Laid-Open Patent Nos. 8-44844 and 8-212350), for example, when the rate of spatial change in pixel value is small, the
subject is hardly or insufficiently distinguished from the background. Hence, it is difficult to stably maintain high extraction precision by automatic processing in a practical use.
When the shadow of the object to be extracted is present in the image in the vicinity of the object, it is hard for either method to extract the image region to be extracted alone and to automatically remove the shadow.
Furthermore, in the method of identifying or recognizing a specific target image, the segmentation processing from other regions in the above-mentioned pre-processing normally constitutes in separable parts and is complicated and hard to attain.
On the recognition technique, also, it is difficult to automatically normalize the size, position, and the like since the size and position of the object to be recognized are not detected in advance. Furthermore, the number of model images having
different sizes that can be prepared is limited due to storage capacity limitations on the database, resulting in poor versatility.
SUMMARY OF THE INVENTION
It is, therefore, the first object of the present invention to provide an image segmentation method, which can precisely and automatically detect the existence range of an intruder in the input image, and can precisely detect the existence range
of the object to be extracted or recognized.
It is the second object of the present invention to provide an image segmentation method which can stably extract an image from the input image even when the reference image and the input image have differences resulting from variations of the
image sensing condition, illumination condition, and the like therebetween.
It is the third object of the present invention to provide an image identification method and apparatus, which can stably and automatically recognize the object to be recognized without being influenced by any differences of the image size and
position of a subject, and are excellent in terms of economy.
It is the fourth object of the present invention to provide an image segmentation method which can perform high-speed image extraction while removing the influences of noise such as shading.
It is the fifth object of the present invention to provide an image segmentation method and apparatus, which can perform image extraction with high tolerance to differences between the reference image and the background portion of the input
image.
It is the sixth object of the present invention to provide an image processing apparatus and method, and a storage medium that stores the method, which can precisely extract a specific image region from a main image which includes a specific
image region to be extracted and a sub image which does not include any specific image region to be extracted.
It is the seventh object of the present invention to provide an image processing apparatus and method, and a storage medium that stores the method, which can precisely extract a specific image region even when the specific image region to be
extracted includes a region similar to an image which is not to be extracted.
It is the eighth object of the present invention to provide an image processing apparatus and method, and a storage medium that stores the method, which can precisely extract a specific image region even when an image which is similar to the
specific image to be extracted but is not to be extracted is present in the vicinity of the specific image region to be extracted.
It is the ninth object of the present invention to provide an image processing apparatus and method, and a storage medium that stores the method, which can precisely extract a specific image region even when the specific image region to be
extracted includes many regions similar to an image which is not to be extracted.
It is the tenth object of the present invention to provide an image processing apparatus and method, and a storage medium that stores the method, which can precisely extract a specific image region even when the specific image region to be
extracted includes hole regions.
It is the eleventh object of the present invention to provide an image processing apparatus and method, and a storage medium that stores the method, which can precisely extract a specific image region at high speed by inputting a rough shape of
the specific region to be extracted in advance.
In order to solve the above problems and to achieve the above objects, the present invention comprises the following arrangement.
More specifically, there is provided an image segmentation method for segmenting an object, which is not present in a reference image, from an input image including the object using the reference image, comprising: the edge intensity distribution
extraction step of extracting edge intensity distributions in the reference image and input image; the direction-classified line detection step of detecting line components in terms of predetermined directions for edges in the reference image and input
image on the basis of the extracted edge intensity distributions; and the image region specifying step of specifying an existence range of the object in the input image on the basis of a distribution of differences between the detected line components in
terms of directions between the reference image and input image.
This image segmentation method can precisely and automatically detect the existence region of an intruder object in the input image, and can precisely detect the existence range of the object to be extracted or recognized.
Since the existence range of an object in the input image is specified on the basis of the distribution of line components in terms of directions, the existence region can be detected more accurately by removing the influences of, e.g., shading.
In order to achieve the above objects, according to the present invention, there is provided an image segmentation method using a reference for segmenting a predetermined image region from an input image input from an external device such as an
image input device, comprising: the edge distribution extraction step of extracting edge distributions in the input image and reference image; the direction-classified line detection step of detecting line distributions in terms of predetermined
directions in the input image and reference image on the basis of the extracted edge distributions; the singular edge extraction step of extracting a singular edge on the basis of a difference between the detected line distributions in units of
directions between the reference image and input image; and the image extraction step of extracting the predetermined image region in the input image on the basis of the extracted singular edge.
This image segmentation method can stably extract an image from the input image even when the reference image and the input image have differences resulting from variations of the image sensing condition, illumination condition, and the like
therebetween.
In order to achieve the above objects, according to the present invention, there is provided an image identification method for identifying an object, which is not present in a reference image, in an input image including the object using a
standard model image representing a predetermined object and the reference image, comprising: the edge intensity distribution extraction step of extracting edge intensity distributions in the reference image and input image; the direction-classified line
detection step of detecting line components in terms of predetermined directions for edges in the reference image and input image on the basis of the extracted edge intensity distributions; the auto-framing step of specifying an existence range of the
object in the input image on the basis of a distribution of differences between the detected line components in units of directions between the reference image and input image; and the model size estimation step of estimating a size with respect to the
standard model image on the basis of the specified existence range of the object, wherein a size of the standard model image is changed to the estimated size, and thereafter, the object is identified on the basis of similarity between the object image
present in the existence range in the input image and the size-changed standard model image.
This image identification method allows stable and automatic recognition of the object to be recognized without being influenced by the image size and position differences of a subject. Also, since standard model images having different sizes
need not be stored, an economical advantage can also be expected.
In order to achieve the above objects, according to the present invention, there is provided an image segmentation method for segmenting an object, which is not present in a reference image, from an input image including the object using the
reference image, comprising: the edge intensity distribution extraction step of extracting edge intensity distributions in the reference image and input image; the direction-classified line detection step of detecting line components in terms of
predetermined directions for edges in the reference image and input image on the basis of the extracted edge intensity distributions; the auto-framing step of specifying an existence range of the object in the input image on the basis of a distribution
of differences between the detected line components in terms of directions between the reference image and input image; and the extraction processing step of performing extraction processing of the object within the specified existence range of the
object.
This image segmentation method can extract an image at high speed while removing the influences of noise such as shading.
In order to achieve the above objects, according to the present invention, there is provided an image segmentation method for segmenting an image region to be extracted from an input image using a reference image that represents a region
approximating a remaining region excluding the image region to be extracted, comprising: the edge intensity distribution extraction step of extracting edge intensity distributions in the reference image and input image; the direction-classified line
detection step of detecting line components in terms of predetermined directions for edges in the reference image and input image on the basis of the extracted edge intensity distributions; the singular contour extraction step of extracting a singular
contour portion of the image region to be extracted on the basis of a distribution of the detected line components in terms of directions in the reference image and input image; and the image extraction step of extracting the image region to be extracted
on the basis of distribution data representing the extracted singular contour portion.
This image segmentation method can extract only a contour inherent to a subject by absorbing differences such as a positional offset, rotational offset, distortion, and the like between the reference image and the background portion of the input
image if they are present, and can achieve image extraction with high tolerance to the differences between the reference image and the background portion of the input image.
In order to achieve the above objects, according to the present invention, there is provided an image segmentation method for segmenting an image region to be extracted from an input image using a reference image that represents a region
approximating a remaining region excluding the image region, comprising: the low-resolution image extraction step of extracting low-resolution image portions in the input image and reference image; the image matching step of performing matching
corresponding points between the input image and reference image; the dominant line map extraction step of segmenting the input image and reference image into a plurality of blocks and detecting dominant line direction components in the blocks; and the
extraction step of extracting the image region on the basis of a degree of matching between a label in terms of directions of each edge of the input image and a label of the dominant line map of the reference image at the edge position.
This image segmentation method can achieve image extraction with high tolerance to differences between the reference image and the background portion of the input image. For example, even when the background portion of the input image and the
corresponding region in the reference image are substantially different scenes but have high similarity, or when the input image and the reference image have different photographing conditions or photographing means, image extraction can be performed
with high precision.
In order to achieve the above objects, according to the present invention, there is provided an image segmentation apparatus for segmenting an image region to be extracted from an input image using a reference image that represents a region
approximating a remaining region excluding the image region, comprising: storage means for storing the reference image; edge extraction means for extracting edge distributions in the input image and reference image; direction-classified line detection
means for detecting line distributions in terms of directions in the input image and reference image on the basis of the extracted edge distributions; corresponding point extraction means for extracting corresponding point information between the
reference image and input image; transformation means for geometrically transforming one of the input image and reference image on the basis of the extracted corresponding point information; singular edge extraction means for extracting a singular edge
on the basis of a line distribution difference in units of directions between the geometrically transformed image, and the other image; and segmentation means for segmenting the image region to be extracted from the input image on the basis of the
extracted singular edge.
This image segmentation apparatus can extract only a contour inherent to a subject by absorbing differences such as a positional offset, rotational offset, distortion, and the like between the reference image and the background portion of the
input image if they are present, and can achieve image extraction with high tolerance to the differences between the reference image and the background portion of the input image.
In order to achieve the above objects, according to the present invention, there is provided an image identification apparatus for identifying an object in an input image including the object which is not present in a reference image using a
standard model image representing a predetermined object and the reference image, comprising: edge intensity distribution extraction means for extracting edge intensity distributions in the reference image and input image; direction-classified line
detection means for detecting line components in terms of predetermined directions for edges in the reference image and input image on the basis of the extracted edge intensity distributions; auto-framing means for specifying an existence range of the
object in the input image on the basis of a distribution of differences between the detected line components in terms of directions between the reference image and input image; and model size estimation means for estimating a size with respect to the
standard model image on the basis of the specified existence range of the object, wherein a size of the standard model image is changed to the estimated size, and thereafter, the object is identified on the basis of similarity between the object image
present in the existence range in the input image and the size-changed standard model image.
This image identification apparatus can stably and automatically recognize the object to be recognized without being influenced by the image size and position differences of a subject since it changes the size of a standard model image to the
required size, and thereafter, identifies an object on the basis of the similarity between an object image present in the existence range in the input image and the standard model image, the size of which has changed. Furthermore, since standard model
images having different sizes need not be stored, an economical advantage can also be expected.
In order to achieve the above objects, according to the present invention, there is provided an image processing apparatus comprising: image input means for inputting a plurality of images including a main image including a specific image region
to be extracted, and a sub image which does not include any specific image region to be extracted; edge distribution extraction means for extracting edge distributions of the plurality of images; main extraction region estimation means for estimating a
main extraction region in the main image on the basis of a difference between the plurality of edge distributions; and region specifying means for extracting or tracing the specific image region on the basis of the main extraction region and the
plurality of images.
This image processing apparatus can precisely extract a specific image region from a main image including the specific image region to be extracted and a sub image which does not include any specific image region.
In order to achieve the above objects, according to the present invention, there is provided an image processing apparatus comprising: image input means for inputting a plurality of images including a main image including a specific image region
to be extracted, and a sub image which does not include any specific image region to be extracted; edge distribution extraction means for extracting edge distributions of the plurality of images; difference edge image extraction means for extracting
difference data of the plurality of edge distributions; outermost contour line extraction means for obtaining an outermost contour line by tracing an outermost line of the difference data; main extraction region estimation means for estimating a main
extraction region in the main image on the basis of the outermost contour line; and region specifying means for extracting or tracing the specific image region on the basis of the main extraction region and the plurality of images.
This image processing apparatus can precisely extract a specific image region even when the specific image region to be extracted includes a region similar to an image which is not to be extracted.
In order to achieve the above objects, according to the present invention, there is provided an image processing method comprising: the image input step of inputting a plurality of images including a main image including a specific image region
to be extracted, and a sub image which does not include any specific image region to be extracted; the edge distribution extraction step of extracting edge distributions of the plurality of images; the main extraction region estimation step of estimating
a main extraction region in the main image on the basis of a difference between the plurality of edge distributions; and the region specifying step of extracting or tracing the specific image region on the basis of the main extraction region and the
plurality of images.
This image processing method can precisely extract a specific image region from a main image including the specific image region to be extracted and a sub image which does not include any specific image region.
In order to achieve the above objects, according to the present invention, there is provided an image processing method comprising: the image input step of inputting a plurality of images including a main image including a specific image region
to be extracted, and a sub image which does not include any specific image region to be extracted; the edge distribution extraction step of extracting edge distributions of the plurality of images; the difference edge image extraction step of extracting
difference data of the plurality of edge distributions; the outermost contour line extraction step of obtaining an outermost contour line by tracing an outermost line of the difference data; the main extraction region estimation step of estimating a main
extraction region in the main image on the basis of the outermost contour line; and the region specifying step of extracting or tracing the specific image region on the basis of the main extraction region and the plurality of images.
This image processing method can precisely extract a specific image region even when the specific image region to be extracted includes a region similar to an image which is not to be extracted.
In order to achieve the above objects, according to the present invention, there is provided a storage medium storing a program including: an image input module of inputting a plurality of images including a main image including a specific image
region to be extracted, and a sub image which does not include any specific image region to be extracted; an edge distribution extraction module of extracting edge distributions of the plurality of images; a main extraction region estimation module of
estimating a main extraction region in the main image on the basis of a difference between the plurality of edge distributions; and a region specifying module of extracting or tracing the specific image region on the basis of the main extraction region
and the plurality of images.
This storage medium stores a program that can precisely extract a specific image region from a main image including the specific image region to be extracted and a sub image which does not include any specific image region.
In order to achieve the above objects, according to the present invention, there is provided a storage medium storing a program including: an image input module of inputting a plurality of images including a main image including a specific image
region to be extracted, and a sub image which does not include any specific image region to be extracted; an edge distribution extraction module of extracting edge distributions of the plurality of images; a difference edge image extraction module of
extracting difference data of the plurality of edge distributions; an outermost contour line extraction module of obtaining an outermost contour line by tracing an outermost line of the difference data; a main extraction region estimation module of
estimating a main extraction region in the main image on the basis of the outermost contour line; and a region specifying module of extracting or tracing the specific image region on the basis of the main extraction region and the plurality of images.
This storage medium stores a program that can precisely extract a specific image region even when the specific image region to be extracted includes a region similar to an image which is not to be extracted.
According to a preferred aspect of the present invention, the image extraction step comprises: the partial region extraction step of extracting a portion of the image region to be extracted as a partial region from the input image; the region
growing step of performing region growing by thresholding similarities between the extracted partial region as a seed and its neighboring regions; and the extraction step of extracting a region obtained by the region growing as the image region to be
extracted.
Hence, even when a portion of the image region to be extracted has similar features of an image to an identical partial region in the reference image, the image region to be extracted can be stably extracted.
According to a preferred aspect of the present invention, the singular contour extraction step includes the step of extracting, as the singular contour portion, edges in the input image which have different line labels in terms of directions in
identical neighboring regions of the input image from those in reference image.
Therefore, only a contour inherent to a subject can be extracted by absorbing differences such as a positional offset, rotational offset, distortion, and the like between the reference image and the background portion of the input image if they
are present.
According to a preferred aspect of the present invention, the singular contour extraction step comprises: the dominant line map extraction step of segmenting the reference image into a plurality of blocks and detecting dominant line direction
components in the blocks; and the line direction comparison step of comparing a label assigned to each of edges of the input image and the dominant direction line component in the block to which that edge belongs, and when the label assigned to the edge
is different from the dominant direction line component in the block to which that edge belongs, the edge is extracted as the singular contour portion of the input image.
Accordingly, only a contour inherent to a subject can be extracted by absorbing differences such as a positional offset, rotational offset, distortion, and the like between the reference image and the background portion of the input image if they
are present.
According to a preferred aspect of the present invention, the image extraction step comprises: the partial region extraction step of binarizing a portion of the image region to be extracted and extracting the binary data as mask data; the
smoothing step of smoothing the extracted mask data; and the singular contour restoration step of restoring the singular contour portion to mask data after the mask data is smoothed.
Hence, partial shape details inherent to a subject can be stably extracted while removing the influences of noise and shading.
According to a preferred aspect of the present invention, the singular contour extraction step includes the step of detecting a occluding boundary line serving as a boundary between the image region to be extracted, and the remaining region, and
determining the detected occluding boundary line as the singular contour portion.
Thus, when edge direction components are hard to detect, for example, when a high-density fine pattern is present on the background or the region to be extracted, or when a plain region without texture crosses a line component having a direction
component different from the boundary direction of the plain region, a contour line inherent to a subject can be stably extracted.
According to a preferred aspect of the present invention, the singular contour extraction step comprises: the dominant line map extraction step of segmenting the reference image into a plurality of blocks and detecting dominant line direction
components in the blocks, extracting boundary points located in the vicinity of a boundary between a block without any dominant direction line component, and a block with the dominant direction line component among the blocks of the input image, and
extracting edges of the input image located at positions closest to the boundary points in a predetermined local region including the boundary point as a portion of the occluding boundary line.
Consequently, even when a plain region without textures crosses a line component having a direction component different from the boundary direction of the plain region, a contour line inherent to a subject can be stably extracted.
According to a preferred aspect of the present invention, the region growing step includes the step of controlling the region growing so that a region growing direction from the edge approximately agrees with a label in terms of directions of
that edge.
Henceforth, contour shape details of the object to be extracted can be prevented from being degraded, and image extraction free from missing portions can be realized.
According to a preferred aspect of the present invention, the image matching step comprises: the corresponding point extraction step of extracting corresponding points between the reference image and input image; the first transformation step of
geometrically transforming one of the input image and reference image on the basis of the extracted corresponding points; and the second transformation step after the geometric transformation, performing color correction of one of the input image and
reference image, so that corresponding pixels in regions including the corresponding points have substantially equal gray levels.
Accordingly, even when the reference image and the background portion of the input image have differences due to different photographing conditions or photographing means, image extraction with high tolerance to such differences can be realized.
According to a preferred aspect of the present invention, the geometric transformation includes global or local processing associated with at least one of a translation, rotation, magnification transformation, and perspective transformation.
Hence, image extraction can be stably attained by absorbing changes in photographing position upon inputting an image due to a shake of a photographing means during photographing, rotation of the sensor plane of the photographing means, different
numbers of pixels when a photographing means used for photographing the reference image is different from that for photographing the input image, magnification differences in photographing, optical characteristics differences such as aberrations, and the
like.
According to a preferred aspect of the present invention, the outermost contour line extraction means comprises: defect detection means for detecting gaps of the difference data; and linking means for linking the gaps.
Therefore, even when an image which is similar to the specific image to be extracted but is not to be extracted is present in the vicinity of the specific image region to be extracted, the specific image region can be precisely extracted.
According to a preferred aspect of the present invention, the outermost contour line extraction means extracts a plurality of outermost contour lines located at uppermost, lowermost, rightmost, and leftmost sides of the difference data, and said
main extraction region estimation means estimates the main extraction region on the basis of a logic operation result of a plurality of binarized outer regions obtained associated with the outermost contour lines.
As a consequence, even when an image which is similar to the specific image to be extracted but is not to be extracted is present in the vicinity of the specific image region to be extracted, the specific image region can be precisely extracted.
According to a preferred aspect of the present invention, the region specifying means extracts or tracks the specific image region on the basis of an initial region obtained by performing thresholding difference data between the main and sub
images, and the main extraction region.
Hence, even when the specific image region to be extracted includes many regions similar to an image which is not to be extracted, the specific image region can be precisely extracted.
According to a preferred aspect of the present invention, the region specifying means grows the initial region as a seed in the main extraction region on the basis of thresholding results of similarities with neighboring regions of the initial
region.
Thus, even when an image which is similar to the specific image to be extracted but is not to be extracted is present in the vicinity of the specific image region to be extracted, the specific image region can be precisely extracted.
According to a preferred aspect of the present invention, the region specifying means comprises: hole determination means for determining a hole region of the specific image region on the basis of difference data between the main and sub images;
and non-hole region extraction means for extracting a region except for the hole region from the initial region.
Hence, even when the specific image region to be extracted includes a hole region, the specific image region can be precisely extracted.
According to a preferred aspect of the present invention, the image processing apparatus comprises information input means for inputting rough information of a shape of the specific image region to be extracted.
Therefore, by inputting a rough shape of the specific image region to be extracted in advance, the specific image region can be precisely extracted at high speed.
According to a preferred aspect of the present invention, the main and sub images are a plurality of images photographed at different times.
According to a preferred aspect of the present invention, the main image is a moving image, and the sub image is a still image.
According to a preferred aspect of the present invention, the outermost contour line extraction step comprises: the gap dete | | |
