 |
Description  |
|
|
BACKGROUND OF THE INVENTION
The present invention relates to an image combining apparatus suitably used in an electronic camera system having a panoramic image sensing mode and, more particularly, to an image combining apparatus for generating a panoramic image by
connecting a plurality of images which partially overlap each other.
As conventional image combining apparatuses for generating a panoramic image (to be also referred to as a combined image hereinafter) by combining a plurality of images obtained by sensing an object image so that the images partially overlap each
other, image combining apparatuses disclosed in, e.g., Japanese Patent Laid-Open Nos. 4-52635 and 5-122606 are known.
An image combining apparatus disclosed in Japanese Patent Laid-Open No. 4-52635 detects information of, e.g., the three-dimensional position, direction, or the like of a camera apparatus, i.e., information for defining the relationship between
adjacent frames or scenes, upon sensing images by an electronic camera, stores such information together with image signals, and reproduces the image signals on the basis of the stored information. With this operation, the image signals are reproduced
while holding the positional relationship between adjacent frames or scenes of the sensed images, and various applications, e.g., reconstruction of an image in a three-dimensional space, and the like, can be achieved.
However, in the conventional image combination represented by Japanese Patent Laid-Open No. 4-52635, upon generating a panoramic image by combining a plurality of images, when the individual images are converted into equal-magnification images on
the basis of zoom magnification information, and the converted image signals are recorded on a frame memory, information of the three-dimensional positions, directions, and the like is simultaneously recorded in correspondence with each image, and the
recorded image signals are merely combined to generate a panoramic image.
More specifically, in the electronic camera system of Japanese Patent Laid-Open No. 4-52635, since a plurality of panoramically sensed images are merely pasted on the frame memory in accordance with the information of their three-dimensional
positions, directions, and the like, the panoramic image obtained by combining these images is nothing but the one obtained by simply laying out the individual images. Accordingly, the joint portions become discontinuous, and a panoramic image with only
poor appearance is obtained.
On the other hand, an image combining apparatus disclosed in Japanese Patent Laid-Open No. 5-122606 senses a plurality of images to be connected to partially overlap each other, and obtains the luminance difference or chromaticity difference
between areas (to be referred to overlapping areas hereinafter) at the end portions of the images to be connected. Upon combination, these images are connected to make the overlapping areas overlap each other, so that the luminance difference or
chromaticity difference between the overlapping areas of the images to be connected assumes 0 or a minimum value.
In this manner, the image combining apparatus is convenient since an image sensing device or the object to be sensed need not be precisely moved and aligned for the purpose of connecting a plurality of images. In addition, since the connecting
position is determined using the luminance difference or chromaticity difference between adjacent images, the images can be accurately combined without impairing connectability.
In the above-mentioned image combining apparatus, when a panoramic image of a wide field angle is to be generated on the basis of a plurality of images sensed so that their image fields partially overlap each other, the following processing is
executed. That is, two images are connected on a plane by performing geometric transformation such as the affine transformation for the two images, so that identical points in the overlapping areas of the two images match each other.
More specifically, when the image of an object is sensed by panning the camera in the horizontal direction, two images are obtained by sensing the image of the object so that an object-side principal point O of the camera lens does not
substantially move during panning, as shown in FIG. 51.
In FIG. 51, I.sub.1 and I.sub.2 represent image sensing planes before and after panning, h.sub.i represents the horizontal field of view of the image sensed before panning, and h.sub.2 represents the horizontal field of view of the image sensed
after panning. If .theta. represents the horizontal field angle of the camera lens, images each having a horizontal field angle .theta. are obtained from the image sensing planes I.sub.1 and I.sub.2. Therefore, if .alpha. represents the overlapping
angle between the two images, the horizontal field of view of the combined image obtained by these two images is (2.theta.-.alpha.).
Assume that a plane having a rectangular frame is placed in front of the camera. This frame is placed in front of the camera at a substantially equal angular position with respect to the two image sensing planes I.sub.1 and I.sub.2 In this case,
images obtained by the two image sensing planes I.sub.1 and I.sub.2 are expected to be images a and b shown in FIG. 52. As for the image sensing plane I.sub.1, since the right side of the frame is closer to the camera, the right side of the image a is
sensed to be vertically larger than its left side. Conversely, as for the image sensing plane I.sub.2, since the left side of the frame is closer to the camera, the left side of the image b is sensed to be vertically larger than its right side. More
specifically, the two straight lines, which are originally parallel to each other, of the frame are sensed as two straight lines which are not parallel to each other in the images a and b.
For this reason, when the two images a and b are combined by performing the affine transformation of translation processing alone (i.e., the affine transformation which does not include enlargement and rotation in the image plane) for these two
images a and b, a combined image ab including two straight lines which cross each other is obtained, as shown in FIG. 53. In this combined image ab, frame lines duplicate themselves in an overlapping portion P.sub.ab resulting in an unnatural image.
This is because images cannot be accurately combined by translation in the image plane since the two image sensing planes I.sub.1 and I.sub.2 are not spatially present in a single plane. Even when image enlargement or rotation within the image plane is
performed, the unnaturalness cannot be eliminated.
In order to alleviate the unnaturalness in the overlapping portion P.sub.ab produced by the image combining apparatus of Japanese Patent Laid-Open No. 5-122606, Japanese Patent Laid-Open No. 5-14751 discloses a panoramic image combining
apparatus, which combines images by performing geometric transformation for projecting a plurality of images sensed to partially overlap each other onto a single columnar surface.
More specifically, the panoramic image combining apparatus of Japanese Patent Laid-Open No. 5-14751 obtains a combined image free from any unnaturalness without double frame lines in the overlapping portion P.sub.ab by temporarily projecting
images a and b obtained by the two image sensing planes I.sub.1 and I.sub.2 onto a common columnar surface.
However, in the conventional image combining apparatuses in Japanese Patent Laid-Open Nos. 122606, 5-14751, and the like, since the individual images are combined by setting and connecting overlapping areas, so that the luminance difference or
chromaticity difference in the overlapping portion of images assumes 0 or a minimum value, i.e., since two areas whose luminance difference or chromaticity difference assumes 0 or a minimum value are simply connected, a panoramic image obtained by
combining the images is merely the one obtained by laying out the individual images, as in the case of Japanese Patent Laid-Open No. 4-52635. Therefore, the joint portions become discontinuous, and a panorama image with poor appearance is obtained. For
example, when the image to be connected rotates from an original image or the enlargement factor of the original image is different from that of the image to be connected, the image quality of the combined image considerably deteriorates.
On the other hand, in the image combining apparatus of Japanese Patent Laid-Open No. 5-14751, which combines images by projecting the individual images onto a columnar surface to geometrically transform them, when a plurality of images obtained
by sensing an object image while panning the camera in both the horizontal and vertical directions are combined, an unnatural image in which frame lines are doubled in the overlapping portion of a combined image is obtained like in the combined image
P.sub.ab in FIG. 53, which is obtained by only translating the sensed images.
When a rectangular frame is sensed at, e.g., four framing positions using the image combining apparatus of Japanese Patent Laid-Open No. 5-14751, four images c.sub.1 to C.sub.4 are obtained, as shown in FIG. 54. When these images c.sub.1 to
c.sub.4 are combined by only translating them without any projection of images onto a columnar surface, an unnaturally combined image c in which frame lines are doubled in overlapping portions P.sub.c is obtained, as shown in FIG. 55. Even when the
images c.sub.1 to c.sub.4 are combined by temporarily projecting the images onto a columnar surface and then translating them, an unnaturally combined image c' in which frame lines are doubled in overlapping portions P.sub.c ' is obtained, as show in
FIG. 56.
This means that even when two images obtained by sensing an object image by panning the camera are to be combined in the horizontal direction, if the camera has been tilted slightly during image sensing, an unnatural image in which frame lines
are doubled in the overlapping portion of the combined image is obtained.
SUMMARY OF THE INVENTION
The present invention has been made to solve the above-mentioned problems, and has as its object to provide an image combining apparatus which can always easily obtain a high-quality panoramic image.
It is another object of the present invention to provide an image combining apparatus, which stores a series of images and image sensing conditions of the series of images in correspondence with each other, so that one of a plurality of combining
means can be adaptively selected on the basis of the image sensing conditions corresponding to the individual images upon combining the images.
According to a preferred aspect of the present invention, corresponding points in the overlapping portions of a series of images to be combined are detected, image sensing parameters used upon executing combining processing are generated on the
basis of the detected corresponding points, and the combining processing is executed using the image sensing parameters.
According to a preferred aspect of the present invention, focal point position information upon image sensing is detected as an image sensing condition. Therefore, a storage means stores the focal point position information in correspondence
with the individual images. A control means discriminates based on the focal point position information corresponding to a series of images to be combined if these images are obtained by short- or long-distance image sensing. If it is determined that
the images are obtained by short-distance image sensing, the control means selects a first combining means for short distances on the basis of the discrimination result; otherwise, the control means selects a second combining means. The series of images
obtained by short-distance image sensing are combined by appropriate combining processing by the first combining means, and the series of images obtained by long-distance image sensing are combined by appropriate combining processing by the second
combining means for long distances.
According to a preferred aspect of the present invention, a transforming means transforms pixel values of the overlapping portions of the individual images on the basis of image sensing conditions corresponding to a series of images to be
combined. With this processing, an image near the connecting portions of adjacent images is transformed.
According to a preferred aspect of the present invention, a detection means detects exposure information upon image sensing as an image sensing condition. Therefore, the storage means stores focal point position information corresponding to the
individual images. The transforming means corrects the density levels of the overlapping portions of the individual images on the basis of the exposure information corresponding to a series of images to be combined.
According to a preferred aspect of the present invention, an image combining means combines a plurality of spherical projection images obtained by a spherical projection transforming means on the basis of the image sensing conditions
corresponding to a series of images to be combined.
According to a preferred aspect of the present invention, the spherical projection transforming means projects and transforms the individual images onto a spherical surface having the focal point position upon image sensing as a center, thereby
generating a plurality of spherical projection images to be combined.
According to a preferred aspect of the present invention, the image combining means combines a plurality of spherical projection images obtained by the spherical projection transforming means, and a plane projection transforming means projects
and transforms the combined image onto a plane again to generate a plane projection combined image.
According to a preferred aspect of the present invention, an adding means adds projection plane type information indicating a spherical projection image to images obtained by the spherical projection transforming means, and adds projection plane
type information indicating a plane projection combined image to an image obtained by the plane projection transforming means.
According to a preferred aspect of the present invention, the image combining means combines a plurality of spherical projection images obtained by spherical projection transforming means to obtain a combined image, and selectively outputs one of
the combined image and the plane projection combined image via an output means in correspondence with the field of view of the combined image.
According to a preferred aspect of the present invention, an optical axis control means controls the directions of the optical axes of a plurality of image sensing means on the basis of image sensing conditions detected by the detection means.
Accordingly, an image is divisionally sensed by the plurality of image sensing means so that the sensed images partially overlap each other.
According to a preferred aspect of the present invention, the angle of the apparatus is detected, and the detected angle information is held together with images sensed in the panoramic image sensing mode. Upon combining these images, a
combining scheme is adaptively selected on the basis of the angle information. With this control, image combination is done by an optimal combining scheme with respect to the angle of the apparatus.
According to a preferred aspect of the present invention, the position of the apparatus is detected, and the detected position information is held together with images sensed in the panoramic image sensing mode. Upon combining these images, a
combining scheme is adaptively selected on the basis of the position information. With this control, image combination is done by an optimal combining scheme with respect to the position of the apparatus.
In order to achieve the above objects, according to another image combining apparatus of the present invention, a condition detection means detects the attitude of the apparatus produced upon movement of the apparatus as moving state information. A storage means stores the moving state information of the apparatus detected by the condition detection means and image sensing conditions as additional information in correspondence with the sensed images. A prediction means predicts the overlapping
area between adjacent images to be combined on the basis of the additional information stored in the storage means. A correspondence detection means detects corresponding points in the overlapping area predicted by the prediction means. An image
combining means combines a series of images on the basis of the corresponding points detected by the correspondence detection means.
According to a preferred aspect of the present invention, the condition detection means detects an angular component of the movement of an image sensing means produced upon rotation of the apparatus during image sensing.
According to a preferred aspect of the present invention, the condition detection means detects the angular component using an angular velocity sensor.
According to a preferred aspect of the present invention, the condition detection means detects a translation component of the apparatus produced upon translation of the apparatus during image sensing.
According to a preferred aspect of the present invention, the condition detection means detects the translation component using an acceleration sensor.
According to a preferred aspect of the present invention, a position detection means detects the position of the apparatus produced upon movement of the apparatus during image sensing. The storage means stores position information obtained by
the position detection means as the additional information.
According to a preferred aspect of the present invention, the position detection means detects the three-dimensional position of the apparatus produced upon movement of the apparatus during image sensing.
According to a preferred aspect of the present invention, an overlapping area detection means sequentially detects the degree of overlapping between sequentially input images by calculations on the basis of the moving state information of the
apparatus detected by the condition (moving information) detection means and image sensing conditions detected by an image sensing condition detection means. A comparison means compares the degree of overlapping detected by the overlapping area
detection means with a predetermined threshold value. A control means makes the operation control of the overall apparatus to set the apparatus in the image sensing state at the timing at which the control means determines based on the comparison result
of the comparison means that the degree of overlapping is equal to the threshold value.
According to a preferred aspect of the present invention, an image sensing situation detection means detects the image sensing situation of a plurality of images to be combined, in accordance with the additional information stored in the storage
means. When it is determined based on the detection result of the image sensing situation detection means that the plurality of images to be combined are obtained by rotation, a switching control means controls a spherical projection transforming means
to generate a spherical projection image by projecting and transforming the plurality of images onto a spherical surface, and controls a switching means to select and output the spherical projection image obtained by the spherical projection transforming
means.
According to a preferred aspect of the present invention, an output means outputs a panoramic image generated by the image combining means to an external device.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the
figures thereof.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram showing the arrangement of an electronic camera system to which an image combining apparatus according to the present invention is applied in the first embodiment of the present invention;
FIG. 2 is a plan view showing the object to be subjected to short-distance panoramic image sensing;
FIG. 3 is a view for explaining the image sensing situation of the short-distance panoramic image sensing;
FIG. 4 is a plan view showing two images obtained by the short-distance panoramic image sensing;
FIG. 5 is a plan view showing a combined image obtained by combining the two images;
FIG. 6 is a plan view for explaining the image sensing situation of long-distance panoramic image sensing;
FIG. 7 is a plan view showing three images obtained by the long-distance panoramic image sensing;
FIG. 8 is a plan view showing a combined image obtained by combining the three images;
FIG. 9 is a view for explaining image data stored in an image memory of the electronic camera system;
FIG. 10 is a block diagram showing the arrangement of an image combining apparatus of the electronic camera system;
FIG. 11 is a block diagram showing the arrangement of an image combining unit in an electronic camera system to which an image sensing apparatus according to the present invention is applied in the first modification of the present invention;
FIG. 12 is a view for explaining the spherical projection processing of the image combining unit;
FIG. 13 is a view for explaining a spherical projection image obtained by the spherical projection processing;
FIG. 14 is a block diagram showing the arrangement of an image combining unit of an electronic camera system to which an image sensing apparatus according to the present invention is applied in the second modification of the present invention;
FIG. 15 is a plan view showing the image to be processed by the image combining unit;
FIG. 16 is a view for explaining the density level correction processing in the image combining unit;
FIG. 17 is a block diagram showing the arrangement of an image combining unit of an electronic camera system to which an image sensing apparatus according to the present invention is applied in the third modification of the present invention;
FIGS. 18A and 18B are views for explaining the processing for controlling the optical axes of two image sensing units;
FIG. 19 is a flow chart showing the processing of an image combining unit of an electronic camera system to which an image sensing apparatus according to the present invention is applied in the fourth modification of the present invention;
FIG. 20 is a flow chart showing the image input processing in the processing of the image combining unit in detail;
FIG. 21 is a flow chart showing spherical projection transforming processing in the processing of the image combining unit in detail;
FIG. 22 is a plan views showing two spherical projection images obtained by the spherical projection transforming processing;
FIG. 23 is a flow chart showing the corresponding point extraction processing in the processing of the image combining unit in detail;
FIG. 24 is a flow chart showing the image combination in the processing of the image combining unit in detail;
FIG. 25 is a plan view showing a combined spherical projection image obtained by the image combination;
FIG. 26 is a flow chart showing the plane projection transforming processing in the processing of the image combining unit in detail;
FIG. 27 is a plan view showing a plane projection combined image obtained by the plane projection transforming processing;
FIG. 28 is a block diagram showing the arrangement of an image combining unit of an electronic camera system to which an image sensing apparatus according to the present invention is applied in the fifth modification of the present invention;
FIG. 29 is a flow chart showing the processing of the image processing unit;
FIG. 30 is a plan view showing two combined images each obtained by combining two spherical projection images by the image combining unit;
FIG. 31 is a plan view showing a combined image obtained by combining the two combined images;
FIG. 32 is a plan view showing an image obtained as a result of plane projection transforming processing of the combined image;
FIG. 33 is a block diagram showing the arrangement of an electronic camera system to which an image combining apparatus according to the present invention is applied in the sixth modification of the present invention;
FIG. 34 is a block diagram showing the arrangement of an angle detector of the electronic camera system;
FIG. 35 is a block diagram showing the arrangement of an electronic camera system in the second embodiment of the present invention;
FIG. 36 is a block diagram showing the arrangement of an angle detector in the electronic camera system;
FIG. 37 is a view for explaining the case wherein panoramic image sensing of a long-distance object is done by panning;
FIG. 38 is a view for explaining a series of images obtained by the panoramic image sensing;
FIG. 39 is a view for explaining a panoramic image obtained by combining the series of images;.
FIG. 40 is a block diagram showing the arrangement of an image combining apparatus of the electronic camera system;
FIG. 41 is a view for explaining the processing for predicting the overlapping area by the image combining apparatus;
FIG. 42 a view for explaining the processing for detecting corresponding points in the overlapping area by the image combining apparatus;
FIG. 43 is a view for explaining the corresponding points obtained by the corresponding point detection;
FIG. 44 is a block diagram showing the arrangement of an electronic camera system according to the seventh modification;
FIG. 45 is a view for explaining the processing for predicting the overlapping area in the electronic camera system;
FIG. 46 is a block diagram showing the arrangement of an image combining apparatus in an electronic camera system according to the present invention in the eighth modification;
FIG. 47 is a view for explaining the case wherein panoramic image sensing of a short-distance object is done by translation according to the eighth modification;
FIG. 48 is a view for explaining a spherical mapped image of the eighth modification;
FIG. 49 is a view for explaining the processing for performing spherical mapping transformation by the image combining apparatus;
FIG. 50 is a view for explaining the processing for predicting the overlapping area by the image combining apparatus;
FIG. 51 is a view for explaining the case wherein two images are sensed by panning a conventional electronic camera system in the horizontal direction;
FIG. 52 is a plan view showing the two images obtained by the electronic camera system;
FIG. 53 is a plan view showing a combined image obtained by combining the two images;
FIG. 54 is a plan view showing four images obtained by sensing a rectangular frame at four framing positions by the electronic camera system;
FIG. 55 is a plan view showing a combined image o | | |
