 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to a double eye image pickup apparatus, and more particularly to a double eye image pickup apparatus which can produce an image at an arbitrary aspect ratio in such a manner that a common subject is taken through a
plurality of image pickup systems such as TV or video cameras with the fields thereof partly overlapping with each other and that a plurality of taken images are combined with each other to obtain an arbitrary aspect-ratio image.
2. Related Background Art
There are methods conventionally known as the panorama image pickup technology for dynamic images using a plurality of image pickup systems, for example as described in Japanese Laid-open Patent Application No. 63-8641. The application describes
such a method that a space is divided into a plurality of spatial zones, information pieces on the spatial zones are simultaneously recorded, and these information pieces are simultaneously reproduced in a reproduction space. Generally, such technology
employs mechanical adjustment of angles of optical axes of the image pickup systems to make fields thereof connected with each other.
There is also an aspect ratio conversion method of trimming top and bottom margins or left and right margins of a screen before outputting, known as an aspect ratio conversion method for changing an aspect ratio of television screen. This is
used for example in converting an aspect ratio of a television screen according to the NTSC method (4:3) into an aspect ratio of a television screen for the HD method or for the ED method (16:9). Since this aspect ratio conversion method is a method
using a part of a taken image, there is no problem caused in respect of the quality of image, but there is a problem of a decrease in horizontal field angle to one third, specifically if a subject is taken through an NTSC camera using an image sensor
with an aspect ratio of 4:3 and the image is output to a monitor for NTSC with an aspect ratio of 4:3.
Meanwhile, some papers suggest double eye image pickup apparatus for obtaining a high-definition image by combining overlap areas in a plurality of images obtained in taking a common subject through a plurality of image pickup systems, for
example as described in Institute of Image Electronics of Japan, preliminary paper 90-03-04, p23-p28. Also, Japanese Laid-open Patent Application No. 5-110926 suggests an image combining apparatus which receives a plurality of images, performs a
coordinate conversion process for the images, combines the processed images and outputs an image with a wider field.
Applying such double eye image pickup apparatus and image combining apparatus, a double eye image pickup apparatus can be constructed for producing an image at an arbitrary aspect ratio by taking images of a common subject using a plurality of
image pickup systems with fields thereof partly overlapping and combining the thus obtained images. That is, a double eye image pickup apparatus can be constructed as shown in FIG. 25, in which two image pickup systems (left pickup system 110.sub.L and
right pickup system 110.sub.R) are used to obtain images of a common subject 101 in a state of the fields partially overlapping and in which a central processing unit (CPU) 120 combines a left image P.sub.L obtained from the left pickup system 110.sub.L
with a right image P.sub.R obtained from the right pickup system 110.sub.R to produce a single image P.sub.L+R at an arbitrary aspect ratio.
SUMMARY OF THE INVENTION
The image pickup systems in the conventional examples had, however, the following problems. Because the angles of the optical axes were set by the mechanical adjustment, it was difficult to make two adjacent images in registration, and boundary
lines were normally conspicuous. Further, it was also difficult to incorporate an occlusion area, which is an area taken by a certain image pickup system but not taken by another image pickup system, appearing when a subject located at a finite distance
is taken, into a final image without a feeling of incompatibility.
It is, therefore, an object of the present invention to provide an image pickup apparatus which can obtain an image at an arbitrary aspect ratio, solving the above problems, correcting the registration, and taking an image in the occlusion area
into consideration.
A double eye image pickup apparatus of the present invention is a double eye image pickup apparatus with a plurality of image pickup systems to perform image pickup with fields thereof at least partly overlapping with each other, which is
characterized by provision of:
means for setting a virtual projection plane in a subject space, based on image signals of overlap portions output from the respective image pickup systems; and
means for projecting the image signals onto the projection plane to combine the image signals into a single image signal.
For the double eye image pickup apparatus of the present invention, it is preferred that an occlusion portion in the image signals is detected and an image signal corresponding to the occlusion area is projected onto the projection plane in the
subject space to be subjected to the process and that, based on position information in the subject space about the image signals, the order of image signals used in the combining process is selected according to the position information.
According to the present invention, an apparatus is arranged with a plurality of image pickup systems to perform image pickup with the fields thereby partly overlapping, in which, using the image signals of overlap portions output from the
respective image pickup systems, corresponding points are detected in the overlap portions, position information is obtained for the images of overlapping areas, the subject is projected onto the virtual projection plane in the subject space, based on
the position information, the image signals are converted into a single image signal using the information of projection plane and the position information, and an angle of convergence of the image pickup systems is changed, whereby an aspect ratio of
image can be arbitrarily changed.
The present invention can provide an image pickup apparatus which can obtain an image at an arbitrary aspect ratio. Further, a high-quality image can be provided by removing a registration shift of the overlapping areas during the combining
process. Further, a high-quality image can be obtained without feeling of incompatibility, taking the occlusion areas in image into consideration.
Incidentally, the aforementioned reference, the Institute of Image Electronics, preliminary papers 90-03-04, p23-p28, discloses nothing about a specific method for obtaining a plurality of images by taking a common subject through a plurality of
image pickup systems with the fields partly overlapping. Further, the aforementioned Japanese Laid-open Patent Application No. 5-110926 discloses nothing about a method for obtaining an image at an arbitrary aspect ratio by combining a plurality of
images with each other. Namely, it fails to show means for specifically constructing the double eye image pickup apparatus as shown in FIG. 25. If the previously described aspect ratio conversion method for trimming upper and lower margins or left and
right margins of screen upon output is employed as the aspect ratio conversion method used in such a double eye image pickup apparatus, a problem of degradation of image quality occurs.
It is thus another object of the present invention to provide a double eye image pickup apparatus which can obtain an image at an arbitrary aspect ratio while minimizing the degradation of image quality.
A double eye image pickup apparatus of the present invention is a double eye image pickup apparatus which can produce an image at an arbitrary aspect ratio by combining a plurality of images obtained by taking a common subject through a plurality
of image pickup systems with the fields partly overlapping, characterized by provision of:
an image combination conversion processing portion for combination-converting a plurality of image signals output from the plurality of image pickup systems, setting virtual image pickup systems a position of a visual point and directions of
optical axes of which are defined by a positional deviation amount of the visual point from the plurality of image pickup systems and a convergence angle of the optical axes, into image signals output from the virtual image pickup systems with arbitrary
object distance and imaging magnification.
The apparatus may comprise:
a calculation control portion for determining a control target value of a convergence angle so that a combination-converted image has a predetermined aspect ratio, according to optical parameters at least representing an imaging magnification and
an object distance of said plurality of image pickup systems;
a convergence angle control portion for performing the convergence angle control according to an output signal from the calculation control portion; and
a drive portion for rotation-driving an optical axis of at least one image pickup system out of said plurality of image pickup systems according to an output signal from the convergence angle control portion.
Also, the apparatus may be so arranged that:
said plurality of image pickup systems each are arranged as an image pickup system having a zooming function; and that
the calculation control portion performs a convergence angle control of said plurality of image pickup systems so that the aspect ratio of the combination-converted image is kept constant in zooming said plurality of image pickup systems.
Since the double eye image pickup apparatus of the present invention is provided with the image combination conversion processing portion for performing such a combination-converting process of a plurality of image signals output from the
plurality of image pickup systems that, setting the virtual image pickup systems a position of a visual point and directions of optical axes of which are defined by a positional deviation amount of the visual point from the plurality of image pickup
systems and a convergence angle of the optical axes, image signals are output from the virtual image pickup systems with arbitrary object distance and imaging magnification, the images obtained by the plurality of image pickup systems with convergence
can be combined into an image on a virtual image plane. The apparatus can provide a combined image corrected in distortion caused by convergence.
Further, since the apparatus is provided with the calculation control portion for determining the control target value of convergence angle so that the combination-converted image has a predetermined aspect ratio, according to optical parameters
at least representing an imaging magnification and an object distance of the plurality of image pickup systems, and the convergence angle control portion for controlling the convergence angle according to an output signal from the calculation control
portion, the convergence control angle which determines the aspect ratio of the combined image is determined by the imaging magnification and object distance, so that the combined image can be obtained at an arbitrary aspect ratio.
Further, the present invention can reveal the following effects.
(1) Since a plurality of image signals output from the plurality of image pickup systems can be used in such a combination-converting process that, setting the virtual image pickup systems the position of the visual point and the directions of
optical axes of which are defined by a positional deviation amount of the visual point from the plurality of image pickup systems and a convergence angle of the optical axes, the image signals are converted into image signals output from the virtual
image pickup systems with arbitrary object distance and imaging magnification, the apparatus can provide a combined image corrected in distortion caused by convergence while minimizing the degradation of image quality of the combined image.
(2) Since the control target value of convergence angle can be determined so that a combination-converted image has a predetermined aspect ratio, according to optical parameters at least representing an imaging magnification and an object
distance of the plurality of image pickup systems, and the convergence angle control can be carried out according to the determined control target value of convergence angle, the apparatus can provide a combined image at an arbitrary aspect ratio.
(3) Since the convergence angle control of the plurality of image pickup systems can be carried out so that the aspect ratio of combination-converted image is kept constant in zooming the plurality of image pickup systems, the convergence control
can be carried out as to follow in zooming, whereby the aspect ratio can be kept constant without feeling of incompatibility.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a drawing to show an aspect ratio conversion processing portion in a double eye image pickup system in the first embodiment according to the present invention;
FIG. 2 is a drawing to show a basic layout of the double eye image pickup optical system in the first embodiment according to the present invention;
FIG. 3 is a drawing to show a basic layout of the double eye image pickup system in the first embodiment according to the present invention;
FIG. 4 is a drawing to show an outline of image input through the double eye image pickup system in the first embodiment according to the present invention;
FIG. 5 is a drawing to show reconstruction of epipolar line for images obtained by the double eye image pickup system in the first embodiment according to the present invention;
FIGS. 6A and 6B are schematic diagrams of epipolar line(s);
FIG. 7 is a drawing to show an outline of a reconstruction process of epipolar line;
FIG. 8 is a drawing to illustrate an outline of a conversion process of an image in occlusion area;
FIG. 9 is a drawing to illustrate an outline of a conversion process of an image in overlap area;
FIG. 10 is a drawing to illustrate an outline of a hidden-line elimination process;
FIG. 11 is a drawing to diagrammatically show an image in overlap area as combination-processed;
FIG. 12 is a drawing to show an outline of areas used in the combination process;
FIG. 13 is a schematic diagram of an image output after the aspect ratio conversion process;
FIG. 14 is a drawing to illustrate an outline of an occlusion area detection;
FIG. 15 is a drawing to show an aspect ratio conversion processing portion in the second embodiment of the present invention;
FIG. 16 is a drawing to show an outline of image conversion in the second embodiment according to the present invention;
FIG. 17 is a drawing to illustrate a setting method of a second virtual projection plane according to the present invention;
FIG. 18 is a drawing to show an outline of a second combining process according to the present invention;
FIG. 19 is a drawing to show a basic layout of image pickup systems in an embodiment of double eye image pickup apparatus according to the present invention;
FIG. 20 is a schematic, structural drawing to show the structure of left image pickup optical system and right image pickup optical system shown in FIG. 19;
FIG. 21 is a block diagram to show the structure of a convergence angle control system in an embodiment of double eye image pickup apparatus according to the present invention;
FIG. 22 is a drawing to illustrate a producing method of control target value of convergence angle in a calculation controlling portion shown in FIG. 21;
FIG. 23 is a drawing to illustrate another producing method of control target value of convergence angle in the calculation controlling portion shown in FIG. 21;
FIG. 24 is a flowchart to show an algorithm of convergence angle control in zooming in an embodiment of double eye image pickup apparatus according to the present invention; and
FIG. 25 is a conceptual drawing to show an example of double eye image pickup apparatus for producing an image at an arbitrary aspect ratio by combining two images obtained in taking a common subject using two image pickup systems with the fields
thereof partly overlapping.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
First described with FIG. 3 is a basic arrangement of a double eye image pickup system according to the first embodiment of the present invention. In FIG. 3, reference numeral 1 denotes a first subject, and 102 and 202 denote first and second
image pickup optical systems having equivalent specifications, which are normally zoom lenses. Numerals 103 and 203 designate image sensors having equivalent specifications, similarly, which are image pickup tubes such as Saticon or solid state image
sensing devices such as CCD. (Although FIG. 3 diagrammatically shows the image sensors of a single plate type (or a single tube type) for brevity, the generality is not lost if the image sensors are arranged in the form of two plate type (two tube type)
through a color separation optical system.)
The optical axes 101 and 201 of the systems cross each other on the first subject, so that the image pickup systems are arranged as inclined at .theta..sub.1 and .theta..sub.2 relative to reference lines F.sub.1 -F.sub.1 ' and F.sub.2 -F.sub.2 '
of rotational angle, each passing through the rotational center of image pickup system. Here, .theta..sub.1 and .theta..sub.2 are arranged and controlled so as to be always equal to each other. Defining .theta..sub.1 and .theta..sub.2 as an angle of
convergence, images are picked up while changing the angle of convergence in accordance with a change in subject distance. The image pickup is conducted at an arbitrary aspect ratio, utilizing a change of overlap area with the change in angle of
convergence.
Next described with FIG. 2 is the constitution and functions of the double eye image pickup system according to the first embodiment of the present invention.
In FIG. 2, 102a, 102b, 102c, 102d and 202a, 202b, 202c, 202d represent lens groups constituting first and second image pickup systems 102 and 202, among which 102b, 202b are zooming lens groups and 102d, 202d are focusing lens groups. Further,
106 and 206 are drive systems (zoom motors) for driving the zooming groups 102b and 202b, and, similarly, 107 and 207 are drive systems (focus motors) for driving the focusing groups 102d and 202d. There are mechanical systems not shown, and drive
systems (convergence angle motors) 104 and 204 for rotating 102 and 103, 202 and 203 together in a plane including the optical axes 101 and 201. Rotational angle information detecting units 105 and 205 are provided for detecting information about the
rotational angle. The detecting units may be constructed as external members such as rotary encoders, or as systems in which a drive system itself can obtain angle information by a driving method, for example as pulse motors. The angle of convergence
of the optical systems is obtained from signals from these detectors. Numerals 108 and 208 denote encoders (zoom encoders) each for obtaining position information along the direction of the optical axis of each lens group in the zooming group 102b or
202b in the image pickup optical system. The focal length f of the image pickup optical system 102 or 202 is obtained from a signal from the corresponding encoder.
Similarly, 109 and 209 are encoders (focus encoders) each for obtaining position information along the direction of the optical axis of each lens group in the focusing group 102d or 202d in the image pickup optical system. These encoders may be
external members such as potentiometers or systems in which a drive system itself can obtain position information along the direction of the optical axis of lens by a driving method, for example like a pulse motor.
Here, the zooming group 102b in the image pickup optical system of FIG. 2 is provided with the encoder (zoom encoder) 108 for preliminarily obtaining position information along the direction of the optical axis of the lens group, and the focal
length f of image pickup optical system 102 is obtained from a signal from the encoder 108. Similarly, the focusing group 102d in the image pickup optical system is also provided with the encoder (focus encoder) 109 for obtaining position information
along the direction of the optical axis of the lens group, and an imaging magnification .beta. of image pickup optical system 102 is obtained from a signal from the encoder 109 together with the signal from the zoom encoder 108. The image pickup
optical system 201 has the same structure and functions.
A control microcomputer (as will be referred to as "controlling CPU") 11 separately controls the drive systems 106, 107, 206, 207 with the signals from the encoders 108, 109, 208, 209, whereby two sets of focal lengths f and imaging
magnifications .beta. of the image pickup optical systems 102 and 202 are always kept coincident.
The construction of an aspect ratio conversion processing portion 10 is next described with FIG. 1. The aspect ratio conversion processing portion is composed of A/D converters 110, 210 for converting analog video signals output from the image
sensors 103 and 203 into respective digital video signals, image memories 111, 211 for storing the digital signals output from the corresponding A/D converters 110, 210, calculation processing portions 12L, 12R for reconstructing an epipolar line of each
video signal, a corresponding point detecting portion 13 for detecting corresponding points in the video signals, a correction conversion processing portion 14 for detecting a three-dimensional position corresponding to each video signal, based on
corresponding point detection signals obtained by the corresponding point detecting portion and parameters of the image pickup systems, a calculation processing portion 15 for making an estimation of occlusion area in an overlap portion, a conversion
processing portion 16 for projecting an image of an occlusion area onto a virtual projection plane, a hidden-line processing portion 17 for writing video signals successively from the far field into an image memory, based on three-dimensional position
information and information of the projection plane, an image memory 311, an image combination processing portion 18 for producing an image from the digital video signals stored in the memories 111, 211 and 311 to output a signal of the produced image to
the outside, an image correlation processing portion 19 for performing a correlation operation of images with the digital video signals stored in the image memories 111, 211 and 311, and a controlling CPU 11 for performing various controls based on
calculation results supplied from the image correlation processing portion 15.
The operation of the aspect ratio conversion processing portion 10 according to the present invention is next described referring to FIG. 1 and FIG. 4. In FIG. 4, the double eye image pickup apparatus according to the present invention is taking
images of subjects 1, 2 and 3 with the optical systems being kept at a convergence angle .theta.. The aspect ratio conversion processing portion 10 performs a conversion process to convert an analog video signal supplied from each image sensor 103, 203
into a digital signal in the first or second A/D converter 110, 210. The controlling CPU 11 receives detection signals from the encoders 108, 208 and 105, 205 in the image pickup optical systems 102 and 202, to perform such a control that the focal
lengths, imaging magnifications and convergence angles are always kept coincident between the image pickup optical systems. The analog video signals input into the respective image sensors 103 and 203 are converted into digital video signals in the
first and second A/D converters 110 and 210, which are temporarily stored in the respective image memories 111 and 211.
The corresponding point detecting portion 13 performs an extraction process of corresponding points using the video signals stored in the image memories 111 and 211. FIG. 6A diagrammatically shows a relation between the convergence and
corresponding points. In FIG. 6A, F.sub.1 and F.sub.2 are rotation centers of the image pickup systems 102 and 202, and 103 and 203 are sensor surfaces. For convenience sake of illustration, the sensor surfaces 103 and 203 are placed in front of
F.sub.1 and F.sub.2. A plane defined by three points of F.sub.1, F.sub.2 and P is called as an epipolar plane, and an intersection 1 between the epipolar plane and the sensor surface 103 is called as an epipolar line. If a corresponding point to point
P is desired to be searched on the sensor surface, the search can be done only on the epipolar line 1 on the sensor surface 103. However, the epipolar line is an oblique line due to the convergence effect, and, therefore, the epipolar line 1 must be
reconstructed in order to perform the search of corresponding point in a smooth manner.
Because the epipolar lines of images focused on the image sensors 103 and 203 as shown in FIG. 4 are inclined as described above, the epipolar lines are reconstructed in the calculation processing portions 12L and 12R to be converted into
horizontal lines as shown in FIG. 6B. An applicable conversion method is one as disclosed in (Fumiaki Tomita and Hironobu Takahashi), "Measures based on boundary line expression of stereo image", Institute of Electronics, Information and Communication
Engineers of Japan, papers D Vol. J71-D, No. 6, pp 1074-1082. The reconstruction of epipolar line is such a conversion process that an image taken through a system in the arrangement as shown in FIG. 4 is converted into an image as taken through a
system in the arrangement shown in FIG. 7.
The video signals converted in the calculation control portions 12L and 12R are again written in the image memories 111 and 211. FIG. 5 schematically shows the images 121 and 221 obtained after the subjects shown in FIG. 4 were taken through the
image pickup optical systems and the reconstruction of epipolar line was carried out for them.
The corresponding point detecting portion 13 first performs the correlation calculation process between the images 121 and 221, detects an overlap area, and thereafter detects corresponding points using an image in the detected overlap area. For
example, the following calculation formula is used for the correlation calculation.
where x' is a coordinate in a direction within the plane of the drawing,
q.sub.1 is an intensity distribution of an image taken by the image pickup optical system 102,
and q.sub.2 is an intensity distribution of an image taken by the image pickup optical system 202.
Overlap areas 112 and 212 are detected based on coordinates providing the maximum correlation value function I'(x.sub.0 ') in formula (1).
The corresponding point detecting portion 13 detects corresponding points between the images after the reconstruction of epipolar line. As shown in FIG. 6B, the reconstructed epipolar lines are horizontal. Ignoring the effect of detection
errors, the detection of corresponding points can be done between the video signals on a common horizontal line, for example on h.sub.L and h.sub.R in FIG. 6B.
One of the detection methods of corresponding points is the template matching method. This method determines corresponding points such that a template is set around a point in a right image and corresponding points are determined by comparison
of similarity in a left image with respect to an image of the template. The corresponding point detecting portion 13 takes a cross-correlation between a pixel value in the image of template and pixel values in the searching image, as expressed by below
formula (2), and determines coordinates to show a maximum value as coordinates of a corresponding point. The maximum value of cross-correlation of formula (2) is 1. ##EQU1##
In above formula (2), L and R represent pixel values in the epipolar-line-reconstructed images 121 and 221, and .sigma.(x',y') the degree of correlation. Also, x', y' represent coordinate values.
The corresponding point detecting portion 13 extracts coordinates corresponding to each pixel value in image 221 from the image 121 by the above-described process and supplies the coordinate values of corresponding points to the correction
converting portion 14. The correction converting portion 14 performs a coordinate conversion process based on coordinate values in the image 121 corresponding to the pixel values in the image 221 obtained in the corresponding point detecting portion 13.
The epipolar-line-reconstructed images are equivalent to images taken by the systems arranged as shown in FIG. 7, and, therefore, the following description is based on the arrangement as shown in FIG. 7.
The correction conversion processing portion 14 performs the following calculation to calculate, based on the principle of triangulation, a position of a point, for example point P (x.sub.0,z.sub.0) in FIG. 7, using a baseline length K,
coordinates x.sub.r ' and x.sub.L ' of a corresponding point, and a distance S' from the rear principal point of lenses to the sensor surface.
The coordinates are defined with the origin at the lens center O.sub.L of the image pickup optical system 202 and the x axis and the z axis being set as shown in FIG. 7.
The calculation processing portion 15 detects an occlusion area in the overlap portions in the images, based on detection signals from the corresponding point detecting portion 13.
As shown in FIG. 5, the epipolar lines h.sub.R and h.sub.L are horizontal in the epipolar-line-reconstructed images 121 and 221. The corresponding point extraction is executed based on formula (2) along the epipolar lines, as described above.
There is, however, no corresponding point detected for an image of the occlusion area. In more detail, no corresponding points are present on the epipolar line h.sub.R to an image area g.sub.1L on the epipolar line h.sub.L in FIG. 5. Therefore,
corresponding points cannot be determined for the area g.sub.1L. It is thus impossible to calculate information of three-dimensional position even if the image of occlusion area is input into the correction conversion processing portion 14. Then
another process is necessary for the image of occlusion area.
In order to process the above image of occlusion area, the calculation processing portion 15 first extracts an occlusion area from the images. The extraction can be executed by the following method. For example, correlation values
.sigma..sub.max (x',y') at corresponding points obtained by the formula (2) are evaluated with respect to a threshold value S as shown in FIG. 14, so that areas satisfying .sigma..sub.max <S are extracted as occlusion areas.
Transferred to the conversion processing portion 16 is position information of regions having corresponding points obtained in the correction conversion processing portion 14 and image information of occlusion areas extracted by the calculation
processing portion 15.
The present embodiment employed the method for detecting corresponding points from the images to obtain the position information, but it is clear that another distance detecting means can be used, such as means for detecting three-dimensional
position information of a subject by projecting a beam of infrared rays or a laser beam onto the subject and detecting reflected light from the subject.
The conversion processing portion 16 combines the three-dimensional position information with the image information of occlusion areas to execute mapping of subject information in the overlap portions.
An outline of the operation is described referring to FIG. 8.
FIG. 8 and FIG. 9 are schematic cross-sectional view of the subjects including the epipolar lines h.sub.L and h.sub.R in FIG. 5. FIG. 8 is a drawing to illustrate the occlusion areas and FIG. 9 a drawing to illustrate an outline of the
combination process of images of overlap areas. In FIG. 9, thick lines f.sub.1 ', f.sub.2 ' and f.sub.3 ' represent areas for which the position information was obtained in the correction conversion processing portion 14.
In FIG. 8, g.sub.1L ' and g.sub.1R ' indicate subject regions corresponding to the occlusion areas g.sub.1L ' and g.sub.1R ' in FIG. 5, for which the position information was not attained, as described previously. It is, however, necessary to
combine images as taking the images of occlusion areas into consideration in order to produce an aspect-ratio-converted image. In the present embodiment the combination process is carried out by the following method. First, three-dimensional position
information of two points at the both ends of each occlusion area, for example three-dimensional position information a' and b' (see FIG. 8) corresponding to two points a, b, is extracted from the images shown in FIG. 5, choosing a farther point out of
the two points a' and b'. Since the point a' is located at a farther position in case of FIG. 8, the point a' is extracted. Then a virtual projection plane (or line) A.sub.1 including the point a' is set, and the subject area g.sub.1L ' corresponding to
the occlusion area is projected onto the projection plane A.sub.1. Namely, the image information of the occlusion area g.sub.1L ' in FIG. 5 is fit into an area A.sub.1L on the projection plane A.sub.1. The same process is executed for the occlusion
area g.sub.1R, so that it is fit into an area A.sub.1R on a projection plane A.sub.2.
According to the above process, mapping is made in the subject space for the images of the overlap areas 112 and 212 in FIG. 5.
FIG. 9 shows regions corresponding to the images on the epipolar lines h.sub.L and h.sub.R in FIG. 5 as mapped by the conversion processing portion 16.
The hidden-line processing portion 17 performs a conversion process of the images of the overlap portions, based on the mapped image information, to write processed data in the image memory 311. In the process a position C of virtual visual
point is first set as shown in FIG. 9. This point may be preliminarily set or externally input. Then setting a virtual sensor plane at the position of visual point, the mapped image information is projected thereonto. For example, a point a
(x.sub.1,z.sub.1) in FIG. 9 is projected at a position determined by the following formula:
where S' is a distance from the lenses to the virtual sensor surface 301. Here, the coordinate axes are the same as those in FIG. 7.
The projection is carried out in order from the farthest subject region onto the sensor surface 301, and the result is written into the image memory 311. In case of FIG. 9, the subject information f.sub.3 ', A.sub.1R on the projection plane
A.sub.2 is projected onto the virtual sensor surface 301 in accordance with formula (4), and then the result is written into the image memory 311. Next, the subject information f.sub.1 ', A.sub.1L on the plane A.sub.1 is projected onto the virtual
sensor plane 301 in accordance with formula (4), and then the result is written into the image memory 311. Further, the information f.sub.2 ' is projected onto the plane 301 in accordance with formula (4), and the result is written into the image memory
311. If there are overlap positions as so projected, later-projected information, i.e., information at a position of smaller z coordinate in FIG. 9, is overwritten over larger x information in the image memory 311. This is diagrammatically shown in
FIG. 10. FIG. 10 schematically shows a state of the enlarged virtual sensor surface 301 in FIG. 9 with the subject information projected thereonto. In FIG. 10, B.sub.1 is a projected image of the subject information on the projection plane A.sub.1,
B.sub.2 a projected image of the subject information on the projection plane A.sub.2, and B.sub.3 a projected image of the subject information f.sub.3. Thus, the images of B.sub.1 ', B.sub.2 ' and B.sub.3 ' are stored in the image memory 311.
FIG. 11 shows an image 320 stored in the image memory 311 after the conversion process in the hidden-line processing portion 17, of the images of the overlap portions.
The image correlation processing portion 19 reads the image information 121, 221, 320 from the image memories 111, 211 and 311, and first performs the correlation calculation, which is the same calculation as formula (1), using the images 121 and
320 to cut a region 122 (FIG. 12) out of the image 121. Then the correlation calculation is performed for the images 221 and 320 to cut a region 222 (FIG. 12) out of the image 221. FIG. 12 schematically shows the cut-out images 122 and 222. The
combination processing portion 18 reads the image 122 from the image memory 111 and similarly reads the image 222 from the image memory 211. The combination processing portion 18 combines the two images with the image 320 read from the image memory 311
to produce an image converted in aspect ratio and output it. A seamless process is conducted to make seam lines inconspicuous between the images. Here is a process of smoothing described as an example. This method is executed as follows. A window W
(i,j) is set about a certain image f (x',y') and an output obtained by the following calculation is given to a point (x',y'). ##EQU2## The size of window is mxn, and for example, the following matrix can be used as the window. ##EQU3## FIG. 13
schematically shows an image 400 as produced by the combination and seamless processes in the combination processing portion 18.
Through the above procedure, an aspect-ratio-converted image is output.
FIG. 15 is a block diagram to show an aspect ratio conversion processing portion 21 in the second embodiment according to the present invention. In FIG. 15, the same members and members with the same functions are denoted by the same reference
numerals as those in the aspect ratio conversion processing portion 10 shown in the first embodiment, and are omitted to describe herein. The operation is also omitted because it is the same as that of the first embodiment. Only constituent portions in
FIG. 15 different from those in the aspect ratio conversion processing portion 10 as described previously are described in the following.
In FIG. 15, a conversion processing portion 21 receives three-dimensional position information of areas for which corresponding points were obtained by a correction conversion processing portion 14, and image information of occlusion areas
obtained by a calculation processing portion 15, and sets a virtual projection plane in a subject space to map the subject information therein.
The operation of the conversion processing portion 21 is next described.
FIG. 16 is a drawing obtained by mapping the regions for which corresponding points obtained in the correction conversion processing portion 14 are present, in the subject space. The conversion processing portion 21 sets a virtual projection
plane, based on depth coordinates (z coordinates) of regions f.sub.1 ', f.sub.2 ', f.sub.3 ' for which corresponding points were obtained. A method to determine the setting position may be one to arrange the regions f.sub.1 ', f.sub.2 ', f.sub.3 ' in
order from the smallest z coordinate, for example as shown in FIG. 17.
Then, a difference of z coordinate between the regions, for example z.sub.1 -z.sub.2 or z.sub.3 -z.sub.1, is obtained and a threshold value T is set for the difference.
the regions f.sub.1 ' and f.sub.3 ' are deemed as being in a same zone and provided with a same label.
In case of the arrangement shown in FIG. 16 where the regions f.sub.1 ' and f.sub.3 ' are given a same label, a virtual projection plane A.sub.3 is set at a position of (z.sub.3 +z.sub.1)/2. A projection plane A.sub.4 is set at a position of
z.sub.2 for the region f.sub.2 '.
Then the regions f.sub.1 ' and f.sub.3 ' for which corresponding points were obtained are projected onto the projection plane A.sub.3. The conversion processing portion 21 receives information of occlusion areas from the calculation processing
portion 15 and, similarly as in the first embodiment, it reads z coordinates of two points a and b at the both ends of an occlusion area, for example g.sub.1L in FIG. 5, to project the information onto a projection plane located at a z-coordinate
position of a farther point out of the two points. FIG. 18 shows an outline of the above process.
Since the regions f.sub.1 ' and f.sub.3 ' are projected on the virtual projection plane A.sub.3 by the above process, a point a' in the subject space, corresponding to the point a in FIG. 5, is at (x.sub.1,(z.sub.1 +z.sub.3)/2). On the other
hand, point b' is at (x.sub.2, z.sub.2), so that an image of occlusion area g.sub.1L in FIG. 5 is mapped on the farther projection plane A.sub.3. Another occlusion area g.sub.1R in FIG. 5 is also mapped on the plane A.sub.3 by the same process.
Next described is a process for the image regions k.sub.1 and k.sub.2 as shown in FIG. 5, other than the overlap regions in the images 121 and 221. For illustration purpose, the description concerns the image 221 in FIG. 5.
The mapping of the image area k.sub.1 other than the overlap area is as follows. First obtained is a point C' in FIG. 18, which has coordinates in the subject space corresponding to a position of a border between the overlapping area 212 and the
non-overlapping area k.sub.1, i.e., the point C in FIG. 5. As shown in FIG. 18, the coordinates of point C' are (0,(z.sub.1 +z.sub.3)/2). The plane A.sub.3 on which the point C' is projected is selected based on the coordinate values, and the image
information of area k.sub.1 in FIG. 5 is mapped onto the virtual projection plane A.sub.3 in FIG. 18. The same process is executed for the area k.sub.2 on the image 121 in FIG. 5 to be mapped on a region k.sub.2 ' in FIG. 18.
All image information of the images 121 and 221 in FIG. 5 is mapped in the subject space by the above serial processing in the conversion processing portion 21.
Following the formula (4) as described in the first embodiment, the combination processing portion 22 projects the subject information mapped on the virtual projection planes A.sub.3 and A.sub.4 in the conversion processing portion 21, onto a
virtual sensor plane 301. A point d' (x.sub.4,(z.sub.1 +z.sub.3)/2) in FIG. 18 is projected at the following point on the sensor surface 301 by the formula (4).
The mapped subject information is projected onto the virtual sensor surface 301 by the above processing. Namely, the subject information is written into the image memory 311 in accordance with projected positions. As for the information as
superimposed on the sensor surface 301, information with smaller z coordinate is overwritten over information with larger z in the image memory 311 similarly as in the first embodiment. After all subject information is written, an aspect-ratio-converted
image is output. Although the first and second embodiments employed the method for automatically setting the virtual projection planes based on the position information of subjects, a method for externally inputting the setting positions of projection
planes, for example arranging such that a man inputs the positions of projection planes based on the image information, may be also employed.
It is of course that the present invention can be applied to image pickup apparatus for producing a high-definition image by superimposing a plurality of obtained images over the entire area.
FIG. 19 is a drawing to show a basic arrangement of image pickup systems in an embodiment of double eye image pickup apparatus according to the present invention.
The do | | |
