 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to a multi-eye image pickup apparatus and,
more particularly, to a multi-eye phototaking method and apparatus which
provide a panoramic image having an arbitrary aspect ratio by combining a
plurality of images.
2. Related Background Art
As a conventional aspect ratio changing scheme of changing the aspect ratio
of a television screen, an aspect ratio changing scheme of trimming the
upper and lower portions or left and right portions of a screen when image
data is output is known. For example, such an aspect ratio changing scheme
is used to change the aspect ratio (4:3) of an NTSC television screen to
the aspect ratio (16:9) of an HD or ED2 television screen.
These aspect ratio changing schemes use part of a phototaken image. For
this reason, especially when an object to be photographed is phototaken by
an NTSC camera using an image sensor with an aspect ratio of 16:9, and the
resultant image is output to an NTSC monitor with an aspect ratio of 4:3,
the horizontal field angle decreases to about 1/3, although no problem is
posed in terms of image quality.
A multi-eye image pickup apparatus for obtaining a high-resolution image by
combining the overlapping regions of a plurality of images obtained by
phototaking one object using a plurality of image taking systems is
disclosed in, e.g., THE JOURNAL OF THE INSTITUTE OF IMAGE ELECTRONICS
ENGINEERS OF JAPAN, 90-03-04, pp. 23-28. Also, a multi-eye image pickup
apparatus for outputting an image with a large field angle by combining
images obtained by coordinates changing processing of a plurality of input
images is disclosed in, e.g., Japanese Laid-Open Patent Application No.
5-110926.
With the use of such a multi-eye image pickup apparatus or its application,
a multi-eye image pickup apparatus can be obtained, which forms an image
having an arbitrary aspect ratio by combining a plurality of images
obtained by phototaking one object to be photographed using a plurality of
image taking systems with portions of the fields of view of the image
taking systems overlapping each other.
As in the multi-eye image pickup apparatus shown in FIG. 7, one object 101
to be photographed is phototaken by using two image taking systems (a left
side image taking system 110.sub.L and a right side image taking system
110.sub.R) with portions of the fields of view of the image taking systems
overlapping each other, and a left side image P.sub.L obtained by the left
side image taking system 110.sub.L and a right side image P.sub.R obtained
by the right side image taking system 110.sub.R are combined with each
other by a central processing unit (CPU) 120, thereby forming one image
P.sub.L+R having an arbitrary aspect ratio.
The above conventional art, however, indicates no method of obtaining an
image having a predetermined aspect ratio. Therefore, with the use of the
scheme of trimming a combined image, a deterioration in image quality
occurs.
In order to solve the above problem, "Multi-eye Image Pickup Apparatus" in
Japanese Patent Application No. 5-223544 (U.S. Ser. No. 08/267,117 filed
on Jun. 24, 1994) is disclosed which performs a phototaking operation by
using a plurality of image taking systems with portions of the fields of
view overlapping. In this image pickup apparatus, a plurality of image
signals output from the respective image taking systems are combined into
one image signal output in a state defined by an arbitrary object distance
and the imaging magnification from an image taking system in which the
viewpoint position and the optical axis direction are defined by the shift
amount of the viewpoint position from each image taking system and the
convergence angle of the optical axis. The present applicant also
disclosed a method of obtaining an image exhibiting little deterioration
in image quality and having undergone correction of distortion caused by
convergence.
The above apparatus and method will be described below. FIG. 8 shows the
basic arrangement of the above apparatus and method. This apparatus is
constituted by two image taking systems, i.e., a right side image taking
system 810.sub.R and a left side image taking system 810.sub.L as first
and second imaging optical systems.
An object plane 801 is a plane common to the two image taking systems. A
left side imaging optical system 811.sub.L and a right side imaging
optical system 811.sub.R are imaging optical systems having equivalent
specifications. In general, zoom lenses are used as these optical systems.
A left side image sensor 812.sub.L and a right side image sensor 812.sub.R
also have equivalent specifications. As these sensors, image pick-up tubes
such as saticons or solid-state image pick-up elements such as CCDs are
used. Optical axes L.sub.L and L.sub.R of these image taking systems are
symmetrically inclined by about .theta., respectively, with respect to a
normal O-O' to the object plane 801 (a point O is located on the object
plane 801) in such a manner that the phototaking fields of view overlap in
a predetermined amount in accordance with the selected aspect ratio of a
screen. Note that 2.theta. is the convergence angle. Object planes
802.sub.L and 802.sub.R are respectively conjugate to the image sensors
812.sub.L and 812.sub.R and are respectively inclined by .theta. with
respect to the object plane 801. Points O.sub.L and O.sub.R are the
intersections between the optical axes L.sub.L and L.sub.R and the object
plane 801. Points C.sub.L and C.sub.R are the principal points of the
imaging optical systems 811.sub.L and 811.sub.R (more specifically, the
principal points on the object side). The imaging optical systems
811.sub.L and 811.sub.R include zoom lens groups and focusing lens groups,
driving systems for driving these lens groups, and encoders for obtaining
position information in the optical axis directions. In addition, each
image taking system includes a mechanical system for rotating the image
taking system within a plane including the optical axis, a driving system,
and an encoder for detecting a rotational angle. A convergence angle
controlling system sets a control target value of the convergence angle in
accordance with an output signal from each encoder and performs
convergence control to obtain an image having a predetermined aspect
ratio.
A method of determining a control target value of the convergence angle
will be described below with reference to FIGS. 8 and 9.
Let .beta. be the imaging magnification of the left side imaging optical
system 811.sub.L and the right side imaging optical system 811.sub.R shown
in FIG. 8, z be the object distance (the distance between the point object
and the point C.sub.L and the distance between the point object and the
point C.sub.R), and 2d be the distance (base line length) between the
point C.sub.L and the point C.sub.R. Assume that the viewpoint is set at a
point on the normal O-O' separated from the object plane 801 toward the
point O' by the distance z, and a virtual plane is set such that the
virtual imaging magnification at the viewpoint is .beta.' (i.e., the
distance between the viewpoint and the image plane is .beta.'z'). In this
case, a temporary imaginary plane I.sub.L+R is obtained by combining an
image plane I.sub.L of the left side image sensor 812.sub.L and an image
plane I.sub.R of the right side image sensor 812.sub.R, as shown in FIG.
9.
Referring to FIG. 9, points A.sub.L, B.sub.L, C.sub.L, and D.sub.L are the
diagonal points of the image plane I.sub.L of the left side image sensor
812.sub.L. These points respectively correspond to points A.sub.L ',
B.sub.L ', C.sub.L ', and D.sub.L ' on the temporary imaginary plane
I.sub.L+R. Points A.sub.R, B.sub.R, C.sub.R, and D.sub.R are the diagonal
points of the image plane I.sub.R of the right side image sensor
812.sub.R. These points respectively correspond to points A.sub.R ',
B.sub.R ', C.sub.R ', and D.sub.R ' on the temporary imaginary plane
I.sub.L+R. In addition, points E.sub.L and F.sub.L are points on the upper
and lower sides of the image plane I.sub.L of the left side image sensor
812.sub.L. These points correspond to the center of the overlapping
region. Points E.sub.R and F.sub.R are points on the upper and lower sides
of the image plane I.sub.R of the right side image sensor 812.sub.R. These
points correspond to the center of the overlapping region. Both the points
E.sub.L and E.sub.R correspond to a point E' on the temporary imaginary
plane I.sub.L+R. Both the points F.sub.L and F.sub.R correspond to a point
F' on the temporary imaginary plane I.sub.L+R.
Assume that the centers of the image planes I.sub.L and I.sub.R are
origins, and the horizontal and vertical directions in FIG. 9 are x- and
y-axes, respectively. In this case, if coordinate systems are defined on
the image planes I.sub.L and I.sub.R, an image point (x.sub.R,y.sub.R) on
the image plane I.sub.R of the right side image sensor 812.sub.R
corresponds to an image point (x.sub.R ',y.sub.R '), on the temporary
imaginary plane I.sub.L+R, which is given by equations (1) and (2):
x.sub.R '={(x.sub.R cos .theta.+.beta.z sin .theta.+.beta.d)/(-x.sub.R sin
.theta.+.beta.z')}.times..beta.'z' (1)
y.sub.R '={y.sub.R /(-x.sub.R sin .theta.+.beta.z')}.times..beta.'z'(2)
In addition, an image point (x.sub.L,y.sub.L) on the image plane I.sub.L of
the left side image sensor 812.sub.L corresponds to an image point
(x.sub.L ',y.sub.L '), On the temporary imaginary plane I.sub.L+R, which
is given by equations (3) and (4):
x.sub.L '={(x.sub.L cos .theta.-.beta.z sin .theta.-.beta.d)/(x.sub.L sin
.theta.-.beta.z')}.times..beta.'z' (3)
y.sub.L '={y.sub.L /(x.sub.L sin .theta.+.beta.z')}.times..beta.'z'(4)
By performing geometrical transform processing like the one indicated by
equations (1) to (4), images on a plurality of image sensors, which cause
convergence, can be combined into an image on one temporary imaginary
plane. Therefore, with the use of an image combining/transform processing
portion (not shown) for performing such geometrical transform processing,
an image having undergone correction of distortion caused by convergence
can be obtained.
In the above method and apparatus, the following problems are posed when an
object to be photographed has a distance distribution in which the object
includes an object at a greatly different distance from the background
from the rest of the objects.
Assume that an object A is located at a distance z.sub.A, as shown in FIG.
8. In this case, if geometrical transform processing based on equations
(1) to (4) is performed, the object position on a temporary imaginary
plane is shifted by the difference between the object distance z and the
object distance z.sub.A. In this case, the positions (x.sub.R ",y.sub.R ")
and (x.sub.L ",y.sub.L "), on an ideal temporary imaginary plane for
forming a combined image free from any position shift, which correspond to
the image point (x.sub.R,y.sub.R) on the image plane of the first image
sensor and the image point (x.sub.L,y.sub.L) on the image plane of the
second image sensor, respectively, are given by
x.sub.R "={[x.sub.R cos .theta.+.beta.z sin
.theta.+.beta.d(z/z.sub.A)]/[-x.sub.R sin .theta.+.beta.z(1-z/z.sub.A) cos
.theta.+.beta.z'(z/z.sub.A)]}.times..beta.'z' (5)
y.sub.R "={y.sub.R /[-x.sub.R sin .theta.+.beta.z(1-z/z.sub.A) cos
.theta.+.beta.z'(z/z.sub.A)]}.times..beta.'z' (6)
x.sub.L "={[x.sub.L cos .theta.-.beta.z sin
.theta.-.beta.d(z/z.sub.A)]/[x.sub.L sin .theta.+.beta.z(1-z/z.sub.A) cos
.theta.+.beta.z'(z/z.sub.A)]}.times..beta.'z' (7)
y.sub.L "={y.sub.L /[x.sub.L sin .theta.+.beta.z(1-z/z.sub.A) cos
.theta.+.beta.z'(z/z.sub.A ]}.times..beta.'z' (8)
Consequently, a positional shift occurs owing to the influence of the
object distance. The corresponding shift amounts are respectively given by
.DELTA.x.sub.R =x.sub.R "-x.sub.R ', .DELTA.y.sub.R =y.sub.R "-y.sub.R '
.DELTA.x.sub.L =x.sub.L "-x.sub.L ', .DELTA.y.sub.L =y.sub.L "-y.sub.L
'(9)
For this reason, an arbitrary point on the object A may be shifted from an
ideal position a by (.DELTA.x.sub.R,.DELTA.x.sub.L) in the x direction,
and by (.DELTA.y.sub.R,.DELTA.y.sub.L) in the y direction, resulting in a
deterioration in the image quality of a combined image.
In addition, since the background is concealed by the object A in the
hatched regions in the overlapping region in FIG. 10, these hatched
regions are not phototaken by both the first and second image sensors,
although the regions are included in the overlapping region. Therefore, a
portion IL in FIG. 10 is phototaken as only an image for the first image
sensor; and a portion IR, as only an image for the second image sensor.
Even if, therefore, images having undergone coordinates changing
processing are combined, the object portion may overlap the background
portion, resulting in a deterioration in the image quality of the combined
image.
SUMMARY OF THE INVENTION
The present invention has been made in consideration of the above
situation, and has as its object to provide an image combining method and
apparatus which can obtain a panoramic image having an arbitrary aspect
ratio with little deterioration in image quality.
According to the present invention, there is provided an image combining
method of phototaking an object to be photographed by using a plurality of
image taking systems with portions of fields of view thereof overlapping,
and forming combined image information by combining pieces of image
information obtained by the image taking systems, comprising the steps of
extracting corresponding point information between the pieces of image
information, calculating distance information by using optical information
in each of the image taking systems and the corresponding point
information, sequentially generating two-dimensional coordinates of a
display image based on the combined image information, changing the
two-dimensional coordinates to two-dimensional coordinates of each piece
of image information by using the optical information and the distance
information, performing compensation processing for each piece of image
information on the basis of the changed two-dimensional coordinates for
each piece of image information, determining an occlusion portion in which
a far-distance portion of the object is concealed by a near-distance
portion, of an overlapping region of phototaken images on the basis of the
corresponding point information, and selecting one of pieces of
compensated image information with respect to the occlusion portion in
forming combined image information by combining the pieces of compensated
image information.
In this case, in selecting compensated image information with respect to
the occlusion portion, compensated image information exhibiting the least
occlusion portion may be selected from the pieces of compensated image
information.
According to the present invention, there is provided an image combining
apparatus for phototaking an object to be photographed by using a
plurality of image taking systems with portions of fields of view thereof
overlapping, and forming combined image information by combining pieces of
image information obtained by the image taking systems, comprising
corresponding points extracting means for extracting corresponding point
information between the pieces of image information, distance information
calculating means for calculating distance information by using optical
information in each of the image taking systems and the corresponding
point information, address generating means for sequentially generating
two-dimensional coordinates of a display image based on the combined image
information, coordinates changing means for changing the two-dimensional
coordinates to two-dimensional coordinates of each piece of image
information by using the optical information and the distance information,
image information compensating means for performing compensation
processing for each piece of image information on the basis of the changed
two-dimensional coordinates for each piece of image information,
determining means for determining an occlusion portion in which a
far-distance portion of the object is concealed by a near-distance
portion, of an overlapping region of phototaken images on the basis of the
corresponding point information, and selecting means for selecting one of
pieces of compensated image information with respect to the occlusion
portion in forming combined image information by combining the pieces of
compensated image information.
In this case, in selecting compensated image information with respect to
the occlusion portion, compensated image information exhibiting the least
occlusion portion may be selected from the pieces of compensated image
information.
In addition, the apparatus may further comprise image correlating means for
dividing each compensated image information into small regions, and
determining an occlusion portion in accordance with a similarity between
corresponding small regions of the respective pieces of compensated image
information. Furthermore, in this case as well, in selecting compensated
image information with respect to the occlusion portion, compensated image
information exhibiting the least occlusion portion may be selected from
the pieces of compensated image information.
Of an overlapping region, a region where a far-distance portion of an
object to be photographed is concealed by a near-distance portion is not
necessarily phototaken by all the image taking systems. For this reason,
if images phototaken by the respective image taking systems are combined,
a deterioration in image quality occurs.
In the image combining method and apparatus of the present invention, in
performing image combining processing of the above regions, an image
obtained by one image taking system is selected. Therefore, no
deterioration in the image quality of a combined image occurs.
When an image taking system exhibiting the minimum concealed region is to
be selected in image selection processing, an actual phototaken image is
used mostly. Therefore, an accurate image closer in appearance to the
actual image can be obtained upon combining processing.
With the use of an image correlation portion, a compensated image having
higher correlation is selected. Even if, therefore, corresponding point
extraction processing is not completely performed, a combined image having
undergone smooth compensation processing and having less distortion can be
obtained.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a view showing the basic positions of image taking systems in a
multi-eye image pickup apparatus according to an embodiment of the present
invention;
FIG. 2 is a schematic view showing the arrangements of right and left side
imaging optical systems in FIG. 1;
FIG. 3 is a block diagram showing the arrangement of a convergence angle
controlling system in the multi-eye image pickup apparatus according to
the embodiment of the present invention;
FIG. 4 is a view for explaining an image formed after combining processing;
FIG. 5 is a block diagram showing the arrangement of an image
combining/changing processing portion in the first embodiment of the
present invention;
FIG. 6 is a block diagram showing the arrangement of an image
combining/changing processing portion in the second embodiment of the
present invention;
FIG. 7 is a view showing the concept of a multi-eye image pickup apparatus
for forming an image having an arbitrary aspect ratio by combining two
images obtained by phototaking one object to be taken using two image
taking systems with portions of the fields of view overlapping each other;
FIG. 8 is a view showing the basic positions of multi-eye image taking
systems;
FIG. 9 is a view for explaining a method of forming a control target value
of the convergence angle; and
FIG. 10 is a view for explaining an image formed after combining processing
.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
FIG. 1 shows the basic positions of image taking systems in a multi-eye
image pickup apparatus according to an embodiment of the present
invention.
This embodiment is basically designed to perform image combining processing
by using the same multi-eye image pickup apparatus disclosed in Japanese
Patent Application No. 5-223544. Referring to FIG. 1, an object plane 1, a
right side object plane 2.sub.R, a left side object plane 2.sub.L, a right
side image taking system 10.sub.R, a left side image taking system
10.sub.L, a right side imaging optical system 11.sub.R, a left side
imaging optical system 11.sub.L, a right side image sensor 12.sub.R, and a
left side image sensor 12.sub.L are respectively identical to the object
plane 801, the object plane 802.sub.R, the object plane 802.sub.L, the
right side image taking system 810.sub.R, the left side image taking
system 810.sub.L, the right side imaging optical system 811.sub.R, the
left side imaging optical system 811.sub.L, the right side image sensor
812.sub.R, and the left side image sensor 812.sub.L shown in FIG. 8.
Therefore, a description of these components will be omitted.
The multi-eye image pickup apparatus of this embodiment includes two image
taking systems, i.e., the left side image taking system 10.sub.L and the
right side image taking system 10.sub.R. In this case, the left side image
taking system 10.sub.L comprises the left side imaging optical system
11.sub.L constituted by a zoom lens, and the left side image sensor
12.sub.L constituted by an image pick-up tube such as a saticon or a
solid-state image pick-up element such as a CCD. The right side image
taking system 10.sub.R has specifications equivalent to those of the left
side image taking system 10.sub.L and comprises the right side imaging
optical system 11.sub.R constituted by a zoom lens, and the right side
image sensor 12.sub.R constituted by an image pick-up tube such as a
saticon or a solid-state image pick-up element such as a CCD.
As the left side image sensor 12.sub.L and the right side image sensor
12.sub.R, a single-tube (single-plate) type sensor like the one shown in
FIG. 1 may be used. Alternatively, a two-tube (two-plate) or three-tube
(three-plate) type sensor may be used through a color separation optical
system.
As described with reference FIG. 8, which shows a similar arrangement,
optical axes L.sub.L and L.sub.R of the left and right side imaging
optical systems 11.sub.L and 11.sub.R are symmetrically inclined by about
.theta., respectively, with respect to a normal O-O" to the object plane 1
in such a manner that the imaging fields of view overlap in a
predetermined amount in accordance with the selected aspect ratio of a
screen. Therefore, the left side object plane 2.sub.L conjugate to the
left side image sensor 12.sub.L and the right side object plane 2.sub.R
conjugate to the right side image sensor 12.sub.R are respectively
inclined by the angle .theta. with respect to the object plane 1.
The arrangements of the left side imaging optical system 11.sub.L and the
right side imaging optical system 11.sub.R, which are based on the basic
positions described above, will be described next with reference to FIG.
2. FIG. 2 shows the schematic arrangements of the left and right side
imaging optical systems shown in FIG. 1.
The left side imaging optical system 11.sub.L comprises lens groups
21.sub.L to 24.sub.L including a zoom lens group 22.sub.L and a focusing
lens group 24.sub.L, a zooming motor 25.sub.L as a driving system for
driving the zoom lens group 22.sub.L, a focusing motor 26.sub.L as a
driving system for driving the from an output signal from the zooming
encoder 41.sub.L. A focusing encoder 42.sub.L is used to obtain the
position information of each lens group arranged in the focusing lens
group 24.sub.L in the optical axis direction (the direction of the optical
axis L.sub.L). The object distance for the left side imaging optical
system 11.sub.L can be obtained from an output signal from the focusing
encoder 42.sub.L. As the zooming encoder 41.sub.L and the focusing encoder
42.sub.L, members to be externally mounted, such as potentiometers, may be
used. Alternatively, a driving system itself, such as a pulse motor, may
detect the position information of each lens in the optical axis direction
by a driving method.
The right side imaging optical system 11.sub.R has the same arrangement as
that of the left side imaging optical system 11.sub.L. Note that a zooming
motor 25.sub.R of the right side imaging optical system 11.sub.R is driven
by the above zooming controlling portion to have the same focal length as
that of the left side imaging optical system 11.sub.L. A focusing motor
26.sub.R of the right side imaging optical system 11.sub.R is driven by
the above zooming controlling portion to focus the right side imaging
optical system 11.sub.R at the same object distance as that at which the
left side imaging optical system 11.sub.L is focused.
A convergence angle controlling system 50 of the multi-eye image pickup
apparatus of this embodiment focusing lens group 24.sub.L, a mechanical
system (not shown) and a driving system (convergence angle motor 27.sub.L)
which serve to integrally rotate the left side imaging optical system
11.sub.L and the left side image sensor 12.sub.L within a plane including
the optical axis L.sub.L, and a convergence angle encoder 28.sub.L for
detecting the rotational angle of the convergence angle motor 27.sub.L. As
the convergence angle encoder 28.sub.L, a member to be externally mounted,
such as a rotary encoder, may be used. Alternatively, a driving system
itself, such as a pulse motor, may detect the rotational angle by a
driving method.
The zooming motor 25.sub.L of the left side imaging optical system 11.sub.L
is driven by a zooming controlling portion (not shown), to which a zooming
signal is externally input, to make the left side imaging optical system
11.sub.L have a predetermined focal length. The focusing motor 26.sub.L is
driven by a zooming controlling portion (not shown), to which a focusing
signal is externally input, to focus the left side imaging optical system
11.sub.L at a predetermined object distance.
A zooming encoder 41.sub.L of the left side imaging optical system 11.sub.L
is used to obtain the position information of each lens group arranged in
the zoom lens group 22.sub.L in the optical axis direction (the direction
of the optical axis L.sub.L). The focal length of the left side imaging
optical system 11.sub.L can be obtained will be described next with
reference to FIG. 3. FIG. 3 is a block diagram showing the arrangement of
a convergence angle controlling system of the multi-eye image pickup
apparatus according to an embodiment.
The convergence angle controlling system 50 is constituted by a calculation
controlling portion 51 for determining a control target value of the
convergence angle such that an image formed upon combining/changing
processing has a predetermined aspect ratio in accordance with optical
parameters representing at least the imaging magnification and the object
distance of the right and left imaging optical systems 11.sub.R and
11.sub.L, and a convergence angle controlling portion 52 for performing
convergence angle control in accordance with an output signal from the
calculation controlling portion 51.
The calculation controlling portion 51 receives an output signal from the
zooming encoder 41.sub.L of the left side imaging optical system 11.sub.L,
an output signal from the focusing encoder 42.sub.L, an output signal from
a zooming encoder 41.sub.R of the right side imaging optical system
11.sub.R, an output signal from a focusing encoder 42.sub.R, and a
controlling signal S from a system controller 60. The convergence angle
controlling portion 52 receives an output signal from the convergence
angle encoder 28.sub.L of the left side imaging optical system 11.sub.L,
an output signal from a convergence angle encoder 28.sub.R of the right
side imaging optical system 11.sub.R, and an output signal from the
calculation controlling portion 51. Note that the convergence angle motor
27.sub.L of the left side imaging optical system 11.sub.L and the
convergence angle encoder 27.sub.R of the right side imaging optical
system 11.sub.R are respectively driven by output signals from the
convergence angle controlling portion 52.
When the zoom lens group 22.sub.L is driven by the zooming motor 25.sub.L
of the left side imaging optical system 11.sub.L, an output signal from
the zooming encoder 41.sub.L, which represents the position information of
each lens group arranged in the zoom lens group 22.sub.L in the optical
axis direction, is input to the calculation controlling portion 51. When a
zoom lens group 22.sub.L is driven by the zooming motor 25.sub.R of the
right side imaging optical system 11.sub.R, an output signal from the
zooming encoder 41.sub.R, which represents the position information of
each lens group arranged in the zoom lens group 22.sub.R, is input to the
calculation controlling portion 51.
When the focusing lens group 24.sub.L is driven by the focusing motor
26.sub.L of the left side imaging optical system 11.sub.L, an output
signal from the focusing encoder 42.sub.L, which represents the position
information of each lens group arranged in the focusing lens group
24.sub.L in the optical axis direction, is input to the calculation
controlling portion 51. When a focusing lens group 24.sub.R is driven by
the focusing motor 26.sub.R of the right side imaging optical system
11.sub.R, an output signal from the focusing encoder 42.sub.R, which
represents the position information of each lens group arranged in the
focusing lens group 24.sub.R in the optical axis direction, is input to
the calculation controlling portion 51.
Upon reception of the controlling signal S commanding rotation of the
optical axes of the left and right side imaging optical systems 11.sub.L
and 11.sub.R from the system controller 60, the calculation controlling
portion 51 forms a control target value of the convergence angle 2.theta.
in accordance with output signals from the zooming encoders 41.sub.L and
41.sub.R and output signals from the focusing encoders 42.sub.L and
42.sub.R, and outputs an output signal representing the formed control
target value to the convergence angle controlling portion 52.
The convergence angle controlling portion 52 compares the output signal
from the calculation controlling portion 51 with output signals from the
convergence angle encoders 28.sub.L and 28.sub.R, and forms output signals
for driving the convergence angle motors 27.sub.L and 27.sub.R so as to
make the output signals from the convergence angle encoders 28.sub.L and
28.sub.R coincide with the formed target value.
In the above multi-eye image pickup apparatus of this embodiment, the
control target value is determined by the same method as that used in the
prior art described with reference to FIG. 9.
Images on a plurality of image sensors, which have convergence, can be
combined into one image on a temporary imaginary plane by performing
geometrical changing processing like the one indicated by equations (1) to
(4). Therefore, with the use of an image combining/changing processing
portion (not shown) for performing such geometrical changing processing,
an image having undergone correction of distortion caused by convergence
can be obtained.
In addition, this image combining/changing processing portion serves as a
portion for combining a plurality of image signals output from the
respective image taking systems into one image signal output in a state
defined by an arbitrary object distance and the imaging magnification from
an image taking system in which the viewpoint position and the optical
axis direction are defined by the shift amount of the viewpoint position
from each image taking system and the convergence angle of the optical
axis.
If the size of the image plane I.sub.L of the left side image sensor
12.sub.L and an image plane I.sub.R of the right side image sensor
12.sub.R is H.times.V (aspect ratio H/V), and the size of a temporary
imaginary plane I.sub.L+R is H'.times.V' (aspect ratio H'/V'), a length H'
of the temporary imaginary plane I.sub.L+R in the horizontal direction can
be calculated by obtaining the x-coordinates of points A.sub.R ' and
B.sub.L ' after changing processing of points A.sub.R and B.sub.L from
equations (1) and (3). In addition, a length V' of the temporary imaginary
plane I.sub.L+R in the vertical direction can be calculated by obtaining
the y-coordinates of the point B.sub.L ' and a point D.sub.L ' after
changing processing of the point B.sub.L and a point D.sub.L from
equations (2) and (4). The aspect ratio of the temporary imaginary plane
I.sub.L+R, which is obtained in this manner, is expressed by equation
(10):
H'/V'=.vertline. sin (.theta.+.omega.)/sin
(.omega.)-2.beta.d/H.vertline..times.H/V (10)
where .omega.=tan.sup.-1 [-H/(2.beta.z)]: half field angle of image taking
system in horizontal direction.
If, therefore, both the size H.times.V of the image planes I.sub.L and
I.sub.R of the left and right side image sensors 12.sub.L and 12.sub.R and
a base line length 2d are constant, equation (10) can be rewritten into
equation (11) below:
.theta.=f(H'/V', .beta., z) (11)
That is, the convergence controlling angle .theta. for determining the
aspect ratio H'/V' of the temporary imaginary plane I.sub.L+R is
determined by an imaging magnification .beta. and an object distance z.
The calculation controlling portion 51 shown in FIG. 3 calculates the
average value of output signals from the zooming encoders 41.sub.L and
41.sub.R and the average value of output signals from the focusing
encoders 42.sub.L and 42.sub.R, and obtains the imaging magnification
.beta. from the two calculated average values by using optical parameters
for the imaging optical systems 11.sub.L and 11.sub.R (e.g., the focal
lengths and principal point intervals of the lens groups constituting the
imaging optical systems 11.sub.L and 11.sub.R). Subsequently, the control
target value of the convergence angle is calculated in accordance with the
imaging magnification .beta. and the object distance z according to
equation (11) in such a manner that the aspect ratio H'/V' of the
temporary imaginary plane I.sub.L+R becomes a predetermined aspect ratio.
The calculated control target value of the convergence angle is sent to
the convergence angle controlling portion 52.
In this case, the average value of the output signals from the zooming
encoders 41.sub.L and 41.sub.R and the average value of the output signals
from the focusing encoders 42.sub.L and 42.sub.R are obtained. However,
the output signal from one of the zooming encoders 41.sub.L and 41.sub.R
and the output from the one of the focusing encoders 42.sub.L and 42.sub.R
may be used. A table of the convergence controlling angle .theta.
corresponding to the imaging magnification .beta. and the object distance
z may be prepared in the calculation controlling portion 51, and the
control target value of the convergence angle may be obtained by referring
to this table. Alternatively, the relationship between the imaging
magnification .beta., the object distance z, and the convergence
controlling angle .theta. may be approximated by a simple polynomial, and
the coefficients of this polynomial may be stored in the calculation
controlling portion 51, thereby calculating the convergence controlling
angle .theta..
In the above-described manner, an image having a predetermined aspect ratio
can be obtained. In this case, since no trimming is required, and an
overlapping region can be set to be as large as possible, a high-quality
image can be obtained. As shown in FIG. 4, an image (temporary imaginary
plane I.sub.L+R) having undergone combining processing exhibits a slightly
distorted rectangular region. For this reason, in displaying an image
having an aspect ratio of 16:9 on an HD monitor or the like, a frame line
may be displayed along the outline of the frame.
FIG. 5 is a block diagram showing the arrangement of the main part of an
image combining apparatus according to the first embodiment of the present
invention. A left side image taking system 510L and a right side image
taking system 510R correspond to the image taking systems whose basic
positions are shown in FIG. 1. The convergence angle .theta. is controlled
by the convergence angle controlling system shown in FIG. 3 to obtain an
image having a predetermined aspect ratio, and right and left images are
simultaneously phototaken. The remaining portion corresponds to an image
combining/changing processing portion. The arrangement and operation of
this portion will be described below.
Image memories 520L and 520R are storing portions for storing left and
right images as digital image data. A parameter storing portion 521 is a
storing portion for storing a base line length 2d, the convergence angle
.theta., an imaging magnification .beta., and an object dist | | |
