 |
Description  |
|
|
BACKGROUND OF THE INVENTION
The present invention relates to an image synthesizing apparatus for
synthesizing a plurality of images, recorded by an electronic camera,
having overlapping portions, on a computer, an image sensing apparatus
used in combination with the image synthesizing apparatus, and an image
synthesizing system using these apparatuses. More specifically, the
present invention relates to automatic determination of timing for
recording an image or images taken in series (image or images to be
combined with an image or images which have already been recorded).
Processing for obtaining a wide angle image by synthesizing a plurality of
images having overlapping portions using a computer is called "panoramic
image synthesizing processing". This processing is to obtain a single
image of stretched scenery or a single wide image.
The resolution of an electronic camera is lower than that of a silver
halide camera or an image scanner (i.e., the number of pixels of the
electronic camera is small), which is a defect of an electronic camera.
Because of this defect, it is necessary to synthesize images recorded by
an electronic camera into a panoramic image in order to obtain a wide
image as well as an image of high resolution. An image of wider angle of
view obtained in one shot with the electronic camera would provide lower
the resolution than an image having normal width. In contrast, by sensing
a single object (original image) in a plurality of shots, or sensing a
scenery in a plurality of shots by shifting image sensing positions in the
horizontal direction, the recorded image as a whole has as high resolution
than that of an image sensed by a scanner or a silver halide camera.
Meanwhile, some panoramic image synthesizing systems have been configured
with electronic cameras and personal computers connected to each other,
which inputs images from the electronic cameras to the personal computers,
and provided at affordable prices. In such systems, it is difficult for
the computers to take full-size images on real-time basis because of the
limitation of transmission paths, such as a parallel interface (I/F) or a
serial I/F, connecting the electronic cameras and the computers.
Therefore, in the conventional systems, images are reduced in size in
order to secure a speed for transmitting images on almost a real-time
basis and inputted by the computers in a short interval, then the images
are provided for a user to confirm an image. Then, after the user confirms
a scene which is desired to obtain a reduced image, a shutter button of
the electronic camera is pressed. Accordingly, it is possible to obtain a
full-size still image after confirming it. Note, the reduced image used
for confirmation is called "confirmation image", hereinafter.
The most difficult process in generating a panoramic image by synthesizing
a plurality of images is to find overlapping portions, i.e., to find a
group of corresponding pixels in the plurality of images (the process is
called "corresponding point extraction", hereinafter). Where corresponding
points are to be found in two images, for example, position relationship
between these two images is detected, thus it is possible to synthesize
the images.
Upon performing corresponding point extraction, characterizing points
and/or resemblance points are needed in these images. In a case where
there are little characterizing points and/or resemblance points in the
images, an error may occur when performing the corresponding point
extraction. As a result, intervention by the user is required in many
cases, and it is difficult to realize a full-automatic corresponding point
extraction process with the conventional technique.
SUMMARY OF THE INVENTION
The present invention has been made in consideration of the above
situation, and has as its object to automatically determine timing for
recording an image or images to be used for panoramic image synthesis.
It is another object of the present invention to provide an image
synthesizing apparatus, an image sensing apparatus, and an image
synthesizing system capable of automatically determining timing for
recording an image or images to be used for panoramic image synthesis by
reduced images generated from sensed images, without requiring an
intervention of a user.
It is still another object of the present invention to provide an image
synthesizing apparatus which receives reduced images, used for determining
timing for recording an image or images, from an image sensing apparatus.
It is still another object of the present invention to provide an image
synthesizing apparatus which generates reduced images, used for
determining timing for recording an image or images, by reducing an image
or images transmitted from an image sensing apparatus.
According to an aspect of the present invention, a user interface is
improved by providing display means for displaying reduced images.
According to another aspect of the present invention, the position
relationship between the first and second images to be synthesized is
recognized by detecting the first overlapping portion between the first
and second reduced images. This is for increasing processing speed for
determining image recording timing.
According to another aspect of the present invention, the second
overlapping portion between the first and second images is detected from
the position relationship between the first and second reduced images,
thereby increasing the processing speed for synthesizing the images.
According to another aspect of the present invention, determination means
for determining whether the size of the first overlapping portion is
appropriate or not is provided, thereby improving precision for
determining image recording timing.
According to another aspect of the present invention, determination means
for determining whether the size of the second overlapping portion is
appropriate or not is provided, thereby improving the synthesis precision.
According to another aspect of the present invention, notification means
for notifying a user of a determination result by the determination means
is provided, thereby improving the user interface.
According to another aspect of the present invention, the second
overlapping portion is roughly obtained by magnifying the first
overlapping portion, edge pixels are extracted in the roughly obtained
second overlapping portion as corresponding points, a mapping relationship
between the first and second images is obtained on the basis of a group of
extracted corresponding points, and the first and second images are
combined by converting either the first or second image on the basis of
the obtained mapping relationship.
According to another aspect of the present invention, position relationship
between the first and second reduced images is detected by comparing pixel
values of the first predetermined number of pixels which are set in the
first reduced image to pixel values of the second predetermined number of
pixels which are set in the second reduced image.
According to another aspect of the present invention, an amount an
overlapping portion between the first and second reduced images is
calculated on the basis of the position relationship between the first and
second reduced images, and whether the ratio of the amount the overlapping
portion to the area of the first or second reduced image is less than a
first threshold or not is determined, and, when the ratio is less than the
first threshold, this information is notified to the user as a timing
suitable for recording an image to be used for image synthesis.
According to another aspect of the present invention, when the amount of
the overlapping portion becomes less than a second threshold which is
smaller than the first threshold, t is information is notified to the user
in that it is not a timing suitable for recording an image to be used for
image synthesis, thereby preventing inappropriate image recording
operation.
According to another aspect of the present invention, the image sensing
apparatus and the image synthesizing apparatus are connected with a
general-purpose interface capable of transmitting images at high speed.
Other features and advantages of the present invention will be apparent
from the following description taken in conjunction with the accompanying
drawings, in which like reference characters designate the same or similar
parts throughout the figures thereof. BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are incorporated in and constitute a part
of the specification, illustrate embodiments of the invention and,
together with the description, serve to explain the principles of the
invention.
FIG. 1 is a block diagram illustrating a configuration of an electronic
camera according to an embodiment of the present invention;
FIG. 2 is a flowchart showing an operation for sensing an image by the
electronic camera shown in FIG.
FIG. 3 is an overall view of an image synthesizing system according to the
embodiment of the present invention;
FIG. 4 is a view showing a configuration of the image synthesizing system
including software and hardware according to the embodiment of the present
invention;
FIGS. 5A and 5B are views showing transition of displayed image as a user
interface according to the embodiment;
FIG. 6 is a flowchart showing a flow of overall control of the image
synthesizing system;
FIG. 7 is a flowchart showing a processing sequence for tracking a relative
position between images in response to the movement of a camera;
FIG. 8A is an explanatory view for explaining the processing for tracking
the relative position between images in response to the movement of the
camera;
FIGS. 8B and 8C are explanatory views for explaining template matching
method for recognizing an overlapping portion between confirmation images
according to the embodiment;
FIG. 9 is a flowchart showing a control sequence for calculating a relative
position;
FIG. 10 is a flowchart showing a processing for checking whether an image
sensing point is appropriate for recording the second image or not;
FIG. 11 is an explanatory view for explaining an appropriate position;
FIG. 12 is a flowchart showing an overall sequence of panoramic image
synthesizing processing;
FIG. 13 is a flowchart showing coordinate transformation of the relative
position;
FIG. 14A is a flowchart showing corresponding point extraction processing;
FIG. 14B is an explanatory view for explaining template matching processing
for detecting a corresponding point between recorded images;
FIG. 15 is a view showing the template image and a matching area in the
corresponding point extraction processing;
FIG. 16 is a view showing a synthesizing process; and
FIG. 17 is a flowchart showing the synthesizing process.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
An image synthesizing system adopting the present invention will be
described in accordance with the accompanying drawings.
The image synthesizing system requires image sensing means and image
synthesizing means. These means do not have to be integrated, and may be
arranged separately to configure the system as a whole. In the system
according to the following embodiment, an electronic camera as the image
sensing means and a personal computer system (so-called PC system) as the
image synthesizing means are separately arranged, and they are connected
via SCSI interface.
<Outline>
In the image synthesizing system, the electronic camera reduces an image
which is currently sensed at real time and transmits it to the PC system,
thereby the reduced image serves for checking an image to be recorded.
Synthesis of images into a panoramic image is automatically performed by
the PC system after taking images from the electronic camera.
First, the PC system requests a confirmation image to the electronic
camera. In response, the electronic camera senses an object, reduces an
obtained image to generate the confirmation image, then sends it to the PC
system. The user presses a shutter button (icon) on a display screen of
the PC system while watching the confirmation image. In response to this
operation, the electronic camera sends the sensed image, i.e., an image
which is an original image of the confirmation image that the user has
just confirmed (first image =left image), to the PC system. Next, the user
instructs the electronic camera to shift the image sensing point via the
PC system in order to obtain a second image having an overlapping portion
with the first image (i.e., the second image is to be synthesized with the
first image). The PC system detects the direction and the amount of
movement of the camera in accordance with the designation, and tracks the
relative position of the second confirmation image with respect to the
first confirmation image. While moving the image sensing point, the
electronic camera keeps transmitting confirmation images to the PC system.
Then, when the user presses the shutter button icon in the PC system, the
electronic camera transmits the currently sensing image (second image) to
the PC system. The PC system stores the relative position of the second
image with respect to the first image at this point. When the user presses
a "synthesis button" icon, the PC system starts panoramic image
synthesizing processing. The PC system first finds overlapping portion
between the first and second images on the basis of information on the
stored relative position, then synthesizes the images.
Thus, in the image synthesizing system,
I: By displaying the relative position of the second image with respect to
the first image by continuously displaying the reduced confirmation images
until the "shutter button" icon, as a designation for recording the second
image is pressed, information about how much and to which direction the
camera has been moved is provided, further, a suggestion to how much and
to which direction the camera is to be moved is given to the user.
II: Furthermore, the system has a function to notify a user of when the
electronic camera has reached an image sensing point suitable for
recording an image for panoramic image synthesis by a sound and/or by an
image on the display. In addition, the system has a function to notify the
user of when the overlapping portion between the first image and an image
to be the second image is too narrow to synthesize the images and when the
two images are not overlapping by a sound and/or an image on the display.
<Configuration of the Electronic Camera>
FIG. 1 is a block diagram illustrating an overall configuration of an
electronic camera 1000 used in the image synthesizing system. To the
electronic camera 1000, a PC system 2000 is connected via a SCSI interface
232.
It should be noted that, regarding the camera used in this image
synthesizing system, if the image synthesizing device in the image
synthesizing system is separately configured from the camera, an ordinary
electronic camera may be used if the camera has capability to communicate
with a computer.
In FIG. 1, reference numeral 200 denotes a lens unit; 202, an iris
diaphragm; 204, an iris diaphragm driver for opening and closing the iris
diaphragm 202; 206, an image sensing device; 208, an analog-digital (A/D)
converter for converting an output from the image sensing device 206 to a
digital signal; 210, a digital signal processor (DSP) which performs
signal processes, such as photometry, colorimetry, and compression, by
using a buffer memory 212.
Further, reference numeral 214 denotes a memory device having a large
capacity for recording sensed images, and can be a magnetic disk drive, an
optical disk drive, a magneto-optical drive, a solid-state memory device
having EEPROM or DRAM whose battery is backed up, for example. As for the
solid-state memory device, there is a so-called memory card. In this
embodiment, a memory card is implemented as the memory device 214.
Reference numeral 216 denotes an interface between the output terminal of
the DSP 210 and the memory device 214.
Reference numeral 218 denotes a synchronizing signal generator (SSG) for
providing clock and synchronizing signals used in the image sensing device
206, the A/D converter 208 and DSP 210; 220, a system controller for
controlling the overall system; 222, a battery as a power source; 224, an
on/off switch; 226, a power supply controller for controlling electric
power supply to each of the circuits 204 to 218.
Reference numeral 228 denotes a shutter switch, which can be held in an
intermediate position, which closes in accordance with press-down
operation of a shutter release button. The system controller 220 executes
image sensing operation (e.g., exposing the image sensing device 206 and
reading signals) in response to the switch 228 being closed.
Reference numeral 232 denotes an external interface (I/F), such as a
bidirectional parallel interface or SCSI interface, which is connected to
a PC system 2000.
FIG. 2 is a flowchart showing a sequence of control operation executed by
the system controller 220 of the camera 1000 when sensing an image.
When the system controller 220 detects that the shutter switch 228 is
closed, it controls the iris diaphragm 202 via the iris diaphragm driver
204 (step S301), thereby exposing the image sensing device 206 (step
S302). Next, a buffer memory 212 is supplied with electric power (step
S303). Then, an electric charge signal is read from the image sensing
device 206 and converted into a digital signal by the A/D converter 208,
then written to the buffer memory 212 via the DSP 210 (step S304). In this
step, the system controller 220 controls the DSP 210 to perform a .gamma.
correction, a knee correction, a white balance adjustment, and further a
compression in a predetermined compression method onto the signal data,
and they write the processed digital data to the buffer memory 212. Next,
the memory device 214 and the interface 216 are supplied with electric
power, and the data stored in the buffer memory 212 is transferred to the
memory device 214 via the DSP 210 and the interface 216 (step S305). Note,
the data may be compressed before being transferred.
As described above, while the shutter switch 228 is held closed, the system
controller 220 is able to transfer a sensed image (non-reduced image) as
well as a reduced confirmation image to the PC system 2000 via the
interface 232. The PC system 2000 sends a confirmation image request
command when requesting the confirmation image and an image request
command when requesting a non-reduced image to the camera 1000 via the
interface 232.
<PC System>
FIG. 3 is an overall view of an image synthesizing system including the PC
system 2000 which is a platform on which the present invention can be
executed.
In FIG. 3, reference numeral 301 denotes a main body of the computer
system; 302, a display for displaying data; 303, a mouse which is a
typical pointing device; 304, a mouse button; and 305, a keyboard.
The above described electronic camera which is supported by a tripod 308 is
connected to the PC system 2000 via a general-purpose interface 232, such
as a bidirectional parallel interface or a SCSI interface, capable of
transmitting an image at high speed.
FIG. 4 is a view showing a configuration of the panoramic image
synthesizing system including software and hardware implemented by the PC
system 2000.
In FIG. 4, reference numeral 509 denotes a hardware; 505, an operating
system (OS) executed on the hardware 509; and 504, an application program
executed on the OS 505. Note, in the blocks configuring the hardware 509
and the OS 505, blocks not necessary for explaining the embodiment of the
present invention (e.g., CPU and memory in the hardware, and a memory
administration system in the OS) are not shown in FIG. 4.
Reference numeral 515 denotes a hard disk for storing files and data; 508,
a file system, configuring the OS, which has a function to make an
application software be able to input and output a file without being
conscious of the hardware; 514, a disk input/output (I/O) interface used
so that the file system 508 reads and writes from/to the hard disk 515;
and 507, a rendering administration system, configuring the OS, which has
a function to make an application software be able to render without being
conscious of the hardware.
Further, reference numeral 513 denotes a video interface for the rendering
administration system 507 to render on the display 302; 506, an input
device administration system, configuring the OS, which has a function to
make an application software be able to receive an input by a user without
being conscious of the hardware; 510, a keyboard interface for the input
device administration system 506 to receive an input from the keyboard
305; and 512, a mouse interface for the input device administration system
506 to receive an input from the mouse 303.
Reference numeral 516 denotes an interface, such as a bidirectional
interface or SCSI interface, which is connected to the electronic camera
1000 for exchanging image data, and the like, via the input device
administration system 506; and 501, an image data administration program.
Further, reference numeral 502 denotes a program to request a confirmation
image to the camera 1000 and control the receipt of a sensed image, i.e.,
a camera driver program; and 503, a data display driver program for
generating a confirmation image (i.e., data-manipulation operation for
reducing sensed images), displaying the confirmation image, and displaying
image data which is already taken and under administration.
It should be noted that a digital signal processor (DSP) of the camera 1000
has a function for reducing an image. In this embodiment, reduction of an
image for obtaining a confirmation image is performed by the DSP of the
camera 1000. The camera driver 502 may have a role to reduce an image.
Reference numeral 517 denotes a panoramic image synthesizing program which
receives sensed images from the image data administration program 501,
performs panoramic image synthesizing processing, and register a resultant
synthesized image to the image data administration program 501. The detail
of the panoramic image synthesizing program will be described with
reference to FIG. 6.
<User Interface>
FIGS. 5A and 5B show graphic user interface implemented by the PC system
2000. More specifically, FIGS. 5A and 5B show transition of an image
displayed in user interface windows 2200 and 2307 which are displayed on
the display 302.
In FIGS. 5A and 5B, reference numeral 2301 denotes a base image (referred
by "first image", hereinafter) with which another image is synthesized,
and reference numeral 2306 denotes an image which is combined to the first
image 2301 (referred by "second image", hereinafter). Further, reference
numeral 2302 is an image to confirm the first image 2301 (referred by
"first confirmation image", hereinafter), and 2303 is an image to confirm
the second image 2306 (referred by "second confirmation image",
hereinafter). The first confirmation image 2302 and the second
confirmation image 2303 are generated from the first image 2301 and the
second image 2306, respectively.
As described above, the user interface of the image synthesizing system has
the window 2307 for displaying the contents of the sensed images (the
first and second images) and the window 2200 for displaying the
confirmation images to show the relative position between the sensed
images.
Note, in the following explanation, when synthesizing images, by using the
left and right images, the first image is occasionally referred to as
"left image" and the second image is occasionally referred to as "right
image" for convenience.
Referring to FIGS. 5A and 5B, an operation sequence of the user interface
for image synthesis is briefly explained.
First, the user presses a shutter button icon 2304 on the display of the
computer while watching the first confirmation image 2302 displayed in the
window 2200. In response to this operation, the first image 2301 is
"recorded" (image data is registered in the file system 508 in practice).
The "recorded" first image 2301 is displayed in the area, as shown in FIG.
5A, in the window 2307. Next, the user shifts the image sensing point of
the electronic camera 1000 by moving the tripod 308 for obtaining the
second image 2306 which overlap the first image 2301. While shifting the
image sensing point, the direction and the amount of shift of the second
confirmation image 2303 which moves as the image sensing point shifts are
calculated by executing the panoramic image synthesizing program 517,
thereby tracking the relative position of the second image 2306 with
respect to the first image 2301. Meanwhile, the second confirmation image
2303 is displayed in the window 2200 at a relative position with respect
to the displayed position of the first confirmation image 2302. In this
manner, the user is informed how much and to which direction the camera
has to be moved. Then, when the user presses the shutter button icon 2304,
the second image 2306 is "recorded" (i.e., the second image is registered
in the file system 508). The "recorded" image 2306 is displayed in the
window 2307 as shown in FIG. 5B.
Next, in response to a press-down operation of a synthesis button icon 2305
by the user, the panoramic image synthesizing processing starts. More
specifically, the relative position between the first image 2301 and the
second image 2307 is obtained on the basis of the information on the
relative position and the images are synthesized by executing the
panoramic image synthesizing program 517. Then, the resultant synthesized
panoramic image is displayed in the window 2307.
Note, there is provided with a function for recognizing that the electronic
camera 1000 detects an image sensing point suitable for recording an image
to be synthesized into a panoramic image while moving the camera 1000 (the
state shown in FIG. 5B) and notifying the user of that by sound or by
using other method, e.g., displaying frame lines of the confirmation
images 2302 and 2303 brightly. As for the detail of this function, refer
to step S2803 in FIG. 10.
<Overall Control Sequence>
A flowchart in FIG. 6 is for explaining an overall sequence of the
panoramic image synthesizing program 517.
At step S2401, whether or not the synthesis button (icon) 2305 is pressed
is checked. When the icon 2305 is pressed, then it is confirmed that a
plurality of images to be synthesized have already been recorded, then the
panoramic image synthesizing processing is performed at step S2415. The
synthesis button (icon) 2305 will be pressed by the user when more than
two images to be synthesized have been recorded.
Whereas, when the synthesis button (icon) 2305 is not pressed or when a
plurality of images to be synthesized have not been recorded even if the
synthesis button (icon) 2305 is pressed, then whether the shutter button
2304 is pressed or not is checked at step S2402. If No at step S2304, then
a sequence fo | | |
