 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
The invention disclosed broadly relates to the field of image processing
and more specifically to image processing in a digital camera for taking
panoramic pictures.
2. Description of the Related Art
Today, panoramic photography is accomplished in various ways. One is to use
a still camera mounted on a tripod to take a succession of shots as the
camera is pivoted around the tripod. In some cameras, a wider than usual
strip of film is exposed with special movable optics.
In other cameras, conventional format film, such as 35 mm film, is masked
during the exposure in the camera to provide a panoramic effect. The
effect is panoramic but the whole exposure is limited by the field of view
through the lens.
Other techniques for creating panoramic photography include to physically
cut and paste together strips of exposed film by carefully aligning
boundaries between edges of film.
The benefits of electronic photography have led to the growth of digital
cameras, that, unlike their film-based counterparts, store images captured
in memory into digital memory such as flash memory. To provide panoramic
photography effects, these digital cameras can interface with personal
computers for joining together two or more images into one image to
provide a panoramic effect by joining edge boundaries of images. One such
system is disclosed in U.S. Pat. No. 6,682,197, by named inventors Omid A.
Moughadam, Stuart R. Ring, and John R. Squilla, entitled "Electronic
Panoramic Camera For Use With An External Processor."
Complicated panoramic digital cameras are available that rely on position
sensors or satellite communications for determining position coordinates.
These position coordinates are used to help combine the panoramic images.
The process of combining scenes taken from different camera orientations
is known as "Image Stitching." One such system is disclosed in U.S. Pat.
No. 5,262,867 by named inventor Kiyonobu Kojima entitled "Electronic
Camera and Device for Panoramic Imaging and Object Searching" issued on
Nov. 16, 1993.
A panoramic camera with a memory device for storing data from a previously
photographed portion of an object and a control device for enabling the
display device to substantially display both the image to be photographed
and the image already photographed and stored in the memory space is
described in U.S. Pat. No. 5,138,460 by named inventors Egawa and Akira
entitled "Apparatus for forming Composite Images" issued on Aug. 11, 1992.
Although these techniques are useful, they are not without their
shortcomings. One shortcoming is the expense and complication arising from
the need for orientation and position sensors for determining picture
orientation. Accordingly, a need exists for a device and method to create
panoramic pictures without the need and expense of position sensors.
Another shortcoming of the current panoramic cameras is how difficult it is
to properly overlap a region between two adjacent frames of a panoramic
image. Too much overlap results in wasting memory in the digital camera
with redundant information.
Another shortcoming with current panoramic cameras is their inability to
guide the user where the correct edge overlap is required for creating a
panoramic picture alignment from two or more images. Accordingly, a need
exists to overcome this problem and to guide the user of a camera using a
method and apparatus to overlap two or more images to provide a correct
panoramic picture.
Another shortcoming of the current panoramic cameras is the requirement to
overlap regions of two or more images. It is common with panoramic image
generation to stitch together two or more images. Problems at the
boundaries of two images include color distortion, perspective distortion,
pixel adjacency and skipped pixels at the edges when joining two or more
images. Accordingly, a need exists for a method and apparatus to overcome
these shortcomings.
Another shortcoming of the current panoramic cameras is their inability to
integrate these mechanisms with other products. The bulk, expense, and
complication make these designs difficult to integrate. Accordingly, a
need exists for a method and apparatus to allow easy integration of
panoramic capabilities into other electronic devices.
Another shortcoming is the inability to easily capture two or more images
to create a panoramic scene without the need for expensive computational
resources in the camera and without the need for expensive position
sensors. Accordingly, a need exists for a method and apparatus to provide
a camera which can capture two or more images so as to create a panoramic
image.
Another shortcoming is the requirement of having two distinct buffers to
transform the captured image from one coordinate system to another
coordinate system, such as rectilinear coordinates to cylindrical
coordinates. The expense and space required for two buffers for coordinate
transformations is undesirable. Accordingly, a need exists to provide a
method and apparatus that can transform images from one coordinate system
to another coordinate system, without the use of two buffers.
Another shortcoming with current panoramic cameras is the perspective of
the series of images relative to each other is lost. One known procedure
uses sub-sampling. The picture is scaled down so that the horizontal
dimension will match the horizontal extent of the view-port. However, the
picture must become a narrow stripe in order to maintain the correct
aspect ratio. This will produce a great loss of vertical resolution.
Another solution is to scroll the panoramic image horizontally by a fixed
amount. In this case no sub-sampling is used, provided that the vertical
dimension of the picture will fit the vertical dimension of the view-port.
Accordingly, a need exists to provide a method and apparatus to create a
moving display of still pictures with perspective without the need to
sub-sample or to horizontal scroll.
Still another shortcoming with current panoramic cameras is the requirement
that all the image processing electronic circuitry for the final stitching
together of one or more images in a series into a single panoramic image
is integrated into the camera. Many times, the added expense, the
additional weight and the additional size of the electronic circuitry make
the camera unwieldy to carry and operate. Moreover, the additional
electronic circuitry makes the camera more expensive to manufacture.
Accordingly, a need exists for a method and apparatus to enable the
preview of a series of adjacent images to form a panoramic, but to enable
the final image stitching processing on a remote device such as a personal
computer.
SUMMARY OF THE INVENTION
A method for correcting color in a system for creating a panoramic image
from a plurality of images taken by a camera. The method comprising the
steps of: receiving a color channel from at least a first image and a
second image; creating an overlap portion between the first image and
second image; and adjusting the color channel for the first image and for
the second image in at least the overlap portion between the first image
and the second image which is independent of motion estimation.
In an alternate embodiment, a system and computer readable medium
corresponding to the above method is described.
BRIEF DESCRIPTION OF THE DRAWINGS
The subject matter which is regarded as the invention is particularly
pointed out and distinctly claimed in the claims at the conclusion of the
specification. The foregoing and other objects, features, and advantages
of the invention will be apparent from the following detailed description
taken in conjunction with the accompanying drawings.
FIG. 1 is a block diagram of a digital still camera according to one aspect
of the present invention.
FIG. 2 is a block diagram of the picture stitching device of FIG. 1 in
another embodiment integrated as a standard cell in a semi-custom
semiconductor device, according to the present invention.
FIG. 3 is a block diagram of a series of individual horizontal still images
comprising a panoramic image with overlapping regions, according to the
present invention.
FIG. 4 is a flow diagram of the overall image acquisition process of the
horizontal images of FIG. 3 using the digital still camera of FIG. 1,
according to the present invention.
FIG. 5 is an illustration of a Previous Picture for demonstrating the
composite overlay and perspective correction of the preview strip the
overall image acquisition process of FIG. 4, according to the present
invention.
FIG. 6 is an illustration of a selected overlap region of FIG. 5, according
to the present invention.
FIG. 7 is an illustration of the corrected perspective of overlap region of
FIG. 6 for aligning the Ideal Next Picture, according to the invention.
FIG. 8 is an illustration of the Current Picture with an Ideal Next Picture
guided by the overlap generated in FIG. 7, according to the present
invention.
FIG. 9 is a high level block diagram of a picture stitching device of FIG.
1, according to the present invention.
FIG. 10 is a detailed flow diagram of the stitching overview of FIG. 4,
according to the present invention.
FIG. 11 is an illustration of alpha blending mixing between the preview
strip and the Current View, according to the present invention.
FIG. 12 is an illustration of the results of alpha blending of FIG. 11 with
a constant ratio, according to the present invention.
FIG. 13 is an illustration of the results of alpha blending of FIG. 11 with
a progressive ratio according to the present invention.
FIG. 14 is an illustration of the results of interlace blending of FIG. 11
using blocks, according to the present invention.
FIG. 15 is an illustration of the results of interlace blending of FIG. 11
using lines, according to the present invention.
FIG. 16 is an illustration of the results of alpha blending with interlace
blending of blocks of FIG. 15, with the vertical dimension equal to the
vertical dimension of the picture according to the present invention.
FIG. 17 is an illustration of the results of alpha blending with interlace
blending of only some of the blocks of FIG. 15, according to the present
invention.
FIG. 18 is an illustration of the areas of measurement for two adjacent
figures, such as images A and images B of FIG. 3, according to the present
invention.
FIG. 19 is an illustration of the individual frame buffers laid out on a
panorama, according to the present invention.
FIG. 20 is an illustration of the panorama of FIG. 19 saved as fixed-sized
strips, according to the present invention.
FIG. 21 is a detailed block diagram of the picture stitching device of FIG.
9, according to the present invention.
FIG. 22 is a block diagram of the Alignment Preview architecture of FIG.
21, according to the present invention.
FIG. 23 is an illustration of the capturing of two rectilinear image planes
and the relationship to a cylindrical circle of a fixed focal point f,
according to the invention.
FIG. 24 is an illustration of pixels in the overlap region of the current
picture of FIG. 23 being projected back to the Previous Picture of FIG. 23
in the overlap region, according to the present invention.
FIG. 25 is an illustration of pixels of the selected overlap region of
source picture of FIG. 7 that cannot be perspectively corrected, according
to the invention.
FIG. 26 is a block diagram of an alternate embodiment of FIG. 9 with two
optional blocks used to improve the perspective correction algorithm
performance, according to the present invention.
FIG. 27 is an example of a limit area copy from a Previous Buffer to a
Current Buffer of FIG. 26 during a first pass, according to the present
invention.
FIG. 28 is an example of a limit area copy from a Previous Buffer to a
Current Buffer of FIG. 26 during a second pass, according to the present
invention.
FIG. 29 is an example of a limit area copy from a Previous Buffer to a
Current Buffer of FIG. 26 during a third pass, according to the present
invention.
FIG. 30 is an example of a limit area copy from a Previous Buffer to a
Current Buffer of FIG. 26 during a fourth pass, according to the present
invention.
FIG. 31 is an example of areas of Current Buffer for which there is no
information, according to the present invention.
FIG. 32 is an illustration of rectilinear and cylindrical images, according
to the present invention.
FIG. 33 is a schematic showing the two phases of the in-place rectilinear
to cylindrical transformation process, according to the present invention.
FIG. 34 is a block diagram of the Displacement Estimation architecture of
FIG. 21 according to the present invention.
FIG. 35 is an illustration of vertical and horizontal downsampling of an
image, according to the present invention.
FIG. 36 is an illustration of the sum-of-absolute differences (SAD) between
a pair of images according to the present invention.
FIG. 37 is an illustration for determining the optimum displacement
corresponding to the minimum local minima in the profile, according to the
present invention.
FIG. 38 is a block diagram of the Color Correction architecture of FIG. 21
according to the present invention.
FIG. 39 is a schematic illustrating the color correction applied at various
regions of the Previous Buffer and Current Buffer, according to the
present invention.
FIG. 40 is a block diagram of the Stitching and Blending architecture of
FIG. 21, according to the present invention.
FIG. 41 is a schematic illustrating the regions of the Previous and Current
Buffer, during image compositing, according to the present invention.
FIG. 42 is a schematic illustrating the regions of the Previous and Current
Buffer being composited over a narrower region, according to the present
invention.
FIG. 43 is a schematic illustrating the regions of the Previous and Current
Buffer contributing to the final panorama, according to the present
invention.
FIG. 44 is a schematic illustrating the Current View within the Panorama,
according to the present invention.
FIG. 45 is a high level block diagram of an architecture for a system using
a motion playback device according to an aspect of the invention.
FIG. 46 is a diagram illustrating a motion picture simulation principle,
according to the invention.
FIG. 47 illustrates cylindrical and planar picture representations of a
panoramic picture, according to the present invention.
FIG. 48 is a detailed block diagram of the motion playback device internal
architecture of FIG. 45, according to the present invention.
FIG. 49 shows an X mapping block architecture of FIG. 49, according to the
present invention.
FIG. 50 shows a Y mapping block architecture of FIG. 49, according to the
present invention.
FIG. 51 is a block diagram of the picture processor of FIG. 49, according
to the present invention.
FIG. 52 shows a vertical subsampling architecture, of the picture processor
of FIG. 50, according to the present invention.
FIG. 53 shows an embodiment of a cubic interpolation system of FIG. 51,
according to the present invention.
FIG. 54 shows an embodiment of a linear interpolation system of FIG. 51,
according to the present invention.
DETAILED DESCRIPTION OF AN EMBODIMENT
However, it should be understood that these embodiments are only examples
of the many advantageous uses of the innovative teachings herein. In
general, statements made in the specification of the present application
do not necessarily limit any of the various claimed inventions. Moreover,
some statements may apply to some inventive features but not to others. In
general, unless otherwise indicated, singular elements may be in the
plural and visa versa with no loss of generality.
1. GLOSSARY OF TERMS USED IN THIS DISCLOSURE
Actual Next Picture--the picture that is actually captured as the next
addition to the set of pictures constituting the Panorama.
Bottom-to-Top Vertical Panorama--a Panorama captured by taking a set of
pictures by rotating the camera up (from bottom to top) between each
capture, with as little horizontal displacement as possible.
Current Buffer--a cylindrically warped version of the current picture.
Current Picture/Current View/Current Frame--a picture displayed on the
camera LCD screen and that is updated in real time. If the LCD screen is
not used or if there is no LCD screen, it is the picture that would be
captured at any given moment if the capture button was pressed.
Ideal Next Picture--in a Left-to-Right Horizontal Panorama, the picture
that would be obtained if the camera was positioned so that the Previous
Picture and Ideal Next Picture have an Overlay Part Length equal to the
Set Overlay Part Length and no vertical displacement. In a Left-to-Right
Horizontal Panorama, the overlay part is on the right of the Previous
Picture and on the left of the Ideal Next Picture.
Ideal Position of the Camera for the Ideal Next Picture--The position of
the camera that allows to capture the Ideal Next Picture.
Image Stitching--the process of digitally combining scenes taken from
different camera orientations.
Left-to-Right Horizontal Panorama--a Panorama captured by taking a set of
pictures by rotating the camera clockwise (from left to right) between
each capture, with as little vertical displacement as possible.
Overlay Part Length--in a Horizontal Panorama, the width of the overlay
part. It is expressed in term of percentage of the whole picture width.
Overlay Part of A Picture--the part of the picture picturing the overlay
zone.
Overlay Zone of Two Pictures--the part of a scene that is present in the
two pictures.
Panoramic--an image with at least one dimension such as height or width
which is greater in dimension of a single capturing device and often
involves a series of images. A picture created from a set of pictures and
that has at least one dimension bigger than the corresponding dimensions
of a source picture.
Preview Strip--a computed image created through digital processing of the
overlay part of the Previous Picture and that strives to predict what the
overlay part of the Ideal Next Picture will look like.
Previous Buffer--a cylindrically warped version of the Previous Picture.
Previous Picture/Previous Frame--a picture that has already been captured
and that is the latest addition to the set of picture constituting the
Panorama.
Right-to-Left Horizontal Panorama--a Panorama captured by taking a set of
pictures by rotating the camera anticlockwise (from right to left) between
each capture, with as little vertical displacement as possible.
Set Overlay Part Length--a constant Overlay Part Length for each pair of
pictures constituting the Panorama. The Set Overlay Part Length is the
fixed length chosen for a given Panorama.
Top-to-Bottom Vertical Panorama--a Panorama captured by taking a set of
pictures by rotating the camera down (from top to bottom) between each
capture, with as little horizontal displacement as possible.
2. PICTURE STITCHING CAMERA
Referring now in more detail to the drawings in which like numerals refer
to like parts throughout several views, shown in FIG. 1 is a block diagram
of a digital still camera 100 according to one aspect of the present
invention. A digital camera comprises one or more optical lenses 102 with
an autofocus/shutter, driver and actuator 104 and associated photometry
interface 108 such as autofocus, auto shutter and contrast control. An
imager 106 such as an CCD or equivalent 104 converts an image projected
through optical lens 102 to a series of pixels 106. Regulators and image
drivers 108 allow regulation of the imager 106. An audio acquisition
device 212, such as microphone, along with audio processing circuitry 214,
allows a user to make aural recordings along with digital images. A
battery 118 with power management circuitry 118 allows the camera 100 to
work as a portable device. A picture processor 116, provides pixel
information to one or more frame buffers 118 coupled to picture stitching
device 120 which is described further below. In this embodiment, the
picture stitching device 120 is implemented as an ASIC. A LCD display 122
or equivalent enables a user to view the image projected through lens 102
into imager 106 and controlled by LCD controller 134. A PAL/NTSC 124
encoder provides an interface to other display types. An image CODEC 132
coupled to picture processor 116 provides known image enhancement effects
for the picture processor 116. A DSP 124 such a STMicroelectronics
ST-20/SH3-DSP is used to control the memory interface and the data I/O 126
such at Infra-Red, Universal Serial Bus or other interfaces. A DRAM
provides execution memory for the DSP 124 to perform Image Stitching
algorithms as described below. An audio output 130 such as a speaker
provides the user aurally playback. All of these components are
representative components of the digital camera 100. Storage media 136
such as Flash memory, diskettes or removable hard drives store each image
and associated audio. In one embodiment, the algorithms carrying out the
steps for Image Stitching described below are stored on the storage media
136 along with captured images and audio. Processing for the images may
occur prior to or after the image is store in storage media 136. The
general operation of a digital camera comprising most elements described
herein is well understood and those skilled in the art.
One or more user inputs via the LCD Controller 128 provides user control
over camera functions such as the orientation of the panoramic e.g.,
horizontal or vertical, and the direction of movement such as a
Left-to-Right Horizontal Panoramic, a Right-to-Left Horizontal Panoramic;
a Top-to-Bottom Vertical Panoramic; and a Bottom-to-Top Vertical
Panoramic. Other user input such as the optional features and desired
effects and to set system parameters such as:
Panorama mode on/off.
Panorama parameters.
Left-to-Right Horizontal mode.
Right-to-Left Horizontal mode.
Top-to-Bottom Vertical mode.
Bottom-to-Top Vertical mode.
Set Overlay Part Length.
Preview display on/off.
Mixing mode parameters.
Alpha blending on/off.
Alpha blending parameters such as alpha blending ratio.
Interlaced block mode on/off.
Interlaced block pattern selection.
In another embodiment, many of the components of FIG. 1 are embedded in as
part of a standard cell in a semi-custom semiconductor device. FIG. 3 is
an block diagram 300 of the picture stitching device of FIG. 2 in another
embodiment integrated as a standard cell in a semi-custom semiconductor
device, according to the present invention. In this embodiment, the
picture stitching device 124 and picture processor 120 are physically
implemented as one unit. The DSP 134, serial I/O 136, image CODEC 132, LCD
preview and display 126, PAL/NTSC encoder 130 and LCD controller & drivers
126 are all integrate as one device 200. To those skilled in the art, it
will be obvious to substitute and modify this exemplary single device 300
to include the picture stitching device 124 and DSP 224 with other
circuitry as well.
3. PANORAMIC CAPTURING OVERVIEW
Turning now to FIG. 3 shown is a series of four images 300 labeled A, B, C,
and D with overlap edges parts ab, bc, cd between. These four images are
joined together to form one Panoramic image. It is important to note that
the Panoramic images can be from Left-to-Right as shown in FIG. 3 or
Right-to-Left for a Horizontal Panoramic. For a Vertical Panoramic, (not
shown) the series of images can run from top- to-bottom or from
bottom-to-top. Accordingly, this present invention is not limited to one
type of horizontal or vertical orientation, and four types of panoramic
images are disclosed: a Left-To-Right Horizontal Panoramic; a
Right-to-Left Horizontal Panoramic; a Top-to Bottom Vertical Panoramic;
and a Bottom-to-Top Vertical Panoramic.
FIG. 4 is a flow diagram 400 of the overall image acquisition using the
digital still camera of FIG. 1, according to the present invention. After
this overall process is described, a more detailed description of the
process steps of FIG. 4 are described below. The digital camera 100
acquires a series of images to form a panoramic image, such as the series
of Left-To-Right Horizontal Panoramic A, B, C, D with overlap edge parts
ab, bc, cd of FIG. 3. The overall image acquisition process begins with
the initialization of the digital camera 100. The user inputs from the LCD
controller setting up the orientation of the panoramic, such as horizontal
or vertical and other system parameters for picture stitching device 124
are set, step 402. The next image is located in the view finder of digital
camera 100. Returning to the example in FIG. 3, if the image is A, it is
the first image, if the image is B,C, or D it is the next image and the
regions of overlap ab, bd, and cd for each image pair is made as a
composite. Once the image either the first frame or the successive frames
with the desired overlap are aligned in the view finder of digital camera
100, the so called Ideal Next Picture, is created. The region of overlap,
called the Preview Strip, is a composite of both the Previous Picture and
the Current Picture. The Preview Strip comprises a perspectively corrected
Previous Picture in the region overlap. The Preview Strip assists the user
in aligning the Current Picture and the Previous Picture. The Current View
is stitched together with the Previous View, in the stitch frame step,
408. A test is made to see if the last image had been stitched. Referring
again to FIG. 3 that is whether image D has been stitched with image C
using overlap region cd. If the last image is not been stitched then the
process returns to step 404 until the last image has been stitched, step
410. Once the last image is stitched final corrections to the overall
image and overall data is made, step 412. After the first frame is
acquired, a perspectively-corrected strip of the Previous Frame is
overlaid on the Current Frame to aid the user in acquiring the Current
Frame. For example, if a Horizontal Panorama is being acquired from
left-to-right, a strip from the right portion of the Previous Frame is
perspectively corrected and overlaid on the left portion of the Current
Frame in the viewfinder. After each frame is acquired, the stitch module
is invoked with the new frame buffer. The stitch module aligns and blends
the new frame into the Panorama, and incrementally saves out the Panorama.
After the last frame is acquired and stitched, the Finalize Stitch module
is called to save out any remaining data.
Turning now to FIGS. 5-8 is an illustration of a series of Left-to-Right
Horizontal Panorama images illustrating the perspective correction of FIG.
4 as seen by the user of the digital camera 100, according to the present
invention. Note the capturing of the Current Picture with an Ideal Next
Picture guided by the overlap generated edge 602. In FIG. 5, an image is
captured in the digital camera 100, this becomes the Previous Picture 500.
A Preview Area 602 is shown from the Previous Picture 502 to user in FIG.
6. FIG. 7 illustrates how the Preview Area 602 is perspectively corrected
or warped 702 to assist the user in aligning the Current Picture 800 in
FIG. 8. Notice the perspectively corrected Preview Area 702 makes aligning
the Current Picture 800 much easier. As shown in FIG. 8, a Preview Strip
702 is generated from the overlay part 602 of the Previous Picture 500.
Note the similarity between the generated Preview Strip 702 and the
Current View 800 overlay part. The computed Preview strip 702 is used as a
visual clue, displayed on the LCD 126 of digital camera 100, to help the
user position the camera 100 until the Current View 800 matches the
Preview Strip 702.
The computed Preview Strip 702 can be displayed on the digital camera 100
in various way to help the user in precisely placing the camera in the
desired position. One way is to alpha blend (with adjustable coefficient)
the preview with the current display of the scene on the LCD 126. In
another embodiment, the interlace some blocks of the preview with the
displayed Current View on the digital camera 100. Further information on
blending is described below.
4. HIGH LEVEL PICTURE STITCHING DEVICE ARCHITECTURE
FIG. 9 is a high level block diagram 900 of a picture stitching device 124
of FIG. 1, according to the present invention. The Frame Buffers 122
comprise a Previous Buffer 902 (or prevBuf) and a Current Buffer 904 (or
curBuf). The Previous Buffer 902 and Current Buffer 904 are coupled to the
picture stitching device 124 through a set of address 906 and data lines
908. A clock input 912 provides timing inputs to the picture stitching
device 124. A picture input from the imager 106, provides the digital
image to the picture stitching device 124. An output labeled out picture
916, provides an output from the picture stitching device 124 to other
devices such as the Image CODEC 132 and DSP 134. An input labeled Control
settings 910, provides parameter input to picture stitching device 124.
Parameter input includes, the size of the Previous Buffer 902 and the
Current Buffer 904, the amount of overlap region between images, types of
color correction, and more. The Previous Buf | | |
