 |
Claims  |
|
|
We claim:
1. A computer system for authoring panoramas, comprising:
memory storing data representations of a plurality of images;
I/O for inputting and outputting data;
function modules in said memory;
a processor cooperating with said memory and I/O for processing
instructions and data from said memory, said I/O and said function
modules;
a pairwise registration function module in said memory and operating said
processor for pairwise registration of said images, said pairwise
registration function module generating output data related to said
pairwise registration of said images, the pairwise registration function
module including a projective matrix, wherein the projective matrix is a
function of image center position, said output data including a Hessian
matrix;
a calibration and global optimization function module in said memory and
operating said processor for calibration and global optimization of said
output data of said pairwise registration module, said calibration and
global optimization module generating output data related to said
calibration and global optimization of said images;
a blending function module in said memory and operating said processor for
generating at least one panorama from said output data from said pairwise
registration module and said calibration and global optimization module;
and
a projection function module in said memory and operating said processor
for forming a panorama from said images.
2. The invention according to claim 1, wherein:
said pairwise registration function module operates said processor to
register a plurality of each of said images that overlap with one another,
by perturbing said images.
3. The invention according to claim 2, wherein:
said pairwise registration function module performs a projective
transformation between a pair of said images, said projective
transformation is estimated by said pairwise registration function module
by performing a gradient-based optimization that minimizes a local error
function.
4. The invention according to claim 3, further comprising:
a user interface module in said memory, and cooperating with said I/O,
operating said processor for initializing said projective matrix of said
pairwise registration function module, said user interface module
operating to process input from a user of the computer system, wherein
said projective matrix is initialized with said camera internal and
external parameters from manual input by a user; and,
said pairwise registration function projective transformation error
function is comprised of the differences in exposures between pixels
comprising the pairs of images, and contains a projective matrix
initialized by camera internal and external parameters relating to said
plurality of images.
5. The invention according to claim 4, wherein:
said internal and external parameters are selected from the group
comprising: image center position of each of said pairs of images, aspect
ratio of said pixels comprising said images, the focal length of the
camera which took said images; the camera orientation with respect to a
common reference frame, camera pan, tilt, roll and skew and brightness and
contrast of said pairs of images.
6. The invention in accordance with claim 4, wherein said user interface
module comprises a virtual reality environment comprising a spherical
coffee table.
7. The invention according to claim 6, wherein:
said user interface module allows the user to select the arrangement of
said images from a plurality of rows or columns of photos.
8. The invention according to claim 3, wherein:
said local error function, eij, is given by the equation
##EQU8##
where s.sub.ij and b.sub.ij, the exposure parameters, represent the
exposure difference, I.sub.i ( ) and I.sub.j ( ) are pixel intensity
values from two images, and A.sub.ij is the overlap area of said images,
the vector X.sub.i represents the homogeneous coordinates of the pixel
locations making up said images, and the matrix M.sub.ij represents the
matrix that transforms coordinates from image j to image i.
9. The invention according to claim 8, wherein:
said pairwise registration function module operates said processor to form
a Gaussian pyramid in said memory from said plurality of overlapping
images, said Gaussian pyramid having a plurality of levels, each
subsequent level being finer than each preceding level, said optimization
of said local error function is performed on progressively finer levels of
said Gaussian pyramid;
said pairwise registration function module operates said processor to
estimate said exposure parameters, s.sub.ij and b.sub.ij, of said local
error function by a correlation-based linear search, and attenuates said
exposure parameters with progressive damping, said progressive damping set
to maximum at the coarsest level of said pyramid and decreasing to zero at
the finest level.
10. The invention according to claim 2, wherein:
said pairwise registration function module operates said processor to form
a Gaussian pyramid in said memory from said plurality of overlapping
images, said Gaussian pyramid having a plurality of levels, each
subsequent level being finer than each preceding level, said optimization
of said local error function is performed on progressively finer levels of
said Gaussian pyramid.
11. The invention according to claim 10, wherein:
the number of pyramid levels of said Gaussian pyramid is computed by said
pairwise registration function module according to the formula:
##EQU9##
where I is the square root of the smaller eigenvalue of the 2.times.2
inertial tensor of the overlap polygon region of said plurality of images,
and I.sub.min is the minimal size of the finest level resolution pyramid
level.
12. The invention according to claim 2, wherein:
a calibration and global optimization function module in said memory and
operating said processor for calibration and global optimization of said
output data of said pairwise registration module, said calibration and
global optimization module generating output data related to said
calibration and global optimization of all of said overlapping images.
13. The invention according to claim 12, wherein:
said calibration and global optimization function module calibrates and
optimizes said output data of said pairwise registration module by
approximating said local error function.
14. The invention according to claim 13, wherein said approximation of said
local error function, eij, is a Taylor Series expansion given by the
formula:
e.sub.ij (M.sub.ij).apprxeq.e.sub.ij.sup.0 +(M.sub.ij
-M.sub.ij.sup.0).sup.T C.sub.ij (M.sub.ij -M.sub.ij.sup.0)
where e.sub.ij.sup.0 is a constant representing the minimal value achieved
by said pairwise registration function module; M.sub.ij is the projective
matrix of overlapping pairs of images, M.sub.ij.sup.0 is the vector
representing the optimal projective matrix, and C.sub.ij is the Hessian
matrix obtained by the pairwise registration function module when
optimizing objective function e.sub.ij.
15. The invention according to claim 14, wherein said calibration and
global optimization function module calculates a global error function for
all overlapping images by application of the chain rule to said output
data of said pairwise registration module, according to the formula:
##EQU10##
and minimizes said global error function, where p.sub.j =the vector for
camera internal parameters for each image j, and q.sub.j =the camera
orientation with respect to a common reference frame for each image j.
16. The invention according to claim 12, further comprising:
said pairwise registration function module operates said processor to
estimate a projective transformation for pairwise registration of images.
17. The invention according to claim 16, wherein:
said projective transformation is initialized by camera internal and
external parameters relating to said plurality of images in said pairwise
registration function, and said projective transformation is initialized
by said output data generated by said calibration and global optimization
module generating output data, in an iterative manner, and,
said calibration and global optimization function module attenuates camera
internal parameters during said calibration and global optimization.
18. The invention according to claim 1, wherein:
said blending function module generating at least one panorama by employing
in the image overlap areas of said output data a multi-resolution weighted
average blending.
19. The invention according to claim 1, further comprising:
a user interface function module in said memory and operating said
processor for accepting input from a human user, and for adjusting the
areas said images overlap;
said blending function module generating at least one panorama by blending
said images where the images overlap, in response to said user interface
module.
20. The invention according to claim 1:
said blending function module generates said panorama from said output data
comprising images having image overlap regions, said blending function
module blending by instructing said processor to construct Laplacian
pyramids stored in said memory for each plurality of images that overlap
in said image overlap regions, said Laplacian pyramids having a plurality
of levels, said blending function module determining the coarsest level
resolution level of the Laplacian pyramids for each of said images, said
blending function module generating a blend mask from the image overlap
region of said plurality of images, said blending function module
constructing a Gaussian pyramid from said blend mask for each said
plurality of images, and blending said images by multiplying said
Laplacian and Gaussian pyramid values for every level of said Laplacian
pyramids for every said plurality of images.
21. The invention according to claim 1, wherein said blending function
module blends the high frequency components of said images that overlap
over a smaller region in the area of overlap than the low frequency
components of said images.
22. The invention according to claim 21, wherein said smaller region is on
the order of 3-5 pixels.
23. The invention according to claim 1, wherein said projection function
module forms said panorama on a geometry that is selected from geometries
consisting of: cubic, polyhedral, cylindrical and spherical geometries,
frusto-conical cones joined at their base with the apexes pointing away
from one another; any quadric surface and geometries using the following
projections: equidistant, equiangular, ellipsoid, Mercator, transverse
Mercator, Oblique Mercator, cylindrical equal-area, Miller cylindrical,
equidistant cylindrical, Cassini, Albers equal-area, Lambert conformal
conic, equidistant conic, bipolar oblique conic conformal, polyconic,
Bonne, azimuthal, orthographic, sterographic, gnomonic, general
perspective, Lambert azimuthal equal-area, azimuthal equidistant,
modified-stereographic conformal, space map projections, space oblique
Mercator, satellite-tracking projections, pseudocylindrical projections,
Van der Grinten, sinusoidal, Mollweide and Eckert IV and VI projections.
24. The invention in accordance with claim 1, wherein said calibration and
global optimization function module calibrates and optimizes said output
data of said pairwise registration module by approximating a local error
function.
25. The invention according to claim 24, wherein said approximation of said
local error function is a Taylor Series expansion given by the formula:
e.sub.ij (M.sub.ij).apprxeq.e.sub.ij.sup.0 +(M.sub.ij
-M.sub.ij.sup.0).sup.T C.sub.ij (M.sub.ij -M.sub.ij.sup.0)
where e.sub.ij.sup.0 is a constant representing the minimal value achieved
by said pairwise registration function module; M.sub.ij is the projective
matrix of overlapping pairs of images, M.sub.ij.sup.0 is the vector
representing the optimal projective matrix, and C.sub.ij is the Hessian
matrix obtained by the pairwise registration function module when
optimizing objective function e.sub.ij.
26. The invention according to claim 25, wherein said calibration and
global optimization function module calculates a global error function for
all overlapping images by application of the chain rule to said output
data of said pairwise registration module, according to the formula:
##EQU11##
and minimizes said global error function, where p.sub.j =the vector for
camera internal parameters for each image j, and q.sub.j =the camera
orientation with respect to a common reference frame for each image j.
27. In a computer having a processor which executes instructions stored in
memory, a method for generating panoramas comprising the steps of:
storing at least two rectilinear images in a computer memory;
generating image output data by performing pairwise registration of said
rectilinear images as an estimate to a projective transformation relating
to said rectilinear images, wherein the projective transformation includes
a projective matrix, the matrix being a function of image center position;
generating image output data by calibration and global optimization of said
rectilinear images from image output data from said pairwise registration,
said image output data including a Hessian matrix;
blending said rectilinear images to create a panorama from said generated
image output data from said pairwise registration and said calibration and
global optimization.
28. The method according to claim 27, wherein:
said step of blending said rectilinear images to create a panorama
comprises the step of identifying a region of overlap between said
rectilinear images from said generated data;
generating a blend mask from said rectilinear images from said generated
data;
locating the boundary of said mask by performing a grassfire transform on
said at least two rectilinear images individually.
29. The method according to claim 27, wherein:
said step of generating data by performing pairwise registration of said
rectilinear images by estimating said projective transformation of said
rectilinear images comprises the steps of:
initializing said projective transformation with camera internal and
external parameters of two rectilinear images that overlap;
forming a Gaussian pyramid from said plurality of overlapping rectilinear
images;
perturbing said overlapping rectilinear images until a local registration
error function is minimized at a first level of said Gaussian pyramid for
said images.
30. The method according to claim 29, wherein:
wherein said camera internal and external parameters are selected from the
group comprising the image center position of each of said pairs of
images, aspect ratio of said pixels comprising said images, the focal
length of the camera which took said images; the camera orientation with
respect to a common reference frame, camera pan, tilt, roll and skew and
brightness and contrast of said images.
31. The method according to claim 29, wherein:
said local registration error function, eij, is given by the equation:
##EQU12##
where s.sub.ij and b.sub.ij, the exposure parameters, represent the
exposure difference, I.sub.i ( ) and I.sub.j ( ) are pixel intensity
values from two images, and A.sub.ij is the overlap area of said images,
the vector X.sub.i represents the homogeneous coordinates of the pixel
locations making up said images, and the matrix M.sub.ij represents the
matrix that transforms coordinates from image j to image i;
further comprising the step of pairwise registration of said overlapping
images on successive iterations of said Gaussian pyramid.
32. The method according to claim 31, further comprising the step of:
attenuating said estimated exposure parameters, sij and bij, in said
equation, at each successive iteration of said Gaussian pyramid by
progressive damping, said damping set to maximum at the coarsest level of
said Gaussian pyramid and decreasing to zero at the finest level of said
Gaussian pyramid.
33. The method according to claim 32, further comprising the step of:
estimating said exposure parameters, sij and bij, according to a
correlation based linear search.
34. The method according to claim 31, further comprising the step of:
finding said first level of said Gaussian pyramid by successively moving
down said Gaussian pyramid until a predetermined number of pixels from
each of said images overlap.
35. The method according to claim 34, further comprising the step of:
determining the number of levels of said Gaussian pyramid of said pairwise
registration of overlapping images according to the formula:
##EQU13##
where I is the square root of the smaller eigenvalue of the 2.times.2
inertial tensor of the overlap polygon region of said plurality of images,
and I.sub.min is the minimal size of the finest level resolution pyramid
level.
36. The invention according to claim 29, wherein:
wherein said calibration and global optimization function module calibrates
and optimizes said output data of said pairwise registration module by
approximating said local error function, eij;
wherein said step of estimating said local error function is an
approximation of said local error function given by the formula:
e.sub.ij (M.sub.ij).apprxeq.e.sub.ij.sup.0 +(M.sub.ij
-M.sub.ij.sup.0).sup.T C.sub.ij (M.sub.ij -M.sub.ij.sup.0)
where e.sub.ij.sup.0 is a constant representing the minimal value achieved
by said pairwise registration function module; M.sub.ij is the projective
matrix of overlapping pairs of images, M.sub.ij.sup.0 is the vector
representing the optimal projective matrix, and C.sub.ij is the Hessian
matrix obtained by the pairwise registration function module when
optimizing objective function e.sub.ij.
37. The invention according to claim 36, further comprising the steps of:
saving said approximated local error function generated during pairwise
registration of said images according to the steps above;
computing, for each and every overlapping images, said calibration and
global optimization of said images, according to the global error formula:
##EQU14##
minimizing said global error function, where p.sub.j =the vector for camera
internal parameters for each image j, and q.sub.j =the camera orientation
with respect to a common reference frame for each image j.
38. The invention according to claim 37, further comprising the steps of:
re-initializing said projective transformation, in an iterative manner,
with said output data generated by said calibration and global
optimization module.
39. The method according to claim 27, wherein:
said step of blending comprises blending high frequency components of said
images that overlap over a smaller region in the area of overlap than low
frequency components of said images.
40. The invention according to claim 39, wherein said blending for high
frequency components of said images is performed over said smaller region
on the order of 3-5 pixels.
41. The invention according to claim 27, wherein said steps further
comprising:
wherein said blending of said rectilinear images to create a panorama
comprises the steps of:
constructing a Laplacian pyramid for the overlapping images in an image
overlap region where said images overlap, said Laplacian pyramids having a
plurality of levels;
determining the coarsest level resolution level of the Laplacian pyramids
for each of said images;
constructing a blend mask from the image overlap region for said plurality
of overlapping images, said blend mask being a Gaussian pyramid
combining said Laplacian pyramid and said Gaussian pyramids to blend said
images in said image overlap region.
42. The method according to claim 41, wherein said step of constructing
said blend mask is performed by locating the boundary of said blend mask
by performing a grassfire transform on at least two overlapping images.
43. The invention according to claim 41, further comprising the steps of:
labeling, during the steps of blending, the pixels comprising each said
overlapping images with values to associate each pixel with a particular
image;
generating a blend mask from the image overlap region of said images, and
computing a boundary to said blend mask from said values of said pixel
labels and from said overlapping images.
44. The invention according to claim 41, wherein said blending function
module generating at least one panorama from said output of said pairwise
registration module and said calibration and global optimization module by
employing in the image overlap areas of said output a multi-resolution
weighted average blending.
45. The invention according to claim 27, further comprising the steps of:
projecting said panorama from said images, after said images have been
pairwise registered, calibrated and globally optimized;
projecting said panorama onto a view surface geometry that is selected from
geometries consisting of: cubic, polyhedral, cylindrical and spherical
geometries, frusto-conical cones joined at their base with the apexes
pointing away from one another; any quadric surface, and geometries using
the following projections: equidistant, equiangular, ellipsoid, Mercator,
transverse Mercator, Oblique Mercator, cylindrical equal-area, Miller
cylindrical, equidistant cylindrical, Cassini, Albers equal-area, Lambert
conformal conic, equidistant conic, bipolar oblique conic conformal,
polyconic, Bonne, azimuthal, orthographic, sterographic, gnomonic, general
perspective, Lambert azimuthal equal-area, azimuthal equidistant,
modified-stereographic conformal, space map projections, space oblique
Mercator, satellite-tracking projections, pseudocylindrical projections,
Van der Grinten, sinusoidal, Mollweide and Eckert IV and VI projections.
46. A computer readable memory storing a program which executes the steps
of:
storing at least a pair of rectilinear images in a computer memory; and
performing pairwise registration of said rectilinear images by calculating
a projective transformation between said rectilinear images, the
projective transformation includes a projective matrix, the matrix being a
function of image center position, said pairwise registration function
module generating a Hessian matrix.
47. The method according to claim 46, wherein:
said step of performing pairwise registration of said rectilinear images by
calculating said projective transformation between said rectilinear images
comprises the steps of:
initializing said projective transformation with camera internal and
external parameters of a pair of said rectilinear images that overlap;
forming a Gaussian pyramid from said pair of overlapping rectilinear
images;
perturbing said overlapping rectilinear images until a local registration
error function is minimized at a first level of said Gaussian pyramid for
said images.
48. The method according to claim 47, wherein:
wherein said camera internal and external parameters are selected from the
group comprising the image center position of each of said pairs of
images, aspect ratio of said pixels comprising said images, the focal
length of the camera which took said images; the camera orientation with
respect to a common reference frame, camera pan, tilt, roll and skew and
brightness and contrast of said pairs of images.
49. The method according to claim 47, wherein:
said local registration error function, eij, is given by the equation:
##EQU15##
where s.sub.ij and b.sub.ij, the exposure parameters, represent the
exposure difference, I.sub.i ( ) and I.sub.j ( ) are pixel intensity
values from two images, and A.sub.ij is the overlap area of said images,
the vector X.sub.i represents the homogeneous coordinates of the pixel
locations making up said images, and the matrix M.sub.ij represents the
matrix that transforms coordinates from image j to image i;
further comprising the step of pairwise registration of said overlapping
images on successive iterations of said Gaussian pyramid.
50. The method according to claim 49, further comprising the step of:
attenuating said estimated exposure parameters, sij and bij, in said
equation, at each successive iteration of said Gaussian pyramid by
progressive damping, said damping set to maximum at the coarsest level of
said Gaussian pyramid and decreasing to zero at the finest level of said
Gaussian pyramid.
51. The method according to claim 50, further comprising the step of:
estimating said exposure parameters, sij and bij, according to a
correlation based linear search.
52. The method according to claim 49, further comprising the step of:
finding said first level of said Gaussian pyramid by iteratively moving
down said Gaussian pyramid until a predetermined number of pixels from
each of said images overlap.
53. The method according to claim 52, further comprising the step of:
determining the number of levels of said Gaussian pyramid of said pairwise
registration of overlapping images according to the formula:
##EQU16##
where I is the square root of the smaller eigenvalue of the 2.times.2
inertial tensor of the overlap polygon region of said plurality of images,
and I.sub.min is the minimal size of the finest level resolution pyramid
level.
54. A computer system comprising:
a memory storing data representations relating to at least two images;
function modules in said memory;
a processor for processing instructions and data from said memory and said
function modules; and
a pairwise registration function module in said memory, and operating said
processor for pairwise registration of said images, said pairwise
registration function module generating output data related to said
pairwise registration of said images, the pairwise registration function
module including a projective matrix, wherein the projective matrix is a
function of image center position, said output data including a Hessian
matrix.
55. The invention according to claim 54, wherein:
said pairwise registration function module operates said processor to
register a plurality of each of said images that overlap with one another,
by perturbing said images.
56. The invention according to claim 55, wherein:
said pairwise registration function module performs said pairwise
registration by estimating a projective transformation relating a pair of
images that overlap.
57. The invention according to claim 56, wherein:
said projective transformation between said pair of images is estimated by
said pairwise registration function module by performing a gradient-based
optimization that minimizes a local error function that is comprised of
the differences in exposures between pixels comprising pairs of
overlapping images, and contains a projective matrix initialized by camera
internal and external parameters relating to said plurality of images.
58. The invention according to claim 57 wherein:
said local error function, eij, is given by the equation
##EQU17##
where s.sub.ij and b.sub.ij, the exposure parameters, represent the
exposure difference, I.sub.i ( ) and I.sub.j ( ) are pixel intensity
values from two images, and A.sub.ij is the overlap area of said images,
the vector X.sub.i represents the homogeneous coordinates of the pixel
locations making up said images, and the matrix M.sub.ij represents the
matrix that transforms coordinates from image j to image i, and;
said pairwise registration function module operates said processor to form
a Gaussian pyramid in said memory from said plurality of overlapping
images, said Gaussian pyramid having a plurality of levels, each
subsequent level being finer than each preceding level, said optimization
of said local error function is performed on progressively finer levels of
said Gaussian pyramid.
59. The invention according to claim 58, wherein:
the number of pyramid levels of said Gaussian pyramid is computed by said
pairwise registration function module according to the formula:
##EQU18##
where I is the square root of the smaller eigenvalue of the 2.times.2
inertial tensor of the overlap polygon region of said plurality of images,
and I.sub.min is the minimal size of the finest level resolution pyramid
level.
60. The invention according to claim 58, wherein:
said pairwise registration function module operates said processor to
estimate said exposure parameters, s.sub.ij and b.sub.ij, of said local
error function by a correlation-based linear search, and attenuates said
exposure parameters with progressive damping, said progressive damping set
to maximum at the coarsest level of said pyramid and decreasing to zero at
the finest level.
61. The invention according to claim 56, further comprising:
a user interface module in said memory and operating said processor for
initializing said projective transformation of said pairwise registration
function module, said user interface module operating to process input
from a user of the computer system, wherein said projective transformation
is initialized with said camera internal and external parameters from
manual input by a user.
62. The invention according to claim 61, wherein:
said internal and external parameters are selected from the group
comprising: image center position of each of said pairs of images, aspect
ratio of said pixels comprising said images, the focal length of the
camera which took said images; the camera orientation with respect to a
common reference frame, camera pan, tilt, roll and skew and brightness and
contrast of said pairs of images.
63. The invention according to claim 61, wherein:
said user interface module allows the user the ability to select the
arrangement of said images.
64. The invention according to claim 63, wherein:
said user interface module comprises a virtual reality environment
comprising a spherical coffee table.
65. The invention according to claim 64, wherein:
said user interface module allows the user to select the arrangement of
said images from a plurality of rows of photos on said spherical coffee
table.
66. The invention according to claim 56, wherein:
a calibration and global optimization function module in said memory and
operating said processor for calibration and global optimization of said
output data of said pairwise registration module, said calibration and
global optimization module generating output data related to said
calibration and global optimization of all of said overlapping images.
67. The invention according to claim 66, wherein:
wherein said calibration and global optimization function module calibrates
and optimizes said output data of said pairwise registration module by
approximating a local error function.
68. The invention according to claim 67, wherein said approximation of said
local error function is a Taylor Series expansion given by the formula:
e.sub.ij (M.sub.ij).apprxeq.e.sub.ij.sup.0 +(M.sub.ij
-M.sub.ij.sup.0).sup.T C.sub.ij (M.sub.ij -M.sub.ij.sup.0)
where e.sub.ij.sup.0 is a constant representing the minimal value achieved
by said pairwise registration function module; M.sub.ij is the projective
matrix of overlapping pairs of images, M.sub.ij.sup.0 is the vector
representing the optimal projective matrix, and C.sub.ij is the Hessian
matrix obtained by the pairwise registration function module when
optimizing objective function e.sub.ij.
69. The invention according to claim 67, wherein said calibration and
global optimization function module calculates a global error function for
all overlapping images by application of the chain rule to said output
data of said pairwise registration module, according to the formula:
##EQU19##
and minimizes said global error function, where p.sub.j =the vector for
camera internal parameters for each image j, and q.sub.j =the camera
orientation with respect to a common reference frame for each image j.
70. The invention according to claim 66, wherein:
said projective transformation initialized by camera internal and external
parameters relating to said plurality of images in said pairwise
registration function is initialized by said output data generated by said
calibration and global optimization module generating output data, in an
iterative manner.
71. The invention according to claim 70, wherein:
said calibration and global optimization function module attenuates camera
internal parameters during said calibration and global optimization.
72. The invention according to claim 66, further comprising:
a blending function module in said memory and operating said processor for
generating at least one panorama from said output data from said pairwise
registration module and said calibration and global optimization module.
73. The invention according to claim 72, wherein:
said blending function module generating at least one panorama from said
output of said pairwise registration module and said calibration and
global optimization module by employing in the image overlap areas of said
output a multi-resolution weighted average blending.
74. The invention according to claim 73, further comprising:
a user interface module in said memory and operating said processor for
accepting input from a human user, and for selecting said images and
adjusting the areas said images overlap;
said blending function module generating at least one panorama from said
output of said pairwise registration module and said calibration and
global optimization module by employing in the image overlap areas of said
output a weighted average blending, in response to said user interface
module.
75. The invention according to claim 73, further comprising:
a user interface module in said memory and operating said processor for
accepting input from a human user, and for selecting said images and
adjusting the areas said images overlap, wherein said user interface
module allows the user the ability to select the arrangement of said
images.
76. The invention according to claim 75, wherein:
said user interface module comprises a virtual reality environment
comprising a spherical coffee table.
77. The invention according to claim 76, wherein:
said user interface module allows the user to select the arrangement of
said images from a plurality of rows of photos.
78. The invention according to claim 73, wherein:
said blending function module generates said panorama from said output data
comprising images having image overlap regions, said image overlap regions
blended by said panorama generating function module by performing a
grassfire transform on data relating to said images to generate a blend
mask of said images.
79. The invention according to claim 73, wherein said blending function
module blends high frequency components of said images that overlap over a
smaller region in the area of overlap than low frequency components of
said images.
80. The invention according to claim 79, wherein said smaller region is on
the order of 3-5 pixels.
81. The invention according to claim 73, wherein:
said blending function module generates said panorama from said output data
comprising images having image overlap regions, said blending function
module blending by instructing said processor to construct Laplacian
pyramids stored in said memory for each plurality of images that overlap
in said image overlap regions, said Laplacian pyramids having a plurality
of levels, said blending function module determining the coarsest level
resolution level of the Laplacian pyramids for each of said images, said
blending function module generating a blend mask from the image overlap
region of said plurality of images, said blending function module
constructing a Gaussian pyramid from said blend mask for each said
plurality of images, and blending said images by multiplying said
Laplacian and Gaussian pyramid values for every level of said Laplacian
pyramids for every said plurality of images.
82. The invention according to claim 73, wherein:
said blending function labels the pixels comprising each said overlapping
images with values to associate each pixel with a particular image, said
blending function module generating a blend mask from the image overlap
region of said images, and computing a boundary to said blend mask from
said values of said pixel labels and from said overlapping images.
83. The invention according to claim 73, further comprising:
a projection function module in said memory and operating said processor
for forming a panorama from said images, said projection function forms
said panorama on a geometry that is selected from geometries consisting
of: cubic, polyhedral, cylindrical and spherical geometries,
frusto-conical cones joined at their base with the apexes pointing away
from one another; any quadric surface, and geometries using the following
projections: equidistant, equiangular, ellipsoid, Mercator, transverse
Mercator, Oblique Mercator, cylindrical equal-area, Miller cylindrical,
equidistant cylindrical, Cassini, Albers equal-area, Lambert conformal
conic, equidistant conic, bipolar oblique conic conformal, polyconic,
Bonne, azimuthal, orthographic, sterographic, gnomonic, general
perspective, Lambert azimuthal equal-area, azimuthal equidistant,
modified-stereographic conformal, space map projections, space oblique
Mercator, satellite-tracking projections, pseudocylindrical projections,
Van der Grinten, sinusoidal, Mollweide and Eckert IV and VI projections.
84. An apparatus for generating a panorama comprising:
means for pairwise registration of images having overlapping regions;
means for calibration and global optimization of said images;
means for blending said images after pairwise registration, calibration and
global optimization of said images;
said pairwise registration function module including a projective matrix,
wherein the projective matrix is a function of image center position; and
means for projecting said blended images into a panorama.
85. The invention according to claim 84, wherein:
said pairwise registration means operates said processor to register a
plurality of each of said images that overlap with one another, through
perturbation of said images.
86. The invention according to claim 85, wherein:
said pairwise registration means estimates a projective transformation
between a pair of images by performing a gradient-based optimization that
minimizes a local error function.
87. The invention according to claim 86, further comprising:
a user interface module in said memory and operating said processor for
initializing said projective matrix of said pairwise registration means,
said user interface module operating to process input from a user of the
computer system, wherein said projective matrix is initialized with said
camera internal and external parameters from manual input by a user; and
the local error function of said pairwise registration means is comprised
of the differences in exposures between pixels comprising the pairs of
images.
88. The invention according to claim 87, wherein:
said projective transformation contains a projective matrix parametrized by
said internal and external parameters, selected from the group comprising:
image center position of each of said pairs of images, aspect ratio of
said pixels comprising said images, the focal length of the camera which
took said images; the camera orientation with respect to a common
reference frame, camera pan, tilt, roll and skew and brightness and
contrast of said pairs of images, and, said projective transformation is
initialized by camera internal and external parameters relating to the
plurality of images.
89. The invention according to cl | | |
