An improved apparatus and method for creating high quality virtual reality panoramas is disclosed that yields dramatic improvements during the authoring and projecting cycles, with speeds up to several orders of magnitude faster than prior systems. In a preferred embodiment, a series of rectilinear images taken from a plurality of rows are pairwise registered with one another, and locally optimized using a pairwise objective function (local error function) that minimizes certain parameters in a projective transformation, using an improved iterative procedure. The local error function values for the pairwise registrations are then saved and used to construct a quadratic surface to approximate a global optimization function (global error function). The chain rule is used to avoid the direct evaluation of the global objective function, saving computation. In one embodiment concerning the blending aspect of the present invention, an improved procedure is described that relies on Laplacian and Gaussian pyramids, using a blend mask whose boundaries are determined by the grassfire transform. An improved iterative procedure is disclosed for the blending that also determines at what level of the pyramid to perform blending, and results in low frequency image components being blended over a wider region and high frequency components being blended over a narrower region. Human interaction and input is also provided to allow manual projective registration, initial calibration and feedback in the selection of photos and convergence of the system.
CROSS-REFERENCE TO RELATED APPLICATIONS
This application Ser. No. 09/160,823, is with commonly assigned utility patent application Ser. No. 09/160,822, filed on Sep. 25, 1998, entitled "Aligning Rectilinear Images in 3D through Projective Registration and Calibration".
An image-based walkthrough system and process that employs pictures, panoramas, and/or concentric mosaics captured from real scenes to present a photo-realistic environment to a viewer. In general, a walkthrough space is divided into a horizontally sectioned grid. Each cell of the grid is assigned at least one source of image data from which an image of a part or all of the surrounding scene as viewed from that cell can be rendered. Whenever the viewer moves into one of the grid cells, the distance between the viewer's currently selected viewing position in that cell, and each picture viewpoint, panorama center, and nearest concentric mosaic wandering circle point, in the cell and its neighboring cells, is computed. An image depicting the scene as would be viewed from the point associated with the closest image data source is then rendered and displayed to the viewer.
A method for operating a data processing system to generate a second image from a first image having partially sampled color values at each pixel. The first image includes a two-dimensional array of pixel values, each of the pixel values corresponding to the light intensity in one of a plurality of spectral bands at a location in the first image. The second image includes a second two-dimensional array of color vectors. Each color vector has a light intensity value for each of the spectral bands. There is one such vector corresponding to each location having a pixel value in the first image. One component of the vector is equal to the pixel value in the first image at that location. The present invention computes the missing color components at each location. The method begins by providing an estimate for each component that is not equal to one of the pixel values from the first image for each vector. An updated estimate for each of the estimates is then generated by optimizing a weighted sum of first and second functions. The first function measures the degree of roughness of the second image, and the second function measures the degree to which the vectors change direction between neighboring locations in the second image.
An image matching method for matching a first image and an overlapping second image includes generating a first set of working layers of the first image and a second set of working layers of the second image. The method determines an overlap between an initial working layer of the first image and an initial working layer of the second image where the initial working layers have a smaller pixel array size and a lower image resolution than the other working layers. The method selects a feature point in the working layer of the first image and determines a position in the working layer of the second image corresponding to the feature point. The method then determines the motion parameters based on the feature point and the position in the first and second images. Finally, the method repeats the selection of a feature point using another working layer of the first image and another working layer of the second image, each of these working layers has a larger pixel array size and a higher image resolution than the initial working layers.
An image processing apparatus comprising image capturing means for capturing a plurality of images, adjacent ones of which partially overlap with each other, overlap region determining means for determining an overlap region, first region determining means for determining a first region within the overlap region determined, first pixel value determining means for determining a pixel value in the first region based on a pixel value of one of the two adjacent images, second pixel value determining means for determining a pixel value in a second region within the overlap region based on respective pixel values of the two adjacent images, and image joining means for joining the two adjacent images with each other by utilizing the determined pixel value in the first region and the determined pixel value in the second region as pixel values in respective regions of the overlap region.
A method for generating a panoramic image includes receiving a first image, dividing the first image into a first portion and a second portion, rotating the first portion of the first image, saving the rotated first portion of the first image in a nonvolatile memory, receiving a second image, dividing the second image into a third portion and a fourth portion, matching an overlapping region between the second portion of the first image and the third portion of the second image, stitching the second portion of the first image and the third portion of the second image to form a first stitched image, rotating the first stitched image, and saving the first stitched image in the nonvolatile memory.
