Image synthesization method - Patent Review 7366360

Description Submit all comments and votes

BACKGROUND OF THE INVENTION

This is a divisional of Application No. 10/347,317, filed on Jan. 21, 2003, now U.S. Pat. 7,085,435 which is a divisional of Application No. 08/719,716, filed on Sep. 25, 1996, now U.S. Pat. No. 6,549,681, issued on Apr. 15, 2003.

1. Field of the Invention

The present invention relates to an image synthesization method for synthesizing a plurality of images, in which the image areas partially overlap each other, in order to create a single synthetic image.

2. Related Background Art

The processing for the synthesization of a plurality of images that partially overlap each other, by using a computer to create a single synthetic image, is generally called panoramic image synthesization. This processing has been developed in response to a demand that it is possible to take a wide picture that constitutes a single image. When an electronic camera is compared with a silver halide camera or a scanner, the low resolution (the small number of pixels) provided by the electronic camera is pointed out as a disadvantage. For an image that is taken by an electronic camera, therefore, panoramic image synthesization is important not only as a means for acquiring a wide image, but also as a means for acquiring an image having a high resolution. Specifically, panoramic image synthesization is effectively demonstrated when a sheet of a document or a page of a magazine is divided into a plurality of segments and images of these segments are taken to acquire data at the similar lebel of a resolution as that afforded by a scanner, or when a scenic view is divided into a plurality of segments and each segment is recorded as a wide angle image at a high resolution.

In panoramic image synthesization, a process for erasing seams where segments overlap is important, and affects the quality of the resultant synthetic image. As a general method, a process for erasing the segment seams shown in FIG. 1A (hereinafter referred to as a "seamless process") is performed. That is, in a location where portions of two images are overlapped, synthesization ratios are gradually changed in consonance with the positions of pixels, and the pixels are added together so that each of the two overlapped image portions constitute 50% of the pixels at the center position. When the overlapped areas are large, a seamless process having a predetermined width is performed, as is shown in FIG. 1B. The seamless process is effective especially for natural images, such as of scenery, and seamless images having a high quality can be provided. The above described conventional technique has the following problems, however. For panoramic synthesization, a method is employed by which matching points in a plurality of images to be synthesized are extracted to determine a position at which to synthesize overlapping images. At this time, an error may occur at the synthesization position. That is, since the minimum unit for which accuracy is possible when matching points are extracted is one pixel, and as accuracy can not be guaranteed if a unit that is smaller than one pixel is employed, an error occurs when a shift of less than one pixel occurs at the pixel synthesization position.

Further, when an image is recorded with an electronic camera, the portion of the image that is located at the periphery of a lens is more or less distorted. This also causes a shift of less than one pixel.

Then, as there is a sharp contrast between paper color, white, and character color, black, in an image, such as a document, in which characters are included, when a document image is synthesized and a seamless process is performed therefor, dual character images can be seen in the portion for which the seamless process has been performed, as is shown in FIG. 1C. And as the characters are sharply contrasted with their background, the shift is easily discernable. For a natural image, however, since the contrast is less distinct than is that for a character image, and since a smooth continuation of the image lines is preferable, the seamless process is effective. Again, however, for images, such as documents, that include characters, in many cases adverse effects are obtained, as has been previously described.

As for an electronic camera, it has been pointed out that low resolution (a small number of pixels) is one of their disadvantages when compared with silver halide cameras or scanners, as is described above. Panoramic image synthesization is important for images recorded by electronic cameras not only for the acquisition of wide angle images but also for the provision of high image resolutions. More specifically, panoramic image synthesization is effective when a single sheet of a document or a page of a magazine is divided into segments and the image segments are recorded to acquire data at the similar level of a resolution as data is obtained at with a scanner, or when a scenic view is divided into segments to acquire a wide angle image at a high resolution.

For panoramic image synthesization, the most important process, and one that is difficult to accomplish, is finding a location where a plurality of images overlap. In essence, this process is one that involves a search for like points (hereinafter referred to as matching points) located in a plurality of images. The process Is hereinafter referred to as a matching point extraction process. The difficulty encountered in performing the matching point extraction process (the error rate) differs, depending on the images being processed. When an overlapped image area includes a unique, characteristic shape that does not exist in other areas, a matching point can be found without any error. However, when a similar pattern exists in an image area other than the overlapped image area (e.g., characters in a document), an incorrect matching point may be extracted.

According to the conventional technique, generally, a user clearly designates a matching point, and based on the designated position, images are synthesized while slight adjustments are performed. Such a conventional example is shown in FIG. 2. When a user selects a plurality of images to be synthesized, the window shown in FIG. 2 is opened. The user designates matching points in two images, and provides marks 21a, 21b, 22a and 22b for these points. Patterns that are nearest the centers of a pair of the marks are examined, a matching positional relationship that applies to both of the marks is acquired, and the points specified by the marks are designated as matching points. A parameter for image synthesization is then acquired by using the matching points, and image synthesization is performed. With the conventional example, however, the following problems are encountered.

(1) Since a user must with considerate accuracy designate matching points for two images, the user must perform a careful comparison of the two images. This imposes a heavy load on the user.

(2) Two matching points are required for image synthesization, and more or less than two points can not be designated. Although only one point is required when an image is shifted only horizontally or vertically, one-point designation is not possible.

(3) Although images can be synthesized more accurately by designating three or more points, this is not possible.

(4) Since the synthesization process is only begun after matching points are designated by a user, to the user the processing period seems overly long.

FIGS. 3A through 3C are diagrams illustrating conventional panoramic image synthesization. In FIGS. 3A and 3B, overlapping portions for two images 201 and 202 are identified, and while the overlapping portions of the images are held in alignment, the images are synthesized to acquire a panoramic image 203.

With such panoramic image processing, however, the following problems are encountered. When an image, such as the panoramic image 203 in FIG. 3B, that is obtained by synthesizing a plurality of images does not have a rectangular shape, the resultant image must be converted into a rectangular shape and into a data form that can be handled by a computer. Therefore, a means is provided for describing a rectangle 204 that encompasses the panoramic image 203, and for filling with a desired color or pattern an area (dummy) 205, of the rectangle 204, in which no image data exists, as is shown in FIG. 3C. An image that is obtained by panoramic image synthesization and that includes a dummy area is, therefore, not a preferable image.

In the panoramic image synthesization processing, matching points are extracted from a plurality of rectangular images, and the images are moved, rotated, or enlarged or reduced so as to position matching points at the same location. Then, an average value for matching pixels is calculated to acquire a synthetic image.

However, as the images are moved, rotated, or enlarged or reduced, the synthetic image does not always have a rectangular shape. To store the synthetic image by using an image file format that is generally employed, a rectangle is described that encompasses the image, and dummy data are provided for a portion of the rectangular area where no image data exists. As a result, rectangular synthetic image data are created, and a synthetic image file is prepared by using an arbitrary image file format.

In this example, however, a problem occurs when an additional image is synthesized by employing the synthetic image. More specifically, a pixel value (density) for the original image data lying within a dynamic range is provided for dummy data for the synthetic image. When another image is to be synthesized with the synthetic image, the pixel value in the synthetic image can not be identified whether it is for the original image data or for the dummy data. Therefore, the following shortcomings have been encountered.

(1) In the matching point extraction process that is generally performed for panoramic image synthesization, dummy data are employed for calculation of matching points. As a result, incorrect matching points are acquired.

(2) During a search for matching points in the matching point extraction process, since a dummy data area is also searched, time is wasted performing unnecessary calculations.

(3) In a process for calculating a pixel value for a synthetic image from the value of a matching pixel, since the pixel value that is calculated includes dummy data, the obtained pixel value for the synthetic image is very different from the pixel value for an original image.

A conventional panoramic image synthesizer that employs the above panoramic image synthesizing technique comprises: matching point extraction means for finding matching points, which are overlapping positions, in images to be synthesized; synthesization parameter calculation means for calculating a parameter that is employed to synthesize images by using the matching points; and synthesization means for synthesizing a plurality of images based on the parameter, which is acquired by the synthesization parameter calculation means, for providing a single image. These means perform the processing when an image is fetched from an electronic camera to a computer. That is, a photographic image is recorded by an electronic camera, image data and associated attribute data for the image data are stored in a memory that is incorporated in the electronic camera. When a panoramic photographic image is to be recorded, an electronic camera is set to a panoramic image photographic mode. In the panoramic image photographic mode, an identifier that indicates one set for a panoramic image is automatically recorded in the attribute data for a photographic image. When the electronic camera is connected to the computer to register the image data and the attribute data, which are stored in the memory incorporated in the electronic camera, in a database in the computer, the attribute data are examined by application software. Then, one set of images is automatically extracted from the attribute data wherein is located the panoramic image photographic mode identifier. In other words, the matching point extraction means, the synthesization parameter calculation means, and the synthesization means are sequentially operated to perform panoramic image synthesization.

In the conventional panoramic image synthesizer described above, a large amount of processing that is performed by the matching point extraction means, the synthesization parameter calculation means and the synthesization means are large, and the period of time for the processing is extended. More specifically, the conventional panoramic image synthesizer performs all of the above described processes when the image data are transmitted to the computer. When the image data include panoramic image data, an extended period of time is required for a process sequence for the acquisition of an image by the computer from the electronic camera, and the registration of it in the database.

As an image manager for managing and for searching for an image, application software, for managing an image file in a file system of a computer, and an image database, for managing and searching for image data separately from a file system of a computer, have been proposed.

A system for managing not only an image but also attribute data for images that are managed is generally employed for the above described image manager. The attribute data are, for example, a title, a memo, another related image, a key word used for a later search, and a date when an image is recorded by an electronic camera. The attribute data are displayed together with an image on a display of the image manager, and are employed to notify a user of the attribute data for the image, and for searching for an image.

When a panoramic image is to be created by synthesizing a plurality of images that are managed by the image manager, the attribute data must again be input relative to the resultant synthetic image, and this system imposes a heavy load on a user.

SUMMARY OF THE INVENTION

It is, therefore, one object of the present invention to provide an image synthesization method whereby the above described problems can be resolved.

It is another object of the present invention to provide an image synthesization method whereby a plurality of images can be easily and effectively synthesized.

To achieve the above objects, according to one aspect of the present invention, an image synthesization method, whereby a plurality of images, each of which has a partially overlapping image area, are synthesized to create a single synthetic image, comprises:

a determination step of inputting a plurality of image data sets that correspond to the plurality of images that are input, and of determining whether or not an image in the partially overlapping image area of each of images that are indicated by the plurality of image data sets includes mainly characters;

an image processing step of performing, for the plurality of image data sets that are input, image processing in consonance with a result of a determination performe