 |
Claims  |
|
|
What is claimed is:
1. A method of processing endoscope images taken by an endoscope having an
insertion section, a bending section provided at a distal end of the
insertion section and an operation section for controlling a movement of
the bending section, comprising the steps of:
entering a first endoscopic image of an object taken by the endoscope
situated in a first position;
entering a second endoscopic image of the object taken by the endoscope
situated in a second position; said second endoscopic image being at least
partially overlapped with said first endoscopic image; and
detecting a positional relation between said first and second positions of
the endoscope.
2. A method according to claim 1, wherein said first and second endoscopic
images are taken by moving the bending section of the endoscope.
3. A method according to claim 2, wherein said bending section of the
endoscope is moved by actuating an operation member provided in the
operation section which is provided at a proximal end of the insertion
section of the endoscope, and said positional relation is detected by
detecting a movement of said operation member.
4. A method according to claim 3, wherein said bending section is moved by
rotating an angle handle provided in the operation section, said angle
handle being coupled with said bending section via an angle wire, and the
positional relation is detected by detecting a rotation of said angle
handle.
5. A method according to claim 4, wherein said positional relation is
detected by detecting a rotation of a pattern disc coupled coaxially with
the angle handle.
6. A method according to claim 5, wherein the rotation of said pattern disc
is detected by a photoelectric sensor.
7. A method according to claim 4, wherein said positional relation is
detected by detecting the movement of the angle wire on which a pattern is
provided.
8. A method according to claim 2, wherein said bending section is bent at a
single position such that a direction of an optical axis of an objective
lens provided in a distal end of the bending section with respect to the
object is changed.
9. A method according to claim 2, wherein said bending section is bent at
two positions such that a direction of an optical axis of an objective
lens provided in a distal end of the bending section with respect to the
object is remained unchanged.
10. A method of processing endoscopic images taken by an endoscope having
an insertion section, a bending section provided at a distal end of the
insertion section, and an operation section for controlling a movement of
the bending section, comprising the steps of:
entering a first endoscopic image of an object taken by the endoscope in a
first position;
entering a second endoscopic image of the object taken by the endoscope in
a second position, said second endoscopic image being at least partially
overlapped with said first endoscopic image;
detecting a positional relation between said first and second endoscopic
images; and
processing said first and second endoscopic images in accordance with said
positional relation to derive geometric information of the object.
11. A method according to claim 10, wherein said processing step comprises
a step of deriving a shift amount represented by a distance between
corresponding two points on the first and second endoscopic images.
12. A method according to claim 11, wherein said distance is detected by
deriving a correlation between small regions in the first and second
endoscopic images.
13. A method according to claim 12, wherein prior to deriving the
correlation, distortion of the first and second endoscopic images is
corrected.
14. A method according to claim 13, wherein after the distortion of the
endoscopic images has been corrected, the endoscopic images are subjected
to an interpolation.
15. A method according to claim 14, wherein said interpolation is carried
out in accordance with a b-spline function.
16. A method according to claim 12, wherein said correlation is detected
with the aid of an electronic correlation calculator.
17. A method according to claim 16, wherein said step of detecting the
correlation comprises a step of deriving correlation values of first
regions in the first and second endoscopic images, a step of selecting
first regions whose correlation values exceed a predetermined threshold
value, a step of deriving correlation values of second regions which are
smaller than said first regions in selected first regions, a step of
detecting a second region having the maximum correlation value, and a step
of deriving an address of the second region having the maximum correlation
value.
18. A method according to claim 16, wherein said correlation is calculated
in accordance with the following equation,
##EQU13##
wherein f.multidot.g is an average of a product of the first and second
images f and g, e,ovs/f/ .multidot.e,ovs/g/ is a product of an average of
f and a average of g, and .sigma..sub.f and .sigma..sub.g are standard
deviations of the images f and g.
19. A method according to claim 12, wherein prior to deriving the
correlation, the first and second endoscopic images are subjected to a
spatial frequency filtering.
20. A method according to claim 12, wherein each of said first and second
endoscopic images is subjected to Fourier transformation and spatial
frequency filtering, successively, and output signals derived by the
spatial frequency filtering are multiplied, and a multiplied signal is
subjected to inverse Fourier transformation.
21. A method according to claim 19 or 20, wherein said spatial frequency
filtering is carried out by extracting high frequency components of the
first and second endoscopic images.
22. A method according to claim 21 wherein the high frequency components
are extracted with the aid of a Laplacian filter.
23. A method according to claim 19 or 20, wherein said spatial frequency
filtering is effected by extracting low frequency components in the first
and second endoscopic images.
24. A method according to claim 23, wherein the low frequency components
are extracted with the aid of an averaging filter.
25. A method according to claim 16, wherein the correlation value C is
derived by the following equation;
##EQU14##
wherein F.sup.-1 represents the inverse Fourier transformation, F and G*
are Fourier transformations of the first and second endoscopic images and
.vertline.F.vertline. and .vertline.G.vertline. are absolute values of F
and G*.
26. A method according to claim 12, wherein said correlation is detected
with the aid of an optical correlation calculator.
27. A method according to claim 26, wherein the step of detecting the
distance comprises producing a holographic film containing a part of the
first endoscopic image, forming a superimposed image of an image of a part
of said holographic film and of an image of a part of the second
endoscopic image, and detecting a position of a point having maximum light
intensity in said superimposed image.
28. A method according to claim 27, wherein said holographic film is
produced by projecting an image of said part of the first endoscopic image
borne on a photographic film onto a raw photographic film together with
reference light, and by developing the raw photographic film to provide
said holographic film.
29. A method according to claim 27, wherein said holographic film is
produced by displaying an incoherent image of the first endoscopic image
on a display device, converting the incoherent image into a coherent
image, projecting a part of the coherent image onto a raw photographic
film together with reference light, and developing the raw photographic
film to provide said holographic film.
30. A method according to claim 11, wherein said processing step comprises
a step of calculating a height of a relevant point of the object in
accordance with the shift amount.
31. A method according to claim 11, wherein said processing step comprises
a step of calculating a distance between two points on the object in
accordance with the shift amount.
32. A method according to claim 10, further comprising a step of displaying
the first and second endoscopic images, a gray image representing
depressions and protrusions of the object, and a three dimensional image
of the object.
33. A method according to claim 32, wherein said displaying step comprises
displaying a distance between two points on a displayed image, said points
being indicated by cursors.
34. A method according to claim 32, wherein said displaying step comprises
displaying a height of a point on a displayed image, said point being
denoted by a cursor. |
|
 |
|
 |
Claims |
|
|
|
Description |
|
|
BACKGROUND OF THE INVENTION
Field of the Invention and Related Art Statement
The present invention relates generally to a technique for processing
images taken with the aid of an endoscope, and more particularly to a
method of deriving three dimensional information from a plurality
endoscopic images.
Heretofore, there have been proposed various types of endoscopes for taking
images within a body and a mechanical construction. There are an optical
endoscope comprising an image guide fiber bundle for transmitting an image
from a distal end to a proximal end of the endoscope, and an electronic
endoscope comprising a solid state image sensor arranged at the distal end
of endoscope. With the aid of such endoscopes, images of objects within a
body can be directly viewed. Further, when the endoscopic images are
required to be recorded, a still camera is provided at an eyepiece section
of endoscope and endoscopic images are recorded on photographic films, or
an image signal supplied from the solid state image sensor is recorded in
an image storing device such as a magnetic and optical image record
devices.
In the known systems for recording the endoscopic images, endoscopic images
are recorded independently from each other. In some cases a plurality of
images are recorded successively, but any relation has not been detected
or determined between these images. Therefore, in case of processing the
recorded images, each image has to be processed separately. Therefore, in
the known endoscopic image processing system, it is impossible to display
a three-dimensional image of an object. It has been proposed to take
stereoscopic images with the aid of the endoscope. To this end, there are
arranged two objective lenses at the distal end of the endoscope and a
pair of image guides within the insertion section. However, it is apparent
that a size, particularly a diameter of such an endoscope is liable to be
large. Further, the two objective lenses could not be arranged such that a
sufficient parallax can be obtained, so that the stereoscopic effect could
be achieved sufficiently. Due to the above mentioned drawbacks, the
stereoscopic endoscope has not been practically manufactured and sold.
SUMMARY OF THE INVENTION
The present invention has for its object to provide a novel and useful
method of processing a plurality of endoscopic images, while these images
can be related to each other in accordance with a positional relation
existing therebetween.
It is another object of the invention to provide a method of processing a
plurality of endoscopic images, in which three dimensional information of
an object is derived in accordance with a positional relation existing
therebetween and a three dimensional image of the object can be displayed.
According to the invention a method of processing endoscopic images taken
by an endoscope having an insertion section, a bending section provided at
a distal end of the insertion section and an operation section for
controlling a bending movement of the bending section, comprises:
entering a first endoscopic image of an object taken by the endoscope
situated in a first position;
entering a second endoscopic image of the object taken by the endoscope
situated in a second position; said second endoscopic image being at least
partially overlapped with said first endoscopic image; and
detecting a positional relation between said first and second positions of
the endoscope.
According to the further aspect of the invention, a method of processing
endoscopic images taken by an endoscope having an insertion section, a
bending section provided at a distal end of the insertion section, and an
operation section for controlling a bending movement of the bending
section, comprises
entering a first endoscopic image of an object taken by the endoscope in a
first position;
entering a second endoscopic image of the object taken by the endoscope in
a second position, said second endoscopic image being at least partially
overlapped with said first endoscopic image;
detecting a positional relation between said first and second endoscopic
images; and
processing said first and second endoscopic images in accordance with said
positional relation to derive geometric information of the object.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic view showing a first embodiment of the endoscopic
image processing system according to the invention;
FIG. 2 is an enlarged view of a distal end of endoscope shown in FIG. 1;
FIGS. 3A and 3B are schematic views depicting a second embodiment of the
system according to the invention;
FIG. 4 is a schematic view illustrating a third embodiment of the system
according to the invention;
FIG. 5 is a schematic view showing a fourth embodiment of the system
according to the invention;
FIG. 6 is a schematic view explaining a shift amount of the distal end of
endoscope;
FIG. 7 is a schematic view showing a measuring method in an image
processing device in the first to fourth embodiments;
FIG. 8 is a block diagram of the image processing device;
FIG. 9 is a schematic view explaining how to correct the image distortion;
FIG. 10 is a block diagram illustrating the distortion corrector;
FIG. 11 is a graph depicting the b-spline function;
FIG. 12 is a schematic view explaining the operation of the correlation
calculator;
FIG. 13 is a block diagram showing a first embodiment of the electronic
correlation calculator;
FIGS. 14, 15 and 16 are block diagrams illustrating second, third and
fourth embodiments of the electronic correlation calculator;
FIGS. 17, 18 and 19 are block diagrams showing first, second and third
embodiments of the spatial frequency filtering device;
FIG. 20 is a block diagram showing a fifth embodiment of the electronic
correlation calculator;
FIG. 21 is a block diagram illustrating an embodiment of the spatial
frequency filtering device shown in FIG. 20;
FIG. 22 is a block diagram showing another embodiment of the spatial
frequency filtering device;
FIG. 23 is a block diagram depicting a sixth embodiment of the electronic
correlation calculator;
FIG. 24 is a schematic view showing a first embodiment of the optical
correlation calculator;
FIG. 25 is a plan view illustrating the mask shown in FIG. 24;
FIG. 26 is a schematic view showing a second embodiment of the optical
correlation calculator;
FIG. 27 is a block diagram of the image display device shown in FIG. 8;
FIGS. 28A, 28B and 28C are schematic plan views showing examples of
display;
FIG. 29 is a cross sectional view showing another embodiment of the
endoscope; and
FIG. 30 is an enlarged view illustrating a distal end of the endoscope
shown in FIG. 29.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
FIG. 1 is a schematic view showing a first embodiment of the endoscopic
image processing system according to the invention. An objective lens 1 is
provided in a distal end of a bending portion 5 of the endoscope. The
objective lens 1 has a function to form an image of an object to be
observed onto an end surface of an image guide 8 formed by an optical
fiber bundle. In the distal end of the bending portion 5 there is also
provided a concave or convex lens 7 which projects light transmitted
through a light guide 6 made of, for instance an optical fiber bundle upon
the object such that the object is illuminated uniformly. The proximal end
of the endoscope is extended up to an eyepiece section 10 provided at a
control section. Then the image of the object formed on the exit surface
of the image guide 8 is picked up via an eyepiece (not shown) and imaging
lens 11 by a television (TV) camera 12. An analog output image signal from
the TV camera 12 is converted into a digital image signal with the aid of
an analog-to-digital (A/D) converter 13. The digital image signal thus
derived is stored in first or second frame memory 14 or 15. The image
signal read out of the first and second frame memories 14 and 15 are
supplied to an image processing device 16 which will be explained later.
An angle wire 17 for steering the bending section 5 into a desired
direction extends in the endoscope and is wound around a rotating drum 18
provided in the control section. The rotating drum 18 is arranged in the
control section coaxially with a pattern disc 19 and angle handle 20.
Along the peripheral portion of pattern disc 19 are formed black and white
equidistant pattern which is detected by a sensor 21 including a
reflection type photosensor. An output signal from the sensor 21 is
amplified by an amplifier 22 and is then supplied to a counter 23. An
output signal of the counter 23 is supplied to first and second latches 24
and 25 and output signals from the latches are supplied to the image
processing device 16. A record command unit 26 including operation switch,
timing circuits, etc. supplies its output signal to the first and second
frame memories 14, 15 and first and second latches 24 and 25.
Now the operation of the above system will be explained. In a first
position of the bending section 5 shown by a solid line in FIG. 1 (in this
position, the bending section is not bent), under the control of a command
supplied from the record command unit 26, a digital image signal picked-up
by the TV camera 12 is stored in the first frame memory 14. At the same
time, a count value of the counter 23 is stored in the first latch 24.
Then the angle handle 20 is rotated slightly and the bending section 5 of
the endoscope is bent as illustrated by a broken line in FIGS. 1 and 2.
In conjunction with the rotation of the angle handle 20, the pattern disc
19 is rotated and the sensor 21 detects the black and white pattern
change. This is to say, the pattern disc 19 and sensor 21 constitute a
rotary encoder which generates an information signal representing an
amount and a direction of rotation. This information signal is supplied to
the counter 23 via the amplifier 22. In the second position of the bending
section 5, the digital image is stored in the second frame memory 15. At
the same time, the count value of the counter 23 is stored in the second
latch 25. In this manner, there are stored in the first and second frame
memories 14 and 15 two images of a part of the object 4 to be observed
which have a parallax shown by A'-B' in FIG. 2, said part of the object 4
being adjacent to a hatched area, and at the same time the positional
relation between these two images has been stored in the first and second
latches 24 and 25.
The digital images are red out of the first and second frame memories 14
and 15 and are processed in the image processing device 16. The count
values stored in the first and second latches 24 and 25 are also supplied
to the image processing device 16 and are converted into angle information
.theta.. In the image processing device 16, various kinds of information
such as a difference in heights between two points on the object 4 and
projection and depression of the object by processing the signals in the
manner which will be explained later.
In the present embodiment, use is made of the optical endoscope having the
image guide, but use may be made of the video endoscope comprising the
solid state image sensor arranged at the distal end of the endoscope. In
such a case, the analog image signals produced by the solid state image
sensor are converted into the digital image signals and then are stored in
the first and second frame memories 14 and 15. This construction will be
explained as a third embodiment (FIG. 4).
Now a second embodiment of the system according to the invention will be
explained with reference to FIGS. 3A and 3B. As illustrated in FIG. 3A, in
opposition to the eyepiece section 10 of the endoscope there is arranged a
still camera 27 and the endoscopic image transmitted to the proximal end
of the image guide is formed on a photographic film 28. In the camera 27
there is arranged a data image on the film 28. The data recording means 29
for forming a data image on the film 28. The data recording means 29
comprises an LED display and is connected to a data record controlling
means 30. In the present embodiment, a slit disc 60 is coupled with the
angle handle 20 and rotary drum 18. A light emitting diode (LED) 32 is
driven by an LED driver 31, and light emitted from LED 32 is made incident
upon a light receiving element 33 formed by phototransistor through slits
formed in the slit disc 60. An output signal from the light receiving
element 33 is supplied via an amplifier 22 to a counter 23 and a count
value in the counter is supplied to the data record controlling means 30.
A release command generating means 34 comprising release switch is also
connected to the data record controlling means 30.
After the film 28 in the still camera 27 has been exposed to the endoscopic
image, the film is developed. Then, as illustrated in FIG. 3B, the image
formed on the developed film is read out by means of a film reading means
35. That is to say, a first slide 37 having the endoscopic image and
position data 36 which represents the position of the bending section 5 of
the endoscope and a second slide 39 including the endoscopic image and
position data 38 of the bending section 5 are read out by the film reading
means 35 comprising a drum type film scanner. An output signal from the
film reading means 35 is supplied via an A/D converter 61 to a data
discriminating means 40 which serves to discriminate the image data and
the position data. The image data 41 and position data 42 are supplied to
the image processing device 16. An output signal from the image processing
device 16 is supplied via a D/A converter 43 to a monitor 44 comprising
the television monitor.
Now the operation of the system according to the second embodiment will be
explained. When the bending section 5 of the endoscope is in a first
position (position represented by the solid line in FIG. 2), a release
command is generated by operating the release command generating means 34,
the count value of the counter 23 is recorded on the photographic film 28
by the data recording means 29 under the control of the data record
controlling means 30. In this case, the count value is recorded as it is.
It should be noted that the count value may be first encoded and an
encoded pattern may be recorded on the film 28. Next, the film 28 is wound
by one frame and the angle handle 20 is rotated so that the bending
section 5 of the endoscope is brought into the second position shown by
the dotted line in FIG. 2. The slit disc 60 is also rotated in conjunction
with the angle handle 20 and the light receiving element 33 detects the
light flux generated from LED 32 and interrupted by the slit disc 60. In
this case, when two pairs of the combination of LED 32 and light receiving
element 33 are arranged separately from each other by 90.degree. with
respect to the slit pattern, it is possible to detect not only the
rotational amount, but also the rotational direction. Therefore, the count
value in the counter 23 is also changed in accordance with the rotation of
the slit disc 60. Then, the release operation is effected with the aid of
the release command generating means 34, and the count value of the
counter 23 is recorded on the film 28 by means of the data recording means
29 and data record controlling means 30. At the same time, the endoscopic
image of the object is exposed on the photographic film 28. In the manner
explained above, there are obtained the first and second slides 37 and 39
having the endoscopic images viewed from the two different positions of
the bending section 5 and the information of these positions. The first
and second slides 37 and 39 are set on the film reading means 35 and the
output signal from the film reading means is converted into the digital
signal by the A/D converter 61. The data discriminating means 40
discriminates the endoscopic image data 41 from the position data 42 which
are then supplied to the image processing device 16.
The image processing device 16 effects calculations on the basis of the
entered data to derive a distance to the object 4 and depressions and
projections of the object. These calculated results are supplied to the
monitor 44 via the D/A converter 43 to display three-dimensional images on
the monitor screen.
FIG. 4 is a schematic view depicting a third embodiment of the system
according to the invention. In this embodiment, the image guide 8,
eyepiece section 10, imaging lens 11 and TV camera 12 in the embodiment
shown in FIG. 1 is replaced by a solid state image sensor 45 arranged in
the distal end of the endoscope. That is to say, the endoscope of the
present embodiment is formed as the video endoscope. The objective lens
not shown is provided at the distal end of the endoscope such that an
image of the object is formed on the solid state image sensor 45. An
output signal from the solid state image sensor 45 is supplied via a video
amplifier 50 to an A/D converter 51. A pattern disc 47 having a conductive
pattern 46 is coupled with the angle handle 20 and a contact brush unit 48
for detecting the pattern 46 is provided. The brush unit 48 is connected
to a drum position detecting means 49. The brush unit 48 comprises three
sets of brushes in order to detect the rotational direction as well as the
rotational angle. These three sets of brushes are deviated from each other
in the rotational direction so that the rotational direction can be
detected in accordance with the order of two sets of brushes which are
short-circuited by the conductive pattern of the pattern disc 47. The drum
position detecting means 49 supplies a record command signal 52 to first
and second frame memories 14 and 15.
Now the operation of the system of the third embodiment will be explained.
During the rotation of the angle handle 20, each time two sets of brushes
in the brush unit 48 are short-circuited via a conductive pattern, the
drum position detecting means 49 send the record command signal 52 to the
first or second frame memory 14 or 15 so that the image signal derived
from the solid state image sensor 45 is stored in the first or second
frame memory. The first and second frame memories 14 and 15 are selected
in an alternate manner. In this manner, in the first and second frame
memories 14 and 15 there are stored two endoscopic images of the object
viewed with a given parallax. The image signals read out of the frame
memories 14 and 15 are supplied to the image processing device 16 which
then derives various kinds of information about the object such as a
distance to the object and depressions and protrusions of object. The thus
derived information signal is displayed on the monitor 44 via D/A
converter 43.
Now a fourth embodiment of the system according to the invention will be
explained with reference to FIG. 5. In the fourth embodiment, the
endoscope is formed as the video endoscope having the solid state image
sensor provided at the distal end of the endoscope. An output signal from
the solid state image sensor 45 is supplied via video amplifier 50 and A/D
converter 51 to the image processing device 16. In order to detect the
position of the bending section 5 of the endoscope, i.e. in order to
derive the parallax, the angle wire 17 has black and white stripe pattern
and amount of movement of the wire is detected by a sensor 21 comprising a
reflection type photosensor. An output signal from the sensor 21 is
supplied via an amplifier 22 to a counter 23 and a count value of the
counter is supplied to a latch 54 and the image processing device 16.
Further, there is provided a record command generating means 55 which is
driven by an operation switch not shown to supply a driving signal to the
frame memory 53 and latch 54.
Now the operation of the fourth embodiment will be explained. In this
embodiment, the image signal is processed by the image processing device
16 which comprises the frame memory and latch and has a high operation
speed. This is to say, during the bending section 5 being in the first
position, the count value of the counter 23 is stored in the latch 54 in
response to the record command generated from the record command
generating means 55, and at the same time the endoscopic image signal
derived by the solid state image sensor 45 is stored in the frame memory
53. When the bending section 5 of the endoscope is in the second position,
the endoscopic image signal and the count value are directly supplied to
the image processing device 16. Then the image processing device 16 can
derive the distance to the object and the condition of depressions and
protrusions of the object in a real time manner by utilizing the signals
stored in the latch 54 and frame memory 53. The calculated data is
displayed on the monitor 44 via D/A converter 43.
Now the operation of the image processing device 16 will be explained by
taking an example for deriving the information which represents absolute
values of heights and magnitudes of the object.
As illustrated in FIG. 6, the bending section 5 of the endoscope is bent by
an angle .theta. and has a radius of curvature r. Since the angle .theta.
is sufficiently small, a distance l of the objective lens positions before
and after the bending, can be approximately expressed by
r.multidot..theta., i.e. l=r.multidot..theta..
FIG. 7 is a schematic view showing a geometric relation between the two
images and object. For the sake of simplicity, the optical axis of the
objective lens 1 is in a plane perpendicular to a line connecting lens
centers O.sub.1 and O.sub.2 before and after the movement. Now it is
assumed that the two endoscopic images obtained at the two positions are
denoted by P.sub.1 and P.sub.2 and a focal length of the objective lens 1
is f. Points A and B on the object are formed at points a.sub.1 and
b.sub.1 on the image P.sub.1 and at points a.sub.2 and b.sub.2 on the
image P.sub.2. Therefore, if the two images P.sub.1 and P.sub.2 are
superimposed, the points a.sub.1 and b.sub.1 are positioned at points
a'.sub.1 and b'.sub.1 on the image P.sub.2. A distance between the points
a'.sub.1 and a.sub.2 is expressed by d.sub.a and a distance between the
points b'.sub.1 and b.sub.2 is denoted by d.sub.b. Further a distance
between the point A on the object and a plane P.sub.o which contains the
centers O.sub.1 and O.sub.2 and is in parallel with the object is
represented by h.sub.A, and a distance between the point B and the plane
P.sub.o is denoted by h.sub.B. Then, there is obtained the following
relation between h.sub.A and d.sub.a due to the similarity of a triangle
AO.sub.1 O.sub.2 and a triangle O.sub.2 a'.sub.1 a.sub.2.
##EQU1##
Therefore, h.sub.A is given by the following equation.
##EQU2##
Similarly h.sub.B is given by the following equation.
##EQU3##
In this manner, it is possible to derive an absolute height h at a point
by calculating a distance d between corresponding points on the two images
P.sub.1 and P.sub.2.
Next, a manner of calculating an absolute value of the distance between two
points will be explained. Now it is assumed that a distance between a
center C.sub.1 of the image P.sub.1 and a.sub.1 is expressed by e.sub.a
and a distance between C.sub.1 and b.sub.1 is denoted by e.sub.b, and
further distances from A and B to a line I which passes through O.sub.1
and is perpendicular to the object are expressed by W.sub.A and W.sub.B,
respectively. Then, the following linear relation is obtained between
W.sub.A and e.sub.a.
##EQU4##
Now, when the equation (2) is replaced in the equation (4), W.sub.A can be
derived by the following equation (5).
##EQU5##
Similarly, W.sub.B is calculated as follows.
##EQU6##
Therefore, a distance W.sub.AB between points of projection of the points
A and B on the plane P.sub.o parallel to the object can be derived by the
following equation.
##EQU7##
In the manner explained above, it is possible to derive an absolute
magnitude of a distance between any two points on the image P.sub.1.
Next, a manner of deriving a distance d between corresponding points on the
two images P.sub.1 and P.sub.2 will be explained. This method is based on
the examination of correlation in a small region in the two images. It is
now assumed that small regions in the two images are expressed by f(r) and
g(r) which are separated from each other by a distance D. That is to say,
g(r)=f(r-D) is satisfied. In the above notations, r represent coordinates
in a two dimensional plane. Then, the correlation between f(r) and g(r)
may be expressed as follows.
.PSI.(s)= .sub.A f(r).multidot.g*(r-s).multidot.dr (8)
In the following explanation, .PSI.(s) is expressed by .PSI.(s)=f(r)*g(r).
When the above equation (8) is Fourier transformed, the following equation
(9) may be obtained.
.PHI.(u)=F(u).multidot.G*(u) (9)
In the equation (9), F(u) is the Fourier transformed equation of f(r) and
G(u) is the Fourier transformed equation of g(r). Since g(r)=f(r-D), the
equation (9) may be rewritten into the following equation
.PHI.(u)=F(u).multidot.F*(u).multidot.e.sup.-.multidot.2.pi.u.multidot.D (
10)
When the equation (10) is inverse Fourier transformed, the following
equation is derived.
.PSI.(s)=R.sub.ff (t)*.delta.(t-D) (11)
In this equation, R.sub.ff (t) is an autocorrelation function of f(r), and
can be expressed as follows.
R.sub.ff (t)=f(r)*f(r) (12)
Further, .delta.(t-D) is the inverse Fourier transformation of
e.sup.-j.multidot.2.pi..multidot.u.multidot.D. The equation (11)
represents that the function .PSI.(s) has a peak at s=D. Therefore, by
deriving the correlation function .PSI.(s) and detecting its peak position
D, it is possible to determine an amount by which g(r) is shifted with
respect to f(r). In this manner the distance d between corresponding
points can be detected by deriving the correlation of the corresponding
small regions selected from the two images P.sub.1 and P.sub.2.
Now the detailed construction of the image processing device 16 will be
explained.
FIG. 8 is a block diagram illustration the whole construction of the signal
processing system of the image processing device 16. The two color image
data 90 sent from the endoscope is stored in an image memory 91. The image
data read out of the image memory 91 is supplied to a color/monochrome
converter 92 and is converted into monochrome image data which is suitable
for measurement. The output signal from the color/monochrome converter 92
is supplied to a distortion corrector 93 and any distortion of the image
is corrected thereby. Then the image data is supplied to a correlation
calculator 94 and a distance between the corresponding points, i.e. the
shift amount is calculated. The calculated shift amount is stored in a
shift amount memory 95. The image data read out of the image memory 91,
the shift amount read out of the memory 95 and the depression and
protrusion information of the object calculated from .theta. are supplied
to an image display device 96 and absolute values of magnitude and height
of a portion in the image are displayed thereon.
(1) Image Entry
The two color image data picked-up by bending the bending section of the
endoscope is stored in the image memory 91. The bending angle .theta. is
entered in the distortion corrector 93 and image display device 96.
(2) Correction of Distortion of Image
The entered image is generally distorted owning to the reason that the
objective lens 1 has a very wide viewing angle and the distal end of
endoscope is tilted. The distortion has to be corrected. Before picking up
the image of object, a standard panel having a number of squares is imaged
under the same condition, and correction values are predetermined for each
picture elements such that the distorted image can be corrected as
illustrated in FIG. 9. Then the distortion of actually picked up
endoscopic images is corrected by using said predetermined correction
values.
FIG. 10 is a block diagram illustrating the construction of the distortion
corrector 94. The value of .theta. is supplied to a memory 104 which
stores correction values for all possible values of .theta.. The
correction values read out of the memory 104 are supplied to an image
memory 105 as address signals under control of which the monochromatic
image signal is stored in the image memory 105 at such locations that any
distortions in the input image can be corrected.
Then, the interpolation is effected for the distortion free image read out
of the image memory 105, so that the deterioration of high frequency
components due to the distortion correction is compensated for.
The interpolation is carried out by using b-spline function similar to the
well-known sinc function. The b-spline function illustrated in FIG. 11 is
expressed as follows.
##EQU8##
(3) Calculation of Correlation
In the calculation of correlation, in right and left images RP and LP shown
in FIG. 12 there are defined detection points KP and calculation regions
each having a center at a detection point. At first there are defined
relatively large regions BE having a relatively small peak in correlation,
and then small regions SE having a large peak in correlation is set in the
regions BE. In this manner, the correlation can be calculated in an
accurate and prompt manner.
Now several embodiments of the correlation calculator realized by
electronic circuits will be explained.
Embodiment 1
FIG. 13 is a block diagram showing a first embodiment of the electronic
correlation calculator. The calculator comprises image memory 107, address
generator 108 and accumulation calculator 109. A calculation region for
the image memory 107 is defined by the address generator 108 and the
correlation is calculated by the accumulation calculator 109. The
calculated result is judged by a discriminator 110. At first a large
region BE is set by the address generator and if a calculated correlation
exceeds a predetermined threshold value, the correlation value and
corresponding address are sent to a controller 112.
The controller 112 controls under the correlation value | | |
