 |
Description  |
|
|
BACKGROUND OF THE INVENTION
1. Field of the Invention
This invention relates to the field of endoscopic surgery. More
specifically the invention relates to obtaining accurate positional
information about an anatomical structure within a patient's body and
using this information to accurately position endoscopic cameras and
surgical instruments within the patient's body.
2. Description of the Prior Art
Systems have been developed to augment a human surgeon's ability to perform
surgery on a patient by providing the surgeon with intraoperative images
of anatomical structures within the patient's body. Typically, these
systems comprise a surgeon specialized form of camera or medical
telescope. Further, a class of these systems, which includes endoscopic
and laparoscopic instruments, has reduced the invasive nature of many
surgical procedures.
This class of systems has two salient characteristics in common: First, the
surgeon using the system cannot directly manipulate the patient's anatomy
with his fingers, and second, the surgeon cannot directly observe what he
is doing. Instead, the surgeon must rely on instruments that can be
inserted through a trocar or through a working channel of an endoscope.
Often, since his hands and attention are fully occupied in performing the
procedure, the surgeon must rely on an assistant to point the endoscopic
camera while the surgery is performed.
To ameliorate the awkwardness of this arrangement, robotic augmentation
devices have been developed for endoscopic surgery. One such device is
described in detail in a copending application entitled "System and Method
for Augmentation of Surgery" Ser. No. 07/714,816 filed Jun. 13, 1991 which
is herein incorporated by reference.
Robotic augmentation devices can potentially greatly assist surgeons during
an operation. Robotic devices do not fatigue. Potentially, they can
position medical telescopes and surgical instruments very accurately and
can perform precise repositioning and repetitive functions. However, in
order for these advantages to be realized, a number of problems need to be
solved. The surgeon still needs to determine what motions the robotic
device is to make and requires a means to communicate with the computer
controlling the robot. In a few cases, such as orthopaedic machining of
bone or pre-planned excision of a tissue volume determined from
preoperative medical images (such as CT or MRI scans), these motions may
be pre-planned. However, in other cases, the surgeon needs to directly
observe the patient's anatomy and interactively specify the motions to be
made relative to anatomical features and the medical telescopes. In these
cases, means of accurately locating anatomical features and instruments
relative to the medical telescopes and to each other and of using this
information to control the robotic augmentation aids are necessary.
A specialized robotic device for stepping a resectoscope through a
preprogrammed sequence of cuts in thranurethral prostatectomies has been
developed. However, this system does not address the problem of providing
the surgeon with a convenient means of controlling the view available
through an endoscopic device or of providing the surgeon with means of
interactively manipulating surgical instruments in response to
intraoperative imaging and other sensory information.
There has been one attempt to provide voice control of a flexible endoscope
in which servomotors attached directly to the control knobs of a
commercial flexible endoscope were activated in response to voice commands
by the surgeon. Difficulties of this approach include: (a) the surgeon (or
an assistant) must still determine which direction to deflect the
endoscope tip to provide a desired view and, consequently, must keep track
of the relationship between the endoscope tip and the anatomical
structures being observed; (b) these corrections must be made continually,
distracting the surgeon from more important matters; and (c) the use of
voice commands for this purpose is subject to errors, potentially
distracting to the surgeon, and may make the use of voice for
communication between the surgeon and operating room personnel more
difficult. Several research efforts are directed to providing improved
mechanisms for flexible endoscopes. These devices do not, however,
simplify the surgeon's problem of controlling the endoscopic camera to
obtain a desired view, either by himself or by communicating with a
skilled operator.
3. Statement of Problems with the Prior Art
Unfortunately, the medical telescopes which are used in minimally invasive
surgery have limited fields of view. As a result, only a small part of the
anatomical Feature hidden inside the patient's body can be viewed at a one
time. Furthermore, surgical telescopes typically provide only a single
vantage point at any one time and it is difficult to provide the desired
view.
Normally, to compensate for this limited field of view, a surgical
assistant operates the telescope, reorienting it to produce many views of
the anatomical feature. While doing this, the assistant must continuously
keep track of the relative orientation between the telescope and the
patient's anatomy in order to be able to quickly and correctly aim the
telescope at the surgeon's request. He or she must also correctly
interpret the surgeon's desires, which are not always evident from the
surgeon's verbal comments.
This creates a number of problems. Surgical procedures of this nature now
require an additional highly-skilled person to assist the surgeon in
manipulating the medical telescope because the surgeon is using both of
his hands performing other tasks. The communication that is required
between the surgeon and the assistant increases the potential for an error
while performing the surgery. The surgeon (and assistant) have to develop
and keep a mental image of the entire hidden anatomical feature because
the telescope can not capture the full image of the feature. Many
telescopes, whether flexible or rigid, provide an oblique view, i.e., the
direction of view is not coincident with the main axis of the telescope.
This further exacerbates the difficulties of correctly aiming the
telescope to achieve a desired view and increases the likelihood that the
surgeon or the assistant could misconstrue the image presented or lose the
orientation of the telescope with respect to the anatomical feature. Human
fatigue contributes to a degradation of positioning of the telescope
and/or of the interpretation of the images that the telescope transmits.
Accordingly, there is a need for a way to obtain accurate and reliable
information about the position and appearance of anatomical features
hidden within a body. There also is a need for an apparatus to accurately
position and orient surgical instruments and/or medical telescopes within
a body and to provide accurate information about their position with
respect to hidden anatomical features. Further, there is a need to provide
a reliable and accurate interface between the surgeon and his surgical
instruments so that he can accurately position these instruments with
respect to an anatomical feature within a body without removing his hands
from his instruments.
OBJECTIVES
An objective of this invention is to provide an improved method to obtain
and display accurate information about the position of an anatomical
feature within a patient's body.
Also an objective of this invention is to provide an improved method of
positioning endoscopic cameras and other surgical instruments within a
patient's body.
A further objective of this invention is to provide an interface for a
surgeon to accurately position an endoscopic camera and/or other surgical
instruments within a patient's body without removing his hands from the
instrument.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a schematic view of a system used for computer augmentation of
surgical procedures.
FIG. 2 is a detail of FIG. 1 showing a distal fine motion rotational
manipulator.
FIG. 3 shows an embodiment of the invention using a stereoscopic
visualization system.
FIG. 4 shows an embodiment of the present invention comprising two robotic
manipulators.
FIG. 5 shows positions in 2D and 3D Cartesian coordinate systems.
FIG. 6 shows the pin-hole mathematical model of a camera.
FIG. 7 shows a method of computing a position in three dimensions using two
nonparallel camera vantage points.
FIG. 8 shows the use of passive visual targets to determine a position of a
surgical instrument.
FIG. 9 shows a method of computing a position in three dimensions using two
parallel camera vantage points.
FIG. 10 shows a method of using oblique medical telescopes.
SUMMARY OF THE INVENTION
The present invention is a method and apparatus for determining positional
information about an object and then using this information to position
instruments in relation to the object. The invention has many applications
but is particularly useful when the object is hidden from view or in a
location that is difficult to access. One preferred embodiment, used in
endoscopic surgery, determines positional information about a designated
anatomical feature which is hidden within a patient's body. The
information is used to position surgical instruments in the body with
respect to the anatomical feature.
The invention first positions an instrument, e.g. a surgical instrument
inserted inside a patient's body, at a desired position relative to a
designated object (anatomical feature). The instrument is capable of
transmitting an image of the object to a computer which then determines
positional information about the object by using various types of image
processing. The information is then related to a human (e.g., a surgeon)
or to a computer controlling a robotic apparatus. The positional
information is used to position or reposition the transmitting instrument
and/or other instruments relative to the designated object.
To further facilitate use of the invention, a number of different output
modes For conveying information from the imaging instruments and computer
to humans in the operating room are provided.
To further facilitate use of the invention, input devices are incorporated
on the inserted instruments so that a human user can input requests to the
system while concurrently manipulating the instrument. Other methods of
inputting requests to the system, such as voice recognition systems, are
also incorporated so that communications with the system does not
interfere with instrument manipulation.
DETAILED DESCRIPTION OF THE INVENTION
Referring to FIG. 1, there is shown a schematic view of a system for use in
computer augmentation of laparoscopic or similar procedures. The system
generally comprises a manipulator apparatus or robot 242, a computer 243,
a drive motor interface 244, a monoscopic monitor 247 with a suitable
image processor 245 and graphics adaptor 246, a stereoscopic monitor 272
with suitable stereo display system 271, and a terminal 248 for connecting
additional input devices to computer 243.
A manipulator similar to the manipulator 242, used in this preferred
embodiment, is described in detail in the copending U.S. application Ser.
No. 07/714,816 filed on Jun. 13, 1991.
Referring to FIGS. 1 and 2, the manipulator 242 comprises a proximal
rectilinear manipulator 6 and a remote center-of-motion distal manipulator
240. The proximal manipulator 6 comprises three mutually orthogonal
sliding motion sections 1, 2, and 3, which provide motion in the X, Y, and
Z directions. Sections 1, 2, and 3 are equipped with computer-controlled
motorized drives 4 connected to motion interface 244 and also have manual
locking clamps 5. The remote center-of-motion distal manipulator 240
comprises rotational sections 7, 250, 251, and 252 to provide
.theta..sub.p .theta..sub.x, .theta..sub.y, and distal .theta..sub.z
rotational motion, and a slide motor 253 adapted to axially slide
instrument 254. These sections are equipped with computer-controlled
motorized drives 249 interfaced to motor interface 244 and have manual
locking clamps 255. Each of the moving sections of manipulator 242 can be
actuated either manually or under computer control and can optionally be
locked by a manual locking device. All the motorized drives 4 and 249 are
controlled by computer 243 through motor interface 244.
Referring to FIG. 2, there is shown a schematic view of the distal fine
motion rotational manipulator 240 with an instrument 241 inserted through
an incision into a patient's body. In the embodiment shown, the distal
manipulator 240 provides a five degree-of-freedom (.theta..sub.p,
.theta..sub.x, .theta..sub.y, .theta..sub.z, and d) remote
center-of-motion wrist, which is supported by the aforementioned proximal
positioning system with three orthogonal linear degrees of freedom (X, Y,
and Z). The proximal linear degrees of freedom are used to place the
center-of-motion M of the remote center-of-motion wrist at the position of
insertion into the patient's body P. Any alternative mechanical structure
(such as a SCARA manipulator, manufactured and sold by IBM) with
sufficient degrees of freedom could be substituted for this purpose.
The four distal revolute degrees of freedom and the sliding degree of
freedom of manipulator 240 give the surgeon a five degree-of-freedom
spherical work volume centered at the insertion point M. These degrees of
freedom may be selectively locked or moved independently (manually or
under computer control) to assist the surgeon in achieving a desired
precise alignment. Furthermore, small motions within the work volume can
be achieved with only small motions of the individual axes. The point M
(i.e., the point at which the surgical instrument enters the patient)
remains unaffected by any motions of the distal manipulator 240. Thus the
manipulator may be moved through its work volume without requiring that
the patient position be moved or that the size of the entry wound be
enlarged.
One consequence of this design is that motion of the proximal manipulator 6
is not needed unless the patient is moved. Consequently, in a preferred
embodiment, the motion of proximal manipulator 6 is disabled by manual
locking and/or disabling of drive motors whenever an instrument is
inserted into the patient. In this mode, the control computer 243
interprets commands requesting motion of manipulator 242 as follows. When
a motion is requested, the control computer 243 attempts to satisfy the
request by moving only distal manipulator 240. If the motion can be
accomplished in more than one way, the computer selects the motion that
minimizes the motion of the most proximal revolute motion section 7 (i.e.,
it minimizes motion of .theta..sub.p). If the motion cannot be
accomplished perfectly, the computer selects the motion of distal
manipulator 240 that most closely approximates the desired motion. Modes
are available to select minimization of positional error of the tip of
instrument 241, orientation error, or weighted combinations thereof. If
the error is greater than a prespecified threshold amount, the control
computer notifies the surgeon using synthesized speech, an audible alarm,
or other means, and makes no motion unless the surgeon explicitly
instructs it to proceed, using voice recognition or other input modality.
One alternative embodiment might seek always to minimize the total motion
of the distal manipulator 240, again forbidding motion of proximal
manipulator 6 whenever a surgical instrument held by the distal
manipulator is inserted into the patient's body. Yet another might permit
small motions of the proximal manipulator, so long as the center-of-motion
M stays within a specified threshold distance (e.g., 3 mm) of the original
value.
If desired, a flexible tip may be added to the distal end of instrument 241
to provide additional degrees of freedom. In the case where a viewing
instrument such as instrument 254 is used, an additional degree-of-freedom
in adjusting the gaze direction may be provided by adding an
adjustable-angle mirror or prism to the distal end of the instrument.
Referring again to FIG. 1, the instrument 254, in the embodiment shown,
includes a video camera 259 and a light source 277 connected to the
instrument via a fiberoptic cable 278. The video output of the camera 259
is fed into the graphics adaptor 246, where it may be optionally mixed
with graphics output from computer 243 and displayed on monitor 247. The
video output from the camera is also optionally fed into the image
processing system 245, which analyzes the image produced by the camera and
provides information to computer 243 about the relative position of the
surgeon's instruments, the camera, and the patient's anatomy. The video
information from the camera may be also optionally supplied to the stereo
display system 271, which can assemble a stereoscopic view of the
patient's anatomy from two or more images taken from different vantage
points and display the image on the stereoscopic monitor 272.
In one preferred embodiment, the stereo display system is a STEREOGRAPHICS
CRYSTALEYES (trademark of StereoGraphics, Inc.) system, where the two
video signals are displayed on a stereoscopic monitor which alternatively
displays the left and right eye image at a frequency of 120 Hz, updating
the video information for each eye 60 times per second. The surgeon wears
stereoscopic liquid crystal (LC) goggles 273, which are synchronized with
the monitor and alternatively block light from entering left and right eye
such that the left eye receives only the video signal from the left camera
and the right eye receives only the information from the right camera. The
frequency of alternation between left and right images is sufficiently
high such that the surgeon perceives no flicker but rather a continuous
stereoscopic image of the patient's anatomy. Other stereo display
technologies are available and may be used.
In the embodiment shown, the surgeon is using a second surgical instrument
260 inside the patient's body, which has passive visual targets 276 placed
on it. These targets 276 are markings on the instrument and are chosen so
as to be easily locatable by the image processing system 245 in the images
supplied by the camera 259.
The set of input/output devices attached to input/output interface 248 of
computer 243 shown in FIG. 1 may include a computer voice recognition and
synthesis system 267, a joystick 268 mounted on the surgical instrument
260 and a sterilized touch screen 269 mounted on monitor 247. In the
preferred embodiment the joystick is a small device, functionally
identical to a 2D or 3D mouse, but designed such that it can be mounted
directly onto a surgical instrument and such that at least two degrees of
freedom of motion can be specified by applying pressure on a small
joystick protruding from the device. One implementation of such a device
uses strain gauges to translate an applied pressure or force into
incremental displacement or velocity information. In another embodiment, a
six degree-of-freedom input device, such as SPACEBALL (A Trademark owned
by Spaceball Technologies, Inc.) could be used to specify motion in any of
the six degrees of freedom. Such a device could be mounted on a surgical
instrument, on the manipulator structure, or at any other convenient
point. One advantage of mounting an input device such as a small joystick
on a surgical instrument is that the surgeon can easily manipulate the
joystick without removing his hands from the surgical instrument, thus
permitting him to provide information to the computer (for example, of a
desired direction of motion of a medical telescope) without interrupting
his work.
The speech recognition and synthesis system 267 includes means of inputting
information to the system, such as a (possibly head mounted) microphone
275, and a means of conveying information to the surgeon, such as a
speaker 274. The speech recognition system 267 is capable of understanding
a vocabulary of instructions spoken by the surgeon and can relate the
information about the commands it has received to the computer 243. The
surgeon may use any of these modalities, either separately or in
combination, to position graphic objects on the monitor 247, to select
commands or operating modes from menus, and to command motions of the
manipulator 242.
Referring to FIG. 3, there is shown an alternative embodiment of the system
for computer augmentation of laparoscopic or similar surgical procedures.
In this embodiment, the surgical instrument 254a is a stereoscopic medical
camera, which incorporates two independent lens systems or optical fibers
and is capable of transmitting two simultaneous images from the patient's
body. The two lenses are separated by a small (known) distance and are
thus able to provide a stereoscopic image. One embodiment of such a device
would comprise two side-by-side fiberoptic bundles or lens systems and one
fiberoptic light channel. The assembly would be surrounded by a suitable
cylindrical casing. The video signals from the two cameras 259a and 259b
are fed into the stereo display system 271 and displayed to the surgeon on
a stereoscopic display monitor 272. Using interface hardware known in the
art, both video signals are also optionally supplied to the image
processing system 245 and the graphics adapter 246.
Another embodiment of the system is shown in FIG. 4, where the system
comprises two manipulators 240a and 240b, carrying surgical instruments
241a and 241b, respectively. In one embodiment, one of the surgical
instruments is a medical telescope, whereas the other instrument is a
surgical tool, such as medical forceps. Since both instruments are
attached to robotic devices, both can be actively positioned under
computer control. On the other hand, as with the single manipulator arm in
the case above, either or both robots can be controlled manually by
releasing, adjusting, and relocking joint axes one at a time. In an
alternative embodiment, both surgical instruments 241a and 241b comprise
medical telescopes or other means of transmitting an image outside of a
patient's body. In such an embodiment, one of the instruments (for
example, 241a) may also comprise a surgical tool such as a miniaturized
surgical forceps. In this case, information from images taken at two
vantage points may be combined to provide precise 3D information to assist
in placing the surgical instrument precisely on the desired portion of the
patient's anatomy.
Referring again to FIG. 1, the image processing system 245 may be used to
locate features on the patient's anatomy of interest to the surgeon. In
this mode, the surgeon would designate a feature of interest by any of a
number of means to be explained below. On the surgeon's command, supplied
via any appropriate input device attached to the terminal 248, the
computer 243 would instruct the image processing system 245 to acquire an
image and precisely locate the designated anatomical feature. In one
embodiment, a reference image of the designated Feature would be acquired
in response to the surgeon's command and stored. Image correlation
techniques would be used to locate the feature during surgery. In an
alternative embodiment, synthetic reference images could be generated from
computer reconstructions of preoperative medical images and models. Once a
feature has been located, the manipulator 242 can be moved to place the
feature at any desired position in the camera field of view. If desired,
an additional image may be acquired, the feature re-located, and a further
adjustment made to refine the desired placement of the camera. This
process may be repeated a number of times to "zero in" on a feature to any
desired accuracy. Each of the foregoing steps is explained below.
As a matter of nomenclature, we will in the following text refer to
positional information in a number of ways. Unless otherwise specified,
the terms "position" and "location" will be used interchangeably. We will
be referring to two-dimensional (2D) and three-dimensional (3D) positions.
When referring to an image obtained by a single monoscopic camera, an
"image location" or "image position" should be understood as a 2D location
within the 2D image. Referring to FIG. 5a, such a location A (within a 2D
image 800) is given as a pair of coordinates (x,t) When the image is
stereoscopic, "image location" or "image position" should be understood as
a 3D location within the volume of the stereoscopic image. Referring to
FIG. 5b, such a location B is described by a triple of coordinates
(x,y,z). We will also refer to positions of anatomical features. Such
features are part of the patient's anatomy and all references to "feature
location" or "feature position" should be understood as 3D positional
information about the feature in question.
In order to use and manipulate images of the patient's anatomy, images must
first be acquired. Referring to FIG. 1, this is done by feeding the live
video signal from camera 259 into the image processing system 245
comprising at least one video digitizer. A video digitizer is a device
capable of converting an analog video signal into a digital signal, which
can be stored in computer memory and arbitrarily modified by the computer.
Conversely, a video digitizer can also convert a digitized (and possibly
modified) video signal back into analog form for display on a standard
monitor.
If positional information is to be extracted from images obtained by a
camera/lens system, a mathematical model of the camera and the lens must
be available to relate image points (i.e., points on the camera's imaging
plane) to the corresponding world points (i.e., 3D locations in the actual
environment). To a good approximation, a perfect camera/lens system can be
modeled as a pin-hole system, illustrated in FIG. 6. The figure depicts a
camera with a lens 600 positioned a distance f in front of the image plane
601. The quantity f is referred to as the focal length of the lens. A
point W=(x,y,z) lying in the plane 602 a distance d=-z in front of the
lens is imaged onto the image plane 601 tit the location C=(x',y'), where
x/d=x'/f and y/d=y'/f.
Given the image coordinates (x',y') of a world point, the above
relationships constitute two equations in three unknowns (x, y, and z) and
are thus not sufficient to recover the 3D coordinates of the corresponding
world point, W. Referring to FIG. 7, the information obtained from n
single image 601a from a first vantage point 600a defines a ray 605a in 3D
space originating at the image point C.sub.a, passing through the lens
center 600a, and extending to infinity. By definition, the actual world
point W lies somewhere on this ray, but additional information is needed
to determine its exact location. If a second image 601b, taken from a
second vantage point 600b (whose position and orientation with respect to
the first vantage point is known), is available, then the corresponding
image point C.sub.b in the second image and the location of the second
vantage point 600b define a second ray 605b in space, such that the world
point W lies on this ray a well. Using known mathematical techniques, the
two rays can be resolved in the same coordinate system and their
intersection can be computed, giving the 3D world coordinates (x,y,z) of
the point W.
Most camera lenses introduce distortions which causes the correspondence of
world and image points to depart from the above pin-hole model. The
process of calibrating the camera/lens system can estimate the nature and
amount of such distortions and the resulting mathematical model can be
used to effectively "undistort" the image points. The pin-hole camera
model can then be applied to the undistorted image. A number of techniques
for calibrating camera/lens systems are known.
As part of the interaction with a two-dimensional image of the patient's
anatomy displayed to the surgeon on a conventional monitor, the surgeon
may wish to designate (i.e., point to) a particular image location within
the displayed image. The surgeon may point to a particular image location
by using any of the following means: (a) by positioning a surgical
instrument equipped with a distinct and clearly visible visual target so
that the image of the visual target on the display coincides with the
desired image location, (b) by manipulating a graphical object on the
screen using an input device mounted on a surgical instrument (such as
joystick 268 in FIGS. 1 and 3 or a similar device), or (c) by manipulating
a graphical object on the screen using a conventional mouse. In method (a)
the visual target may consist of a brightly colored spot or a known
geometric pattern of such spots at a known position on the instrument
(e.g., pattern 276 in FIGS. 1 and 3). The use of a bright color, distinct
from any color naturally occurring inside the patient's body, greatly
simplifies the problem of locating artificial visual targets and lessens
the chances of erroneous location of such targets. Such spots on the
surgical instrument can be located using known image processing
techniques, involving thresholding (to isolate the spots from the rest of
the image) and computationally determining the centers of the so obtained
thresholded regions. In methods (b) and (c) the position of the feature of
interest is taken as the final position of the graphical object.
Once the 2D coordinates of an image location have been specified to
computer 243, the computer can confirm the location by marking the
location with a graphical object superimposed on the image. In one
embodiment of this method of confirming an image location to the surgeon,
2D cross-hair cursors or 2D box cursors can be used to show the location
of interest in the image. The "image", in this context, can be either a TV
camera image or a computer generated graphical rendition of the anatomical
area of interest.
We have so far described a variety of method for the surgeon to specify a
particular 2D location of interest in a monoscopic image. We next discuss
methods, such as image processing, to determine positional information
about three-dimensional anatomical features and/or surgical instruments in
the patient's body.
Referring to FIGS. 1 and 3, if a stereoscopic display (live or static) of
the patient's anatomy is available during the surgical procedure, then a
surgeon can designate the desired 3D anatomical feature of interest by
manipulating a 3D stereoscopic graphical object (cursor) on the
stereoscopic display 272 until the graphical object is coincident with the
desired anatomical feature. Any of the appropriate aforementioned input
devices and modalities 248 (such as the surgical tool mounted joystick or
trackball, voice, etc.) can be used to specify the desired motion of the
graphical object within the stereoscopic volume of the image.
If the actual physical size of a designated object is known, its distance
from the viewing instrument may be estimated from the size of its image,
as seen by the viewing instrument. Since we know that the feature lies on
a ray originating at the center of inn age of the feature and passing
through the vantage point as shown in FIG. 7, the position of the feature
relative to the viewing instrument may then be computed. Let the size of
the feature in the image be l, let the corresponding actual size of the
feature be s, and let f denote the focal length of the camera. The
distance z from the camera lens to the feature of interest can then be
computed as z=(f.times.s)/l.
Referring to FIG. 8, in one embodiment, where passive visual targets 701 on
a surgical instrument 700 are used, the position of a 3D feature (e.g., a
surgical instrument 700) can be determined as follows: At least three non
collinear circular spots 701 of known diameter s are marked on the
surgical instrument 700 (FIG. 8a). Since the surgical instrument may have
an arbitrary orientation with respect to the camera, these spots will in
general appear on the image plane as ellipses 705 (FIG. 8b). However, the
length of the major axis of each ellipse l will be the same as the
diameter of the circular image that would be seen if the corresponding
circular spot were presented at that same distance from the lens in such a
manner that the plane in which it lies is perpendicular to the view axis
of the camera. Let the length of the major axis of the observed ellipse as
it appears in the image be l (FIG. 8b). Then the distance of the spot from
the camera lens can be computed from z=(f.times.s)/l. Having performed
this computation for at least three spots and knowing the position of the
spot pattern with respect to the tip of the surgical instrument suffices
to compute the 3D location of the tip of the surgical instrument with
respect to the camera. Other techniques, known in the art, permit
calculation of the position and orientation, relative to the camera, of a
pattern of five dots from the 2D positions of their centroids in the image
obtained. Other patterns of dots or other visual targets can be used as
well. The 3D location of the tip of the instrument relative to the camera
may then be readily computed from the known position of the tip relative
to the visual target.
Additionally, stereo image processing may be used to precisely locate 3D
anatomical features. In one embodiment, image processing can be used in
conjunction with a stereoscopic camera to locate an anatomical feature.
Referring to FIG. 3, surgical instrument 254a is a stereoscopic medical
camera, comprising of two independent lens systems or optical fibers and
is capable of transmitting two simultaneous images from the patient's
body. The lenses are separated by a small (known) distance d, as shown in
FIG. 9. The 3D position of the anatomical feature relative to the camera
tip can be computed from the pin-hole camera model (FIG. 6). Specifically,
if the image plane locations of the center of the feature of interest in
the two images are denoted by f.sub.1 =(x.sub.1, Y.sub.1) and f.sub.2
=(x.sub.2, y.sub.2), as shown in FIG. 9, then the distance z of the
feature center from the camera lens can be computed as z=(f.times.d)/c,
where c=.sqroot.(x.sub.2 -x.sub.1).sup.2 +(y.sub.2 -y.sub.1).sup.2 and f
denotes the focal length of the camera. Image correlation techniques or
other image processing techniques known to the art may be used to locate
features in images.
Referring again to FIG. 9, in another embodiment, using only a monocular
camera, image processing techniques can be used to determine the position
of an anatomical feature in three dimensions as follows: A first image
601a of the anatomical feature is acquired and a reference representation
(such as a multi-resolution image pyramid representation known in the
image processing art) is stored. The manipulator 242 is used to displace
the camera lens tip 600a laterally by a known amount d, and a second image
601b is acquired. The center of the feature of interest W is located in
the second image, using the reference representation of the feature, by
means of correlation techniques (such as multi-resolution normalized
correlation methods known in the art) and the 3D displacement of the
anatomical feature from the camera tip may be computed as in the case
above. Specifically, if the image plane locations of the feature of
interest W in the two images 601a and 601b are denoted by f.sub.1
=(x.sub.1, y.sub.1) and f.sub.2 =(x.sub.2, y.sub.2), respectively, then
the distance z of the feature from the camera lens can be computed as
z=(f.times.d)/c, where c=.sqroot.x.sub.2 -x.sub.1).sup.2 +(y.sub.2
-y.sub.1).sup.2 and f denotes the focal length of the camera.
In another embodiment, the physical constraint of maintaining minimal
translational motion of the telescope with respect to the port of entry
into the patient's body may preclude laterally displacing the telescope to
obtain a second image, as described | | |
