Methods and apparatus for registering CT-scan data to multiple fluoroscopic images

Claims

What is claimed is:

1. A method for registering two dimensional fluoroscopic images with a three dimensional model of a surgical tissue of interest, comprising steps of:

(a) generating a three dimensional model of a surgical tissue of interest using a plurality of images of the surgical tissue of interest;

(b) obtaining at least two, two dimensional fluoroscopic images representing at least two views of the surgical tissue of interest, the images containing markers for associating an image coordinate system with a surgical coordinate system;

(c) detecting the presence of first contours of the surgical tissue of interest in each of the at least two views;

(d) deriving bundles of lines in three dimensional space that pass through the detected first contours; and

(e) iteratively matching points in three dimensional space on second contours obtained from the three dimensional model with the bundles of lines in three dimensional space until the three dimensional model is registered within the surgical coordinate system to a predetermined level of accuracy.

2. A method as in claim 1, wherein the step of generating a three dimensional model includes preliminary steps of performing a plurality of scans through the surgical tissue of interest to generate a plurality of image slices, and processing the image slices to generate a three dimensional model of the surface of the surgical tissue of interest.

3. A method as in claim 1, wherein the step of obtaining at least two, two dimensional fluoroscopic images includes steps of positioning a calibration device within a field of view of a fluoroscopic imaging system, the calibration device containing radio-opaque markers having a predetermined shape, and processing the two dimensional images to locate edges of images of the markers.

4. A method as in claim 3, wherein the step of positioning includes a step of translating the calibration device through the field of view using a robot manipulator, the calibration device being translated along a first ruled surface and along a second ruled surface that is generally parallel to the first ruled surface, the first and second ruled surfaces each being generally perpendicular to an optical axis of the fluoroscopic imaging system.

5. A method as in claim 4, wherein the calibration device has a generally circular cylindrical rod-like shape having the radio-opaque markers spaced apart along a longitudinal axis of the calibration device.

6. A method as in claim 3, wherein the step of positioning includes a step of mounting the calibration device within a field of view of a camera that forms a portion of the fluoroscopic imaging system.

7. A method as in claim 4, wherein the calibration device has a first set of radio-opaque markers located in a first plane, and a second set of larger radio-opaque markers located in a second plane, and wherein the first plane is located further from a source of imaging radiation than the second plane and is generally parallel to the second plane, the first and second planes each being generally perpendicular to an optical axis of the fluoroscopic imaging system.

8. A method as in claim 1, wherein the step of iteratively matching includes steps of:

defining a distance between points on the surfaces of the model and a beam of lines in three dimensional space that approach the surfaces; and

finding a pose of the surfaces that minimizes a distance to the lines from the points.

9. A method as in claim 1, wherein the step of iteratively matching includes a step of minimizing a sum of positive measures of a Euclidean distance between each point and its corresponding line.

10. A method as in claim 9, wherein the step of minimizing employs one of a non-linear Levenberg Marquardt minimizer, or a Linear minimizer followed by constrained minimization, using a sum of squared distances between points and lines.

11. A method as in claim 9, wherein the step of minimizing employs a statistically Robust minimizer that weights each squared distance between a point and a line in the sum with a function that favors distances that are similar to a median of the distances.

12. A method as in claim 1, wherein the first contours are two dimensional contours and the second contours are apparent contours.

13. A method as in claim 1, wherein the surgical coordinate system is comprised of a robot coordinate system.

14. A robotically-assisted surgical system, comprising:

a robot having an effector that is controllably positioned within a robot coordinate system;

a first imaging device for obtaining two dimensional radiographic images a tissue of interest, the images containing markers obtained from a calibration device that is located within a field of view of the first imaging device and that is separate from the tissue of interest the markers being used for associating an image coordinate system with the robot coordinate system;

a second imaging device for obtaining images of slices through the tissue of interest;

a data processor having an input coupled to an output of the first and second imaging devices and an output coupled to the robot, said data processor generating a three dimensional model of the tissue of interest from the output of the second imaging device, said data processor further detecting a presence of two dimensional contours of the tissue of interest in each of at least two views of the tissue and deriving bundles of lines in three dimensional space that pass through the detected two dimensional contours; said data processor operating to iteratively match points in three dimensional space, that are located on three dimensional apparent contours that are associated with a surface of the three dimensional model, with the bundles of lines in three dimensional space until the three dimensional model is registered with the robot coordinate system to a predetermined level of accuracy, wherein said surface apparent contours are boundaries between visible and invisible surface triangles.

15. A robotically-assisted surgical system as in claim 14, wherein said calibration device is adapted for being robotically translated, external to a patient who comprises the tissue of interest, and within a field of view of said first imaging device, said calibration device having a rod-like shape and being comprised of a radiolucent material containing a plurality of radio-opaque markers that are spaced apart along a longitudinal axis of said calibration device.

16. A robotically-assisted surgical system as in claim 15, wherein said calibration device is robotically translated along a first plane and along a second plane that is generally parallel to the first plane, the first and second planes each being generally perpendicular to an optical axis of the first imaging device, wherein said radio-opaque markers are each sized so as to be capable of producing an image when said calibration device is translated within the field of view of the first imaging device.

17. A robotically-assisted surgical system as in claim 14, wherein said calibration device is adapted for being positioned within a field of view of said first imaging device and external to a patient who comprises the tissue of interest, said calibration device being comprised of a radiolucent body portion containing a first set of radio-opaque markers located in a first plane and a second set of larger radio-opaque markers located in a second plane, and wherein the first plane is located further from a source of imaging radiation of the first imaging device than the second plane and is generally parallel to the second plane, the first and second planes each being generally perpendicular to an optical axis of the first imaging device, said calibration device further comprising a plurality a robot reference locations on a surface thereof.

18. A system for registering two dimensional fluoroscopic images with a three dimensional model of a tissue of interest, comprising:

a model processor for generating a three dimensional model of a tissue of interest using as input a plurality of images taken through the tissue of interest;

an imaging system for obtaining, in cooperation with a calibration device, at least two, two dimensional images representing at least two views of the tissue of interest, the images containing calibration markers for associating an image coordinate system with a second coordinate system; and

an image processor having inputs coupled to outputs of said model processor and said imaging system for detecting the presence of first contours of the tissue of interest in each of the at least two views, said image processor operating to derive bundles of lines in three dimensional space that pass through the detected first contours, and further operating to match points on second contours obtained from the three dimensional model with the bundles of lines in three dimensional space until the three dimensional model is registered within the second coordinate system to a predetermined degree of accuracy.

19. A system as in claim 18, wherein said model processor uses a plurality of image slices obtained from a plurality of scans through the tissue of interest, and processes the image slices to generate a three dimensional model of the surface of the tissue of interest.

20. A system as in claim 18, wherein said system further comprises a robot manipulator for translating said calibration device within a field of view of said imaging system, said calibration device comprising said calibration markers, and wherein said image processor processes the two dimensional images to locate images of said calibration markers within said two dimensional images.

21. A system as in claim 18, wherein said system further comprises a robot manipulator for translating said calibration device within a field of view of said imaging system, said calibration device comprising said calibration markers, wherein said robot manipulator is controlled for translating said calibration device within a first ruled surface and within a second ruled surface that is spaced apart from the first ruled surface.

22. A system as in claim 20, wherein a composite image of said calibration markers is comprised of a plurality of images of said calibration device located at a plurality of different locations within the field of view of said imaging system.

23. A system as in claim 18, wherein said calibration device has a generally circular cylindrical rod-like shape, and wherein said calibration markers are comprised of radio-opaque material and are spaced apart from one another along a longitudinal axis of said calibration device.

24. A system as in claim 18, wherein said calibration markers are comprised of radio-opaque material, wherein said calibration device comprises a first set of said calibration markers that are disposed in a first plane and a second set of larger calibration markers that are located in a second plane, wherein said first plane is located closer to a source of imaging radiation of said imaging system than said second plane and is generally parallel to said second plane, and wherein said first plane and said second plane are each generally perpendicular to an optical axis of said imaging system.

25. A system as in claim 18, wherein the second coordinate system is comprised of a robot coordinate system.

26. A system as in claim 18, wherein the tissue of interest is comprised of bone, and wherein said imaging system comprises means for obtaining images of bone through overlying tissue.

27. A method for generating a three dimensional model of a tissue of interest and for aligning the three dimensional model with two dimensional images of the tissue of interest, comprising steps of:

performing a plurality of imaging scans through the tissue of interest to generate a plurality of two dimensional image slices;

processing the image slices to generate a three dimensional model of a surface of the tissue of interest;

obtaining a plurality of two dimensional images representing at least two views of the tissue of interest, the at least two views also comprising images of calibration markers for associating an image coordinate system with a second coordinate system;

detecting first contours of the tissue of interest in each of the at least two views;

deriving bundles of lines in three dimensional space that pass through the detected first contours; and

matching points on second contours obtained from the surface of the three dimensional model with the bundles of lines until the three dimensional model is aligned within the second coordinate system to a predetermined level of accuracy.

28. A method as in claim 27, wherein the step of processing the image slices to generate a three dimensional model is comprised of the steps of:

extracting outer contours of the tissue of interest for each image slice to obtain at least one and typically a plurality of polygonal curves representing contours of the tissue of interest.

replacing the polygonal curves with approximating polygonal curves containing fewer vertices, such that the approximating polygonal curves do not deviate from the original polygonal curves by more than a pre-specified threshold, wherein each polygonal curve is represented using an ordered array of vertices, and each vertex is indexed with its position in the array;

building a surface model comprised of triangles that contains every vertex and every edge of the approximating polygonal curves, the step of building a surface model examining in turn pairs of consecutive image slices and constructing a surface slab that contains the approximating polygonal curves extracted from both image slices; and

combining the surface slabs to form a resulting surface by removing duplicate references to vertices belonging to polygonal curves shared by any two slabs.

29. A method as in claim 28, and further comprising a step of:

approximating the resulting surface with a surface containing fewer triangles by the use of a surface simplification technique.

30. A method as in claim 28, wherein the step of extracting comprises steps of:

selecting a plurality of points within an image slice in the vicinity of a structure of interest;

constructing a polygonal curve linking the selected points; and

modifying the polygonal curve to minimize an expression that combines a measure of curve length, a measure of average curvature of the curve, a measure of an image potential average curvature of the curve, and a measure of an image potential.

31. A method as in claim 30, wherein the image potential equals the squared norm of the image gradient, where if I(x,y) is the image intensity of a pixel of location (x,y), the image gradient grad (I) is a vector whose first coordinate is the derivative of I with respect to x, and whose second coordinate is the derivative of I with respect to y.

32. A method as in claim 28, wherein the step of replacing comprises steps of:

for each polygonal curve, iterating the following steps (a)-(c) until a maximum deviation threshold is respected,

(a) computing the maximum deviation between any vertex and a line segment obtained by joining the last vertex with the first vertex;

(b) if the maximum deviation computed is larger than the threshold, splitting the ordered array of vertices into two arrays; and

(c) considering two polygonal curves defined with the resulting split arrays.

33. A method as in claim 28, wherein the second contours are surface apparent contours defined as boundaries between visible and invisible surface triangles, wherein a visible triangle is one that a ray cast from a focal point of the lines to the triangle centroid makes an obtuse angle with the triangle normal direction, wherein surface apparent contours can be constructed as a set of closed, oriented, non-planar polygonal curves.

34. A method as in claim 27, wherein multiple images of calibration markers are superimposed to form a calibration marker grid, the images being obtained for a plurality of different locations of a calibration device within a field of view of a tissue imaging system.

35. A method as in claim 27, wherein images of calibration markers are processed using Thin Plate Spline functions to determine lines in three dimensional space representing a path of imaging radiation.

FIELD OF THE INVENTION

This invention relates generally to robotics and medical imaging techniques and, in particular, to methods for registering a robot to a planned trajectory for assisting in surgical procedures.

BACKGROUND OF THE INVENTION

Computers are increasingly used to plan complex surgeries by analyzing preoperative Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) scans of a patient. In order to execute the surgical plan, it is important to accurately align or register the three dimensional preoperative data to an actual location of the anatomical features of interest during surgery.

One conventional technique for performing this type of registration is to attach a stereo-tactic frame or fiducial markers to the patient, and to precisely locate the frame or markers both prior to and during surgery.

For example, a conventional registration protocol includes implanting three metallic markers or pins in a patient's femur, one proximally in the trochanter and two distally in the condyles, near the knee. The insertion of the pins requires minor surgery. A CT-scan image of the patient is subsequently acquired. By analyzing the CT data, the surgeon decides upon the size and location of the implant that best fits the patient's anatomy. During surgery, the metallic pins are exposed at the hip and knee. The patient's leg is attached to a surgical robot device that must then locate the exposed pins. A registration, or coordinate transformation from CT space to robot space, is computed using the locations of the three pins as a Cartesian frame. The accuracy of this registration has been measured to be better than one millimeter. However, the use of such pins as markers is not always desirable, as they may cause significant patient discomfort, and the required surgical procedure to insert and subsequently remove the pins results in inconvenience and additional cost to the patient.

An alternative technique is to perform anatomy-based registration that uses anatomical features of the patient, generally bony features, as markers for registration.

Conventional methods for anatomy-based registration of three dimensional volume data to projection data include three techniques, described by Lavallee et al. in "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps", proceedings of Geometric Methods in Computer Vision, SPIE vol. 1570, pages 322-336, 1991; by Lee in a PhD Thesis on "Stereo Matching of Skull Landmarks", from Stanford University in 1991; and by Feldmar et al. in Technical Report No. 2434, "3D-2D projective registration of free-form curves and surfaces" from INRIA, Sophia Antipolis, 1994.

In the approach of Lavallee et al., "Matching 3-D smooth surfaces with their 2-D projections using 3-D distance maps", calibrated video images are used to register a model of a vertebra to its projections. A hierarchical volume is built that is used to query the closest point from anatomical surfaces to projection lines. Also defined is a negative distance to address the situation of lines intersecting the surface.

In the approach described by Lee, "Stereo Matching of Skull Landmarks", stereo pairs of radiographs are used to track in real time the position of a patient's skull during radiotherapy delivery. Localized bony features that are segmented from a CT-scan are employed for this purpose.

In the approach described by Feldmar et al., "3D-2D projective registration of free-form curves and surfaces", surfaces are registered to projected contours. This is accomplished by defining image-to-surface correspondences and by minimizing a least squares criterion using iterative methods. The criterion incorporates contour and surface normals. This method accommodates errors in calibration by allowing optimization of the camera parameters.

Conventional methods for performing geometric calibration of images include a two-plane method described by Martin in "Camera Models Based on Data from Two Calibration Planes", published in Computer Graphics and Image Processing, 1981, volume 17, pages 173-180, and a method described by Champleboux et al. in "Accurate Calibration of Cameras and Range Imaging Sensors: The NPBS Method" published in ICRA Conference Proceedings, 1992, pages 1552-1557.

The calibration of distortion-free radiographs was investigated by Brown, "Registration of planar film radiographs with computed tomography", in Mathematical Methods in Biomedical Image Analysis, pages 42-51, San Francisco, Calif., June 1996, IEEE.

Most of the above described techniques, with the notable exception of Lee's, uses high quality three dimensional and projection images for experiments, such as high resolution CT scans of dry bones, or simulations of radiographs using video images. However, such high quality data is typically only available in a controlled laboratory test, and is superior to the data that would normally be clinically available. For example, typical CT slices also show soft tissue, present notable artifacts, and are of wide and unequal spacing to minimize the x-ray doses delivered to the patient. A precise segmentation of such data presents a very challenging problem. Furthermore, most fluoroscopic images that are obtained with commonly available clinical devices are characterized by a narrow field of view (FOV), typically with a maximum FOV of 100 mm, and include significant noise and distortion.

As such, there exists a need to provide an improved system and method for accomplishing an anatomy-based registration of three-dimensional data (model data) obtained from a scan, such as a CT scan or an MRI scan, to two dimensional projection data, such as x-ray data, enabling the registration of a surgical robot to a preoperative treatment plan.

OBJECTS AND ADVANTAGES OF THE INVENTION

It is thus a first object and advantage of this invention to provide a system and method for anatomy based registration of a three-dimensional CT-scan to two dimensional x-ray projection data.

It is a further object and advantage of this invention to provide a system and method for anatomy based registration of a three-dimensional CT-scan to two dimensional x-ray projection data, enabling the registration of a surgical robot to a preoperative treatment plan.

A further object and advantage of this invention is to provide a method for geometrically calibrating x-ray projection images using a calibration device that includes radio-opaque markers, wherein in one embodiment the calibration device is manipulated by the robot.

Another object and advantage of this invention is to provide a method for processing a three dimensional model and a set of inverse projection three dimensional lines to reduce the complexity of the task of registering the model to the lines by making the task a succession of sub-tasks of registering points to lines.

Another object and advantage of this invention is to provide an efficient, reliable, and clinically viable method for registering a set of three dimensional points to a set of three dimensional lines.

SUMMARY OF THE INVENTION

The foregoing and other problems are overcome and the objects and advantages are realized by methods and apparatus in accordance with embodiments of this invention.

In accordance with the teachings of this invention, and in one embodiment, an image calibration device is attached to a robot end effector. A conventional fluoroscope and video image digitization system and software are used to acquire fluoroscopic images while the robot manipulates the calibration device along multiple surfaces (not necessarily planar), approximately perpendicular to a fluoroscope optical axis. A plurality of radio-opaque mark