System and method of registration of three-dimensional data sets including anatomical body data

What is claimed is:

1. An image data registration system comprising:

an image data storage unit for storing a first data set of three-dimensional image data associated with a predetermined portion of an anatomical body with reference to a first coordinate frame;

an image data acquisition and storage device for obtaining and storing a second data set of three-dimensional image data associated with a surface of said predetermined portion of said object with reference to a second coordinate frame, said image data acquisition and storage device comprises a surface sensing unit for sensing the outer skin surface of said predetermined portion of the body so as to obtain said second data set of three-dimensional image data; and

an image data processor for registering said first and second data sets to generate a matched image data set in which said first coordinate frame is relatively aligned with said second coordinate frame.

2. The system of claim 1, wherein said first data set of three-dimensional image data comprises three-dimensional anatomy image data.

3. The system of claim 1, wherein said image data processor is operable for selecting a predetermined subset of data points from said second data set.

4. The system of claim 3, wherein said image data processor is further operable for matching said subset of data points to all possible corresponding subsets of data points associated with said first data set which results in a set of hypothesized transformation data sets.

5. The system of claim 4, wherein said image data processor is further operable for sequentially mapping said first data set to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said first data set into the second coordinate frame associated with said second data set.

6. The system of claim 1 further comprising:

an imaging device for providing a real-time image of said predetermined portion of the object relative to a third coordinate frame; and

an image output device for producing an output image corresponding to a combination of said matched image data set and said real-time image.

7. The system of claim 6, wherein said image data processor is further operable for calibrating said real-time image with said matched data set so that said third coordinate frame is aligned with said second coordinate frame.

8. The system of claim 7, wherein said image data processor is further operable for compensating for movements associated with the object in order to maintain the alignment of said matched image data set and said real-time image in said output image.

9. The system of claim 7, wherein said image data processor is further operable for compensating for movements associated with said real-time imaging means in order to maintain the alignment of said matched image data set and said real-time image in said output image.

10. A surgical image registration system comprising:

a first storage device for storing three-dimensional anatomy image data associated with a predetermined portion of a patient's body with reference to a first coordinate frame;

a surface scanning unit for scanning the skin surface of said predetermined portion of the patient's body so as to obtain three-dimensional surface image data;

a second storage device for storing said three-dimensional surface image data with reference to a second coordinate frame; and

an image data processor operable for registering said anatomy image data and said surface image data to produce a matched image data set in which said first coordinate frame is relatively aligned with said second coordinate frame.

11. The system of claim 10 further comprising:

an imaging device for providing a real-time image of said predetermined portion of the patient's body relative to a third coordinate frame; and

an image output device for producing an output image corresponding to a combination of said matched image data set and said real-time image.

12. The system of claim 11, wherein said image data processor is further operable for calibrating said real-time image with said matched data set so that said third coordinate frame is aligned with said second coordinate frame.

13. The system of claim 12, wherein said image data processor is further operable for compensating for movements associated with the patient's body in order to maintain the alignment of said matched image data set and said real-time image in said output image.

14. The system of claim 12, wherein said image data processor is further operable for compensating for movements associated with said real-time imaging means in order to maintain the alignment of said matched image data set and said real-time image in said output image.

15. The system of claim 11, wherein said real-time imaging device comprises a real-time video camera.

16. The system of claim 11, wherein said image output device comprises a video display monitor.

17. The system of claim 11, wherein said image output device comprises a head-mounted display device.

18. The system of claim 11, wherein said image output device comprises a surgical microscope having a video projection overlay feed.

19. The system of claim 11, wherein said image output device comprises a transparent projection screen.

20. The system of claim 10, wherein said three-dimensional anatomy image data comprises prerecorded magnetic resonance imaging (MRI) data.

21. The system of claim 10 wherein said three-dimensional anatomy image data comprises prerecorded computed tomography (CT) data.

22. The system of claim 10, wherein said three-dimensional anatomy image data is segmented into tissue types.

23. The system of claim 10, wherein said surface scanning unit comprises a device for illuminating and reading a sequence of laser lines across the skin surface of said predetermined portion of the patient's body.

24. The system of claim 10, wherein said image data processor is operable for selecting a predetermined subset of data points from said surface image data.

25. The system of claim 24, wherein said image data processor is further operable for matching said subset of data points to all possible corresponding subsets of data points associated with said anatomy image data which results in a set of hypothesized transformation data sets.

26. The system of claim 25, wherein said image data processor is further operable for sequentially mapping said anatomy image data to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said model data into the second coordinate frame associated with said surface image data.

27. A method of registering three-dimensional image data sets in a surgical imaging system, comprising:

storing three-dimensional anatomy image data associated with a predetermined portion of a patient's body with reference to a first coordinate frame;

scanning the skin surface of said predetermined portion of the patient's body so as to obtain three-dimensional surface image data;

storing said three-dimensional surface image data with reference to a second coordinate frame;

registering said anatomy data and said surface image data to produce a matched image data set in which said first coordinate frame is relatively aligned with said second coordinate frame;

generating a real-time image of said predetermined portion of the patient's body relative to a third coordinate frame; and

producing an output image corresponding to a combination of said matched image data set and said real-time image.

28. The method of claim 27, wherein said scanning further comprises illuminating and reading a sequence of laser lines across the skin surface of said predetermined portion of the patient's body.

29. The method of claim 27, wherein said registering further comprises selecting a predetermined subset of data points from said surface image data.

30. The method of claim 29, wherein said registering further comprises matching said subset of data points to all possible corresponding subsets of data points associated with said anatomy image data and producing a set of hypothesized transformation data sets.

31. The method of claim 30, wherein said registering further comprises sequentially mapping said anatomy image data to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said anatomy image data into the second coordinate frame associated with said surface image data.

32. The method of claim 31, wherein said sequential mapping further comprises performing a least-squares minimization between said anatomy image data and said surface image data.

33. The method of claim 32, wherein said sequential mapping further comprises randomly perturbing said solution data set and repeating said least-squares minimization in order to avoid local minima and solve for a global minima.

34. The method of claim 27 further comprising calibrating said real-time image with said matched data set so that said third coordinate frame is aligned with said second coordinate frame.

35. The method of claim 34, wherein said calibrating further comprises compensating for movements associated with the patient's body in order to maintain the alignment of said matched image data set and said real-time image in said output image.

36. The method of claim 34, wherein said calibrating further comprises compensating for movements associated with said real-time imaging means in order to maintain the alignment of said matched image data set and said real-time image in said output image.

37. The method of claim 27, wherein said three-dimensional anatomy image data comprises prerecorded magnetic resonance imaging (MRI) data.

38. The method of claim 27, wherein said three-dimensional anatomy image data comprises prerecorded computed tomography (CT) data.

39. The method of claim 27, wherein said three-dimensional anatomy image data is segmented into tissue types.

40. A method of registering image data, comprising the steps of:

storing a first data set of three-dimensional image data associated with a predetermined portion of an object with reference to a first coordinate frame;

obtaining and storing a second data set of three-dimensional image data associated with a surface of said predetermined portion of said object with reference to a second coordinate frame; and

registering said first data set and said second data set to generate a matched image data set in which said first coordinate frame is relatively aligned with said second coordinate frame, wherein

said registering comprises:

generating a first data set of image data with reference to a first coordinate frame;

generating a second data set of image data with reference to a second coordinate frame;

selecting a predetermined subset of data points from said first data set;

matching said subset of data points to all possible corresponding subsets of data points associated with said second data set and producing a set of hypothesized transformation data sets; and

sequentially mapping said second data set to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said second data into the first coordinate frame associated with said first data set, wherein

said sequential mapping further comprises performing a least-squares minimization between said first and second data sets, and randomly perturbing said solution data set and repeating said least-squares minimization in order to avoid local minima and solve for a global minima.

41. A method of registering three-dimensional data sets, comprising:

generating a first data set of image data with reference to a first coordinate frame;

generating a second data set of image data with reference to a second coordinate frame;

selecting a predetermined subset of data points from said first data set;

matching said subset of data points to all possible corresponding subsets of data points associated with said second data set and producing a set of hypothesized transformation data sets; and

sequentially mapping said second data set to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said second data into the first coordinate frame associated with said first data set, wherein

said sequential mapping further comprises performing a least-squares minimization between said first and second data sets, and randomly perturbing said solution data set and repeating said least-squares minimization in order to avoid local minima and solve for a global minima.

42. A system for registering three-dimensional data sets, comprising:

means for generating a first data set of image data with reference to a first coordinate frame;

means for generating a second data set of image data with reference to a second coordinate frame;

means for selecting a predetermined subset of data points from said first data set;

means for matching said subset of data points to all possible corresponding subsets of data points associated with said second data set and producing a set of hypothesized transformation data sets; and

means for sequentially mapping said second data set to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said second data into the first coordinate frame associated with said first data set, wherein

said sequential mapping means further comprises means for performing a least-squares minimization between said first and second data sets, and means for randomly perturbing said solution data set and repeating said least-squares minimization in order to avoid local minima and solve for a global minima.

43. An image data registration system comprising:

an image data storage unit for storing a first data set of three-dimensional image data associated with a predetermined portion of an object with reference to a first coordinate frame;

an image data acquisition and storage device for obtaining and storing a second data set of three-dimensional image data associated with a surface of said predetermined portion of said object with reference to a second coordinate frame; and

an image data processor for registering said first and second data sets to generate a matched image data set in which said first coordinate frame is relatively aligned with said second coordinate frame, wherein said image data processor comprises:

means for generating a first data set of image data with reference to a first coordinate frame;

means for generating a second data set of image data with reference to a second coordinate frame;

means for selecting a predetermined subset of data points from said first data set;

means for matching said subset of data points to all possible corresponding subsets of data points associated with said second data set and producing a set of hypothesized transformation data sets; and

means for sequentially mapping said second data set to each of said hypothesized transformation data sets to form a solution data set, said solution data set representative of a highly accurate transformation of said second data into the first coordinate frame associated with said first data set, wherein

said sequential mapping means further comprises means for performing a least-squares minimization between said first and second data sets, and means for randomly perturbing said solution data set and repeating said least-squares minimization in order to avoid local minima and solve for a global minima.

Description Submit all comments and votes

BACKGROUND OF THE INVENTION

The invention relates in general to a system and method of image data registration, and more particularly to a system and method of registering three-dimensional surgical image data utilized in image guided surgery and frameless stereotaxy.

Neurosurgical procedures, such as biopsy or tumor extraction, require highly precise localization on the part of the surgeon, in order to attain the desired extraction of material while minimizing collateral damage to adjacent structures. The problem is exacerbated by the fact that the localization is three dimensional in nature, and often requires localizing a structure deeply buried within the cranium. While methods exist (e.g. MRI, CT) for imaging and displaying the 3D structure of the cortex, this still leaves the surgeon with the problem of relating what she sees on the 3D display with the actual anatomy of the patient.

Conventional solutions typically involve presurgically attaching a stereotactic frame to the patient's skull, then imaging the skull and frame as a unit. This allows the surgeon to locate, from the 3D images, the location of the tumor or other target relative to a coordinate system attached to the stereotactic frame, and thus to the patient's head. As well, the frame typically includes a movable armature that allows the positioning of a probe at any orientation relative to the patient. This lets the surgeon mark a planned angle of entry to access the tumor, thus localizing the expected extraction of material.

Unfortunately, the use of stereotactic frames is both cumbersome to the surgeon, and involves considerable discomfort to the patient, who must wear the device for several days between imaging and surgery. In addition, such frames can have limited flexibility, especially should surgical plans have to change in the middle of the procedure, e.g. if the line of attack is found to pass through critical regions, such as the motor strip.

Accordingly, it is an object of the present invention to provide an imaging method and system which registers sets of three-dimensional image data of an object.

It is another object of the present invention to provide a surgical imaging method and system which registers clinical data, such as segmented MRI or CT reconstructions, with surface data associated with a portion of a patient's body.

It is yet another object of the present invention to provide a surgical imaging method and system which generates real-time, adaptive, enhanced visualizations of the patient in the operating room so as to accommodate dynamic image-guided surgical planning and image guided surgical procedures, such as biopsies or minimally invasive therapeutic procedures.

SUMMARY OF THE INVENTION

The present invention in one embodiment provides an image data registration system and method of storing a first data set of three-dimensional image data associated with a predetermined portion of an object as model image data with reference to a first coordinate frame; obtaining and storing a second data set of three-dimensional surface image data associated with a surface of the predetermined portion of the object with reference to a second coordinate frame; and registering the first data set and the second data set to generate a matched image data set in which the first coordinate frame is aligned with the second coordinate frame.

In another embodiment of the present invention, there is provided a method of registering three-dimensional data sets, comprising generating a first data set of image data with reference to a first coordinate frame; generating a second data set of image data with reference to a second coordinate frame; selecting a predetermined subset of data points from the first data set;

matching the subset of data points to all possible corresponding subsets of data points associated with the second data set and producing a set of hypothesized transformation data sets; and sequentially mapping the second data set to each of the hypothesized transformation data sets to form a solution data set, the solution data set representative of a highly accurate transformation of the second data into the first coordinate frame associated with the first data set.

The present invention more particularly provides a surgical method and system which performs the registration of clinical sensory data with the corresponding position of the patient's body on the operating table at the time of surgery, using methods from visual object recognition, which do not require the use of a previously attached stereotactic frame. The method has been combined with an enhanced visualization technique, in which there is displayed a composite image of the 3D anatomical structures with a view of the patient's body.

This registration enables the transfer to the operating room of preoperative surgical plans, obtained through analysis of the segmented 3D preoperative data, where they can be graphically overlaid onto video images of the patient. Such transfer allows the surgeon to apply carefully considered surgical plans to the current situation and to mark landmarks used to guide the progression of the surgery. Extensions of the method and system include adaptively re-registering the video image of the patient to the 3D anatomical data, as the patient moves, or as the video source moves.

BRIEF DESCRIPTION OF THE DRAWINGS

The file of this patent contains at least one photograph executed in color. Copies of this patent with color photographs will be provided by the United States Patent and Trademark Office upon request and payment of the necessary fee.

FIG. 1 shows a block diagram of an image data registration system in accordance with the present invention;

FIG. 2 shows a flow outline of the data registration technique utilized in accordance with the present invention;

FIG. 3 shows exemplary results of registering laser data derived from laser scan lines of a head to the skin surface segmented from an MRI data set; and

FIG. 4 shows exemplary final results of the system of the present invention as a combined video/MRI visualization in which a surgeon is provided with an enhanced visualization view inside of the head of a patient.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

For purposes of illustration, an exemplary embodiment of the present invention will be described with reference to a craniotonomy procedure. It will be appreciated that the described image registration method and system of the present invention is not limited to use only with cranial structures, but in fact may be used for registering three-dimensional image data for sets of other objects including other portions of an anatomical body.

Accordingly, with reference to FIG. 1, there is shown an image data registration system 100 in accordance with the present invention. The system 100 operates in accordance with the following exemplary overview for cranial surgery. A patient requiring surgical therapy is initially scanned by a three-dimensional, high resolution, internal anatomy scanner, such as Magnetic Resonance Imaging (MRI) or Computed Tomography (CT). It will be appreciated that any form of volummetric imaging techniques, such as PET, SPECT, etc., can also be used. If a brain atlas is available, or if a previous, segmented scan of the patient is available, the current scan is registered to this prior scan, and the match is used to drive fast, accurate, automated segmentation of the current scan, and to identify changes since the reference scan was taken, to better identify the pathology. If no previous information is available, then the current scan is segmented to produce organ surface data and other anatomical structures such as ventricles and tumor tissue using conventional automated algorithms. This is typically done by training an intensity classifier on a user selected set of tissue samples, where the operator uses knowledge of anatomy to identify the tissue type. Once initial training is completed, the rest of the scans can be automatically classified on the basis of intensities in the scanned images, and thus segmented into tissue types. Conventional automatic methods for removing gain artifacts from the sensor data can be used to improve the segmentation. This 3D anatomical reconstruction is referred to as the model, and is represented relative to a model coordinate frame. For simplicity, the center of the coordinate system can be taken as the centroid of the points.

The patient is then placed in an operating room, which is equipped with a laser range scanner for obtaining depth data of the patient's skin surface where the surgery is to be performed; and enhanced visualization equipment, such as a video or digital camera, mixer and display monitor, a head-mounted display with trackable landmarks, an operating room microscope with video projection overlay feed, along with microscope-mounted trackable landmarks, or transparent projection screens along with screen mounted trackable landmarks, medical instrument holders containing trackable landmarks. The operating table may also contain fixed raised landmarks that will remain viewable and in the same position during surgery, and landmark tracking equipment.

Prior to draping, the patient is scanned by the laser range scanner. The 3D locations of any table landmarks are also calculated to identify their location relative to the patient. The current MRI or CT scan is automatically registered to the patient skin surface depth data obtained by the laser range scanner. This provides a transformation from MRI/CT to patient. The position and orientation of a video camera relative to the patient is determined by matching video images of the laser points on an object to the actual 3D laser data. This provides a transformation from patient to video camera. The registered anatomy data is displayed in enhanced visualization to "see" inside the patient. In particular, the two previously computed transformations can be used to transform the 3D model into the same view as the video image of the patient, so that video mixing allows the surgeon to see both images simultaneously. Alternatively, the images are combined with a surgical microscope or transparent imaging panel in order to augment the line-of-sight view of the surgeon with the MRI data. The patient is draped and surgery is performed.

The enhanced visualization does not interfere with the surgeon, nor does it require any procedures different from that to which the surgeon is accustomed. Rather, the system provides the surgeon with additional visualization information to greatly expand the limited field of view. The location of table landmarks can be continually tracked to identify changes in the position and attitude of the patient's head, relative to the visualization camera. Visualization updates are performed by re-rendering based on this tracking. Viewer location is continually tracked to identify any changes in the position of the viewer. In the case of a stationary video camera, this is unnecessary, though in the case of head-mounted displays such tracking is necessary. Visualization updates are performed by re-registration. If landmarks are used for tracking, re-registration is unnecessary. Updates are performed simply by re-rendering based on the tracked position information. Medical instruments may be tracked to align them with predetermined locations as displayed in the enhanced visualization. In general, the surgical procedure is executed with an accurately registered enhanced visualization of the entire relevant anatomy of the patient, and thus with reduced side effects.

With reference back to FIG. 1, the patient's body 102 is positioned on the operating table 104. The system 100 obtains three-dimensional image data from the skin surface of the patient with the utilization of a 3D laser scanning unit 106, which includes both a laser striping device 108 and a laser camera 110. Exemplary apparatus for carrying out the laser striping is described in U.S. Pat. Nos. 4,498,778 and 4,628,469, incorporated herein by reference. In essence, the system utilizes uses a plane of laser light and a video camera to obtain three dimensional measurements of the patient's skin, and uses the "structured light" method of obtaining the desired measurements. This method is based on the principal of triangulation for measurement.

The 3D laser scanning unit 106 controls the laser striping device 108 to generate a laser beam which is optically spread out to form a plane of light 112, which is projected in front of the laser camera 110 at an angle to the optical axis of the camera. The plane of light is formed, for example, by a laser beam reflected off of an oscillating mirror or a passive lens as is well known.

The laser camera 110 is placed at an angle to the plane of light such that a portion of the plane is in the camera field of view (FOV). As this plane of light strikes an object, such as the patient's skin, the diffuse reflection appears on the video camera image as a line of light 114. In other words, when an object is placed in this visible region such that it intersects the laser plane, points in the camera image plane illuminated by the laser unambiguously correspond to fixed 3D scene points. This is most easily appreciated by considering the case of objects on a flat support plane, with the plane of laser light striking the support plane at an oblique angle. When only the support plane is imaged, the laser plane makes a straight line in the image, and the camera can be oriented so that this line is vertical, for example. When an object is placed on the support plane, the imaged intersection of laser plane and the object is deflected from the previously recorded position, by an amount that is a direct function of the height of the object above the plane. By measuring this deflection, the distance to the observed point can be computed.

The projection of the stripe in the plane of the illumination onto the focal plane of the imaging device is unique. Each sample point in the image plane corresponds to a unique point in the plane of the laser illumination. This correspondence can be determined through a transform which, in turn, can be determined by scanning a known 3D shape. The laser scanning unit uses this unique mapping between image plane points and 3D points in space to determine the 3D coordinates points on the surface of the patient's skin illuminated by the laser. When the laser is moved, a different cross-section of the skin under the scanner can be measured. With multiple scans and the acquisition of multiple points within each scan, a sample grid is converted into 3D measurements. The density is only limited by the number of scans taken and the number of samples in each scan taken from the scanning unit. In the surgery example, approximately 20 scans are taken with between 100 and 200 3D points measured in each.

The laser scanning unit 106 used could be substituted for by any of a number of surface 3D scanners. There are numerous other conventional methods, including laser radar and moire fringe analysis, that could be used in the system 100. Other non-contact sensor types, including ultrasound or radar, are possible, as are a wide array of contact (probing) types of measurement devices. All that is required is the derivation of a modest number of accurate skin surface 3D points.

The system 100 also includes a real-time video camera 116 for providing a real-time image of the patient to which the 3D anatomy data will be registered. In one embodiment, the real-time video camera may be the same camera as the laser camera 110. An image data processor 118 serves to register the 3D surface data to the 3D anatomy data which has been prestored in an image data storage unit 120. A scan converter 122 is used to take the terminal outputs of the processor and convert them into video signals. Some computer processors provide direct video outputs, and thereby wouldn't require a scan converter. In an exemplary embodiment, the image data processor is an IBM RS6000 or IBM PVS used in conjunction with a Sun Sparc 10. A video mixer 124 mixes the video images from the processor, scanning unit and real-time video camera and thereafter fed to a video monitor 126 for live video visualizations enhanced with conventional computer graphics.

Prior to using the aforementioned hardware in a surgical setting, a calibration of the system is desirable. Calibration of the laser scanning unit 106 including the laser striping unit 108 and the cameras 110, 116 is performed using scan data from a precisely machined shape referred to as a gauge. This known shape, along with the images of the laser scans from it, can be used to precisely calibrate the laser instrument such that all subsequent measurements, anywhere in the operating range of the scanning unit will result in accurate 3D measurements as measured relative to some fixed reference frame. Since this reference frame is arbitrary, vertices of the laser system calibration gauge are used to define the frame.

The real-time video camera is also calibrated using the same gauge. The camera calibration is used to determine the appropriate viewpoint for rendering the registered 3D images prior to being mixed with the video images. The calibration determines the focal length of the camera lens, the position of the focal point, and the orientation of the image plane relative to the 3D laser reference frame. Since the laser calibration provides a mapping between every 2D point on the image plane and their corresponding 3D point on the laser plane, each measurement from objects in front of the scanning unit provides such a image/laser plane point correspondence and can be used to find the camera model.

In the case where the laser camera is used as the real-time video camera, si