3D VISUALIZATION AND ANALYSIS
3D VISUALIZATION AND ANALYSISIMAGE PROCESSING, ANALYSIS, AND VISUALIZATION:
METHODOLOGIES: AND TECHNIQUESCURRENT PERSPECTIVE
Jayaram K. Udupa, Ph.D.
Medical Image Processing Group
Department of Radiology
University of Pennsylvania
423 Guardian Drive – 4th Floor Blockley Hall
Philadelphia, Pennsylvania – 19104-6021
Telephone : (215) 662-6780
Fax : (215) 898-9145
jay@mipg.upenn.edu
Synopsis: This paper gives a comprehensive overview of the various image operations currently available in 3D imaging and identifies the main current hurdles.
ABSTRACT
3D imagingComputer aided processing, visualization, and analysis of images is the science of the methods of defining, visualizing, manipulating, and analyzing 3D object information captured in multidimensional images acquired from multiple imaging modalities. The purpose of this papert of the course is to give a comprehensive overview of this science to non-specialities and to identify the major challenges currently faced in its medical applications. It describes these 3D image-relateding operations under four groups: preprocessing, visualization, manipulation, and analysis. Preprocessing operations are aimed at extracting extracting, or improving the extraction of of, object information in given images. Visualization operations facilitate seeing and comprehending objects in their full dimensionality. Manipulation operations allows altering object structures and relationship among objects. Analysis operations deals with methods of quantifying object information. This e papercourse describes the concepts underlying these operations in a systematic manner. It draws two main conclusions: (1) There are two classes of approaches to 3D imaging - those based directly on image intensities and those based on object shape. (2) There are two main groups of challenges related to object delineation and validation.
1. INTRODUCTION
1.1 SCOPEPurpose, Scope
The main purpose of this paper is to give an overview of the current status of the science of Computer Aided Visualization and Analysis (abbreviated from now on by CAVA) of multidimensional images. 3D imaging, to identify the major challenges currently faced, and to point out the opportunities available for advancing the science. This review It describes schematically and systematically the main CAVA 3D imaging operations currently used and illustrates operations them with medical examples. It delineates the main concepts without detailing theory and algorithms and attempts to clarify some common misconceptions.
Its The intended audience of this review is developers of 3D imaging methods and software, developers of 3D CAVA imaging applications and medical researchers, and clinicians involved in CAVA interested in 3D imaging applications. The description presentation assumes some familiarity with medical imaging modalities and a knowledge of the rudimentary concepts related to digital images.
1.2 BACKGROUNDBackground
The main purpose of 3D imaging CAVA in medicine is to produce qualitative and quantitative information about an object/object system under study by using a computer,: given multiple multimodality medical images pertaining to the object system under study. Currently many imaging modalities are available from including digital radiography, computerized tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single position photositron emission computed tomography, (SPECT), and ultrasound, and many other emerging modalities such as optical tomography, magneto-encephalography, and magnetic resonance elastography. (US), to produce qualitative and quantitative information about an object/object system under study. The types of objects that are studied include rigid (e.g., bones), deformable (e.g., musclessoft tissue structures), static (e.g., skull), dynamic (e.g., lungs, heart, joints), and conceptual (e.g., activity regions in PET, SPECT, and functional MRI, and isodose surfaces in radiation therapy).
Multidimensional dimensionalities are possible for current medical image acquisitions.
2D : A digital radiograph, a tomographic CT/MR/PET/SPECT/US slice.
3D : A volume of tomographic slices of a static object.
4D : A time sequence of 3D images of a dynamic object.
5D : A 4D image of a dynamic object for a range of parameters (e.g., MR spectroscopic images of a dynamic object).
It is not feasible at present to acquire 4D and 5D images in a truly realistic manner, hence approximations are made. Two- and three-dimensional (2D and 3D) images are ubiquitous in radiology currently; e.g., a digital or digitized radiograph (2D), and a volume of tomographic slices of a static object. Four-dimensional (4D) images may be thought of as comprised of sequences of 3D images representing a dynamic object system; e.g., a sequence of 3D CT images of the thorax representing a sufficient number of time points of the cardiac cycle. Although the speed of image acquisition of modern CT scanners is rapidly increasing and ultrasound scanners (which are inherently real time) are overcoming the limitations of data acquisition in volume, it is not feasible at present to acquire 4D images in a realistic manner. This, however, is likely to change in the next 2-3 years. Images of dimensionality greater than 4 are also possible, for example a sequence of 4D images of a dynamic object over a range of parameters (such as frequency in MR spectroscopy), although the routine production of such images is not practical at present. In most applications, the object system of study consists of at most a couple of static objects. For example, a MRI 3D study of a patient head may focus on three 3D objects: white matter, grey matter, and cerebrospinal fluid (CSF)CSF.
Most background material on 3D imaging CAVA is scattered, and a systematic textbook on the subject is not available at present. [1-3] are edited books that may help the reader to start off on this subject. The references listed at the end are only representative and not exhaustiveBooks cited in [1-3] may help the reader to start off in this area and explore specific areas more deeply. The references listed at the end of this paper are only representative and may also help the reader in delving deeper into specific sub-areas within CAVA.
Some comments are in order regarding the choice of the phrase Computer Aided Visualization and Analysis, or CAVA, vis-à-vis another important area, which has come to be commonly known as CAD, or Computed Aided Diagnosis. In our view, CAVA is a much broader activity than CAD and predates CAD. While the latter focuses mainly on diagnosis, the former encompasses activities related to the development of new visualization, image processing, and quantitative image analysis tools to facilitate CAD, to facilitate new therapeutic strategies, and to help in basic clinical research, education, and training. CAVA is not a widely used terminology, and this area is badly in need of a name. Although there is some overlap in terms of operations between CAVA and CAD, we will focus only on CAVA in this paper.
Classification of CAVA Operations
CAVA operations may be broadly classified into four groups: PREPROCESSING, for defining the object system, meaning to create a geometric model of the objects in the system in the computer; VISUALIZATION, for viewing and comprehending the object system; MANIPULAT'ION, for altering the objects (for example, virtual surgery); ANALYSIS, for quantifying information about object system. CAVA operations are highly interdependent. For example, some form of visualization is essential to facilitate the other three classes of operations. Similarly, object definition through an appropriate set of preprocessing operations is vital to the effective visualization, manipulation, and analysis of the object system. We use the phrase "CAVA" to collectively refer to these four classes of operations.
Monoscopic or stereoscopic video display monitor of a computer workstation is presently the most commonly used viewing medium to facilitate CAVA operations. However, other media, such as holography and head-mounted displays, are also available. Unlike the 2D computer monitor, holography offers a 3D medium for viewing. The head-mounted displays are basically two tiny monitors presented in front of the two eyes through a helmet-like device that is worn over the head. This offers the sensation of being free from our natural surrounding and immersed in an artificial environment. By far, however, the computer monitor is the most commonly used viewing medium, mainly because of its superior flexibility, speed of interaction, and resolution over other media.
A generic representation of CAVA systems is shown in Figure 1. A workstation with appropriate software implementing CAVA operations forms the core of the system. Depending on application, a wide variety of input/output devices are utilized. Considering the core of the system (independent of input/output), the following categorization of CAVA systems can be made: (i) Physician display consoles/workstations provided by imaging device vendors. (ii) Image processing/visualization workstations supplied by other independent vendors. (iii) CAVA software provided by other software vendors. (iv) Open source and controlled open source software freely available via Internet.
Terminology
For brevity and later reference, some terms that are frequently used in this paper are defined below. See Figure 2. The 2D, 3D,... region corresponding to the body region that is imaged is divided into rectangular elements (see Figure 2). These are usually referred to as pixels for 2D space and voxels for 3D space. We will refer to them as voxels for all dimensionalities. For 2D space, the voxels are usually squares, and for 3D space, they are cuboids with a square cross section.
1.3 CLASSIFICATION OF 3D IMAGING OPERATIONS
3D Imaging operations can be broadly classified into the following groups:
PREPROCESSING : For defining the object system, meaning to create a geometric
model of the objects in the system in the computer.
VISUALIZATION : For viewing and comprehending the object system.
MANIPULAT'ION : For altering the objects (for example, virtual surgery).
ANALYSIS : For quantifying information about object system.
These operations are highly interdependent. For example, some form of visualization is essential to facilitate the other three classes of operations. Similarly, object definition through an appropriate set of preprocessing operations is vital to the effective visualization, manipulation, and analysis of the object system. We use the phrase "3D imaging" to collectively refer to these four classes of operations.
Monoscopic or stereoscopic video display monitor of a computer workstation is the most commonly used viewing medium for images. However, other media, such as holography and head-mounted displays, are available. Unlike the 2D computer monitor, holography offers a 3D medium for viewing. The head-mounted displays are basically two tiny monitors presented in front of the two eyes through a helmet-like device that is worn over the head. This offers the sensation of being free from our natural surrounding and immersed in an artificial environment. By far, however, the computer monitor is the most commonly used viewing medium, mainly because of its superior flexibility, speed of interaction, and resolution over other media.
A generic representation of 3D imaging systems is shown in Figure 1. A workstation with appropriate software implementing 3D imaging operations forms the core of the system. Depending on application, a wide variety of input/output devices are utilized. Considering the core of the system (independent of input/output), the following categorization of 3D imaging systems can be made.
Physician display consoles provided by imaging device (CT, MR scanners) vendors. Image processing/visualization workstations supplied by other independent vendors.
3D imaging software supplied independent of workstations.
University-based 3D imaging software often freely available via Internet.
Systems provided by scanner manufacturers and workstation vendors are usually focussed in their solution, but may cost $50K-$150K. If you have the know-how of accessing and installing (and running) the software, very effective solutions are available inexpensively under the last category above. For example, you can configure a complete system running on modem PCs (such as a Pentium 300) with performance on par with (or even surpassing) the performance of the costly systems under the previous categories for under $5K.
1.4 TERMINOLOGY
For brevity and later reference, some terms that are frequently used are defined below. See Figure 2.
The 2D, 3D,... region corresponding to the region of the body that is imaged is divided into rectangular elements (see Figure 2). These are usually referred to as pixels for 2D and voxels for 3D. We will refer to them as voxels for all dimensionalities. For 2D, the voxels are usually squares, and for 3D, they are cuboids with a square cross section.
Scene: Short form for a multidimensional image of an object system. We think of it as a rectangular array of voxels together with a value assigned to each voxel.
Scene domain: The region represented by the rectangular array of voxels of the scene.
Scene intensity: The value assigned to the voxels.
Pixel size: The length of the side of the square cross section of the voxel.
Scanner coordinate system: An origin plus an orthogonal axes system affixed to the imaging device.
Scene coordinate system: An origin plus an orthogonal axes system affixed to the scene. The origin is usually assumed to be the upper left corner of the first slice of the scene, and the axes are the edges of the scene domain converging at the origin.
Object coordinate system: An origin plus an orthogonal axes system affixed to the object or object system.
Display coordinate system: An origin plus an orthogonal axes system affixed to the display device.
Rendition: A 2D image depicting the object information captured in a scene or of an object system.
In the rest of this paper, we will describe the commonly used CAVA operations under Preprocessing and Visualization in detail, but owing to space limitations, only briefly summarize the operations available under Manipulation and Analysis.
1.5 OBJECT CHARACTERISTICS IN IMAGES
There are two important object characteristics whose realistic handling is vital in all 3D imaging operations.
GRADED COMPOSITION
Most objects in the body have a heterogeneous material composition. in addition to this, imaging devices introduce a blurring in the acquired images because of various limitations. As a result, regions corresponding to the same object display a gradation of scene intensities. Figure 3 shows a CT slice of a knee wherein both patella and femur exhibit this property. That is, the region corresponding to bone in these objects has not just one CT number but a gradation of values.
HANGING-T'OGETHERNESS (GESTALT)
In spite of this gradation, when we view images, voxels seem to hang together (form a gestalt) to form an object. For example, the high intensity voxels of the patella do not (and should not) hang together with (seem to belong to) the group of similar voxels in the femur, although voxels with dissimilar intensities in the femur hang together.
2. PREPROCESSING
The aim of Ppreprocessing operations is to output computer object models, or another set of scenes which facilitate creating computer models of objects,: given a set of input scenes., to output computer object models or another set of scenes, which facilitate creating computer models of objects. The most commonly used operations are: volume of interest, filtering, interpolation, registration, and segmentation. These are described in detail in this section.
2.1 VOLUME OF INTERESTVolume of Interest (VOI)
This operation converts a given scene to another scene. Its purpose is to reduce the amount of data by specifying a region of interest (ROI) and a range of intensity of interest (IOI).
An ROI is specified by indicating a rectangular box that delimits the scene domain in all dimensions. An IOI is specified by indicating an intensity interval. Within this interval, scene intensities are transferred unaltered to the output. Outside the interval, they are set to the lower and upper limits. For the CT 3D scene of a head shown in Figure 4 (left), a 3D rectangular box is specified to indicate the ROI. The IOI is indicated as an interval on a histogram of the scene. The corresponding slice in the output scene is shown on the right. This operation can often reduce storage requirement for scenes by a factor of 2 to 510. It is advisable to use this as the first operation in any sequence of 3D imaging operationsCAVA operations.
Challenge: To completely automate the VOI operation, and to do it in an optimal fashion, explicit delineation of objects in images first becomes necessary.
Filtering
2.2 FILTERINGFiltering
This operation converts to a given scene to another scene. Its purpose is to enhance wanted (object) information and to suppress unwanted (noise/background/other object) information in the output scene. Two classes of filtering operations are available.e:
Suppressing : Unwanted information is suppressed, hopefully without affecting
wanted information.
Enhancing : Wanted information is enhanced, hopefully without affecting unwanted information.
The most commonly used suppressing filter is a smoothing operation mainly used for suppressing noise. Here, a voxel v in the output scene is assigned an intensity which is a weighted average of the intensities of voxels in the neighborhood of v in the input scene [4]. Methods differ as to how the neighborhoods are determined and how the weights are assigned [5]. Another commonly used method is median filtering. In this method, the voxel v in the output scene is assigned a value, which is simply the middle value (median) of the intensities of the voxels in the neighborhood of v in the input scene when they are arranged in an ascending order.
In another method [5, 65], often used in processing MR images, a process of diffusion and flow is considered to govern the nature and extent of smoothing. The idea here is that in regions of low rate of change of intensity, voxel intensities diffuse and flow into neighboring regions. This process is prevented by voxels at which the rate of change of intensity is high. There are parameters (of the conductance function) in the method which control the extent of diffusion that takes place. and the limits of the magnitude of the rate of change of scene intensity that are to be considered "low" and "high". The method is quite effective in overcoming noise and yet sensitive enough not to suppress subtle details and to blur edges.
The most commonly used enhancing filter is an edge enhancer [1-34]. Here, the intensity of v in the output is the rate of change of intensity at v in the input. If we think of the input scene as a function, then this rate of change is given by the magnitude of the gradient of the function. Since this function is not known in analytic form, various digital approximations for this operation are used. The gradient has a magnitude (the rate of change) and a direction in which this change is maximum. For filtering, the direction is usually ignored, although it is used in operations relating to creating creating three-dimensional renditions depicting surfaces and tissue interfaces. Methods differ in how the digital approximation is arrived at, which is extensively studied in computer vision [76].
Figure 35 shows an example of a smoothing filter (b) and an edge- enhancing filter (c) applied to a slice of a CT MR scene of a head (a).
Challenge: : Most existing suppressing filters often also suppress object information and enhancing filters enhance unwanted information. To minimize these effects, explicit incorporation of object knowledge into these operations becomes necessary.
2.3 INTERPOLATIONnterpolation
This operation converts a given scene to another scene. Its purpose is to change the level of discretization (sampling) of the input scene. Interpolation becomes necessary in the following situations: (i) To change the nonisotropic discretization of the input scene to an isotropic or to a desired level of discretization. (ii) To represent longitudinal scene acquisitions of a patient in a registered common scene coordinate system. (iii) To represent multimodality scene acquisitions in a registered common scene coordinate system. (iv) In reslicing the given scene. Two classes of methods are
available: scene-based and object-based.
Scene-based: The iIntensity of a voxel v in the output scene is determined based on the intensity of voxels in the neighborhood of v in the input scene. Methods differ in how the neighborhoods are determined and in the form of the functions of the neighboring intensities used to estimate the intensity of v [3, 6, 71-3]. The simplest situation in 3D interpolation is to estimate new slices between slices of the input scene keeping the pixel size of the output scene the same as that of the input scene. This leads to a 1D interpolation problem: estimate the scene intensity of any voxel v in the output scene from the intensities of voxels in the input scene on the two sides of v in the z direction (the direction orthogonal to the slices, see Figure 2). In nearest neighbor interpolation, v is assigned the value of the voxel that is closest to v in the input scene. In linear interpolation, two voxels v1 and v2 on the previous and next adjoining slices, with the same x, y coordinates as that of v, are considered, one on each side of v, are considered. The value of v is determined based on the assumption that the input scene intensity changes linearly from the intensity at v1 to that at v2. In higher order (such as cubic) interpolations, more neighboring voxels on the two sides of v are considered. When the size of v in the output scene is different in all its dimensions from that of voxels in the input scene, the situation becomes more general than that described above. Now, intensities are assumed to vary linearly or as a higher order polynomial in each of the three directions in the input scene.
Object-based: Object information derived from scenes is used in guiding the interpolation process. In one extreme [8, 98], the given scene is first converted to a "binary" scene (a scene with only two intensities, 0 and 1) by a segmentation operation (described later). The idea is that the 1-valued voxels represent the object of interest and the 0-valued voxels denote the rest of the scene domain. The "shape" of the region represented by the 1-valued voxels (the object) is then utilized to create an output binary scene with similar shape [9, 10] via interpolation. This is done by first converting the binary scene back into a (grey-valued) scene by assigning to every voxel in this scene a value that represents the shortest distance of the voxel to the boundary between 0- and 1-valued voxels. The 0-voxels are assigned a negative distance and the 1-voxels are assigned a positive distance. This scene is then interpolated using a scene-based technique and subsequently converted back to a (interpolated) binary scene by setting a threshold at 0. On the other extreme, the shape of the intensity profile of the input scene is itself considered as an "object" to guide interpolation so that this shape is retained as best as possible in the output scene [101]. To interpolate a 2D scene by this method, for example, it is converted into a 3D surface of intensity profile wherein the height of the surface represents pixel intensities. This (binary) object is then interpolated using the objectshape-based method just described. Several methods in between these extremes exist [112, 123]. The object shape-based methods have been shown to produce more accurate results [8-101] than most of the commonly used scene-based methods.
Figure 46 shows an example of binary object-based (shape-based) interpolation of a child's skull at "coarse" and "fine" levels of discretization derived from CT data. The original 3D scene was first thresholded to create a binary scene. This binary scene was then interpolated at coarse (Figure 46a) and fine (Figure 46b) levels and then surface rendered.
Challenge: To identify specific object information and to incorporate thisit into interpolation.
With such information, interpolation accuracy can be improved.
2.4 REGISTRATIONRegistration
This operation takes as input two scenes or two objects and outputs a transformation which when applied to the second scene or second object matches it as best as possible to the first. Its purpose is to combine scene/object information from multiple scenes/ modalities/protocols, to determine change, growth, motion, displacements of objects, and to aid in object identification. Two classes of methods are available: scene-based and object-based.
Scene-based: To match two scenes S1 and S2, a rigid transformation - translation and rotation - (often also scaling) is calculated for one of the scenes S2 such that the transformed scene [pic] matches in intensity pattern as best as possible the other scene S1 [13, 14]. Methods differ in the matching criterion used and in how the optimum translation and rotation are determined among the infinite possible translations and rotations [15]. Methods are also available for situations where objects undergo elastic (non-rigid) deformation [16]. In the latter case, the transformation is output usually as a deformation field wherein each voxel in S2 has a vector assigned to it which indicates to where that voxel would move after registration.
Figure 57 shows an example of scene-based rigid registration. Input consisted of a set of two4 3D scenes corresponding to the MR scans of a Multiple Sclerosis (MS) patient at two4 longitudinal time instances. Approximately the same slice in each of these time instances is shown on the top. Since the scenes are not registered, it is difficult to see how the disease (hyperintense lesions around the ventricles) has progressed. The slices at the same location after 3D registration are shown at the bottom. The progression of the disease is now readily discerned. Note that, upon registration (either rigid or non-rigid), the s3 scene S2s should be interpolatedare resliced by (reslicing is discussed under Visualization) using a scene-based interpolation method to obtain [pic]slices at the same location..
Object-based: In these methods, two scenes are registered based on object information extracted from the two scenes. For the match to be effective, the two objects should be as similar as possible. For example, to match a MR and a PET 3D scene of the head of a patient, we may use the outer skin surface of the head computed from each of the two scenes and match the two surfaces [17]. Instead of, or in addition to, the surface, landmarks (points, curves, ridges, planes, etc.) observable in and computable from both scenes as well as implanted objects can also be used [18-20]. The best translation and rotation parameters required to match the two objects are determined by minimizing some measure of "distance" between the two (sets of) objects. Methods differ in how "distances" are defined and in how optimum solutions are computed.
Figure 8 shows an example of rigid object-based registration. The application here is kinematic analysis of the midtarsal joints of the foot from a sequence of MR 3D scenes of a patient foot acquired for different positions of the foot. Two different positions of the joint (consisting of 4 bones - talus, calcaneus, navicular, and cuboid) are shown. Motion (translation and rotation) of each bone from one position to the other is determined by matching the same bone in the two positions.
Instead of rigid transformations, deformable matching operations can also be used on objects [21-, 232]. These may beare more appropriate than rigid matching for non-rigid soft-tissue structures. Usually a global approximate rigid matching is done first followed by local deformations for finer matching. Deformable registration is also used to match computerized brain atlases to a given patient brain scene data set [243]. Some object information has to be first identified in the scene. This has several potential applications in functional imaging, neurology, and neurosurgery, as well as in object definition itself, and is a very actively pursued area of research.
Challenge: : Scene-based methods require intensity patterns in the two scenes to be similar for optimum performance. This is often violated. Converting scenes into fuzzy (non-binary) object descriptions that retain object gradation can may potentially overcome this problem and but may still retain the strength of scene-based methods. Deformable fuzzy object matching seems natural and appropriate of the in most situations. This requires the development of fuzzy mechanics theory and algorithms. Deformable registration is an actively pursued area of research at present.
.
2.5 SEGMENTATIONSegmentation
This operation outputs from a givenGiven a set of scenes, this operation outputs computer models of object information captured in the scenes. Its purpose is to identify and delineate objects. Segmentation consists of two related tasks: Recognition and Delineation.
Recognition: This consists of determining roughly the objects' whereabouts in the scene. For example, in Figure 3, this task involves determining "this is the femur, this is the patella,...” This does not involve the precise specification of the region occupied by the object. It is a high-level act of indicating, for example, in an MR image of human brain, “this is the white matter object, this is the grey matter object, this is the CSF object, etc.” Human-assisted recognition can be accomplished, for example, by using the mouse cursor to point at object regions or to specify seed points.
Delineation: This involves determining the objects' precise spatial extent and composition including gradation. In Figure 3 again, ifIf bone is the object system of interest in a scene of the knee for example, then delineation consists of defining the spatial extent of the femur, tibia, fibula, and patella separately, and for each voxel in each object, specifying an objectness value. Once the objects are defined separately, the bones femur and the patella can be individually visualized, manipulated, and analyzed.
Object knowledge usually facilitates its recognition and delineation. Ironically, this implies that segmentation is required for effective object segmentation. As we have observed, segmentation is needed for carrying out most of the preprocessing operations in an optimum fashion. We shall note later that segmentation is an essential operation for most visualization, manipulation, and analysis tasks also. Segmentation is, therefore, the most crucial among all 3D CAVA imaging operations and also the most challenging.
Knowledgeable humans usually outperform computer algorithms in the high-level task of recognition. However, carefully designed computer algorithms outperform humans in the precise, accurate, and efficient delineation. Clearly, human delineation that can account for graded object composition (which comes from natural object material heterogeneity, noise, and various artifacts such as partial volume effects) is impossible. Most of the challenges in completely automating segmentation may be attributed to the shortcomings in computerized recognition techniques, and to the lack of delineation techniques that can handle graded composition composition (which comes from natural object material heterogeneity, noise, and various artifacts such as partial volume effects) and the tendency of voxels to hanging-together in space (as a fuzzy cloud)ness of voxels in the presence of this gradation.
Approaches to Recognition. Two classes of methods are available for recognition.
Automatic (kKnowledge- and Atlasmodel-based): Artificial intelligence methods are used to represent knowledge about objects and their relationships [24-26]. Preliminary delineation is usually needed by these methods to extract object components and to form and to test hypotheses related to whole objects.
A carefully created "atlas" consisting of a complete description of the geometry and relationship of objects is often used [27, 28, -29]. Some delineation of object components in the given scene is necessary for these methods. This information is used for determining the mapping necessary to transform voxels or other geometric elements from the scene space to the atlas. The information is also used in the opposite direction to deform the atlas so that it matches the delineated object components in the scene. Alternatively, statistical shape models (mostly in 2D space) have also been used to facilitate recognition [29].
Human-assisted: Often a simple human assistance is a sufficient recognition aid in solving a segmentationthe recognition problem. This assistance may be in the form of specification of several "seed" voxels inside the 3D region occupied by the object or on its boundary; or indication of a box (or other simple geometric shapes) that just encloses the object and that can be quickly specified; or a click of a mouse button to accept a real object (say a lesion) or to reject a false object.
Approaches to Delineation: Numerous A large variety of methods are available for delineation. Often, delineation is itself considered to be the total segmentation problem, as such its solutions are considered to be equivalent to solutions to the entire segmentation problem. It is, however, helpful to distinguish between recognition and delineation for understanding, and hopefully solving, the difficulties encountered in segmentation. Approaches to delineation can be broadly classified as followsboundary-based, region-based, and hybrid.
Boundary-based : These methods output an object description in the form of a boundary surface that separates the object from the background. The boundary description may be as a hard set of primitives - points, polygons, surface patches, voxels, etc., or as a fuzzy set of primitives such that each primitive has a grade of "boundariness" associated with it.
Region-based: T hese methods produce an object description in the form of the region occupied by the object. The description may be simply as a (hard) set of voxels voxels, in which case or as a fuzzy set such that each voxel in the set has a grade of "objectness" assigned to it. In the hard method, each voxel in the set is considered to contain 100% object materiamaterial, or as a fuzzy set, in which case lmembership in each voxel. In the fuzzy method, this may be any number between 0 and 100%. Prior information about object classes is more easily captured via boundaries than in region description. Hybrid methods attempt to combine information about boundaries and regions in seeking a solution to the delineation problem.
If we combine the approaches for recognition and delineation, there are 12 8 possible classes of approaches to segmentation. We will now examine these strategies briefly.
Hard boundary-based, automatic methods: The most common technique in this class is thresholding and isosurfacing [30-33]. In these methods, a threshold scene intensity is specified, and the surface that separates voxels whose intensity is above the threshold from those whose intensity is below the threshold is computed. Methods differ in how the surface is represented and computed, and in whether or not connectedness of the surface is taken into account. The surface may be represented in terms of voxels, voxel faces, points, triangles, and other surface elements. The idea of connectedness can be explained with respect to the bones in Figure 3. If connectedness is not used, the surface obtained from this scene at a threshold appropriate for bone will consist of both the femur and the patella. With connectedness, each of these objects can be represented as a separate surface if they are separated in the 3D scene. In Figure 6, the isosurface is connected and is represented as a set of faces of voxels [30].
In addition to scene intensity threshold, intensity gradient has also been used in defining boundaries [34].
Fuzzy boundary-based automatic methods: Concepts related to fuzzy boundaries, such as connectedness, closure, and orientedness that are well established for hard boundaries are difficult and not yet developed. However, computational methods that identify only those voxels of the scene that are in the vicinity of the object boundary and that assign to them a grade of boundariness have been developed [35, 36]. These use scene intensity and/or intensity gradient to determine boundary gradation.. Figure 9 shows a 3D rendition of the fuzzy boundary of the femur and patella in the CT scene shown in Figure 3, detected by such a method.
Hard, boundary-based, human-assisted methods: The degree of human assistance in these methods ranges from fully manual boundary tracing (manual recognition and delineation) to specifying just a point inside the object or on its boundary (i.e., manual recognition and automatic delineation [37-42]. In routine clinicalmany applications, manual boundary tracing is perhaps the most commonly used method. On the other hand, boundary detection methods requiring simple user assistance based on intensity [37, 38] and gradient criteria [39-42], have been developed. These methods cannot be guaranteed to work correctly in large applications routinely.
There are many user-assisted methods that fall in-between the extremes indicated above and that require different degrees of human assistance for each scene to be segmented [43-49]. In view of the inadequacy of the minimally-user-assisted methods referred to above, much effort is currently focussed on doing recognition more-or-less manually and delineation more-or-less automatically. These methods go under various names: active contours or snakes [43-45], active surfaces [46, 47], and live-wire/live-lane [48, 49].
In active contour/surface methods, an initial boundary is specified, say by indicating a rectangle or a rectangular box close to the boundary of interest. The boundary is considered to have certain stiffness properties. Additionally, the given scene is considered to exert forces on the boundary dependent on intensity gradients. For example, a voxel exerts a strong attractive force on the boundary if the rate of change of intensity at the voxel is high.. Within this static mechanical system, the initial boundary deforms and eventually arrives at a shape for which the combined potential energy is the minimum. Unfortunately, the steady-state shape is usually impossible to compute. Further, whatever shape is accepted as an alternative may not match with the desired boundary. In this case, further correction of the boundary is needed. In assessing how good such segmentation methods are, it is important to determine their precision (repeatability) and efficiency (the number of scenes that can be segmented per unit time). Such evaluations have not generally been done for published methods. s.
The principles underlying live wire/lane methods [48, 49] are different from those for active boundary methods. In live wire/lane methods [48, 49]these methods, every pixel edge is considered to represent two oriented and directed edges whose orientation is opposite to each other. The "inside" of the boundary is considered to be to the left of the directed edge, and its outside to the right. A cost is assigned to every directed edge based on several factors, which is inversely related to the "boundariness" of the edge. Several local features, such as intensity to the left (inside) and to the right (outside), and intensity gradient and its direction, are used for determining the cost. . TIn 2D live wire, the user initially selects a point (pixel vertex) vo on the boundary of interest. The computer now shows a "live-wire" segment from vo to the current mouse cursor position v, that which is an oriented path (a connected sequence of directed pixel edges) whose total edge cost is the smallest of all paths from vo to v. As the user changes v through mouse movement, the optimal path is computed and displayed in real time. In particular, if v is on or close to the boundary, the live wire snaps onto the boundary (see Figure 610). v is now deposited and becomes the new starting point, and the process continues. Typically 2-5 points are sufficient to segment a boundary such as that of athe boundary of the talus of a foot in an MR image as shown in Figure 610. This method and its derivatives are shown to be 2-3 times faster and and statistically significantly more repeatable than manual tracing [48]. Its 3D version [49] is about 3-15 times faster than manual tracing. Note that, in this method, recognition is manual but delineation is automatic.
Active shape/appearance model based methods [50, 51] have gained popularity recently. They differ from active contour methods in an essential way in that statistical information about the boundary shape, its variation, the intensity pattern in the vicinity of the boundary, and its variation are all captured in the model to assist in segmentation. A statistical model is built first from a set of training images by identifying the same homologous set of landmark points on the boundaries and by creating a mean boundary shape from the landmarks and also by quantifying the variation of the training boundaries around this mean shape. Subsequently, to segment a given image, the model is utilized to search for a location where the model would best fit the image, and then the mean boundary is deformed locally by moving its landmarks so that the deformed mean boundary matches the local boundary intensity pattern as best as possible. This local adjustment is done in such a way that the deformed mean boundary still falls within the boundary variation observed in the training images and captured in the model. Figure 7 illustrates segmentation of the talus in the MR image of Figure 6 by using the active shape method.
We are not aware of any published fuzzy, boundary-based, human-assisted methods.
HHaard, region-based, automatic methods: The most common ly used among these methods is thresholding. A voxel is considered to belong to the object region if its intensity is at or above a lower threshold and at or below an upper threshold. If the object is the brightest in the scene (e.g., bone in CT images), then only the lower threshold needs to be specified. The threshold interval is conveniently specified on a scene intensity histogram as shown in Figure 11 (middle). The thresholded object is shown as a binary scene on the right. The original scene is on the left.
Another commonly used method is clustering. Suppose we determine multiple values associated with every voxel, say the T2 and proton density (PD) values through MR imaging of the head of a patient. Consider a 2D histogram (often also called a scatter plot) of these values - a plot of the number of voxels in the given 3D scene for each possible (T2, PD) value pair. Figure 812(a) shows a pair of T2, PD slices and (b) shows such a histogram. The 2D space of all possible (T2, PD) value pairs is usually referred to as a feature space. The idea in clustering is that feature values corresponding to the objects of interest cluster together in the 2D histogram (feature space). To segment an object, therefore, we need to just identify and delineate this cluster. In other words, the problem of segmenting the scene is converted into the problem of segmenting the 2D scene representing the 2D histogram. For the cluster indicated in Figure 812(b), the object segmented is shown in Figure 812(c). INote that, in addition to the (T2, PD) values at every voxel, we may also use computed features such as the rate of change of T2 and of PD values at every voxel (or texture measures). In this case, our feature space would be 4-dimensional. There is a well-developed body of techniques in the area of pattern recognition [520] for automatically identifying clusters, and these have been extensively applied to medical images [531-57].
One of the popular cluster identification methods is the k-nearest neighbor (kNN) technique [520]. Suppose our problem is segmenting the white-matter (WM) region in a 3D MR scene of a patient's head, wherein, for each voxel we have imaged the T2 and the PD value. The first step in this method is to identify two sets XWM and XNWM of points in the 2D feature space, corresponding to WM and non-WM regions. These will be used as the basis for determining whether or not a voxel in the given scene belongs to WM. These sets are determined through a "training" set. Suppose we previously segmented manually one or more scenes. Each voxel in the WM and non-WM regions in each scene contributes a point to the respective set XWM or XNWM. The next step is to select a value for k, say k = 7. k is a parameter in the method and remains fixed. For each voxel v in the given scene to be segmented, its location P in the feature space is determined. Then 7 points that are "closest" to P from the sets XWM and XNWM are determined. If a majority of these (> 4) are from XWM, then v is considered to be in WM, otherwise v does not belong to WM. Note that the step of obtaining XWM and XNWM need not be repeated for every scene to be segmented.
Note that thresholding is essentially clustering in a 1D feature space. All clustering methods have parameters whose values need to be somehow determined. If these are fixed in an application, the effectiveness of the method in routine processing cannot be guaranteed and usually some user assistance becomes necessary eventually for each study.
Examples of other non-clustering methods are [58-59].
Fuzzy, region-based, automatic methods: The simplest among these methods is fuzzy thresholding [5860], as illustrated in Figure 913, which is a generalization of the thresholding idea. It requires the specification of four intensity thresholds. If the intensity of a voxel v is less than t1 t1 or greater than t4, , objectness of voxel v is 0. Its objectness is 1 (100%) if its intensity lies between t2 and t3. For other intensity values, objectness takes on in-between values. Other functional forms have also been used. Figure 104 shows a rendition of bone and soft-tissue identified in this fashion from the CT scene of a Figure 3.patient’s knee.
Many of the clustering methods can be generalized to output fuzzy object information. For example, in the kNN method described previously, if m out of k points closest to P areis from XWM, then the objectness (white-matterness) of v is m/k. Note that tThe above fuzzy generalization of thresholding is a form of fuzzy clustering. One approach to more generalized fuzzy clustering is the fuzzy c-means method [59, 6061]. Its application has been investigated for segmenting brain tissue components in MR images [62, 613]. Roughly the idea is as follows. Suppose that there are two types of tissues to be segmented, say WM and GM, in a 3D MR scene, and let the feature space be 2D (T2 and PD values). Actually, we need to consider three classes: WM, GM and everything else. In the 2D scatter plot of the given scene, therefore, our task is to define three clusters, corresponding to these three classes. The set X of points to which the given scene maps in the feature space can be partitioned into three clusters in a large (although finite) number of ways. The idea in hard c-means method is to choose that particular cluster arrangement for which the sum (over all clusters) of squared distances between points in each cluster and the cluster center is the smallest. In the fuzzy c-means method, each point in X is allowed to have an objectness value that lies between 0 and 1., Now the number of cluster arrangements is infinite. The distance in the criterion to be minimized is modified by the objectness value. Algorithms have been described for both methods to find clusters that approximately minimize the criterion described above.
As with hard clustering methods, the effectiveness of fuzzy clustering methods in routine applications cannot be guaranteed, as such some user assistance on a per-scene basis is usually needed.
Hard, region-based, human-assisted methods: The simplest among these techniques is manual painting of regions using mouse-driven paintbrush [62]. This is the region dual of manual boundary tracking.
In contrast to this completely manual recognition and delineation scheme, there are methods in which recognition is manual but delineation is automatic. Region growing is a popular method in this group [63]-65]. At start, the user specifies a "seed" voxel within the object region, say by using a mouse pointer on a slice display. A set of criteria for inclusion of a voxel in the object is also specified; for example, (1) the scene intensity of the voxel should be within an interval t1 to t2; (2) the mean intensity of voxels included in the growing region at any time during the growing process should be within an interval t3 to t4; (3) the intensity variance of voxels included in the growing region at any time during the growing process should be within an interval t5 to t6. Starting with the seed voxel, its 3D neighbors (usually the closest 6, 18 or 26 neighbors) are examined for inclusion. Those that are included are marked so that they will not be reconsidered for inclusion later. The neighbors of the voxels selected for inclusions are then examined, and so on. The process continues until no more voxels can be selected for inclusion. .I
Note that, if we use only criterion (1), and if t1 and t2 are fixed during the growing process, then this method outputs essentially a connected component of voxels satisfying a hard threshold interval. Note also that, for any combination of criteria (1) to (3), or if t1 to t6 are not fixed, it is not possible to guarantee that the set of voxels (object) O(vl) obtained with a seed voxel vl is the same as object O(v2), where v2 [pic] v1 is a voxel in O(v1). This lack of theoretical robustness is a problem with most many region-based methods.
Fuzzy, region-based, human-assisted methods: We already pointed out that the fuzzy region-based methods described earlier eventually need human assistance. In this sense, those methods fall in this category. A recent method, which by design takes human assistance, is the fuzzy connectedness technique [66]. In this method, recognition is manual and involves pointing at an object in a slice display. Delineation is automatic and takes into account both the characteristics of intensity gradation and how these gradations hang together in an object. graded composition and hanging-togetherness. It has been effectively applied in several large applications [65, 66]including multiple sclerosis lesion quantification [67-70], MR angiography [71], and soft tissue display for craniomaxillofacial surgery planning [72]. A brief outline of the method is given below. In this method, every pair (v1, v2) of
We think of near-by voxels in the voxel array as having a fuzzy adjacency relation which indicates their spatial nearness. It is independent of any scene intensity values. The strength of this relationship (varying between 0 and 1) between two voxels is a non-increasing function of the distance between them. Roughly, fuzzy adjacency captures the blurring characteristic of imaging devices.
We think of near---by voxels in a scene as havingis assigned a fuzzy affinity relation which indicates how v1 and v2 they hang together locally in the same object. The strength of this relation (varying between 0 and 1) between any two voxels is a function of how close v1 and v2 are spatially and how similar are their fuzzy adjacency as well as their scene intensity values. As a simple example, this function may be the product of the strength of their adjacency and (l- | I(vl) - I(v2) |) where I(v1) and I(v2) are the intensity values of voxels v1 and v2 scaled in some appropriate way to the range 0 to 1. Affinity expresses the degree to which voxels "hang together" locally to form a fuzzy object. Of course, the intent here is that this is a local property. Voxels that are far apart will have negligible affinity as per the above function. The real "hanging-togetherness" of voxels in a global sense is captured through a fuzzy relation called fuzzy connectedness, defined as follows. To each pair of voxels (vl, v2), a strength of connectedness is assigned as follows. There are numerous possible paths between v1 and v2. A path from v1 to v2 is a sequence of voxels starting from v1 and ending on v2, the successive voxels being near-by with a certain degree of adjacency. The "strength" of a path is simply the smallest of the affinities associated with pairs of successive voxels along the path. The strength of connectedness between v1 and v2 is simply the largest of the strengths associated with all possible paths between v1 and v2. A fuzzy object is a pool of voxels together with a membership (between 0 and 1) assigned to each voxel that represents its "objectness". The pool is such that the strength of connectedness between any two voxels in the pool is greater than a small threshold value, a(typically about 0.1) and the strength between any two voxels, one in the pool and the other not in the pool, is less than the threshold value.. Obviously, computing fuzzy objects even for the simple affinity function exemplified above is computationally impractical if we proceed straight from the definitions. But the theory allows us to simplify the complexity considerably for a wide variety of affinity relations so that fuzzy object computation can be done in practical time (about 15-20 minutes for a 256(256(64 16 bit/voxel 3D scene on a Sparc 20 workstation). A wide variety of application-specific knowledge of image characteristics can be incorporated into the affinity relation [65, 66].
Figure 115 shows an example of fuzzy-connected segmentation (in 3D) of WM, GM, CSF and MS lesions in a T2-PD scene pair. Figure 16 shows a (maximum intensity projection) rendering (left) of a contrast enhancedn MRA data set and a rendition of thea 3D fuzzy connected vessel tree (right)with the arteries and veins separated.
Hybrid strategies: These methods (e.g., [67, 68]) try to synergistically combine the strengths of boundary-based and region-based methods, with the hope of overcoming their individual weaknesses by the strength of the other. The current trend seems to be toward combining model based (such as active appearance) methods with region-based methods.
detected from a point specified on the vessel. Here fuzzy connectedness is used to remove the clutter that obscures the vessel display.
Challenges: (1) To develop general segmentation methods that can be easily and quickly adapted to a given application. (2) To keep human-assistance required on a per-scene basis to a minimum. (3) To develop fuzzy methods that can realistically handle uncertainties in data and prior knowledge about objects. . (4) To assess the efficacy of segmentation methods (discussed later).
3. VISUALIZATION
These operations create renditions of given scenes or object systems. Their purpose is given a set of scenes or objects, to create renditions that facilitate the visual perception of object information given a set of scenes or objects. Two classes of approaches are available: scene-based and object-based.
3.1 SCENE-BASEDScene Based
In these approaches, renditions are created directly from given scenes. Two further subclasses may be identified: slice mode and volume mode.
Slice mode
Methods differ based on what is considered to constitute a "slice" and how this information is displayed. What: natural slices (axial, coronal, sagittal), oblique/curved slices. How: montage, roam through/fly through, gray scale/pseudo color.
As an example of the slice mode of display, Figure 12 shows a pseudo color display of the "same" MR slice of a patient's brain obtained at two different longitudinal time instances (and subsequently registered) for a multiple sclerosis patient. The two slices corresponding to the two time instances are assigned red and green hues. Where the slices match perfectly or where there has been no change (in say, the lesions), the display shows yellow because of the combination of red and green. At other places, either red or green shows up.
Volume mode
In this mode, what is displayed and how it is done are as follows. What: surfaces, interfaces, intensity distributions. How: maximum intensity projection (MIP) surface rendering, volume rendering.
In any combination of these methods, a technique of projection is needed to go from the higher-dimensional scene to the 2D screen of the monitor. Two approaches are pursued - ray casting [35], which consists of tracing a line perpendicular to the viewing plane from every pixel in the viewing plane into the scene domain, and voxel projection [69] which consists of directly projecting voxels encountered along the projection line from the scene onto the view plane. Either of these projection methods may be used in the following rendering methods. Voxel projection is generally considerably faster than ray casting.
Maximum Intensity Projection (MIP): In these methods [70, 71], the intensity assigned to a pixel in the rendition is simply the maximum of the scene intensity encountered along the projection line. Figure 11 (left) shows its application to MRA. This is the simplest 3D rendering technique. It is most effective when the objects of interest are the brightest in the scene, have simple 3D morphology, and have minimal gradation of intensity values. Contrast enhanced CT angiography and MRA [72] are ideal applications for this method. By appropriately mapping the scene intensity onto a new gray scale, so that the intensities in the object of interest appear with the highest intensity in the new scale irrespective of whether they are intermediate or low value in the given scene, the MIP idea can be generalized (GMIP [73]) to render objects of any intensity. The main advantage of MIP is that it requires no segmentation. However, the ideal conditions mentioned earlier frequently go unfulfilled, due (for example) to the presence of other bright objects such as clutter from surface coils in MR angiography, bone in CT angiography, or other obscuring vessels that may not be of interest. Consequently, some segmentation eventually becomes necessary.
Scene-based surface rendering: In these methods [74], object surfaces are portrayed in the rendition directly from the given scene. A threshold interval has to be specified to indicate the object of interest in the given scene. Clearly, the speed of rendering is of utmost importance here since the idea is that object renditions are created interactively directly from the scene as the threshold is changed. Instead of thresholding, any automatic, hard, boundary- or region-based segmentation method can be used. However, in this case, the parameters of the method will have to be specified interactively and sufficient speed of segmentation and rendition should be possible for this mode of visualization to be useful. Although at present rendering based on thresholding can be accomplished in about 0.01 sec - 0.1 sec on a 2 GHz Pentium PC using appropriate algorithms in software [75], more sophisticated segmentation methods (such as kNN) may not offer interactive speed.
The actual rendering itself consists of the following three basic steps: projection, hidden part removal, shading. These are needed to impart a sense of three-dimensionality to the rendered image that is created. Additional cues for three-dimensionality may be provided by techniques such as stereoscopic display, motion parallax by rotation of the objects, shadowing, and texture mapping. If ray casting is the method of projection chosen, then hidden part removal is done by stopping at the first voxel encountered along each ray that satisfies the threshold criterion [76]. The value (shading) assigned to the pixel (in the viewing plane) corresponding to the ray is determined as described below. If voxel projection is used, hidden parts can be removed by projecting voxels from the farthest to the closest (with respect to the viewing plane) and always overwriting the shading value, which can be achieved in a number of computationally efficient ways [77, 78].
The shading value assigned to a pixel p in the viewing plane is dependent on the voxel v that eventually projected onto p. The faithfulness with which this value reflects the shape of the surface around v depends to a large extent on the surface normal vector estimated at v. Two classes of methods are available for this purpose: object-based and scene-based. In object-based methods [79, 80], the vector is determined purely from the geometry of the shape of the surface in the vicinity of v. In scene-based methods [76], the vector is considered to be the gradient of the given scene at v; that is, the direction of the vector is the same as the direction in which scene intensity changes most rapidly at v. Given the normal vector N at v, the shading assigned to p is determined usually as [fd(v, N, L) + fs(v, N, L, V)] fD(v). Here fd is the diffuse component of reflection, fs is the specular component, fD is a component that depends on the distance of v from the viewing plane, and L and V are unit vectors indicating the direction of the incident light and of the viewing rays. The diffuse component is independent of the viewing direction, but depends only on L (as a cosine of the angle between L and N). It captures the scattering property of the surface. The specular component captures surface shininess. It is maximum in the direction of ideal reflection R whose angle with N is equal to the angle between L and N. This reflection dies off as a cosine function on either side of R. By weighting the three components in different ways, different shading effects can be created.
Scene-based volume rendering: In scene-based surface rendering, a hard object is implicitly created and rendered on the fly from the given scene. In scene-based volume rendering, a fuzzy object is implicitly created and rendered on the fly from the given scene. Clearly, surface rendering becomes a particular case of volume rendering. The central idea in volume rendering is to assign an opacity to every voxel in the scene that can take on any value in the range 0% to 100%. The opacity value is determined based on the objectness value at the voxel and how prominently we wish to portray this particular grade of objectness in the rendition. This opacity assignment is specified interactively via an opacity function, such as the one shown in Figure 9. Here, the vertical axis now indicates % opacity. Every voxel is now considered to transmit, emit, and reflect light. Our goal is to determine the amount of light reaching every pixel in the viewing plane. The amount of transmission depends on the opacity of the voxel. Its emission depends on its objectness and hence on opacity. The greater the objectness, the greater is the emission. Its reflection depends on the strength of the surface that is present in it. The greater the strength the greater is the reflection. Just as in surface rendering, there are three basic steps: projection, hidden part removal, shading or composting. The principles underlying projection are identical to those described under surface rendering.
Hidden part removal is much more complicated than for surface rendering. In ray casting, a common method is to discard all voxels along the ray from the viewing plane beyond a point at which the "cumulative opacity" is above a high threshold (say 90%) [81]. In voxel projection, in addition to a similar idea, a voxel can be discarded if the voxels surrounding it in the direction of the viewing ray have "high" opacity [36]. The shading operation, which is more appropriately called composting, is more complicated than for surface rendering. It has to take into account all three components - transmission, reflection, and emission. We may start from the voxel farthest from the viewing plane along each ray and work toward the front, calculating for each voxel its output light. The net light output by the voxel closest to the viewing plane is assigned to the pixel associated with the ray. Instead of this back-to-front strategy, calculations can be done from front-to-back, which is actually shown to be faster than the former [36].
As with surface rendering, voxel projection methods for volume rendering are substantially faster than ray casting. Figure 10 shows the CT knee data set rendered by using this method, wherein two types of tissues – bone and soft tissue - have been identified.
Object Based
In these methods of visualization, objects (hard or fuzzy) are explicitly defined first and then rendered. In difficult segmentation situations, or when segmentation is time-consuming or when it has too many parameters, it is not practical to do direct scene-based rendering. The intermediate step of completing object definition then becomes necessary. Depending on whether the object definition method is hard or fuzzy, we have object-based surface and volume rendering methods, respectively.
Object-based surface rendering: These methods take as input hard object descriptions and create renditions. The methods of projection, hidden-part removal, and shading are similar to those described under scene-based surface rendering. The only difference here is that, a variety of surface description methods have been investigated by using voxels [69, 77, 82, 83], voxel faces [78, 84, 85], triangles [31, 32, 38, 86] and other surface patches. Therefore the projection methods that are appropriate for the specific surface elements have been developed. Figure 4b shows a rendition of a child’s skull based on voxels while the same object is rendered by using triangles in Figure 13. Object-based surface rendering of large surfaces consisting of 10 million voxels/triangles can be done at the rate of about 10-20 frames per sec on modern PCs (2 GHz Pentium) entirely in software, which is about 10-30 times faster than rendering the same surfaces in hardware rendering engines [75, 87].
Object-based volume rendering: These methods take as input fuzzy object descriptions, which are in the form of a set of voxels wherein each voxel has an objectness value and a number of other parameters, such as gradient magnitude, associated with it [36]. Since the object description is more compact than the original scene, and since additional information to speed up computation can be stored as part of the object description, volume rendering based on fuzzy object descriptions can be done at interactive speeds on modern PCs entirely in software. Figure 14 shows a fuzzy object rendition of the data set in Figure 13. Figure 15 (a) shows a rendition of craniofacial bone and soft tissue both of which have been defined separately by using the fuzzy connected methods described earlier. If we use a direct scene-based volume rendering method by using the opacity function illustrated in Figure 9, then the skin would become inseparable from other soft tissues and would always obscure the rendition of muscles.
Misconceptions in visualization: Several statements relating to visualization appear frequently in the literature, which are not true. The following occur most frequently. (1) Surface rendering equated to thresholding: Clearly, thresholding is only one (actually the simplest) of the many available hard region- and boundary-based segmentation methods, the output of any of which can be surface rendered. (2) Volume rendering not requiring segmentation: The phrase "volume rendering" is very general and is used in different senses. This is clarified below. In whatever sense it is taken, the above statement is not true. The only useful volume rendering/visualization technique that does not require segmentation of any sort is maximum intensity projection. The opacity assignment schemes described in Figure 9 and under SCENE-BASED visualization are clearly fuzzy segmentation strategies, and they face the same problems that any segmentation method encounters. Note how clearly unconvincing it is to call opacity functions such as the one shown in Figure 9 not representing segmentation, but its particular manifestation when t1 = t2 and t3 = t4, which corresponds to thresholding, indeed segmentation. (3) The meaning of volume rendering: The meaning conveyed by this phrase varies quite a bit in its use in the literature. Frequently, it is used to refer to any scene-based rendering techniques, hard as well as fuzzy. It is also used to refer to object-based rendering techniques. In this general sense, clearly the slice mode of visualization is also captured within this phrase. It is better to reserve this term to refer to only fuzzy object rendering whether it is done via scene-based or object-based methods but not to hard object rendering methods.
Challenges: (1) Note that the preprocessing operations and subsequently the visualization operations can be applied in many different sequences to get to the desired end result. For example, the sequence filtering(interpolation(segmentation(rendering may produce significantly different renditions from those produced by interpolation(segmentation( filtering(rendering. Considering the different methods for each operation and the parameters associated with each operation, there is a large number of ways of producing the desired end results. A systematic study is needed to understand what combination of operations is best for a given application. Usually, whatever fixed combination is provided by the CAVA system is taken to be the best for the application. (2) Objective comparison of visualization methods. In view of (1), this becomes a daunting task. (3) Realistic tissue display including color, texture and surface properties.
MANIPULATION
The main purpose of these operations is, given an object system, to create another object system by changing objects or their relationship in the system. The main motivation for these operations is to simulate surgery procedures based on patient-specific scene data, and to develop aids for interventional and therapy procedures. Compared to Preprocessing and Visualization, work on Manipulation is in its infancy. Operations to cut, separate, add, subtract, move, mirror, stretch, compress, bend objects and their components have been developed. The idea here is that the user directly interacts with an object-based surface or volume rendition of the object system to execute these operations. Clearly, for the operations to be practically useful, they should be executable at interactive speeds.
ANALYSIS
The main purpose of these operations is, given a set of scenes or an object system, to generate a quantitative description of the morphology, architecture, and function of the object system. The goal of many 3D imagingCAVA applications is analysis of an object system. Although visualization is used as a visual aid, that in itself is not always the end goal. As such, many of the current application-driven works are in analysis. As in other operations, we may classify them into two groups: scene-based and object-based. In scene-based methods, quantitative description is based directly on scene intensities. Examples of measures include ROI statistics, density, activity, kinetics, perfusion, and flow. Object structural information derived from another modality is often used to guide the selection of regions for these measurements. In object-based methods, quantitative description is obtained from the objects based on their morphology, architecture, how they change with time, or on the relationship among objects in the system, and how that changes. Examples of measures include distance, length, curvature, area, volume, kinematics, and mechanics.
In summary, there seems to be a dichotomy that is pervasive in all CAVA operations, namely that these operations may be scene based or object based. The challenges encountered in most CAVA operations are attributable to the difficulties encountered in image segmentation since object information is vital for their VOI definition, filtering, interpolation, registration, detection, realistic rendering, manipulation, and analysis. Currently most of the efforts in CAVA technology are focused on segmentation, registration, and analysis. Future research in image segmentation is likely to explore more hybrid strategies to investigate how best to combine prior object knowledge and information that can be best extracted from the given image.
Slice Mode
Methods differ based on what is considered to constitute a "slice" and how is this information displayed.
What : natural slices How : montage
(axial, coronal, sagittal) roam through/fly through
oblique/curved slices grey scale/pseudo color
Figure 17 shows a montage display of the natural slices of a CT scene. Figure 18 demonstrates the extraction of an oblique slice from a CT scene of a child's head guided by a 3D display. This reslicing operation illustrates how visualization is needed for visualization itself. Figure 19 shows a pseudo color display of the "same" MR slice of a patient's brain obtained at two different longitudinal time instances (and subsequently registered) for a multiple sclerosis patient. The two slices corresponding to the two time instances are assigned red and green hues. Where the slices match perfectly or where there has been no change (in say, the lesions), the display shows yellow because of the combination of red and green. At other places, either red or green shows up.
Volume Mode
In this mode, what is displayed and how it is done are as follows.
What : surfaces How : surface rendering
Interfaces volume rendering
intensity distributions maximum intensity projection (MIP)
In any combination of these methods, a technique of projection is needed to go from the higher-dimensional scene to the 2D screen of the monitor. For 4- and higher-dimensional scenes, their 3D "cross sections" should be first determined before these techniques can be applied. Projections are then applied for going from 3D to 2D. Two approaches are pursued (see Figure 20) - ray casting [35], which consists of tracing a line perpendicular to the viewing plane from every pixel in the viewing plane into the scene domain, and voxel projection [73] which consists of directly projecting voxels encountered along the projection line from the scene onto the view plane. Either of these projection methods may be used in the following rendering methods. Voxel projection is generally considerably faster than ray casting.
Maximum Intensity Projection (MIP)
In these methods [74, 75], the intensity assigned to a pixel in the rendition is simply the maximum of the scene intensity encountered along the projection line. Figure 16 (left) shows its application to MRA. This is the simplest 3D rendering technique. It is most effective when the objects of interest are the brightest in the scene, have simple 3D morphology, and have minimal gradation of intensity values. Contrast enhanced CT angiography and MRA are ideal applications for this method, and consequently, MIP is commonly used in these applications [76], shading value. There are several computationally efficient ways of achieving this [73, 80-82].
The shading value assigned to a pixel p in the viewing plane is dependent on the voxel v
that eventually projected onto p. The faithfulness with which this value reflects the shape of the surface around v depends to a large extent on the surface normal vector estimated at v. Two classes of methods are available for this purpose: object-based and scene-based. In object-based methods [83, 84], the vector is determined purely from the geometry of the shape of the surface in the vicinity of v. In scene-based methods [79], the vector is considered to be the gradient of the given scene at v, that is the direction of the vector is the same as the direction in which scene intensity changes most rapidly at v. Given the normal vector N at v, the shading assigned to p is determined usually as [fd(v, N, L) + fs(v, N, L, V)] fD(v). Here fd is the diffuse component of reflection, fs is the specular component, fD is a component that depends on the distance of v from the viewing plane, and L and V are unit vectors indicating the direction of the incident light and of the viewing rays. The diffuse component is independent of the viewing direction, but depends only on L (as a cosine of the angle between L and N). It captures the scattering property of the surface. The specular component captures surface shininess. It is maximum in the direction of ideal reflection R whose angle with N is equal to the angle between L and N. This reflection dies off as a cosine function on either side of R. By weighting the three components in different ways, different shading effects can be created.
Volume-Rendering
In scene-based surface rendering described above, a hard object is implicitly created and rendered on the fly from the given scene. In scene-based volume rendering, a fuzzy object is implicitly created and rendered on the fly from the given scene. Clearly, surface rendering becomes a particular case of volume rendering. Further, volume rendering in this mode is generally much slower than surface rendering, requiring typically 3 sec - 20 sec even on specialized hardware [77]. Its main advantage is that it does not require segmentation of any sort. However, the ideal conditions mentioned above are frequently not fulfilled, for example, due to the presence of other bright objects such as clutter coming from surface coils in MRA and bone in CTA and other obscuring vessels which may not be of interest to us. Consequently, some segmentation eventually becomes necessary.
Surface Rendering
In these methods [78], object surfaces are portrayed in the rendition. A threshold interval has to be specified to indicate the object of interest in the given scene. Clearly, the speed of rendering is of utmost importance here since the idea is that object renditions are created interactively directly from the scene as the threshold is changed. Instead of thresholding, any automatic, hard, boundary- or region-based method can be used. However, in this case, the parameters of the method will have to be specified interactively and sufficient speed of segmentation and rendition should be possible for this mode of visualization to be useful. Although at present rendering based on thresholding can be accomplished in about 0.03 sec - 0.25 sec on a Pentium 300 using appropriate algorithms in software [62], more sophisticated segmentation methods (such as kNN) may not offer interactive speed.
The actual rendering itself consists of the following three basic steps: projection, hidden part removal, shading. These are needed to impart a sense of three dimensionality to the rendered image that is created. Additional cues for three dimensionality may be provided by techniques such as stereoscopic display, motion parallax by rotation of the objects, shadowing, and texture mapping.
If ray casting is the method of projection chosen, then hidden part removal is done by stopping at the first voxel encountered along each ray that satisfies the threshold criterion [79]. The value (shading) assigned to the pixel (in the viewing plane) corresponding to the ray is determined as described below. If voxel projection is used, hidden parts can be removed by projecting voxels from the farthest to the closest (with respect to the viewing plane) and always overwriting the rendering engines.
The central idea in volume rendering is to assign an opacity to every voxel in the scene that can take on any value in the range 0% to 100%. The opacity value is determined based on the objectness value at the voxel and how prominently we wish to portray this particular grade of objectness in the rendition. This opacity assignment is specified interactively via an opacity function, such as the one shown in Figure 13. Here, the vertical axis now indicates % opacity. Every voxel is now considered to transmit, emit, and reflect light. Our goal is to determine the amount of light reaching every pixel in the viewing plane. The amount of transmission depends on the opacity of the voxel. Its emission depends on its objectness and hence on opacity. The greater the objectness, the greater is the emission. Its reflection depends on the strength of the surface that is present in it. The greater the strength the greater is the reflection.
Just as in surface rendering, there are three basic steps: projection, hidden part removal, shading or compositing. The principles underlying projection are identical to those described under surface rendering.
Hidden part removal is much more complicated than for surface rendering. In ray casting, a common method is to discard all voxels along the ray from the viewing plane beyond a point at which the "cumulative opacity" is above a high threshold (say 90%) [85]. In voxel projection, in addition to a similar idea, a voxel can be discarded if the voxels surrounding it in the direction of the viewing ray have "high" opacity [36].
The shading operation, which is more appropriately called compositing, is more complicated than for surface rendering. It has to take into account all three components - transmission, reflection, and emission. We may start from the voxel farthest from the viewing plane along each ray and work toward the front, calculating for each voxel its output light. The net light output by the voxel closest to the viewing plane is assigned to the pixel associated with the ray. Instead of this back-to-front strategy, calculations can be done from front-to-back, which is actually shown to be faster than the former [36].
As with surface rendering, voxel projection methods are substantially faster than ray casting. Figure 21 shows the CT knee data set illustrated in Figure 3 rendered using this method, wherein three types of tissues - bone, fat, and soft tissue - have been identified.
3.2 OBJECT-BASED
In these methods of visualization, objects (hard or fuzzy) are explicitly defined first and then rendered. In difficult segmentation situations, or when segmentation is time-consuming or when it has too many parameters, it is not practical to do direct scene-based rendering. The intermediate step of completing object definition then becomes necessary. Depending on whether the object definition method is hard or fuzzy, we have object-based surface and volume rendering methods, respectively.
Surface Rendering
These methods take as input hard object descriptions and create renditions. The methods of projection, hidden-part removal and shading are similar to those described under scene-based surface rendering. The only difference here is that, a variety of surface description methods have been investigated using voxels [73, 80, 82], points, voxel faces [30, 81, 86, 87], triangles [31, 38, 88] and other surface patches. Therefore the projection methods that are appropriate for the specific surface elements have been developed. Figure 22 shows a rendition based on voxel faces of the craniofacial skeleton of an agnathia patient using CT data. Figure 8 shows renditions of the bones of the foot using the same method based on MR data.
Volume Rendering
These methods take as input fuzzy object descriptions which are in the form of a set of voxels wherein each voxel has an objectness value and a number of other parameters, such as gradient magnitude, associated with it [36]. Since the object description is more compact than the original scene, and since additional information to speed up computation can be stored as part of the object description, volume rendering based on fuzzy object description can be done at interactive speeds even on PCs such as a Pentium 300 entirely in software. In fact, the speed of rendering now (2 sec - 15 sec) is comparable to the speed of scene-based volume rendering using specialized hardware engines. Figure 23 shows a fuzzy object rendition of the data set in Figure 22. Figure 24(a) shows a rendition of craniofacial bone and soft tissue both of which have been defined separately using the fuzzy connected methods described earlier. Note that, if we use a direct scene-based volume rendering method using the opacity function illustrated in Figure 13, then the skin becomes inseparable from other soft tissues and always obscures the rendition of muscles as illustrated in Figure 24(b).
Misconceptions in Visualization
Several statements relating to visualization appear frequently in the literature which are not true. The following occur most frequently.
(1) Surface rendering equated to thresholding: Clearly, thresholding is only one (actually the simplest) of the many available hard region- and boundary-based segmentation methods, the output of any of which can be surface rendered.
(2) Volume rendering not requiring segmentation: The phrase "volume rendering" is very general and is used in different senses. This is clarified below. In whatever sense it is taken, the above statement is not true. The only useful volume rendering/visualization technique that does not require segmentation of any sort is maximum intensity projection. The opacity assignment schemes described in Figure 13 and under SCENE-BASED visualization are clearly fuzzy segmentation strategies, and they face the same problems that any segmentation method encounters. Note how clearly unconvincing it is to call opacity functions such as the one shown in Figure 13 not representing segmentation, but its particular manifestation when t1 = t2 and t3 = t4, which corresponds to thresholding, indeed segmentation.
(3) The meaning of volume rendering: The meaning conveyed by this phrase varies quite a bit in its use in the literature. Frequently, it is used to refer to any scene-based rendering techniques, hard as well as fuzzy. It is also used to refer to object-based rendering techniques. In this general sense, clearly the slice mode of visualization is also captured within this phrase. It is better to reserve this term to refer to only fuzzy object rendering whether it is done via scene-based or object-based methods but not to hard object rendering methods.
Challenges: (1) Note that the preprocessing operations and subsequently the visualization operations can be applied in many different sequences to get to the desired end result. For example, the sequence filtering(interpolation(segmentation(
rendering may produce significantly different renditions from those produced by interpolation(segmentation( filtering(rendering. Considering the different methods for each operation and the parameters associated with each operation, there is a large number of ways of producing the desired end results. Figure 25 shows five different sequences of such operations applied to a CT scene. A systematic study is needed to understand what combination of operations is best for a given application. Usually, whatever fixed combination is provided by the 3D imaging system is taken to be the best for the application. (2) Objective comparison of visualization methods. In view of (1), this becomes a daunting task. (3) Realistic tissue display including color, texture and surface properties.
4. MANIPULATION
The main purpose of these operations is: given an object system, to create another object system by changing objects or their relationship in the system. The main motivation for these operations is to simulate surgery procedures based on patient-specific scene data, and to develop aids for interventional and therapy procedures. Compared to Preprocessing and Visualization, work on Manipulation is in its infancy. Therefore, we will not discuss this in detail. Two classes of operations are being developed: rigid, deformable.
4.1 RIGID
Operations to cut, separate, add, subtract, move, and mirror objects and their components have been developed, using both hard [82, 89-91] and fuzzy object definitions [92]. The idea here is that the user directly interacts with an object-based surface or volume rendition of the object system to execute these operations. Clearly, for the operations to be practically useful, they should be executable at interactive speeds. This is indeed possible using both hard as well as fuzzy object definitions [82, 92] on a machine such as a Pentium 300.
Figure 26 shows a craniomaxillofacial osteotomy and segmental movement procedures done on a child's skull using a hard object definition and the associated manipulation operations [82].
4.2 DEFORMABLE
Operations to stretch, compress, bend, etc. are being developed.
Mechanical modeling of soft-tissue structures including muscles, tendons, ligaments, and capsules is complicated because forces generated by them and their behavior under external forces are difficult to determine especially in a patient-specific fashion. Therefore, past attempts have considered properties of soft-tissues to build generic models. Generic models based on local deformations are used as aid in facial plastic surgery [93] quite independent of the underlying bone and muscle, but treating only the skin surface. The use of multiple layers - skin, fatty tissue, muscle, and bone, where bone does not move - has been explored in different combinations to model facial expression animation [94-96]. Although attempts have been made to model soft-tissue in this fashion and in capturing their mechanical properties - for example, how a musculo-tendon actuator functions [97] or what the force-length function is [98] - no attempt seems to have been made in integrating hard-tissue (bone) changes with the soft-tissue modifications in a model. The reasons for this are: the lack of adequate visualization tools, the lack of tools to simulate osteotomy procedures and the lack of tools to integrate soft-tissue models with hard-tissue changes.
Challenges: This area is open for further research. Considering that most of the tissues in the body are deformable and movable, and the fact that object information in images is inherently fuzzy, basic fuzzy mechanics theories and algorithms need to be developed to carry out patient-specific object manipulation and analysis.
5. ANALYSIS
The main purpose of these operations is: given a set of scenes or an object system, to generate a quantitative description of the morphology, architecture, and function of the object system.
The goal of many 3D imaging applications is analysis of an object system. Although visualization is used as a visual aid, that in itself may not always be the end goal. As such, many of the current application-driven works are in analysis. Since they tend to be very application-specific, we will not go into a detailed discussion of analysis methodologies, but give an outline of the classes of analysis operations that are common among applications. As in other operations, we may classify them into group: scene-based and object-based.
5.1 SCENE-BASED
In these methods, quantitative description is based directly on scene intensities. Examples, of measures include ROI statistics, density, activity, perfusion and flow. Object structural information derived from another modality is often used to guide the selection of regions for these measurements.
5.2 OBJECT-BASED
In these methods, quantitative description is obtained from the objects based on their morphology, architecture, how it changes with time, or on the relationship among objects in the system, and how that changes. Examples of measures include distance, length, curvature, area, volume, kinematics, kinetics, and mechanics.
Challenges: Considering the fact that object information in images is fuzzy, fuzzy morphometry and mechanics theories and algorithms are needed to realistically analyze object information in images.
SOURCES OF DIFFICULTY IN 3D IMAGING
There are mainly two types of difficulties, relating to object definition and validation. Object definition difficulties have already been discussed in detail under Preprocessing. Difficulties in validation are discussed below. There are two types of entities that need to be validated: qualitative, quantitative.
6.1 VALIDATION: Qualitative
The purpose of qualitative validation is to compare visualization methods for a given task. Observer studies and ROC analysis can be used for this purpose [99]. A major challenge here is how to sift among the large number of combinations of operations, methods and their parameter settings, so that a small number of methods can be selected for formal ROC analysis.
6.2 VALIDATION: Quantitative
The purpose of quantitative validation is to assess the precision, accuracy, and efficiency of the measurement process.
Precision: This refers to the reliability of the method. This is usually easy to establish. It consists of repeating the measurement process and assessing variation using coefficient of variation, correlation coefficient, Kappa statistic, analysis of variance, etc. All steps that involve subjectivity have to be repeated (including for example, how the patient is positioned in the scanner).
Accuracy: This refers to how the measurement agrees with truth. We may handle accuracy for recognition and delineation separately. Establishing recognition accuracy needs histological assessment of object presence/absence for small objects. For big (anatomic) objects, an expert reader can provide truth. One can then apply ROC analysis.
Establishing delineation accuracy needs a point-by-point assessment of object grade. Truth is very difficult to establish. Usually the following surrogates of truth are used: physical phantoms, mathematical phantoms, manual (expert) delineation, simulation of mathematical objects and burying them in actual images, comparison with a process whose accuracy is known.
Efficiency: This refers to the practical viability of the method, say in terms of the number of studies that can be processed per hour. This has two components - computer time and operator time. The former does not matter as long as it is not impractical. The latter however is very crucial in determining whether or not a method is practically viable no matter how precise and accurate it is. This aspect of validation is usually ignored. It should be conducted and its statistical variation should be analyzed and reported.
In summary, generally there are two approaches to 3D imaging: scene-based and object-based. There are mainly two types of difficulties related to object definition and validation of delineation. There are also numerous challenges, as outlined throughout this paper, for mathematicians, engineers, physicists, and physicians.
REFERENCES
[ 1] A. Kaufman (ed.): A Tutorial on Volume Visualization. Los Alamitos, CA: IEEE Computer Society Press, Inc.
[ 2] K. Höhne, H. Fuchs and S. Pizer (eds.): 3D Imaging in Medicine, Algorithms, Systems, Applications. Berlin: Springer-Verlag, 1990.
[ 31] J. Udupa JK, and G. Herman GT (Eeditorss.).: 3D Imaging in Medicine. 2nd Edition. Boca Raton, FL: CRC Press, : Boca Raton, FL, 20001991.
[ 2] Sonka M, Hlavac V, Boyle R. Image Processing, Analysis, and Machine Vision. 2nd Edition. Brooks/Cole Publishing Company: Pacific Grove, CA, 1999.
[ 3] Yoo, TS (editor). Insight Into Images. A.K. Peters Ltd.: Wellesley, MA, 2004.
[ 4] J.K. Udupa JK, and R. Goncalves R: “Imaging transforms for visualizing surfaces and volumes.,” Journal of Digital Imaging, 1993; 6(4):213-236, 1993.
[ 5] G. Gerig G, O. Kübler O, R. Kikinis R, , F.A. Jolesz, FA.: “Nonlinear anisotropic filtering of MRI data.,” IEEE Transactions on Medical Imag, 1992;ing, 11:221-232, 1992.
[ 6] Saha PK, Udupa JK. Scale-based image filtering preserving boundary sharpness and fine structure. IEEE Trans Med Imag, 2001; 20(11):1140-1155.W.K. Pratt: Digital Image Processing: New York, NY: Wiley, 1991.
[ 7] M.R. Stytz MR, and R.W. Parrott RW. : “Using kriging for 3-D medical imaging.,” Computerized Medical ImagingComput Med Imag Graph 1993 and Graphi;cs, 17(6):421-442, 1993.
[ 8] S.P. Raya SP, and J.K. Udupa JK.: “Shape-based interpolation of multidimensional objects.,” IEEE Transact ions on Medical Imag 1990;ing, 9(1):32-42, 1990.
[ 9] W.E. Higgins WE, C. Morice C, and E.L. Ritman EL. : “Shape-based interpolation of tree-like structures in three-dimensional images.,” IEEE Transactions on Medical Imag 1993;ing, 12(3):439-450, 1993.
[10] G.T. Herman, J. Zheng and C.A. Bucholtz: “Shape-based interpolation,” IEEE Computer Graphics and Applications, 12(3):69-79, 1992.
[101] G.J. Grevera GJ, and J.K. Udupa JK. : “Shape-based interpolation of multidimensional grey-level images.,” IEEE Trans actions on Medical Imag 1996ing; , 15(6):882-892, 1996.
[112] D.T. Puff DT, D. Eberly D, and S.M. Pizer SM. : “Object-based interpolation via cores. ,” Iin Proceeding of SPIE: Medical Imaging’94, Image Processing 1994;, 2167:143-150, 1994.
[123] A. Goshtasby A, D.A. Turner DA, and L.V. Ackerman LV. : “Matching tomographic slices for interpolation.n,” IEEE Trans actions on Med ical Imag 1992;ing, 11(4):507-516, 1992.
[134] R.P. Woods RP, J.C. Mazziotta JC, and S.R. Cherry SR. : “MRI-PET registration with automated algorithm.,” Journal of Compututer Assististed Tomogrraphy , 1993; 17:536-546, 1993.
[14] Wells III WM, Viola P, Atsumi H, Nakajuma S, Kikinis R. Multi-modal volume registration by maximization of mutual information. Med Image Anal 1996; 1(1):35-51.
[15] Goshtasby AA. 2-D and 3-D Image Registration. Wiley Intersciences: Hoboken, NJ, 2005.
[15] C. Studholme, D.L.G. Hill and D.J. Hawkes: “Automated 3D registration of MR and CT images of the head,” Medical Image Analysis, 1:163-175, 1996.
[16] M.I. Miller MI, G.E. Christensen GE, Y. Amit Y, and U. Grenander U. : “Mathematical textbook of deformable neuroanatomies. ,” Proceedings of the Nat’ional Academy of Sciences of the United States of A 1993;America , 90:11944-11948, 1993.
[17] C.A. Pelizzari CA, G.T.Y. Chen GTY, D.R. Spelbring DR, R.R. Weichselbaum RR, and C-T. Chen C-T. : “Accurate three-dimensional registration of CT, PET, and/or MR images of the brain. ,” Journal of Computmputer Assist ed Tomography, 1989; 13:20-26, 1989.
[18] K.D. Toennies KD, , J.K. Udupa, JK, G.T. Herman GT, I.L. Wornom IL, and S.R. Buchman SR.: “Registration of three-dimensional objects and surfaces. ,” IEEE Compututer Graphics and Appli cations, 1990; 10(3):52-62, 1990.
[19] K.S. Arun KS, T.S. Huang TS, and S.D. Blostein SD.: “Least-squares fitting of two 3-D point sets.,” IEEE Transnsactions on Pattern Analysis andAMI 1987 Machine Intelligence;, 9:698-700, 1987.
[20] J.B.A. Maintz JBA, P.A. van den Elsen PA, and M.A. Viergever MA. : “Evaluation of ridge seeking operators for multimodality medical image matching. ,” IEEE Transactions on pattern PAnalysis and Machine I 1996ntelligence; , 18:353-365, 1996.
[21] R. Bajcsy R, and S. Kovacic S. : “Multiresolution elastic matching.,” Comput omputer Vision ision, Graph raphics and Imag mage Proc 1989rocessing;, 46:1-21, April 1989.
[22] Ng L, Ibanez L. Medical image registration, Chapter 10. In: Yoo TS (Editor). Insight Into Images. AK Peters Ltd.: Wellesley, MA, 2004.J.C. Gee, C. Barillot, L. Le Briquer, D.R. Haynor and R. Bajcsy: “Matching structural images of the human brain using statistical and geometrical image features,” SPIE Proceedings, 2359:191-204, 1994.
[23] Avants B, Sundaram T, Duda JT, Gee JC, Ng L. Non-rigid image registration, Chapter 11. In: Yoo TS (Editor). Insight Into Images. AK Peters Ltd.: Wellesley, MA, 2004.
[243] A.C. Evans AC, W. Dai W,, L. Collins L, , P. Neelin P, and S. Marrett S. : “Warping of a computerized 3-D atlas to match brain image volumes for quantitative neuroanatomical and functional analysis.,” SPIE Proc eedings, 1991; 1445:236-246, 1991.
[254] L. Gong L, and C. Kulikowski C. : “Comparison of image analysis processes through object-centered hierarchical planning. ,” IEEE Transactions on Pattern Analysis and Medicine I 1995;ntelligence, 17(10):997-1008, 1995.
[25] S. Raya: “Low-level segmentation of 3-D magnetic resonance brain images – a rule-based system,” IEEE Transactions on Medical Imaging, 9(3):327-337, 1990.
[26] D. Collins D, and T. Peters T. : “Model-based segmentation of individual brain structures from MRI data. ,” SPIE Proc eedings,1992, 1808:10-23, 1992.
[27] J. Gee J, M. Reivich M, and R. Bajcsy R. : “Elastically deforming 3D atlas to match anatomical brain images. s,” Journal of Computer Assisted Tomogr 1993; raphy, 17(2):225-236, 1993.
[28] M.I. Miller, G.E. Christensen, Y. Amit and U. Grenander: “Mathematical textbook of deformable neuroanatomies,” Proceedings of National Academy of Sciences of the United States of America, 90(24):11944-11948, 1993.
[289] G.E. Christensen GE, R.D. Rabbitt R, and M.I. Miller MI. :3 “3-D brain mapping using a deformable neuroanatomy. ,” Physsics in Medicine and Biol 1994ogy, 39:609-618, 1994.
[29] Cootes TF, Taylor CJ, Cooper DH, Graham J. Active shape models – Their training and application. Comput Vision Imag Und 1995; 61(1):38-59.
[30] J. Udupa JK, S. Srihari S, and G. Herman GT. : “Boundary detection in multidimensions.,” IEEE Transaction s on Pattern Analysis and Machine Intelligence, 1982; PAMI-4:41-50, 1982.
[31] G. Wyvill G, C. McPheeters C, and B. Wyvill B.: “Data structures for soft objects. ,” The Visual Comput 1986;er , 2(4):227-234, 1986.
[32] W. Lorensen W, and H. Cline H. : “Marching cubes: A high resolution 3D surface construction algorithm. ,” Computer Graph 1989;ics, 23(3):185-194, 1989.
[33] A. Doi A, and A. Koide A. : “An efficient method of triangulating equi-valued surfaces by using tetrahedral cells.,” IEICE Transactions on Communications Electronics Information and Syst 1991;ems, E-74(1):214-224, 1991.
[34] H. Liu H. : “Two- and three-dimensional boundary detection. ,” Comput omputer Graph raphics and Imag mage Procrocessing, 1977; 6:123-134, 1977.
[35] M. Levoy M. : “Display of surfaces from volume data.,” IEEE Comput er Graph ics and Appli 1988;cations, 8(3):29-37, May 1988.
[36] J. Udupa JK, and D. Odhner D. : “Shell rendering. ,” IEEE Comput er Graph ics and Appli 1993;cations , 13(6):58-67, 1993.
[37] J. Udupa JK, , H. Hung H, and K. Chuang K. : “Surface and volume rendering in 3D imaging: A comparison. ,” J ournal of Digital Imag 1990ing; , 4:159-168, 1990.
[38] A. Kalvin A. : Segmentation and Surface-Based Modeling of Objects in Three-Dimensional Biomedical Images, Ph.D. Thesis., Department of Computer Science. New York University: New York, NY, March 1991.
[39] G.T. Herman GT, and H.K. Liu HK. : “Dynamic boundary surface detection. ,” Comput er Graphics and Image Proc 1978;essing, 7:130-138, February 1978.
[40] D. Pope D,, D. Parker D, D. Gustafson D, and P. Clayton P. : “Dynamic search algorithms in left ventricular border recognition and analysis of coronary arteries. ,” IEEE Proceedings of Computers in Cardiol 1984;ogy , 9:71-75, 1984.
[41] A. Amini A, T. Weymouth T, and R. Jain R. : “Using dynamic programming for solving variational problems in vision. ,” IEEE Transaction on Pattern Analysis and Machine Intelligence, 1990; 12(9):855-867, 1990.
[42] D. Gieger D, A. Gupta A, L. Costa L, and J. Vlontzos J. : “Dynamic programming for detecting, tracking and matching deformable contours. ,” IEEE Trans actions on Pattern Analysis and Machine Intelligence, 1995; 17(3):294-302, 1995.
[43] M. Kass M, A. Witkin A, and D. Terzopoulous D. : “Snakes: Active contour models.,” Int’nternational Journal of Computer Vision 1987; , 1(4):321-331., 1987.
[44] S. Lobregt S, and M. Viergever M. : “A discrete dynamic contour model. ,” IEEE Trans sactions on Medical Imaging, 1995; 14(1):12-24, 1995.
[45] L. Cohen L. : “On active contour models.,” Computer Vision, GGraphics and Image Processing: Image Understanding 1991; , 53:211-218, 1991.
[46] I. Cohen, I, L.D. Cohen LD, and N. Ayache N. : “Using deformable surfaces to segment 3-D images and infer differential structures. ,” Computer Vision, Graphics, and Image Processing: Image Understanding, 1992; 56(2):242-263, .September 1992.
[47] T. McInerney T, and D. Terzopoulous D.. : “A dynamic finite element surface model for segmentation and tracking in multidimensional medical images with application to cardiac 4D image analysis. ,” Comput erized Med ical Imag ing and Graphics 1995; , 19(1):69-93, January 1995.
[48] Falcao A, Udupa JK, Samarasekera S, Sharma S, Hirsch BE, Lotufo R. User-steered image segmentation paradigms: Live wire and live lane. Graph Mod Imag Proc 1998; 60(4):233-260.
A.X. Falcao, J.K. Udupa, S. Samarasekera and B.E. Hirsch: “User-steered image boundary segmentation,” SPIE Proceedings, 2710:279-288, 1996.
[49] Falcao A, Udupa JK, Miyazawa FK. An ultra-fast user-steered image segmentation paradigm: Live-wire-on-the-fly. IEEE Trans Med Imag 2000; 19(1)55:62.
A. Falcao and J.K. Udupa: “Segmentation of 3D objets using live wire,’ SPIE Proceedings, 3034:228-235, 1997.
[50] R.O. Duda and P.E. Hart: Pattern Classification and Scene Analysis. New York, NY: John Wiley & Sons, 1973.Cootes T, Taylor O. Modelling object appearance using the grey-level surface. In: E. Hancock (Editor). 5th British Machine Vision Conference, pp. 479-488. BMVA Press: York, England, 1994.
[51] Cootes T, Page G, Jackson C, Taylor C. Statistical grey-level models for object location and identification. Image and Vision Comput 1996; 14(8):533-540.
J.C. Bezdek, L.O. Hall and L.P. Clarke: “Review of MR image segmentation techniques using pattern recognition,” Medical Physics, 20(4):1033-1048, 1993.
[52] Duda RO, Hart PE, Stork DG. Pattern Classification. John Wiley & Sons: New York, NY, 2001.
[53] Bezdek JC, Hall LO, Clarke LP. Review of MR image segmentation techniques using pattern recognition. Med Phys 1993; 20(4)1033-1048.
[54] Clarke LPI, Velthuizen RP, Camacho MA et al. MRI segmentation: Methods and application. J Magn Res Imag 1995, 13(3):343-368.
L. Clarke, R.P. Velthuizen, M.A. Camacho, J.J. Heine, M. Vaidyanathan, L.O. Hall, R.W. Thatcher and M.L. Silbiger: “MRI segmentation: Methods and applications,” Journal of Magnetic Resonance Imaging, 13(3):343-368, 1995.
[53] M. Vannier, R. Butterfield, D. Jordan and W. Murphy: “Multi-spectral analysis of magnetic resonance images,” Radiology, 154:221-224, 1985.
[54] H. Cline, W. Lorensen, R. Kikinis and F. Jolesz: “Three-dimensional segmentation of MR images of the head using probability and connectivity,” Journal of Computer Assisted Tomography, 14(6):1037-1045, 1990.
[55] R. Kikinis R, M. Shenton, M, F. Jolesz F. , G. Gerig, J. Martin, M. Anderson, D. Metcalf, C. Guttmann, R. McCarley, W. Lorensen and H. Cline: “Quantitative analysis of brain and cerebrospinal fluid spaces with MR images. ,” J ournal of Magn etic Res onance Imag 1992; ing, 2:619-629, 199.2.
[56] J.R. Mitchell JR, S.J. Karlick SJ, D.H. Lee DH, and A. Fenster: A. “Computer-assisted identification and quantification of multiple sclerosis lesions in MR imaging volumes in the brain. ,” J ournal of Magn etic Res onance Imag ing, 1994; 4:197-208,. 1994.
[57] M. Kohn M, N. Tanna N, G. Herman GT, S. Resnick, P. Mozley, R. Gur, A. Alavi, R. Zimmerman et al. and R. Gur: “Analysis of brain and cerebrospinal fluid volumes with MR imaging. Part I: Methods, reliability and validation. ,” Radiol 1991;ogy, 178:115-122, 1991.
[58] M. Kamber, R. Shingal, D.L. Collins, G.S. Francis and A.C. Evans: “Model-based 3-D segmentation of multiple sclerosis lesions in magnetic resonance brain images,” IEEE Transactions on Medical Imaging, 14:442-453, 1995.
[59] W.M. Wells, III, W.E.L. Grimson, R. Kikinis and F.A. Jolsez: “Adapative segmentation of MRI data,” IEEE Transactions on Medical Imaging, 15:429-442, 1996.
[5860] R. Drebin R, L. Carpenter L, and P. Hanrahan: P. “Volume rendering. ,” Computer Graph 1988ics; , 22:65-74, 1988.
[5961] J.C. Bezdek JC : Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum: New York, N.Y.:NY, Plenum, 1981.
[60] Bezdek JC. A convergence theorem for the fuzzy ISODATA clustering algorithms. IEEE Trans PAMI 1980, 2:1-8.
[61] Pham DL, Prince JL. Adaptive fuzzy segmentation of magnetic resonance images. IEEE Trans Med Imag 1999; 18:737-752.
[62] J.K. Udupa JK, D. Odhner D, , S. Samarasekera S, R. Goncalves, K. Iyer, K. Venugopal and et al.S. Furuie: “3DVIEWNIX: An open transportable, multidimensional, multimodality, multiparametric imaging software system. ,” SPIE Proceedings, 1994; 2164:58-73., 1994.
[63] P.J. Burt, T.H. Hong and A. Rosenfeld: “Segmentation and estimation of region properties through co-operative hierarchical computation,” IEEE Transactions on System Man and Cybernetics, SMC-11:802-809, 1981.
[634] T.H. Hong TH, and A. Rosenfeld: A. “Compact region extraction using weighted pixel linking in a pyramid. ,” IEEE Trans actions on Pattern Analysis and MMachine Intelligence, 1984; PAMI-6:222-229., 1984.
[65] S. Dellepiane and F. Fontana: “Extraction of intensity connectedness for image processing,” Pattern Recognition Letters, 16:313-324, 1995.
[646] J. Udupa JK, and Samarasekera S.: “Fuzzy connectedness and object definition: Theory algorithms and applications in image segmentation. ,” Graph Graphical Mod odels and Image mage Proc 1996rocessing;, 58(3):246-261, 1996.
[65] Udupa JK, Samarasekera S. Fuzzy connectedness and object definition: Theory, algorithms, and applications in image segmentation. Graph Mod Image Proc 1996; 58(3):246-261.
[66] Udupa JK, Saha PK. Fuzzy connectedness in image segmentation. IEEE Proc 2003; 91(10):1649-1699.
[67] J.K. Udupa, L. Wei, S. Samarasekera, Y. Miki, M.A. van Buchem and R.I. Grossman: “Multiple sclerosis lesion quantification using fuzzy connectedness principles,” IEEE Transactions on Medical Imaging, 16(5):598-609, 1997.Chu S, Yuille A. Region competition: Unifying snakes, region growing and Bayes/MDL for multi-band image segmentation. IEEE Trans PAMI 1996; 18:884-900.
[68] S. Samarasekera, J.K. Udupa, Y. Miki and R.I. Grossman: “A new computer-assisted method for enhancing lesion quantification in multiple sclerosis,” Journal of Computer Assisted Tomography, 21(1):145-151, 1997Chakraborty A, Staib L, Duncan J. Deformable boundary finding in medical images by integrating gradient and region information. IEEE Trans Med Imag 1996; 15:859-870.
[69] Y. Miki, R.I. Grossman, S. Samarasekera, J.K. Udupa, M.A. van Buchem, B.S. Cooney, S.N. Pollack, D.K. Kolson, M. Polansky and L.J. Mannon: “Clinical correlation of computer assisted enhancing lesion quantification in multiple sclerosis,” American Journal of Neuroradiology, 18:705-710, 1997.Frieder G, Gordon D, Reynolds R. Back-to-front display of voxel-based objects. IEEE Comput Graph Appl 1985; 5:52-60.
[70] Napel S, Marks MP, Rubin GD et al. CT angiography with spiral CT and maximum intensity projection. Radiol 1992; 185:607-610.
[M.A. van Buchem, J.K. Udupa, F.H. Heyning, M.P. Boncoeur-Martel, Y. Miki, J.C. McGowan, D.L. Kolson, M. Polansky and R.I. Grossman: “Quantitation of macroscopic and microscopic cerebral disease burden in multiple sclerosis,” American Journal of Neuroradiology, 18:1287-1290, 1997.
[71] Brown DG, Riederer SJ. Contrast-to-noise ratios in maximum intensity projection images. Magn Res Med 1992; pp. 130-137.
J.K. Udupa, D. Odhner, J. Tian, G. Holland and L. Axel: “Automatic clutter-free volume rendering for MR angiography using fuzzy connectedness,” SPIE Proceedings, 3034:114-119, 197.
[72] J.K. Udupa, J. Tian, D.C. Hemmy and P. Tessier: “A pentium-based craniofacial 3D imaging and analysis system,” Journal of Craniofacial Surgery, 8(5):333-339, 1997Hertz SM, Baum RA, Owen RS, Holland GA, Logan DR, Carpenter JP. Comparison of magnetic resonance angiography and contrast arteriography in peripheral arterial stenosis. Amer J Surg 1993; 166(2):112-116..
[73] G. Frieder, D. Gordon and R. Reynolds: “Back-to front display of voxel-based objects,” IEEE Computer Graphics and Applications, 5:52-60, 1985.Grevera GJ, Udupa JK. GMIP-generalized maximum intensity projection. SPIE Proc 2004; 5467:636-645.
[74] Goldwasser S, Reynolds R. Real-time display and manipulation of 3-D medical objects: The voxel machine architecture. Comput Vision Graph Imag Proc 1987; 39(4)1-27.
D.G. Brown and S.J. Riederer: “Contrast-to-noise ratios in maximum intensity projection images,” Magnetic in Resonance and Medicine, pp. 130-137, 1992.
[75] Grevera GJ, Udupa JK, Odhner D. An order of magnitude faster surface rendering in software on a PC than using dedicated rendering hardware. IEEE Trans Vis Comput Graph 2000; 6(4):335-345.
S. Schreiner, C.B. Paschal and R. L. Galloway: “Comparison of projection algorithms used for the construction of maximum intensity projection images,” Journal of Computer Assisted Tomography, 20:56-67, 1996.
[76] S. Napel, M.P. Marks, G.D. Rubin, M.D. Dake, C.H. McDonnell, S.M. Song, D.R. Enzmann and R.B. Jeffrey: “CT angiography with spiral CT and maximum intensity projection,” Radiology, 185-607-610,1992.
[77] S.M. Hertz, R.A. Baum, R.S. Owen, G.A. Holland, D.R. Logan and J.P. Carpenter: “Comparison of magnetic resonance angiography and contrast arteriography in peripheral arterial stenosis,” American Journal of Surgery, 166:112-116, 1993.
[78] S. Goldwasser and R. Reynolds: “Real-time display and manipulation of 3-D medical objects: The voxel machine architecture,” Computer Vision, Graphics and Image Processing, 39(4):1-27, 1987.
[7976] K.H. Höhne KH, and R. Bernstein R. : “Shading 3D images from CT using gray-level gradients. ,” IEEE Transactions on Med ical Imag 1986ing, 5:45-47, 1986.
[7780] R. Reynolds R, D. Gordon D, and L. Chen L. : “A dynamic screen technique for shaded graphics display of slice-represented objects.,” Comput omputer Vision ision Graph raphics and Imag Image Proc 1987rocessing, 38:275-298, 1987.
[78] Herman GT, Udupa JK. Display of 3-D information in 3-D digital images: Computational foundations and medical application. IEEE Comput Graph Appl 1983; 3:39-46.
[79] Chen LS, Herman GT, Reynolds RA et al. Surface rendering in the cuberille environment. IEEE Comput Graph Appl 1985; 5:33-43.
[81] G. Herman and L. Liu: “Display of three-dimensional information in computed tomography,” Journal of Computer Assisted Tomography, 1:155-160, 1977.
[82] J. Udupa and D. Odhner: “Fast visualization, manipulation, and analysis of binary volumetric objects,” IEEE Computer Graphics and Applications, 11(6):53-62, 1991.
[83] L.S. Chen, G.T. Herman, R.A. Reynolds, et al.: “Surface rendering in the cuberille environment,” IEEE Computer Graphics and Applications, 5:33-43, 1985.
[804] D. Gordon D, and R.A. Reynolds RA.: “Image-space shading of three-dimensional objects.,” Comput omputer Vision ision, Graph Graphics and Imag mage Proc 1985Processing;, 29:361-376, 1985.
[815] M. Levoy M. : “Display of surfaces from volume data, ,” ACM Transactions on Graph 1990;ics, 9:245-271, 1990.
[826] Udupa JK, Odhner D. Fast visualization, manipulation and analysis of binary volumetric objects. IEEE Comput Graph Appl 1991; 11(6):53-62.
[83] Lacroute P. Analysis of a parallel volume rendering system based on the shear-warp factorization. IEEE Trans Vision Comput Graph 1996; 2(3):218-231.
[84] Udupa JK. Multidimensional digital boundaries. CVGIP: Graph Models Image Proc 1994:50(4):311-323.
J.K. Udupa: “Multidimensional digital boundaries,” CVGIP: Graphical Models and Image Processing, 50(4):311-323, 1994.
[857] E. Artzy E, G. Frieder G, and G. Herman GT. : “The theory, design, implementation and evaluation of a three-dimensional surface detection algorithm. ,” Comput omputer Graph raphics and Imag mage Processing, 1981; 15:1-24, 1981.
[868] J.-H. Chuang J-H, and W.-C. Lee W-C. : “Efficient generation of isosurfaces in volume rendering. ,” Computer & Graph 1995ics; , 19(6):805-812, 1995.
[879] Grevera GJ, Udupa JK, Odhner D. T-shell rendering and manipulation. SPIE Proc 5744, in press.C. Cutting, B. Grayson, F. Bookstein, L. Fellingham and J. McCarthy: “Computer-aided planning and evaluation of facial and orthognathic surgery,” Computers in Plastic Surgery, 13:449-461, 1986.
[90] S. Trivedi: “Interactive manipulation of three-dimensional binary images,” The Visual Computer, 2:209-218, 1986.
[91] V.V. Patel, M.W. Vannier, J.L. Marsh and L.-J. Lo: “Assessing craniofacial surgical simulation,” IEEE Computer Graphics and Applications, 16(1):46-54, 1996.
[92] D. Odhner and J.K. Udupa: “Shell manipulation: Interactive alteration of multiple-material fuzzy structures,” SPIE Proceedings, 2431:35-42, 1995.
[93] A. Paouri, N.M. Thalmann and D. Thalmann: “Creating realistic three-dimensional human shape characters for computer-generated films,” in Proceedings of Computer Animation’91, Tokyo: Springer-Verlag, pp. 89-100, 1991.
[94] K. Waters: “A muscle model for animating three dimensional facial expression,” Proceedings of SIGGRAPH’87, 21(3):17-24, 1987.
[95] S. Platt and N. Badler: “Animating facial expressions,” Proceedings of SIGGRAPY’81, pp. 245-252, 1981.
[96] D. Terzopoulos and K. Waters: “Techniques for realistic facial modeling and animation,” in Proceedings of Computer Animation’91, Tokyo: Springer-Verlag, pp. 59-74, 1991.
[97] F.E. Zajac, E.L. Topp and P.J. Stevenson: “A ‘dimensionless’ musculotendon model,” Proceedings of IEEE Engineering in Medicine and Biology, 1986.
[98] S. Jianhua, M.N. Thalmann and D. Thalmann: “Muscle-based human body deformations,” in Proceedings of CAD/Graphics’93, Beijing, China, pp. 95-100, 1993.
[99] M.W. Vannier, C.F. Hildebolt, J.L. Marsh, et al.: “Cranio-synostosis: Diagnostic value of three-dimensional CT reconstruction,” Radiology, 173:669-673, 1989.
FIGURE CAPTIONS
FIGURE 1: A schematic representation of 3D imaging CAVA systems.
FIGURE 2: Illustration used for defining the basic terminology.
FIGURE 3: A CT slice of a knee illustrating graded composition of intensities and their hanging-togetherness. The voxels in the same object, for example, the femur, have considerably different values assigned to them. In spite of this gradation of values, we have no difficulty in identifying the voxels as belonging to the same object (hanging-togetherness).
FIGURE 4a: Illustration of ROI and IOI Operations. A CT slice of a head with rectangular ROI indicated on it.
FIGURE 4b: A histogram of the intensities of the scene shown in Figure 4(a) with an IOI indicated.
FIGURE 4c: The slice in the output scene corresponding to the slice in Figure 4(a) produced by this volume of interest operation.
FIGURE 35a: Illustration of suppressing and enhancing filtering operations. A CTAn MRI slice of a human head.
FIGURE 35b: The slice in Figure 35(a) after applying a smoothing diffusion filter [6]. Note that noise is suppressed in regions of uniform intensity but edges are also not blurred.
FIGURE 35c: The slice in Figure 35(a) after applying an edge- enhancing filter. Note that regions of uniform intensity are not enhanced since the gradient in these regions is small. However, the boundaries (especially of the skin and bone ) are enhanced.
FIGURE 46a: Illustration of shape-based interpolation. A CT scene of a child’s skull was thresholded to create a binary scene first. This scene was then interpolated via the shape-based method at a coarse resolution and was surface rendered.
FIGURE 46b: The binary scene was interpolated (shape-based) at a fine resolution and then surface rendered.
FIGURE 5a7: Illustration of intra modality scene-based registration.; Top: An MRI (PD) slice situated approximately at the same location in the head of a multiple sclerosis patient. The slicenes were acquired at 24 different time instances. Bottom: The slice at the same location obtained after 3D registration of the twofour scenes and reslicing of the scenes to get the same slice. The progression of the hyperintense lesions with time is now readily seen which was not apparent prior to registration.
FIGURE 85b: Motion of the bones of the foot defined in MR scenes for the two positions shown are determined by registering the respective bone surfaces in the two positionsIllustration of inter modality scene-based registration. An MRI (PD) slice of a galactosemia patient’s brain and his corresponding PET slice before registration overlaid (left) and after registration in 3D space (right).
FIGURE 9: A rendition of the fuzzy boundaries of “bone” detected in the data of Figure 3 using both intensity and gradient criteria.
FIGURE 610: Illustration of the live-wire method on a MR slice of a foot. The boundary of interest is that of the talus. The contour shown represents one live-wire segment from the first point on the far right to the second point onf the far left.
FIGURE 711: Illustration of thresholding by specifying an intensity interval on a histogram (middle) of a scene of the head (left). The segmented object is shown as a binary scene on the rightactive shape model based boundary segmentation on the MR slice of Figure 6. The two contours shown represent the initial model contour (yellow) and the final delineated boundary (green).
FIGURE 812a: A T2 and a PD slice of a MR scene of a patient’s head.
FIGURE 812b: The scatter plot of the slices shown in Figure 12(a) together with a cluster outline indicated for CSF.
FIGURE 812c: The segmented binary slice showing CSF.
FIGURE 913: Fuzzy thresholding. If the voxel value is less than t1 or greater than t4, the voxel’s objectness is 0. If the voxel value is less thanlies between t2 and t3, the objectness is 100%. For other values, the objectness lies between 0 and 100%.
FIGURE 104: A rendition of bone and soft-tissue which are segmented by using fuzzy thresholding of the CT scene in Figure 3of a subject’s knee.
FIGURE 15a: A slice of a T2-PD scene pair of a patient’s head.
FIGURE 15b: A slice of the union of 3D fuzzy connected white matter and grey matter objects detected from the scene in (a).
FIGURE 15c: A slice of the 3D fuzzy connected CSF object detected from the scene in (a).
FIGURE15d: A slice of the union of 3D fuzzy connected multiple sclerosis lesions detected from the scene in (a).
FIGURE 116a: A (maximum intensity projection) 3D rendition of an MRA scene.
FIGURE 116b: A volume (maximum intensity projection) 3D rendition of the 3D fuzzy connected vessels detected from the scene rendered in (a).
FIGURE 11c: A volume rendition of the arteries and veins separated via fuzzy connectedness.
FIGURE 17: A montage display of a 3D CT scene of the head of a patient.
FIGURE 18: Illustration of reslicing guided by a 3D display. The CT data pertain to a patient with a craniofacial disorder. A plane is selected interactively via the 3D display to indicate the orientation of the slice plane. The slice corresponding to the oblique plane is shown on the right.
FIGURE 129a: Illustration of pseudo color display. Approximately the same MR slice of a patient’s head taken at two different times, one displayed in green and the other in red. Where there is a match, the display appears yellow. Green and red indicate mismatch.
FIGURE 129b: The same slice displayed after a 3D scene-based registration. Now green and red indicate either a registration error or change in an object such as a lesion between the two time instances.
FIGURE 20: Projections are created for rendition either by ray casting from the viewing plane to the scene or by projecting voxels from the scene to the viewing plane.
FIGURE 21: Scene-based volume rendering of CT knee data of Figure 3, showing bone, fat, and soft-tissue.
FIGURE 1322: An object-based surface rendition of the skull of a child by using triangles to represent the surfacewith agnathia.
FIGURE 1423: An object-based volume rendition of the data of Figure 1322. A fuzzy object representation was first obtained by using fuzzy thresholding illustrated in Figure 913.
FIGURE 1524a: Object-based volume rendering of bone and soft-tissues. Bone and soft tissue structures (muscles) were detected as separate 3D fuzzy connected objects from a craniofacial 3D CT scene. Skin is effectively peeled off because of its weak connectedness to muscles.
FIGURE 24b: The scene in (a) rendered directly using scene-based volume rendering via the opacity function illustrated in Figure 3 separately for bone and soft tissue. Here, skin becomes inseparable from muscles (since they have similar CT numbers) and hence obscures the rendition of the muscles.
FIGURE 25: Skull from CT data rendered using five different sequences of preprocessing and visualization operations.
FIGURE 26: Examples of virtual surgery operations (rigid manipulations) on a child’s skull rendered from CT data. The operation mimics an osteotomy procedure used in craniomaxillofacial surgery for advancing the frontal bone.
FIGURE 1: A schematic representation of 3D imagingCAVA systems.
FIGURE 2: Illustration used for defining basic terminology.
FIGURE 2: Illustration used for defining the basic terminology.
[pic][pic][pic]
FIGURE 3: (a) Illustration of suppressing and enhancing filtering operations. An MRI slice of a human head. (b) The slice in Figure 3(a) after applying a smoothing diffusion filter [6]. Note that noise is suppressed in regions of uniform intensity but edges are not blurred. (c) The slice in Figure 3(a) after applying an edge-enhancing filter. Note that regions of uniform intensity are not enhanced since the gradient in these regions is small. However, the boundaries are enhanced.
[pic][pic]
FIGURE 4: (a) Illustration of shape-based interpolation. A CT scene of a child’s skull was thresholded to create a binary scene first. This scene was then interpolated via the shape-based method at a coarse resolution and was surface rendered. (b) The binary scene was interpolated (shape-based) at a fine resolution and then surface rendered.
[pic]
FIGURE 5: (a) Illustration of intra modality scene-based registration. Top: An MRI (PD) slice situated approximately at the same location in the head of a multiple sclerosis patient. The scenes were acquired at 3 different time instances. Bottom: The slice at the same location obtained after 3D registration of the two scenes and after reslicing of the scenes to get the same slice. The progression of the hyperintense lesions (arrows) with time is now readily seen which was not apparent prior to registration.
[pic]
FIGURE 5: (b) Illustration of inter modality scene-based registration. An MRI (PD) slice of a galactosemia patient’s brain and his corresponding PET slice before registration overlaid (left) and after registration in 3D space (right).
[pic]
FIGURE 6: Illustration of the live-wire method on a MR slice of a foot. The boundary of interest is that of the talus. The contour shown represents one live-wire segment from the first point on the far right to the second point on the far left.
[pic]
A CT slice of a knee illustrating graded composition of intensities and hanging-togetherness. The voxels in the same object, for example, the femur, have considerably different values assigned to them. In spite of this gradation of values, we have no difficulty in identifying the voxels as belonging to the same object (hanging-together ness).
FIGURE 4a: Illustration of ROI and IOI operations. A CT slice of a head with a rectangular ROI indicate on it.
[pic]
FIGURE 7: Illustration of active shape model based boundary segmentation of the MR slice of Figure 6. The two contours shown represent the initial model contour (red) and the final delineated boundary (green).
[pic]
(a)
[pic][pic]
(b) (c)
FIGURE 8: (a) A T2 and a PD slice of a MR scene of a patient’s head. (b) The scatter plot of the slices shown in (a) together with a cluster outline indicated for CSF. (c) The segmented binary slice showing CSF.
[pic]
FIGURE 54b: (b) Illustration of inter modality scene-based registration. An MRI (PD) slice of a galactosemia patient’s brain and his corresponding PET slice before registration overlaid (left) and after registration in 3D space (right).A histogram of the intensities of the scene shown in Figure 4(a) with an IOI indicated.
[pic]
FIGURE 6: Illustration of the live-wire method on a MR slice of a foot. The boundary of interest is that of the talus. The contour shown represents one live-wire segment from the first point on the far right to the second point on the far left.
[pic]
FIGURE 7: Illustration of active shape model based boundary segmentation of the MR slice of Figure 6. The two contours shown represent the initial model contour (yellow) and the final delineated boundary (green).
FIGURE 4c: The slice in the output scene corresponding to the slice in Figure 4(a) produced by this volume of interest operation.
FIGURE 5a: Illustration of suppressing and enhancing filtering operations. A CT slice of a human head.
FIGURE 5b: The slice in Figure 5(a) after applying a smoothing filter. Note that noise is suppressed in regions of uniform intensity but edges are also blurred.
FIGURE 5c: The slice in Figure 5(a) after applying an edge enhancing filter. Note that regions of uniform intensity are not enhanced since the gradient in these regions is small. However, the boundaries (especially of the skin and bone) are enhanced.
[pic]
[pic][pic]
FIGURE 8: (a) A T2 and a PD slice of a MR scene of a patient’s head. (b) The scatter plot of the slices shown in (a) together with a cluster outline indicated for CSF. (c) The segmented binary slice showing CSF.
FIGURE 6a: Illustration of shape-based interpolation. A CT scene of a child’s skull was thresholded to create a binary scene first. This scene was then interpolated via the shape-based method at a coarse resolution and was surface rendered.
FIGURE 6b: The binary scene was interpolated (shape-based) at a fine resolution and then surface rendered.
FIGURE 7: Illustration of scene-based registration. Top: A MR (PD) slice situated approximately at the same location in the head of a multiple sclerosis patient. The slices were acquired at 4 different time instances. Bottom: The slice at the same location obtained after 3D registration of the four scenes and reslicing of the scenes to get the same slice. The progression of the hyperintense lesions is now readily seen which was not apparent prior to registration.
FIGURE 8: Motion of the bones of the foot defined in MR scenes for the two positions shown are determined by registering the respective bone surfaces in the two positions.
FIGURE 9: A rendition of the fuzzy boundaries of “bone” detected in the data of Figure 3 using both intensity and gradient criteria.
FIGURE 10: Illustration of the live-wire method on MR slice of a foot. The boundary of interest is that of the talus. The contour shown represents one live-wire segment from the first point on the far right to the second point on the far left.
FIGURE 11: Illustration of thresholding by specifying an intensity interval on a histogram (middle) of a scene of the head (left). The segmented object is shown as a binary scene on the right.
FIGURE 12a: A T2 and a PD slice of a MR scene of a patient’s head.
FIGURE 12b: The scatter plot of the slices shown in Figure 12(a) together with a cluster outline indicated for CSF.
FIGURE 12c: The segmented binary slice showing CSF.
FIGURE 9: Fuzzy thresholding. If the voxel value is less than t1 or greater than t4, the voxel’s objectness is 0. If the voxel value lies between t2 and t3, the objectness is 100%. For other values, the objectness lies between 0 and 100%.
FIGURE 13: Fuzzy thresholding. If the voxel value is less that t1 or greater than t4, the voxel’s objectness is 0. If the value is between t2 and t3, the objectness is 100%. For other values, the objectness lies between 0 and 100%.
[pic]
FIGURE 104: A rendition of bone and soft-tissue which are segmented by using fuzzy thresholding of the CT scene of a subject’s knee.
in Figure 3.
[pic][pic]
[pic]
FIGURE 15a: A slice of T2-PD scene pair of a patient’s head.
FIGURE 15b: A slice of the union of 3D fuzzy connected white matter and grey matter objects detected from the scene in (a).
FIGURE 15c: A slice of the 3D fuzzy connected CSF object detected from the scene in (a).
FIGURE 15d: A slice of the union of 3D fuzzy connected multiple sclerosis lesions detected from the scene in (a).
FIGURE 11: (6a) : A (maximum intensity projection) 3D rendition of an MRA scene.
(FIGURE 16b): A volume (maximum intensity projection) 3D rendition of the 3D fuzzy connected vessels detected from the scene rendered in (a). (c)
FIGURE 17: A montage displayvolume rendition of the arteries and veins of a 3D CT scene of the head of a separated via fuzzy connectedness.
[pic] [pic]patient.
FIGURE 12: (a) Illustration of pseudo color display. Approximately the same MR slice of a patient’s head taken at two different times, one displayed in green and the other in red. Where there is a match, the display appears yellow. Green and red indicate mismatch. (b) The same slice displayed after a 3D scene-based registration. Now green and red indicate either a registration error or change in an object such as a lesion between the two time instances
FIGURE 18: Illustration of reslicing guided by a 3D display. The CT data pertain to a patient with a craniofacial disorder. A plane is selected interactively via the 3D display to indicate the orientation of the slice plane. The slice corresponding to the oblique plane is shown on the right.
FIGURE 19a: Illustration of pseudo color display. Approximately the same MR slice of a patient’s head taken at two different times, one displayed in green and the other in red. Where there is a match, the display appears yellow. Green and red indicate mismatch.
FIGURE 19b: the same slice displayed after a 3D scene-based registration. Now green and red indicate either a registration error or change in an object such as a lesion between the two time instances.
[pic]
FIGURE 20: Projections are created for rendition either by ray casting from the viewing plane to the scene or by projecting voxels from the scene to the viewing plane.
FIGURE 21: Scene-based volume rendering of CT knee data of Figure 3, showing bone, fat, and soft-tissue.
FFIGURE 1322: An object-based surface rendition of the skull of a child with agnathiaby using triangles to represent the surface.
[pic]
FIGURE 14: An object-based volume rendition of the data of Figure 13. A fuzzy object representation was first obtained by using fuzzy thresholding illustrated in Figure 9.
[pic]
FIGURE 23: An object-based volume rendition of the data of Figure 22. A fuzzy object representation was first obtained using fuzzy thesholding illustrated in Figure 13.
FIGURE 1524a: object-based volume rendering of bone and soft tissues. Bone and soft tissue structures (muscles) were detected as separate 3D fuzzy connected objects from a craniofacial 3D CT scene. Skin is effectively peeled off because of its weak connectedness to muscles.
FIGURE 24b: The scene in (a) rendered directly using scene-based volume rendering via the opacity function illustrated in Figure 3 separately for bone and soft tissue. Here, skin becomes inseparable from muscles (since they have similar CT numbers) and hence obscures the rendition of the muscles.
FIGURE 25: Skull from CT data rendered using five different sequences of preprocessing and visualization operations.
FIGURE 26: Examples of virtual surgery operations (rigid manipulations) on a child’s skull rendered from CT data. The operation mimics an osteotomy procedure used in craniomaxillofacial surgery for advancing the frontal bone.
-----------------------
% objectness
intensity
0
% objectness
100
t4
t3
(a)
(c)
(b)
z
(a)
(a)
(c)
(b)
x
y
Scene domain
Viewing direction
(b)
t2
User
Output
Input
Workstation
Imaging devices
Printers
Model generators
Robotic manipulators
Storage devices
Preprocessing
Visualization
Manipulation
Analysis
Imaging devices
Pointing devices
Simulators
Storage devices
(b)
(a)
FIGURE 2: Illustration used for defining basic terminology.
(c)
(b)
(a)
(c)
t1
v
u
w
x
z
r
y
scene domain
viewing direction
abc: scanner coordinate system
xyz: scene coordinate system
uvw: object coordinate system
rst: display coordinate system
c
t
b
a
s
................
................
In order to avoid copyright disputes, this page is only a partial summary.
To fulfill the demand for quickly locating and searching documents.
It is intelligent file search solution for home and business.
Related searches
- financial planning and analysis career
- financial planning and analysis jobs
- financial planning and analysis description
- financial planning and analysis pdf
- financial planning and analysis examples
- financial planning and analysis training
- financial planning and analysis skills
- financial planning and analysis course
- summary and analysis example
- financial planning and analysis certification
- financial planning and analysis role
- financial planning and analysis process