The%20Uses%20of%20Object%20Shape%20from%20Images%20in%20Medicine

1
The Uses of Object Shape from Images in Medicine

• Stephen M. Pizer
• Kenan Professor
• Medical Image Display Analysis Group
• University of North Carolina
• Credits Many on MIDAG, especially
• Daniel Fritsch, Guido Gerig, Edward Chaney,
  Elizabeth Bullitt,
• Stephen Aylward, George Stetten, Gregg Tracton,
  Tom Fletcher, Andrew Thall, Paul Yushkevich,
  Nikki Levine, Greg Clary, David Chen

2
Object Representation in Medical Image Analysis

• Extract an object from image(s) segmentation
• Radiotherapy
• Tumor plan to hit it
• Radiosensitive normal anatomy
• plan to miss it
• Surgery
• Plan to remove it
• Plan to miss it
• During surgery, view where it is
• effect of treatment
• Radiology
• View it to judge its pathology

PD MRA T2 T1 Contrast
3
Image Guided Planning of Radiotherapy

• Planning in 3D
• Extracting normal anatomy
• Extracting tumor
• Planning beam poses

4
Object Representation in Medical Image Analysis

• Registration (find geometric transformation that
  brings two images into alignment)
• Radiotherapy
• Fuse multimodality images (3D/3D) for planning
• Verify patient placement (3D/2D)
• Surgery
• Fuse multimodality images (3D/3D or 2D) for
  planning
• Fuse preoperative (3D) intraoperative (2D)
  images
• Radiology
• Fuse multimodality images (3D/3D) for diagnosis

5
Object Representation in Medical Image Analysis

• Shape Volume Measurement
• Make physical measurement
• Radiotherapy
• Measure effect of therapy on tumor
• Radiology, Neurosciences
• Use measurement in science of object development
• Find how probable an object is
• Radiology, Neurosciences
• Use measurement as quantitative input to
  diagnosis
• Use measurement in science of object development
• Use as prior in object extraction
• E.g., extract the kidney shaped object

6
Object Shape Volume Measurement
Neurofibromatosis (Gerig, Greenwood)
Infant Ventricle from 3D U/S (Gerig, Gilmore)
7
Object Extraction (Segmentation)

• Approach 1 preanalyze, then fit to model
• Neurosurgery (MR Angiogram), Radiology (CT)
• Vessels, ribs, bronchi, bowel via tube skeletons
• Cardiology (3D Ultrasound)
• Geometry via clouds of medial atoms
• Fit appropriately labeled clouds to 3D LV model
• Cardiac Nuclear Medicine (2D Gated Blood Pool
  Cine)
• Extract LV, with previous frame providing model
• Extraction via deformable m-rep model
• Shape from extracted LV analyze shape series
• Surgery, Radiation Oncology (Multimodality MRI)
• Extract tumor, using local shape characteristics

8
Extracting Trees of Vessels via Skeletons
(Aylward, Bullitt)
9
Presenting Ribs via Tube Skeletons (Aylward)
10
Presenting Bronchi and Lung Vessels via Tube
Skeletons (Aylward)
11
Presenting Small Bowel via Tube Skeletons
(Aylward)
12
Presenting Blood Vessels Supplying a Tumor for
Embolization (Bullitt)
Full tree, 2D Subtree, 2D 3D, from 2 poses
13
Heart Model (G. Stetten)
14
Statistical Analysis of Medial Atom Clouds (G.
Stetten)
15
LV Tube Identified by Medial Atom Statistical
Analysis (G. Stetten)
sphere
slab
cylinder
16
Mitral Valve Slab Identified by Medial Atom
Statistical Analysis (G. Stetten)
sphere
slab
cylinder
17
Automatic LV Extraction via Mitral Valve/LV Tube
Axis (G. Stetten)
18
Gated Blood Pool Cardiac LV Cine Shape Analysis
(G. Clary)
Example sequence 4-sided medial elliptical
analysis
19
Object Extraction (Segmentation)

• Approach 2 deform model to optimize reward for
  image match reward for shape normality
• Radiation Oncology (CT or MRI)
• Abdominal, pelvic organs
• Deform m-reps model
• Neurosciences (MRI or 3D Ultrasound)
• Internal brain structures
• Spherical harmonics boundary model
• Deformable m-reps model
• Neurosurgery (CT)
• Vertebrae

20
M- Reps for Medical Image Object Extraction and
Presentation (Chen, Thall)
21
Displacements from Figurally Implied Boundary
Boundary implied by figural model
Boundary after displacements
22
Vertebral M-reps Model
23
Vertebral M-reps Model Spinous Process Figure
24
Cerebral Ventricle M-reps Model
25
Extraction with Object Shape as a Prior

• Brain structures (Gerig)

26
Registration

• Registration (find geometric transformation that
  brings two images into alignment)
• Radiotherapy
• Fuse multimodality images (3D/3D) for planning
• Verify patient placement (3D/2D)
• Surgery
• Fuse multimodality images (3D/3D or 2D) for
  planning
• Fuse preoperative (3D) intraoperative (2D)
  images
• Radiology
• Fuse multimodality images (3D/3D) for diagnosis

27
Image Guided Delivery of Radiotherapy

• Patient placement
• Verification of plan via portal image
• Calculation of new treatment pose

28
Finding Treatment Pose from Portal Radiograph
and Planning DRR
29
Medial Net Shape Models
Medial nets, positions only
Medial net
30
Image Match Measurment of M-rep
31
Registration Using Lung Medial Object Model
Reference Radiograph (Levine)
Medial nets, positions only
Medial net
32
Radiograph/Portal Image Registration (Levine)
Intensity Matching Relative to Medial Model
Medial net
33
Shape Volume Measurement

• Find how probable an object is
• Training images Principal components
• Global vs. global and local
• Correspondence

Hippocampi (Gerig)
34
Modes of Global Deformation
Training set
Mode 1
x xmean b1p1
Mode 2
x xmean b2p2
Mode 3
x xmean b3p3
35
Shape Volume Measurement

• Shape Measurement
• Modes of shape variation across patients
• Measurement amount of each mode

Hippocampi (Gerig)
36
Multiscale Medial Model

• From larger scale medial net,
• interpolate smaller scale medial net and
  represent medial displacements

b.
37
Summary What shape representation is for in
medicine

• Analysis from images
• Extract the anatomic object-shaped object
• Register based on the objects
• Diagnose based on shape and volume
• Medical science via shape
• Shape and biology
• Shape-based diagnostic approaches
• Shape-based therapy planning and delivery
  approaches

38
Shape Sciences

• Medicine
• Biology
• Geometry
• Statistics
• Image Analysis
• Computer Graphics

39
The End
40
Options for Primitives

• Space xi for grid elements
• Landmarks xi described by local geometry
• Boundary (xi ,normali) spaced along boundary
• Figural nets of diatoms sampling figures

41
Figural Models

• Figures successive medial involution
• Main figure
• Protrusions
• Indentations
• Separate figures
• Hierarchy of figures
• Relative position
• Relative width
• Relative orientation

42
Figural Models with Boundary Deviations

• Hypothesis
• At a global level, a figural model is the most
  intuitive
• At a local level, boundary deviations are most
  intuitive

43
Medial Atoms

• Imply boundary segments
• with tolerance
• Similarity transform equivariant
• Zoom invariance implies width-proportionality of
• tolerance of implied boundary
• boundary curvature distribution
• spacing along net
• interrogation aperture for image

44
Need for Special End Primitives

• Represent
• non-blobby objects
• angulated edges, corners, creases
• still allow rounded edges , corners, creases
• allow bent edges
• But
• Avoid infinitely fine medial sampling
• Maintain tangency, symmetry principles

45
Coarse-to-fine representation

• For each of three levels
• Figural hierarchy
• For each figure,
• net chain, successively smaller tolerance
• For each net tile,
• boundary displacement chain

46
Multiscale Medial Model

• From larger scale medial net
• Coarsely sampled
• Smooother figurally implied boundary
• Larger tolerance
• Interpolate smaller scale medial net
• Finer sampled
• More detail in figurally implied boundary
• Smaller tolerance
• Represent medial displacements

47
Multiscale Medial/Boundary Model

• From medial net
• Coarsely sampled, smoother implied boundary
• Larger tolerance
• Represent boundary displacements along implied
  normals
• Finer sampled, more detail in boundary
• Smaller tolerance

48
Shape Represn in Image Analysis

• Segmentation
• Find the most probable deformed mean model, given
  the image
• Probability involves
• Probability of the deformed model
• Probability of the image, given the deformed model

49
Medialness medial strength of a medial
primitive in an image

• Probability of image deformed model
• Sum of boundariness values
• at implied boundary positions
• in implied normal directions
• with apertures proportional to
• tolerance
• Boundariness value
• Intensity profile distance from mean (at scale)

50
Shape Repn in Image Analysis

• Segmentation
• Find the most probable deformed mean model, given
  the image
• Registration
• Find the most probable deformation, given the
  image
• Shape Measurement
• Find how probable a deformed model is

51
Object ShapeRepresentations for Medicine to
Manufacturing

• Figural models, at successive levels of tolerance
• Boundary displacements
• Work in progress
• Segmentation and registration tools
• Statistical analysis of object populations
• CAD tools, incl. direct rendering