Stereoscopic Video Overlay with Deformable Registration

1
Stereoscopic Video Overlay with Deformable
Registration

• Balazs Vagvolgyi
• Prof. Gregory Hager
• CISST ERC
• Dr. David Yuh, M.D.
• Department of Surgery
• Johns Hopkins University

2
The CASA Project
Todays Surgical Assistant A Simple Information
Channel
3
The CASA Project
Preoperative Imagery
Virtual fixtures with da Vinci Robot
Information Fusion with da Vinci Display
Stereo surface tracking
Task graph execution system
Stereo tool tracking
Tissue Classification
HMM-based Intent Recognition
Ultrasound
Capabilities of a Context-Aware Surgical
Assistant (CASA)
4
The CASA Project
Preoperative Imagery
Information Fusion with da Vinci Display
Stereo surface tracking
Stereo tool tracking
Developing a Context-Aware Surgical Assistant
(CASA)
5
Information Overlay

• Problem setting
• Given pre-operative scan data from a suitable
  imagingmodality
• Video sequence from a stereo endoscope
• Add value
• Overlay underlying anatomy on the stereo video
  stream (x-ray vision)
• Include annotations or other information tied to
  imagery

Key Problem Nonrigid registration of organ
surface to data
add kidney picture
6
Inputs What Do We Know?

1. Pre-operative 3D model- most probably
   volumetric- only a portion of it will be visible
   on the endoscope- anatomy will be deformed
   during the surgical procedure
2. Camera system properties can be measured-
   optical stereo calibration- local
   brightness/contrast/color response
3. Stereo image stream- 3D surface can be
   reconstructed- texture information
4. A guesstimate of modelendoscope 3D
   relationship- We can guess where to start
   searching i.e. patient position

7
Outputs What Do We Generate?

1. Position of 3D model registered to stereo image
2. Model deformed to the current shape of anatomy
3. Rendering a synthetic 3D view on the stereo
   stream
4. Everything done real-time

Original Image
Stereo Data
Deformed Mesh
8
All this in a flow chart
3D model
stereo video stream
2D
3D
3D texturetracking
Stereo imagepre-processing
Building andoptimizingdisparity map
DeformableRegistration to3D surface
Imageoverlay
Recognizingdeformations
optical parameters
9
Classical Stereo Vision The Problem

• Blocks of each image are compared using SAD
• Optimization for each block independently on
  entire depth range
• Very fast implementation (GPU)
• Lousy results

Small Vision Systemfrom Videre Design(w/o
structured light)
10
Solution 1 Lighting and Multi-Scale

• Input images downsized to several scale levels
  (½, ¼, )
• Each scale processed with the same algorithm
• Propagate coarse search results to the finer
  scale
• Quality of disparity map is better
• Even faster than single scale computation
• Requires structured light

SVL implementation(using structured light)
11
Solution 2 Dynamic Programming

• Solve a (spatially) global optimization with
  regularization
• O(D) min SAD(D) Smooth(D)
• GLOBAL optimum found in polynomial time

12
Solution 2 Dynamic Programming

1. Defining the recursive cost function
2. Memoization
3. Finding lowest cost path, which is the disparity
   map (DM in red)

Smoothness
Error
13
Dynamic Programming on Images

• Minor issue previous approach applies to
  scanline
• Approximate DP applied to entire image- 3D
  disparity space (D)- Cost function (C)-
  Memoization (P)

14
Dynamic Programming Results
15
Dynamic Programming In Vivo Results
Stereo recordings from the da Vinci robot
Focal length of 700 pixels 5mm baseline
Distance to surface of 55mm to 154mm.
Textured 3D Model
Raw Disparity Map
16
Surface to 3D Model Registration

• Inputs
• point cloud from the stereo surface modeler
• point cloud generated from a model or volume
  image
• Outputs- transformation to register the 3D
  model to the 3D surface

17
Results Rigid Registration
Current algorithm usesIPC with modificationsto
account for occlusionsdue to viewpoint (z-buffer)
Complete system (stereoplus registration)
operatesat 5 frames/second
18
From Rigid to Deformable

• Calculate residual errors in z direction
• Define a spring-mass system
• Perform local gradient descent

19
Deformable Registration Results

Final registration error of lt 1mm exceptfor the
area where the tool enters the image
20
Coming in CASA
Tool Tracking
Tissue Surface Classification
21
Thank you!
22
Telemanipulation with Integrated Laparoscopic
Ultrasound for Hepatic Surgery

• Collaboration between JHU and Intuitive Surgical,
  Inc.

Needle insertion demonstrates alignment
Registered 3D ultrasound volume swept
w/autonomous robot motion
Ultrasound probe examining artificial lesion in
porcine liver with registered 2D ultrasound
overlay