3D Medical Volume Reconstruction Using Web Services

1
3D Medical Volume Reconstruction Using Web
Services
2
Speaker

• Andrew Shirk
• Research Programmer
• Automated Learning Group, NCSA
• D2K Web Service Architect Lead Developer

3
Credits and Acknowledgements

• Research Team Members at NCSA, UIUC
• Rob Kooper
• Andrew Shirk
• Sang-Chul Lee
• Peter Bajcsy
• Research Team Members at UIC
• Robert Folberg
• Amy Lin
• Funding provided by
• National Institutes of Health (NIH)
• National Laboratory for Advanced Data Research
  (NLADR)
• National Center for Supercomputing Applications
  (NCSA)

4
Outline

• 3D Medical Volume Reconstruction
• Background
• Problem overview
• Workflow
• Motivation for Using Web Services
• Image Registration Application
• User Interface Applet
• System Architecture (Client Tier)
• D2K (Data to Knowledge) Background Terminology
• D2K Web Service Interface
• System Architecture (Service Tier)
• Demo
• Summary

5
Outline

• 3D Medical Volume Reconstruction
• Background
• Problem overview
• Workflow
• Motivation for Using Web Services
• Image Registration Application
• User Interface Applet
• System Architecture (Client Tier)
• D2K (Data to Knowledge) Background Terminology
• D2K Web Service Interface
• System Architecture (Service Tier)
• Demo
• Summary

6
3D Medical Volume Reconstruction Background

• Part of ongoing work between the NCSA (ALG) and
  UIC
• The web services research and development at
  NCSA, represented by Andrew Shirk and Rob Kooper,
  has focused on developing tools for remote
  collaboration and distributed dataflow processing
  using web services
• The image processing research at NCSA,
  represented by Dr. Bajcsy and Sang-Chul Lee, is
  focused on developing tools for automatically
  processing and analyzing images of cancerous
  tissue cross sections
• How to automate 3D volume reconstruction?
• How to optimize 3D volume reconstruction
  decisions and parameter selection?
• What are the validation approaches for evaluating
  3D volume accuracy?
• What image techniques are applicable to CLSM
  intensity adjustment?
• The research at UIC, represented by Dr. Folberg
  and Dr. Lin, is focused on the study of the
  unique vascular networks produced by uveal
  (interior eye) and cutaneous (skin) melanomas
• What mechanisms are responsible for the networks?
• How do they evolve in the different types of
  melanomas?

7
3D Medical Volume Reconstruction Overview

• Goal is to reconstruct 3D medical volume from
  specimen cross section images
• Cross section images are produced using high
  resolution microscopy (LSCM)
• Internal structures of specimen sample are of
  primary importance
• Scanning each specimen cross section requires
  multiple scans in a grid pattern
• Each scan is performed at multiple confocal
  depths
• Yields multiple, virtual, cross sections for
  each physical cross section
• Cross section images require significant
  pre-processing
• Each plane of scans must be mosaicked (stitched)
  together to recreate larger cross section
• Low quality cross section scans must be
  eliminated
• Scans are color and intensity corrected to
  enhance salient features of sample
• Segments are detected and centroids are computed
• Pre-processing must be automated
• Adjacent cross section images require
  registration with one another
• Corrects for shear, translation, and rotation
  problems that occur when specimen is sliced
• After registration has taken place, the goal of
  reconstructing the volume can be achieved

8
3D Medical Volume Reconstruction Task Workflow

• Specimen Preparation
• Data Acquisition
• Image Pre-Processing
• Best Scan Selection
• Image Registration
• Volume Reconstruction
• Visualization Analysis

9
3D Medical Volume Reconstruction Task Workflow

• Specimen Preparation
• Data Acquisition
• Image Pre-Processing
• Best Scan Selection
• Image Registration
• Volume Reconstruction
• Visualization Analysis

10
3D Medical Volume Reconstruction Task Locations

• UIC
• UIC
• NCSA
• NCSA
• UIC
• NCSA
• UIC / NCSA

11
Motivation for Using Web Services

• Ease of integrating geographically distributed
  resources and expertise
• Medical expertise - Medical School, UIC
• Image analysis expertise - NCSA, UIUC
• High end computing resources - NCSA
• Development speed/cost
• Tools are easy to use
• Tools are free
• Short collaborative cycles
• Upgrades take place behind service interface
• Service interface does not need to change
• Loose coupling between presentation layer and
  business logic
• Flexibility of technology choice when
  implementing user interface
• No binding to vendor or programming language

12
Outline

• 3D Medical Volume Reconstruction
• Background
• Problem overview
• Workflow
• Motivation for Using Web Services
• Image Registration Application
• User Interface Applet
• System Architecture (Client Tier)
• D2K (Data to Knowledge) Background Terminology
• D2K Web Service Interface
• System Architecture (Service Tier)
• Demo
• Summary

13
User Interface Applet

• Goal is to successfully register the two cross
  section images using semi-automated techniques
• Images of two complete volumes are accessible via
  dropdown menus
• Images are lazily loaded upon user request from
  web server
• Each image has been automatically
• Mosaicked
• Color and intensity corrected
• Segmented
• Segments and centroids are packed with image data
• Compressed (2.25 MB / 15.25 MB)
• Decompression takes place after arrival at applet
• Medical expert picks points or segments for use
  by registration processing algorithm
• At least 3 pairs needed for successful
  registration
• Use centroids? option should be checked for
  segment selection

14
User Interface Applet

• Right click an image to
• Zoom in/out
• Adjust gamma
• Adjust band display
• Press Reset to clear point/region selections

15
User Interface Applet

• Press Compute to submit registration job to D2K
  Web Service for processing
• Chosen images and selected registration points
  are sent as job execution parameters (a.k.a. D2K
  Module properties)
• A new window will appear for monitoring job
  status
• Submitted, started, finished timestamps
• When job finishes, View Result button will
  enable if there are results
• Resize window to get larger view of result
• Look in log panel if job did not succeed

16
Tradeoffs of Image Registration System Design

• Tradeoffs were made to ensure a responsive
  interactive experience using limited client-side
  storage and processing resources
• Image data transfer ? full image vs. image
  pyramids
• Tradeoff increased initial download time (full
  image) for fast panning and zooming (image
  pyramids)
• Image pyramids require continuous downloading
  during panning and zooming
• We chose to transfer entire image first
• Image segmentation computation ? compute on
  server vs. compute on client
• Tradeoff increased image size (compute on
  server) for reduced client CPU requirements
• We chose to compute segments on server
• Image compression ? compress before transferring
  vs. transfer uncompressed
• Tradeoff decreased image size (compress before
  transferring) for increased client CPU
  requirements
• We chose to compress before transferring
• Image transformation ? compute during job
  processing vs. compute on client
• Tradeoff increased network traffic for massively
  reduced client CPU requirements
• Result image must be transferred back to the
  client
• We chose to perform transformation during job
  processing

17
System Architecture (Client Tier)

• Medical expert requests the image registration
  applet from the web server
• Applet manages user specified registration
  parameters
• Applet invokes D2KServiceProxy operations for
  submitting job requests, monitoring job status,
  and retrieving job results
• D2KServiceProxy is a convenience class for
  Java-based clients
• Looks like any other class
• Wraps implementation used to interact with the
  D2K Web Service
• Currently delegates to stub classes
• Stubs handle object serialization (marshalling)
  and SOAP message creation and transmission

18
System Architecture (Client Tier)

• Medical expert requests the image registration
  applet from the web server
• Applet manages user specified registration
  parameters
• Applet invokes D2KServiceProxy operations for
  submitting job requests, monitoring job status,
  and retrieving job results
• D2KServiceProxy is a convenience class for
  Java-based clients
• Looks like any other class
• Wraps implementation used to interact with the
  D2K Web Service
• Currently delegates to stub classes
• Stubs handle object serialization (marshalling)
  and SOAP message creation and transmission

19
D2K Data to Knowledge

• D2K is a dataflow processing system that
  integrates analytical data mining methods for
  prediction, discovery, and anomaly detection with
  data management and information visualization
• Facilitates KDD process

20
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

21
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

22
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

23
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

24
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

25
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

26
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

27
D2K Components

• D2K Toolkit
• Development environment (GUI) for building data
  mining applications
• Visually link together code modules
• D2K Itinerary
• Data flow graph stored in an XML file
• D2K Module
• An itinerary node implemented as a Java class
• Contains module properties, similar to JavaBean
  properties.
• Different types of modules are provided for
  different tasks.

28
D2K Components

• D2K Infrastructure
• Itinerary Execution engine

29
D2K Components

• D2K Infrastructure
• Itinerary Execution engine
• D2K-Driven Applications
• Applications that make use of the D2K
  Infrastructure
• Toolkit is a D2K-Driven app

30
D2K Components

• D2K Infrastructure
• Itinerary Execution engine
• D2K-Driven Applications
• Applications that make use of the D2K
  Infrastructure
• Toolkit is a D2K-Driven app
• D2K Server
• Special kind of D2K-Driven app
• Wraps the infrastructure to provide remote
  itinerary and module execution
• Used by the Toolkit to distribute module execution

31
D2K Components

• D2K Infrastructure
• Itinerary Execution engine
• D2K-Driven Applications
• Applications that make use of the D2K
  Infrastructure
• Toolkit is a D2K-Driven app
• D2K Server
• Special kind of D2K-Driven app
• Wraps the infrastructure to provide remote
  itinerary and module execution
• Used by the Toolkit to distribute module
  execution
• D2K Web Service
• Provides a generic programmatic interface for
  executing itineraries
• Communicates with D2K Servers over socket
  connections using D2K Specific protocols.

32
D2K Web Service Interface
33
D2K Web Service Interface
34
D2K Web Service Interface
35
D2K Web Service Interface
36
D2K Web Service Interface
37
System Architecture (Service Tier)

• D2K Web Service and constituent parts are general
  purpose
• In no way are specific to an itinerary or client
  application
• Single endpoint can service many client
  applications
• D2K Web Service has many roles
• Job broker and monitor (primary role)
• Job processing is delegated to eligible D2K
  Servers using D2K specific protocols over TCP
  socket communications
• Proxy threads running in D2KWS process monitor
  job activity in remote processes and persist
  state changes
• Policy enforcer
• Operations, itineraries, jobs, and server sets
  (servers) are access controlled
• Central storage/archive
• Itinerary definitions, module classes, job
  execution parameters, job results, job resources
• Module class server
• D2K Servers lazily load classes from D2K Web
  Service at job runtime
• Requested classes (and class dependencies!) are
  archived when job finishes execution
• Archive available to clients for deserializing
  results

38
System Architecture (Service Tier)

• D2K Web Service and constituent parts are general
  purpose
• In no way are specific to an itinerary or client
  application
• Single endpoint can service many client
  applications
• D2K Web Service has many roles
• Job broker and monitor (primary role)
• Job processing is delegated to eligible D2K
  Servers using D2K specific protocols over TCP
  socket communications
• Proxy threads running in D2KWS process monitor
  job activity in remote processes and persist
  state changes
• Policy enforcer
• Operations, itineraries, jobs, and server sets
  (servers) are access controlled
• Central storage/archive
• Itinerary definitions, module classes, job
  execution parameters, job results, job resources
• Module class server
• D2K Servers lazily load classes from D2K Web
  Service at job runtime
• Requested classes (and class dependencies!) are
  archived when job finishes execution
• Archive available to clients for deserializing
  results

39
System Architecture (Service Tier)

• RDMS maintains state for service
• Accounts, Roles
• Itinerary metadata
• Id, name, description, owner, version info, etc.
• Job metadata
• Id, name, status, owner, submit date, start date,
  etc.
• Server Set metadata
• Id, name, description, constituent servers,
  owner, etc.
• Access policies
• Itineraries, jobs, and server sets
• D2K Servers are job processors
• Head process accepts job requests
• If server is not busy, a worker process is
  started on new port and port number is returned
  to web service
• D2K Web Service connects to new process for job
  execution
• Job hand off is complete once itinerary
  definition has been pushed to worker process
• Resources needed for itinerary execution are
  loaded over the network from D2KWS by worker
  processes

40
Outline

• 3D Medical Volume Reconstruction
• Background
• Problem overview
• Workflow
• Motivation for Using Web Services
• Image Registration Application
• User Interface Applet
• System Architecture (Client Tier)
• D2K (Data to Knowledge) Background Terminology
• D2K Web Service Interface
• System Architecture (Service Tier)
• Demo
• Summary

41
Outline

• 3D Medical Volume Reconstruction
• Background
• Problem overview
• Workflow
• Motivation for Using Web Services
• Image Registration Application
• User Interface Applet
• System Architecture (Client Tier)
• D2K (Data to Knowledge) Background Terminology
• D2K Web Service Interface
• System Architecture (Service Tier)
• Demo
• Summary

42
Summary

• The presented prototype system for 3D medical
  volume reconstruction using web services was
  successfully used in a collaboration between UIC
  and NCSA researchers
• A web services based approach had several
  advantages
• Eased logistics of collaboration
• Medical experts were able to perform
  computationally intensive operations using
• Large image data
• Sophisticated 3D volume reconstruction methods
• Medical experts did not have invest in
• Computational and storage resources necessary to
  perform tasks solely on the desktop
• Development of image analysis software
• Future improvements to the user interface will
  not require alteration of the back-end service
• Improvements to registration algorithms will be
  immediately available to client application

43
Outline

• 3D Medical Volume Reconstruction
• Background
• Problem overview
• Workflow
• Motivation for Using Web Services
• Image Registration Application
• User Interface Applet
• System Architecture (Client Tier)
• D2K (Data to Knowledge) Background Terminology
• D2K Web Service Interface
• System Architecture (Service Tier)
• Demo
• Summary

44
Q and A

• Questions?
• Links to further reading
• 3D medical volume reconstruction prototype system
• http//i2k.ncsa.uiuc.edu/MedVolume
• This presentation
• http//algdocs.ncsa.uiuc.edu/PR-20050715-1.ppt
• Feature Based Registration of Fluorescent LSCM
  Imagery Using Region Centroids
• Sang-Chul Lee Peter Bajcsy
• Proceedings of SPIE Conference on Medical
  Imaging, February 12-17, 2005, San Diego, CA.
• http//algdocs.ncsa.uiuc.edu/PB-20050212-2.pdf
• Complexity of Registration Decisions during 3D
  Medical Volume Reconstructions
• Peter Bajcsy, Sang-Chul Lee, Rob Kooper, David
  Clutter
• Understanding Complex Systems Symposium, May
  16-19, 2004, UIUC
• http//algdocs.ncsa.uiuc.edu/PR-20040516-1.pdf