Exploring large data spaces in Virtual Reality

1
Exploring large data spaces in Virtual Reality

• Robert G. Belleman
• Section Computational Science
• University of Amsterdam
• robbel_at_science.uva.nl

2
Overview

• Large data spaces
• Interactive exploration environments
• Interaction techniques in Virtual Reality
• A test case
• vascular reconstruction in a virtual environment

3
High Performance Computing

• High Performance Computing
• computing power increases (Moores Law)
• storage capacity increases
• Result data spaces get increasingly large and
  complex, multi-dimensional, time dependent.

4
What are data spaces?

• Roughly speaking
• Data sets
• the files or states that are generated
• Parameter spaces
• the number of variables and their allowed
  freedom (range, resolution) in a program

5
Examples of large data sets

• Medical images (i.e. CT, (f)MRI, PET)
• 512 x 512 at 16 bits slices in common use
• tens to hundreds slices 5Mb to5Gb per scan
• 1024 x 1024 at 16 bit in very nearfuture 20Mb
  to 20Gb per scan
• time variant scans

6
more examples ...

• Simulation experiment results
• FEM, MD, lattice Boltzmann
• often dimensionality gt 3(e.g. time variant)
• multiparameter data fields
• gigabytes of data per run

7
more examples ...

• Measurements
• high-speed data acquisition devices particle
  accelerators, microbeam scanners, DNA scanners,
  CLSM
• Financial data, etc.
• Terabytes of data per experiment is no longer an
  exception!

8
Parameter spaces

• Simulation of complex systems
• intractable a certain timestep in a simulation
  can only be reached by starting at t0
• NP complete time and space requirements grow
  exponentially with problem size
• Explicit simulation by a guided search through
  parameter space required (non-deterministic
  algorithms SA, CA, NN, LBM, etc.)

9
Examples of parameter spaces

• Molecular dynamics
• picosecond timeresolution
• docking involves searchthrough large
  problemspaces

10
more examples

• Finite Element Methods (FEM)
• large (hierarchically) structured meshes
• Lattice Boltzmann methods (LBM)
• large 3D (hierarchically) structured grids
• large parameter spaces
• Optimization problems in general

11
From data to knowledge

• Analysis of data spaces is often difficult
• no analysis methods known, or ill-posed
• size of data sets too large or too complex
• Often data analysis or simulation runs can take
  days, sometimes weeks!

12
Bring in the expert

• Presentation is often the only way to obtain
  insight (note not limited to visualization)
• Is it possible to make short cuts? E.g. by
  putting an expert in the loop?

13
HITL

• Human In The Loop
• a.k.a. interactive exploration
• a.k.a. exploratory analysis
• a.k.a. computational steering
• a.k.a. problem solving environments
• a.k.a. virtual laboratory

14
Interactive Exploration Environments

• Goal providing an interactive environment that
  allows for the exploration of large data spaces.
• Distinction between static and dynamic
  environments.

15
Interactive Static Exploration Environments (ISEE)

• Exploring large time-invariant datasets
• Multi-modal data representation
• visualization
• sonification
• haptification?

16
Interactive Dynamic Exploration Environments
(IDEE)

• Exploring dynamically changing data from living
  simulations
• Changing parameters What if...?
• Requires time management

17
Time management
Synchronous (lockstep)
Asynchronous
18
Prerequisites for an IEE

• Why Virtual Reality?
• Quality presentation
• Informative, avoid clutter
• Rapid update rate for continuous perception
• gt 10 fps for vision
• gt 20 cps for sound
• gt 1000 cps for haptics

19
Prerequisites for an IEE

• Intuitive interaction
• increased functionality requires a well
  considered user interface
• Real-time feedback lt 0.1 sec delay
• These often conflict one another.

20
VR interaction techniques

• XiVE X in Virtual Environments
• There is no WIMP for VEs.
• XiVE swallows GUIs into a VE
• allows existing applicationsto be used in VEs
  with nochanges

21
VR interaction techniques

• Context Sensitive Speech Recognition
• Interaction with visual constructs can be hard in
  a VE.
• Speech is a different modality
• Adding context decreases WER (?)
• Fast, intuitive interaction
• come here, make blue, increase size by 200

22
VR interaction techniques

• SCAVI Speech, CAVE and Vtk Interaction
• Direct interaction with Vtk actors using
  pointer or voice
• select, drag, scale, rotate, copy, paste, etc.
• event handlers when in focus, when dragged, when
  selected, when spoken to, etc.

23
VR interaction techniques

• GEOPROVE Geometric Probes for VEs
• Measurements in VR
• Uses probes consisting of markers

24
So how does all this work?Lets look at a test
case...
25
Simulated vascular reconstructionin a virtual
operating theatre
26
Overview

• Interactive virtual environments for the
  exploration of
• Multi-dimensional datasets
• Multi-parameter spaces (computational steering)
• Visualization and interaction in Virtual Reality
  (VR)
• Applied to a test casesimulated vascular
  reconstruction in VR

27
VRE

• The Virtual Radiology Explorer (VRE)
• Static exploration of 3D medical datasets
• Virtual Reality (VR) interface
• CAVE at SARA, Amsterdam
• Portable ImmersaDesk
• Surface/volume rendering
• Virtual endoscopy
• PACS data and computinginterface
• Data storage and processingon parallel system
  (IBM SP2)

28
Vascular disease

• StenosisTreatment thrombolysis, balloon
  angioplasty, stent placement, endarterectomy,
  bypass
• AneurysmTreatment shunt, bypass

29
The problem

• Best treatment often not obvious
• read the parameter space
• Human body is a complex structure
• read the data space
• A treatment is not always best under all
  situations
• read combination of both

30
Pre-operative planning
31
Traditional treatment ofvascular disease
32
Interactive simulated vascular surgery
33
The Virtual Laboratory

• Shared use of distributed computing
  resourceshigh performance computers, scanners,
  algorithms, etc.
• Connected via high performance networks
• Common infrastructure the Virtual Laboratory
• Multi-disciplinary scientific experimentation
• Problem solving environments (PSE)
• Time/location independent scientific
  experimentation
• Collaborative scientific research

For additional information...
DutchGrid initiative http//vlabwww.nikhef.nl/
34
Simulated Vascular Reconstruction

• Simulated vascular reconstruction
• Patient specific angiographydata
• Fluid flow simulationsoftware
• Simulation of reconstructivesurgical procedure
  in VR
• Interactive visualization ofsimulation results
  in VR
• Pre-operative planning
• Explore multiple reconstructionprocedures

35
Preprocessing

• Segmentation of patient specific MRA/CTA scan
• Isolates region of interest
• Lattice Boltzmann grid generation
• Defines solid and fluid nodes, inlet and outlet
  conditions

36
Fluid flow simulation

• Lattice Boltzmann Method (LBM)
• Lattice based particle method
• Regular lattice, similar to CT or MRI datasets
• Spatial and temporal locality
• Ideal for parallel computing
• Allows irregular 3D geometry
• Validated with experimentsand FE simulations
• Non-compressiblehomogeneous fluid,laminar flow
• Velocity, pressure and shearstress calculated
  fromparticle densities

37
Interactive exploration in VR

• Visualize simulation results
• Flow field, pressure, shear stress
• Real time
• Interactive exploration
• VR interaction to locateregions of interest
• Interactive grid editing
• Simulate vascularreconstruction procedure

38
Interactive exploration in VR

• Quantification in VR GEOPROVEGeometric Probes
  for Virtual Environments

39
Interactive vascular surgery
40
Summary

• Test case shows example of a Problem Solving
  Environment (PSE)
• Shared use of distributed resources
• Time/location independent collaborative
  experimentation
• PSEs open new possibilities for collaborative
  scientific research
• Grid initiatives (Globus)
• Virtual Environments provide intuitive interface
  for the exploration of multi-dimensional datasets
  and parameter spaces

41
Questions?

• For more info
• http//www.science.uva.nl/robbel/
• or email
• robbel_at_science.uva.nl