Progressive Transmission and Rendering of Foveated Volume Data

1
Progressive Transmission and Rendering of
FoveatedVolume Data

• Chen Chen, Zhiyong Huang, Hang Yu, Ee-Chien Chang
• School of Computing
• National University of Singapore

2
Introduction - motivation

• Problem
• Large volume data provided by modern scanners
  like CT and MRI
• Slow network speed comparing to the large size of
  volume data
• Solution
• Wavelet foveation
• Progressive transmission and rendering
• coarse-to-fine
• region-based

3
Introduction - wavelet foveated
volume

• Foveated volume is a non-uniform sampled volume
  whose resolution is highest at the fovea (user
  specified ROI) and falls off as the distance to
  the fovea increases priority selection
• Wavelet foveated volume is constructed by select
  ROI at each level of details in the volumes
  wavelet form - compression

4
Proposed method - overview

• Transform to Wavelet Form
• Divide into 83 block, run-length encode (RLE)
  each block
• Client request ROI (x,y,z),s
• Generate Fw
• Reorder data blocks of Fw
• Progressive transmission
• Progressive Rendering

5
Proposed method - overview

• Client-server volume rendering system
• Server Side
• Input Cw ROI(x,y,z),s
• Output Fw
• Client Side
• Input Fw
• Output Iw1 -gt Iw2 -gt -gt Iwfinal
• Two transmission and rendering schemes

6
Proposed method - overview
Server Site
Client Site
Wavelet Transform Cw
request
Generate Fw from Cw and ROI
User Input ROI(x,y,z),r
Reorder data blocks in Fw according to the schemes
Progressive Render RB or CTF
Progressive transmission
Progressive rendering
Display
7
Wavelet transform

• Divide the volume into blocks of size 2k3
• Apply generic Haar wavelet transform on each
  block to obtain 7 detail blocks and 1 average
  block
• Group 8 adjacent blocks to again obtain a block
  of size 2k3
• Repeat this procedure until the average volume
  reaches size m3

8
Wavelet transform - run-length encoding
(RLE)

• A more compact representation
• RLE is performed block by block
• Compressed block will be a list of value(v) and
  length(l) pairs
• v Integer value of a coefficient
  l Number of successive v
• Example
• L1,1,1,1,0,0,5,0,0,0,0,0,0,0,0
• R(1,4),(0,2),(5,1),(0,8)

9
Derive foveated volume

• Cw, ROI(x,y,z),s gt Fw
• Layer 0 image average image of Cw
• Layer i image IWT(layer i-1 image, 7 layer i-1
  detail images)
• ROI in ith layer ROI in Layer i image

10
Derive foveated volume

• Center of ROI in ith layer is
• Size of ROI in ith layer is s3
• Boundary of ROI in ith layer can be defined as

11
Derive foveated volume

• 4th ROI (9,9),6 6,11,6,11
• Layer 3 details (i,j) (3,5,3,5)

12
Progressive transmission - motivation

• Data size 10243, Fovea size 483, Average
  image size 163
• MaxlayerLog2(1024/16)6
• Total amount of data 243(571)497,664
  coefficients
• Reorder data blocks
• Progressive transfer

13
Progressive transmission - two schemes

• Coarse-to-fine
• Description Transmit a rough average image first
  and refine the fovea from outer ROIi to inner
  ROIi1
• Implementation Send the wavelet foveated data
  from inner layer to outer layer progressively
• Region-based
• Description Transmit full resolution ROImaxlayer
  first and expand the image from ROImaxlayer-1 to
  ROI0 with decreasing resolution
• Implementation Data blocks and parent data
  blocks of ROImaxlayer-i1 will be sent in ith
  iteration

14
Progressive transmission - illustration in 2D

• 2D image size 128128, m16, k8,
  ROI(86,86),45, maxlayer3
• ROI0(10,10),450,15 blk_index0,1
• ROI1(21,21),450,31 blk_index0,1
• ROI2(43,43),4520,63 blk_index1,3
• ROI3(86,86),4564,107 blk_index4,6

15
Progressive transmission - illustration in 2D
16
Progressive transmission - coarse-to-fine
17
Progressive transmission - coarse-to-fine
18
Progressive transmission - region-based
19
Progressive rendering - rendering equation

• For each voxel
• intensity value
• opacity value
• The final intensity result reaches the viewer of
  each pixel will be Eulerian Sum over accumulated
  opacity
• Rendering equation to render a subvolume with
  same intensity and opacity value

20
Progressive rendering - rendering equation

• For 2 subvolumes Sa and Sb with size na and nb,
  voxels in Sa and Sb have , , and ,
  respectively. Sa is in front of Sb, that is, Sa
  is over Sb. The intensity of Sa and Sb and
  Sfinal can be defined as

21
Region-based - partial volume reconstruction

• Starting coefficient
• Ending coefficient
• Wavelet coefficients at ith layer detail image
• begin
• end
• Reconstruction starts from layer 0, that is, the
  average image and move out to higher level of
  details until the desired resolution is reached

22
Region-based

• Center fovea is reconstructed using partial
  volume reconstruction and rendered at iteration 1
• 6 subvolumes are rendered Top, Bottom, Left,
  Right, Front, Back with resolution scale 2(1-i),
  i is the number of iteration
• 6 sub volumes are rendered separately and combine
  the rendered result to the previous rendering
  image

23
Region-based - image composition

• Top, bottom, left and right are added to the
  previous rendering result image at their right
  position
• Front and back image needs image composition with
  the previous rendering result image (front) over
  (inner) over (back)
• over operator can be defined as given image
  pixel A(da, aa) and B(db, ab)
• C A over B
• dc da aadb
• ac aa ab

24
Region-based - image composition
25
Coarse-to-fine

• Start rendering from ROI0 to ROImaxlayer
• Each ROIi except ROImaxlayer can be divided into
  7 subvolumes top, bottom, left, right, front,
  back and center
• Center subvolume is the average image of the
  inner ROI and its rendering result, accumulated
  density value and opacity value will be kept for
  next iteration

26
Coarse-to-fine

• ROIi is constructed by center part of ROIi-1 and
  level i detail coefficients
• Number of voxels each coefficient in ROIi
  represent is 2maxlayer-i
• Rendering equation for each voxel in ROIi is
• Rendering result of front and back subvolume RFi
  and RBi will be take down for next iteration
• Rendering result of iteration i1 will be
  combined with RFi and RBi in iteration i1

27
Results - direct rendering
28
Results - region-based
29
Results - region-based
30
Results - coarse-to-fine
31
Results - coarse-to-fine
32
Results - time
33
Results - coarse-to-fine
34
Results - coarse-to-fine
35
Results - region-based
36
Results - region-based
37
Results - coarse-to-fine
38
Results - coarse-to-fine
39
Results - region-based
40
Results - region-based
41
Conclusion and future work

• Two progressive transmission and rendering
  schemes, region-based and coarse-to-fine
• Blockwise RLE used in wavelet compression
• Wavelet foveation
• Image composition
• Large memory is still needed
• cut the data set exceeding a certain amount into
  smaller data sets and build location map to
  indicate actual position of each data segments
• Cache can be used to store frequently used
  blocks, for example, those high level details