3D GFX in MPEG-4 tutorial CGI

1
3D GFX in MPEG-4 tutorialCGI04 (Crete, GR,
2004-06-16)Module 2 Shapes

• Patrick Gioia France Télécom RD, FR
• Francisco Morán Universidad Politécnica de
  Madrid, ES
• Mikaël Bourges-Sévenier Mindego, US
• Mahnjin Han Samsung Advanced Institute of
  Technology, KR
• Gauthier Lafruit Interuniversity
  Micro-Electronics Centre, BE
• Alain Mignot SGDL Systems, CA
• Alexandru Salomie Vrije Universiteit Brussel, BE
• Michael Steliaros Superscape, UK

2
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

3
Polygons

• Approximation of target surface
• Method tesselation with planar facets
• Quality first order (linear) ? no smoothness
  (C0 continuity)
• Mesh definition IFS (Indexed Face Set)
• Connectivity list of faces , Pn in0, in1,
  in2, ,
• ? arbitrary topology (non-manifold, open, higher
  genus, etc.)
• Geometry list of vertices , Vk (xk, yk,
  zk),
• Mesh coding
• Connectivity (lossless) triangle strips,
  trianglevertex trees, etc.
• Geometry (lossy) coordinate quantisation
  prediction from conn.

4
Polygons LODs (Levels Of Detail)

• LOD concept
• 1976 Clark introduced the idea
• Main interest rendering efficiency
• Taxonomy of LOD extraction techniques
• Static vs. dynamic
• Global vs. local
• Progressive vs. hierarchical LODs
• Successful simplification techniques
• 1996 Hoppes edge collapses
• 1997 Garlands quadrics ? qslim
• Progressive 3D mesh coding
• 1996 Hoppes PM (Progressive Mesh)
• 1998 IBMs PFS (Progressive Forest Split)

5
Polygons mesh simplification example
6
Polygons in MPEG-4 (pre-AFX)

• MPEG-4 (1999) IFSs
• Based on VRML97
• Arbitrary topology meshes
• Properties (normals, colours and textures)
• MPEG-4 Amd.1 (2000) 3DMC (3D Mesh Coding)
• 40-501 compression of IFSs by IBMs TS
  (Topological Surgery)
• Incremental transmission and rendering
• Progressive coding by IBMs PFS
• Error resilience by SAIT

7
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

8
Patches

• Approximation of target surface
• Method tesselation with predefined curved
  patches
• Quality higher order (polynomic/rational) ? Cn
  continuity
• Mesh definition
• Connectivity regular grid of quads. or
  triangles
• ? planar topology
• Geometry list of control points , Pk (xk,
  yk, zk),
• Mesh coding
• Connectivity (lossless) implicit
• Geometry (lossy) coordinate quantisation
  prediction from conn.

9
Patches example (bicubic Béziers)
Tensor product of cubic Bézier curves
?Single patch (4x4 control points) vs. two
patches (4x7 control points)?
10
Patches in MPEG-4 (AFX)

• MPEG-4 Part 16 (2003) NURBS
• Based on VRML97 Amd., originally proposed by
  blaxxun
• Support for NURBS curves and patches
• Specific nodes for Béziers curves and patches
  (for increased efficiency)
• Support for free-form deformations

11
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

12
SSs definition
SS limit of recursive refinement of base
control mesh
SS inherently define hierarchically nested LODs
13
SSs (Subdivision Surfaces)

• Approximation of target surface
• Method tesselation with curved patches
• Quality higher order ? Cn continuity
• Mesh definition
• Connectivity list of triangles/quads., e.g.,
  , Tn in0, in1, in2,
• ? arbitrary (manifold) topology
• Geometry list of control points , Pk (xk,
  yk, zk),
• Mesh coding
• Connectivity (lossless) as for polygonal
  (manifold) mesh
• Geometry (lossy) as for polygonal (manifold)
  mesh

14
SSs advantages over polygons and patches
Polygons Are the simplest approach (linear
approximation) Can resolve fine details and
handle arbitrary topologies Lead to
unstructured, huge meshes Patches Are a more
powerful approach (higher order
approximation) Are convenient for coarse and
smooth models Need cumbersome trimming and
stitching mechanisms SSs Connect and unify the
two extremes above Provide multi-resolution
handles for hierarchical coding/editing
15
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

16
Plain SSs classic schemes
Catmull-Clarks (1978) quadrilateral, primal,
approximating, C2
Loops (1987) triangular, primal, approximating,
C2
Dyns butterfly (1990) triangular, primal,
interpolating, C1
17
Plain SSs extended Loop scheme (1/3)

• Motivation real-time performance on low spec.
  terminals
• High-speed rendering means triangles due to
  commodity HW optimised SW, but
• designers like working with quads.
• Problems with Catmull-Clarks scheme
• Rendering in real-time requires triangles for
  visibility culling and HW-based or optimised SW
  renderers
• The best that can be done is triangulation prior
  to rendering
• This defeats the whole memory efficiency
  argument, since both mesh sets (quad.-based and
  triangulated) must be maintained
• More importantly, the triangulation is out of the
  designers control
• For consistent real-time rendering on lightweight
  devices
• A guaranteed triangulation is essential
• Topology must be maintained in native format

18
Plain SSs extended Loop scheme (2/3)

• Loop subdivision is more client friendly
• Triangle-based input mesh triangle meshes at
  all levels, already in native format needed for
  rendering
• but it is not very designer friendly ?
  extension of Loops scheme through edge
  visibility tagging
• Still uses Loops scheme mostly ? renderer
  friendly
• Allows flexibility of quads, but with exact
  triangulation control
• Addresses Loop material boundary problem at
  extraordinary vertices
• Such vertices are common due to the way
  designers tend to work
• This problem occurs with colour, texture or any
  material discontinuity

19
Plain SSs extended Loop scheme (3/3)

• Extended Loop subdivision stencils
• Identical to Loop if no invisible edges ? C2
• Local subdivision matrix is not constant in
  general but
• recently proven to converge also in the regular
  quad. case

(a) interior smooth vertex (b) interior dart
vertex (c) boundary/crease vertex (d) visible
edge (e) invisible edge (f) boundary/crease edge
20
Plain SSs normal control (1/2)

• Piecewise smooth surfaces
• Smooth (interior) vs. sharp/boundary edges and
  vertices?
• Concave? vs. convex? corner vertices
• NYUs edge/vertex tagging
• Complete set of schemes (C-C, Loop) in a unified
  implementation
• Sound normal control (flat-spot C2)

21
Plain SSs normal control (2/2)
22
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

23
WSSs (Wavelet/detailed SSs)

• Wavelet-based MRA (Multi-Resolution Analysis)
• 1st generation wavelets classic nD signals
• 2nd generation wavelets Lounsberys extension
  to 2-manifolds
• Arbitrary vs. subdivision (semi-regular)
  connectivity meshes
• Base mesh extraction and remeshing techniques
  needed
• Applications of MRA
• High compression
• 1993 Shapiros zerotrees (hierarchical
  organisation of wavelet coefficients)
• 1996 Saids SPIHT (Set Partitioning In
  Hierarchical Trees)
• 1997 Kolarovs coding of scalar functions
  defined on 2-manifolds
• 1999 Moráns and Khodakovskys hierarchical
  coding of surfaces
• Editing/animation
• 1994 Finkelsteins multiresolution curves
• 1997 Zorins interactive multiresolution mesh
  editing

24
WSSs subdivision prediction mechanism

• Price (requirements)
• Base mesh extraction
• Subdivision scheme predictor
• Details (3D vectors) prediction errors ?
  remeshing
• Prize (advantages)
• Predictive coding ? immediate (if smooth target
  mesh)
• Zerotree/SPIHT-based coding ? not quite...

25
WSSs detail hierarchy and quantisation

• Edge-based detail hierarchy
• Based on CalTechs PGC (Progressive Geometry
  Compression)

• Local normal frame n, t1, t2 tied to edge e
• Normal deserves more bits
• Tangential components can be treated worse
• Normal can be approximated
• Tied to coarser LOD for multiresolution editing

26
WSSs of mesh and mess sizes

• Mesh sizes bit/vertex
• Raw (valid for up to 217 ? 131 k triangles) 144
• Basic coding (triangle strips geometry
  quantisation) ? 50
• Single resolution coding (Toumas) 15-20
• Progressive coding (Rossignacs) 22-28
• Approximation distances/errors
• Point to vertex, plane, surface
• Surface to surface (Lp)
• L1 average
• L2 RMS
• ...
• L? maximum (Hausdorffs)
• Most common L2 relative to bounding box diagonal

27
WSSs rate vs. distorsion curves
28
SSs in MPEG-4 (AFX)

• MPEG-4 Part 16 (2003) plain wavelet SSs
• Plain SSs for mesh smoothing
• Considered schemes Catmull-Clark, extended
  Loop, butterfly
• No details are added but
• normal control achievable through edge/vertex
  tagging of initial control mesh
• Wavelet/detailed SSs for surface approximation
• Possibly tagged base mesh
• Details are added after each subdivision step,
  which are
• wavelet-transformed according to one of several
  possible schemes
• Most suited for multi-resolution
  editing/animation
• Most suited for view-dependent transmission

29
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

30
MeshGrid components
31
MeshGrid hierarchical RG
RG is hierarchical

• Hierarchical animation is possible
• Applies Dyns 4-point interpolatingscheme for
  curves

32
MeshGrid hierarchical CW
CW is hierarchical
33
MeshGrid CW features

• Regular 4-connectivity
• Triangulation on the fly using connectivity
  rules
• Triangles, Quadrilaterals, Pentagons, Hexagons,
  Heptagons
• Compact storage using a 3D extension of Freeman
  chain code

34
MeshGrid relationship between CW and RG
35
MeshGrid examples
36
MeshGrid in MPEG-4 (AFX)

• MPEG-4 Part 16 (2003) MeshGrid
• Combines a surface (CW) with a volumetric (RG)
  description
• Multi-scalable resolution (CW), global shape
  (RG), refinement (offsets)
• Versatile animation possibilities vertex-level,
  grid-level, offset-level
• Surface defined as a non-uniform connectivity
  net with higher density
• High curvature areas
• Locations that are deformed during the
  animation
• Allows view-dependent partitioning in ROIs
  (Regions Of Interest)

37
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

38
CGD (Computational Graceful Degradation)

• Hints on rendered scene complexity from server to
  client
• What is rendered scene complexity?
• Viewpoint distance ? perceived size of objects
• Mesh complexity number of vertices/triangles
• Rendering mode texture complexity coming
  soon to this same theater
• What kind of hints?
• Necessary processing power f (covered pixels,
  triangles, rendering mode)
• What can the client do about?
• Viewpoint nothing much
• Mesh complexity interpret or request coarser
  LOD
• Rendering mode texture complexity ignore
  users crave for luxury

39
View-dependent WSSs
but adaptive transmission is better!
Progressive/hierarchical transmission is good
Client must be able to read bitstream parts
independently? bitstream must have markers ask
for specific parts OR transmit back viewing
parameters and received data
40
View-dependent WSSs
41
View-dependent MeshGrid

• View-dependent decoding
• Downstream
• Receive and decode ROIs
• Backchannel
• Request ROIs or send cache info.

• Encoding(decompose model in LODs)
• Partition each resolution level in one or several
  ROIs according to complexity
• Encode surface locally in each ROI

42
View-dependent MeshGrid example
43
Advanced adaptive techniques in MPEG-4

• MPEG-4 Amd.1 (2000) CGD
• MPEG-4 Part 16 (2003) View-dep. WSSs and
  MeshGrid
• Reduction of transmitted data volume for any 3D
  modelling paradigm
• Backchannel
• Use depends on application type
• Server-driven client sends status and server
  decides
• Client-driven client sends requests and server
  replies (if possibe)
• Upstream data depends on parameter/command type
• General parameters (field of view, frame rate)
  low frequency updates
• Navigational parameters (position, orientation)
  high frequency updates
• Paradigm-specific parameters (WSSs, MeshGrid,
  moreToCome?)

44
3D object modelling paradigms

• Surfaces (2,5D)
• Primitives
• Polygons for arbitrary meshes
• LODs (Levels Of Detail) for progressive 3D mesh
  coding
• Patches for smooth surfaces
• SSs (Subdivision Surfaces)
• Inherently hierarchical 3D surface coding (gt
  progressive 3D mesh coding)
• Plain SSs for mesh smoothing
• WSSs (Wavelet SSs) for surface approximation
• MeshGrid
• Easily hierarchical 3D surface coding
• Advanced adaptive techniques
• CGD (Computational Graceful Degradation)
• View-dependent 3D surface coding (WSSs and
  MeshGrid)
• Backchannel
• Solids (3D)

45
Solids

• Solid primitives
• Solid models the arithmetic of forms

46
Solids solid primitives
47
Solids algebraic shape
Implicit equation
Simplest solid primitive
for
Quadrics
Quartics
48
Solids Implicit node
Implicit NDT SFGeometryNode exposedField
SFVec3f bboxSize 2.0 2.0 2.0 exposedField MFFlo
at c 10 coeffs for quadrics, 35
coeffs for quartics exposedField
SFBool solid TRUE exposedField
SFBool dual FALSE exposedField MFInt32 densities
( 1, 2)
The cylinder and its dual
Various quartics (4th degree equations)
Animation by interpolation of coefficients values
49
Solids Quadric node
Quadric NDTSFGeometryNode field
SFVec3f bboxSize 2.0 2.0 2.0 exposedField SFVec
4f P0 -1 0 0 1 exposedField SFVec4f P1 1 0 0
1 exposedField SFVec4f P2 0 1 0
0 exposedField SFVec4f P3 0 0 1
0 exposedField SFVec4f P4 0 1 0
1 exposedField SFVec4f P5 0 0 1 1 exposedField
SFBool solid FALSE exposedField
SFBool dual FALSE exposedField MFInt32 densitie
s ( 1, 2)  
10 geometric control points
50
Solids building blocks
Predefined quadrics
Exact geometry, solid and holes by building
blocks Canadian Space Agency Model of SMT
(Special Purpose Dexterous Manipulator - Task
Verification Facility - Manipulator Test bed )
Solid modelling by juxtaposition of predefined
quadrics
51
Solids arithmetic of forms
F Q1 Q2 Q3
A cube multiplication of 3 degenerated quadrics
Multiplication of two forms
Examples of solid operations
52
Solids arithmetic operators
 
 
53
Solids ternary logic operators
Dual of Reciprocal Kleene implication
 
54
Solids density filters
 
 
55
Solids SolidRep nodeand arithmetic expression
SolidRep NDTSFGeometryNode exposedField
SFVec3f bboxSize 2.0 2.0 2.0 exposedField SF3DNod
e solidTree NULL exposedField MFInt32 densityLi
st
DEF Egg_script Script Solid Tree
definition field SFNode Egg USE solid_node
primitives field SFNode White
container field SFNode Yolk matter inside
code url "vrmlscript function
initialize() Egg.solidTree ( White Yolk
) "
Density filtering
Different densities of the egg cut by a box
56
Solids applications
Architecture
Mechanics
Virtual models
Biotechnology
57
Solids main benefits

• Exact geometry
• Embedded topology
• Compactness

21 Kb
37 Kb
1.1 Mb
407 Kb
58
Solids in MPEG-4 (AFX) ???

• MPEG-4 Part 16 (2003) Implicit, Quadrics,
  SolidRep
• Algebraic surfaces solids
• Algebraic surfaces by coefficients
• Quadrics by geometric control points
• Arithmetic of forms
• Arithmetic, logic (ternary) and filter operators
• Integration of algebraic solids and polygonal
  primitives
• Arithmetic expressions through language
• Independent of the rendering mode (polygonal,
  pixel-based, ray-tracing, etc.)

59
Conclusions

• MPEG-4 has lots of interoperating 3D modelling
  tools
• Surfaces polygons, patches, SSs, WSSs, MeshGrid
• Solids building blocks and powerful language
• usable with lots of flexibility
• Progressive (or, even better, hierarchical)
  multi-resolution content
• View-dependent content, for possibly
  client-driven scenarios
• in lots of potential applications!
• Generic 3D objects, from cartoonish to
  highly-detailed
• Virtual character animation
• Interactive simulations and high-resolution
  virtual reality
• Accurate CAD/CAM/AEC models and scientific data
  visualization
• Terrain fly-over with adaptive (view-dependent)
  refinements
• You name it!