ContextDriven Visualization for Augmented Reality

1
Context-Driven Visualization for Augmented Reality

• Dieter Schmalstieg
• Technische Universität Graz

2
Mobile Augmented Reality
Inertial sensor
camera
HMD
tracked touchpad
GPS

• Tracking
• dGPS
• Inertial
• Vision (marker)

notebook WLAN GPRS phone
3
Handheld Augmented Reality

• Low Cost
• Robust and fool-proof
• Large variety of devices
• Intuitive user interface
• Self-contained operation
• Networking support
• Tracking support

4
Example Signpost

• Wide area 3D navigation system

5
Modeling Requirements for AR

• AR uses large complex models
• AR uses geometric, visual models
• AR does not use normal computer graphics models
• Visual realism not important
• Rendering performance not important because only
  limited amount of detail interesting
• Semantic structure important for application
  goals (situation, context)

6
Overview

• Problem Data structures and algorithms for
  visualization driven by context
• Such as location, situation, user prefs etc.
• Background scene graph
• Context-driven scene graph traversal
• Background Magic lenses
• Context-driven magic lenses

7
Scene Graph

• Solution for complex graphics problems
• High-level data structure
• Objects, not drawings
• Directed Acyclic graph
• Hierarchy for
• Geometric aggregation e.g., articulated body
• Semantic aggregation, e.g., parts-engine

8
Coins Software Architecture

• Based on older Open Inventor standard
• Hierarchical composition
• Composite pattern
• Decorator pattern
• Work done in traversals
• Visitor pattern
• Dual-dispatch on node/traversal type
• Rendering traversal
• Reflection, garbage collection
• Scripting language

9
Example
10
Context-driven traversal

• Problem
• Information scattered throughout scene graph
• Finding information requires full search
• Manipulating may require changing SG in many
  places
• Acerbated by mix-and-match scene graph
  composition, even at runtime!
• Need to delegate behavior modification to local
  nodes and their interaction with near nodes

Where on earth is the blue node in the SG?
11
Example

• Indoor AR navigation
• 2 views
• Head up display
• World in miniature
• Rendered from same, consistent model
• No copy!

12
Contribution

• Generic parameterization during traversal
• add generic parameter passing technique
• binding of parameters happens during traversal
• based on existing standard scene graph Open
  Inventor
• scene graph becomes a template structure
• evaluated in the last possible moment

13
Traversal and state

• State stack

• Scene graph

1
3
2

• transformation
• material parameters
• light sources

• state contains specialized information

14
Add general state - context

• Add 3 things
• Context state
• Nodes accessing context
• Affecting changes in the scene graph

context
15
Operations on context

• Context maps string gt strings
• foreground gt 1 0 0
• position gt 10 -5 0
• node references
• Possible operations
• add
• set
• remove
• clear

16
Implementation

• Deriving from dedicated classes in Open Inventor
  framework.
• SoContextElement
• Stores the context data
• SoContext
• Node to write to context
• SoContextReport
• Node to read the context
• SoContextSwitch
• Reads context and selects children to traverse

17
Setting and retrieving simple context

• SoContext
• index foreground,background
• value 0 1 0, 0 0.5 0
• DEF REPORT SoContextReport
• index foreground
• Material
• diffuseColor USE REPORT.value

implicit conversion!
18
Templated scene graph

• Multiple references to one graph
• Templates for
• attributes of nodes
• sub graphs
• Individual references set template parameters

val
19
Examples mobile augmented reality
world in miniature
augmentation
generic model
20
Examples mobile augmented reality
augmentation
WIM
model
model
wireframe
Z
doorstyle
model
Wall style
R2
R1
generic model
wall
door
21
Discussion

• Context driven traversal allows local behavior in
  semantic space of SG
• It does not allow local behavior
• In 2D screen space
• In 3D object space
• ? We want that as well!

22
Example Information Filtering

• Avoid display clutter for mobile AR user
• Display only relevant information BOTH
• Important for user
• Within field of view
• Explicitely determined in object space, per
  object
• We want filtering implicit as part of traversal,
  per pixel

Höllerer 2001
23
Example for geometric locality

• Subsurface infrastructure visualization
• (collaboration with Grintec Gesmb)

24
Magic Lenses

• Local behavior in screen space
• Bier 1993 2D case
• Used to
• reveal hidden information
• enhance data of interest
• suppress distracting information
• Developed in 2D as interaction tools

25
3D Magic Lenses, contd

• Viega 1996 HW Clipping planes
• Schmalstieg 1998 Stenciling
• Flat vs volumetric lenses
• Flat lenses are a simplification of volumetric
• Implemented using clipping planes

26
3D Magic Lenses contd

• Ropinski 2005
• Uses a more robust multi-pass technique
• Dual z-buffer by clever use of shadow buffer
• Programmed as HW shader object space algorithm
• Single lens only

27
3D Magic Lenses contd

• Mendez, ISMAR 2006
• Enhancement to Ropinski 2005
• Dual-depth buffer using float textures
• Completely general overlapping lenses

28
3D Magic Lenses contd

• Mendez, ISMAR 2006
• Three render passes
• Render behind and next to lens
• Render inside lens
• Render in front and next to lens
• As many lenses as available float textures but
  render passes

29
Putting It All Together

• Context-driven Magic Lenses
• Objects organized in context families
• Define 1 render style per (lens,context family)
• Annotate (existing) scene graph with context
• Targeted as an interaction tool and not as a
  rendering technique

30
Context Families

• Sub scene graphs (objects) are assigned
  context information
• Make smart sub graphs who know who they are
• Correctly styled regardless of their position in
  the graph
• Styles are also arbitrary sub-graphs
• Context-sensitive traversal concatenates styles
  and objects in right order
• Lens nodes create screen/object space masks

Objects affected according to their context
information
31
Building Dynamic Sub Graphs

• Context-sensitive traversal concatenates styles
  and objects in right order
• Styling sub graphs are also just concatenated

root content
32
CSML Dynamic Sub Graphs

• Styling contributions are correctly concatenated
  in case of intersecting lenses (see lt2gt)
• But no general solution for non-commutative style
  sub graphs yet

lt1gt wrong
lt2gt correct
33
Applications

• X Ray Vision

34
Applications

• Information Enhancement

35
Applications

• Virtual cutout

36
Applications

• Multiple Magic Lenses

37
(No Transcript)
38
Thanks!

• Collaborators
• Gerhard Reitmayr
• Erick Mendez
• Denis Kalkofen
• Michael Knapp
• Sponsors
• Austrian Science Fund FWF
• FFG BRIDGE
• Grintec GesmbH
• www.studierstube.org