Introduction to: Interactive Entertainment

1
Mixed Reality
Charles Hughes School of EE CS (SEECS) Media
Convergence Lab University of Central
Florida collaboration with IST (Stapleton) and
Digital Media (Moshell, Wirth)and CS
(Micikevicius, Pattanaik) and CREOL (Rolland) and
Psychology (Sims) And Film (Van Wagenen)
2
MR Background Projects
3
What is Mixed Reality?
4
Mixed Reality is Strictly Between Real and Virtual
Augmented Virtuality
Augmented Reality
REAL
VIRTUAL
5
The Dream of Mixed Reality
As Visceral as a Theme Park
As Immersive as Military Simulation
As Intuitive as Play
As Meaningful as Education
As Interactive as Video Games
As Compelling as Motion Pictures
6
Imagination Compelling
7
Mixed Reality Continuum
Milgrams Reality-Virtuality Continuum
Augmented Reality (AR)
Augmented Virtuality (AV)
P.T. Barnum Reality-Imagination Continuum
Aristotles Media- Imagination Continuum
Physical Reality
Virtual Reality
Compelling Mixed Reality (The Play)
Film
Traditional Theme park
Novel
Magic Show
8
Making MR Compelling

• Real and virtual must mutually occlude -- its
  not good enough to treat real as backdrop
• If augmented reality, then virtual must blend
  into real (illumination is a big deal)
• Real and virtual should affect each other its
  not a one-way street
• Audio and show effects are crucial

9
Artistic Convention

• What is on the page is only one fourth of the
  story. It is like an iceberg where
  three-quarters of the story you dont see, it is
  beyond the page.--Earnest Hemingway

10
Imagination Picks up Where Technology Leaves Off
I want the reader to burn a hole in the page
with their imagination and lose themselves within
the story.--Stephen King
11
Experiential Marketplace
(1997 Annual Worldwide Revenues)

• Simulation Training (3.3 Billion)
• Theme Parks (7 Billion)
• Arcade (14 Billion)
• Home Video Games (14 Billion hardware and
  software)
• Museums (12-18 Billion)
• Sports and Recreation (23 Billion)
• Tourism (50 Billion)
• Conventions Conferences (1.1 Trillion)

12
Transforming Technological Novelty into
Compelling Media
13
Technology Supporting MR
14
CAVE
15
Optical See-Through HMD
16
Video See-Through HMD
17
Magnetic Tracking
Interference
18
Acoustical / Inertial Tracking
Line-of-sight
19
Optical Tracking (HiBall)
Line-of-sight
20
Blue Screen (registration)
Consistent lighting
21
Green Screen Clips
22
Examples of MR
23
Retro-Reflective Screen
24
Medical Application
25
Actual Use of Reflective Drape
26
Distributed AR Environment

• Collaborative environments
• Information exchange through 3D objects
  manipulation

27
MR Arcade / MR MOUT
Layering of Virtual and Real
28
MOUT Training
29
Now thats Immersion!!
30
MR Aquarium
31
MR Aquarium Plans
32
Aqua Gauntlet
33
MR Cartoon
34
Nolie Our MR Cartoon Character
35
Real Backlot
Vehicles
Scenery
Actors
Lighting
Visual Capture
Props Dressing
Graphics
Special Effects
Audio
Stage Management
Scripts
36
Virtual Backlot
Vehicular Simulation
World building
Avatars
Illumination
Point of View
Virtual Assets
Graphics
Physical Modeling
Sound Synthesis
Dungeon Master
Simulation
37
Sample of Research Issues

• Tracking
• Magnetic, Acoustical, Optical
• Markers (shape, color)
• Outdoors
• Registration
• Sorting Real and Virtual
• Mutual Occlusion
• Rendering
• Real-Time Illumination
• Level-of-Detail
• Scenario
• Scripting
• Communication / Coordination

38
Creative End

• Three most important things are
• Story, Story, Story
• How do we script multilinear stories?
• What are poetics of MR stories?
• Audioscape
• 3D sound
• hypersonic sound
• Show Effects

39
Show Control Devices
40
Optical Tracking

• Marker Placement
• Felix Hamza-Lup et al.

41
Markers

• Marker type
• passive
• active
• Some marker attributes
• field of emission (for active)
• occlusion
• size and shape
• inter-marker distance constraints

42
Question 1

• How to place a given set of markers onto an
  irregular object such that the chance that the
  object is seen from different angles is maximized?

43
The Quiescent Algorithm

• Quiescence (resting state) minimum potential
  energy state of a set of points
• 3 step algorithm for placing a set of markers on
  a complex object
• choose an intermediary regular surface (e.g
  sphere, cylinder)
• apply optimization technique to distribute
  markers on this surface (e.g Simulated Annealing)
• apply texture mapping to distribute markers on
  the final surface

44
Shape Detection

• The object is represented as a 3D mesh
• The extent of elongation of the mesh is given by
  the eigenvalues of the dispersion matrix.

di pi-p , i?1,n where n is of vertices in
the mesh.
Dispersion matrix A?didiT Diagonalize
DV-1AV, where Dij?j eigenvalues, V mx. of
eigenvectors
45
Surface Selection

• If ?i / ?j , i ? j gt 10 , we use a cylinder, else
  use a sphere.
• Ex

46
Simulated Annealing

• Markers modeled as electrons
• Minimization of potential maximizes distance
  between neighbors
• Cost Function

• Cooling Schedule Tnew 0.995Told
• 150 Moves before lowering T
• Initial Temp 400 Stop when T ? 0.7
• p(?E)natural exp(-?E/kT)
• p(?E)adj (exp(- ?E/T) / (1exp(- ?E/T))

47
After Simulated Annealing

• Marker distribution after simulated annealing on
  the sphere

48
Markers Mapping

• Last step
• Markers mapping from the surface of the
  intermediary object to the surface of the
  irregular object.
• Using the normal from the intermediate surface

Intermediary surface
Object
49
Question 2

• How can we minimize the number of markers while
  maintaining the maximum coverage condition and
  the viewpoint constraint ?
• viewpoint constraint at least k markers must be
  visible from that viewpoint (tracking system
  dependent)

50
Viewpoint Algorithm

• 4 step alg. for placing a set of markers on a
  complex object
• each polygon is assigned a different number
• a set of viewpoints around the object is selected
• compute for each polygon the number of viewpoints
  that see it gt TriangleList
• While ViewpointsList ?Ø
• Add a marker on the highest count triangle
• Check all viewpoints marker count
• If marker count ? k
• remove viewpoint from ViewpointsList
• update TriangleList

51
Computing Viewpoint

• M // initial triangles list (the polygonal mesh
  for the object)
• VP // viewpoints position and orientation list
• begin
• forall ti ? M do assign(i,ti) // create
  TriangleList
• optimize (VP) // simulated annealing
• forall vj ? VP do
• forall ti ? M do
• check visibility (vj ti) // check if visible
  from that viewpoint
• update M // node triangle number,
  viewpoints count
• while (VP ??)
• tchoose(M) // returns the triangle with
  ti.count max
• add_marker(t) // add a marker on this triangle
• forall vj ? VP do // check_limit() returns
  seen from a viewpoint
• if check_limit(vi) ? k do
• remove vi from VP
• update M
• end

52
Viewpoint Algorithm Analysis

• Space complexity
• double linked lists and arrays
• O(n)
• Time Complexity
• viewpoints (m) ltlt triangles (n)
• O(nm) , if m?n O(n2)

53
Experimental Results Quiescent algorithm

• Input
• randomly generated 3D triangular mesh
• 30 markers
• Output
• VRML 3D scene
• Green pos. after simulated annealing
• Red final maker position
• Sphere as intermediary object

54
Experimental Results Quiescent algorithm

• Input
• randomly generated 3D triangular mesh
• 24 markers
• Output
• vrml 3D scene
• Green pos. after simulated annealing
• Red final maker position
• Cylinder as intermediary object

55
Experimental Results Viewpoint algorithm

• Input
• randomly generated 3D triangular mesh
• 30 viewpoints
• k3 (at least 3 markers visible from each
  viewpoint)
• Output
• vrml 3D scene
• White viewpoints position
• Red final maker position
• (!) Only 8 markers used.

56
Algorithm Efficiency Assessment

• Problems
• Markers attributes e.g. field of emission
• Objects with cavities
• Visual assessment
• Prototype tracking probe
• Active Markers Infrared Emitting Diodes

57
Prototype semispherical probe

• 360 degrees field of regard in azimuth and 90
  degrees elevation
• Position
• Accuracy 0.225mm
• Precision 0.02mm
• Orientation
• Accuracy 0.6 degrees

58
Virtual Forests
59
Goals

• Evolve according to a verifiable biological model
  in faster than real-time
• Be dynamically alterable by changing biological
  and tree model parameters at run time
• Deploy at Orlando Science Center as part of
  experience to learn more about ecology of
  Southern pine forests

60
Generating Forests
61
L-System Rules

• ? FA(1)
• p1 A(k) ? /(?)(?)FA(k 1)(?)FA(k1)
• min1, (2k 1)/k2
• p2 A(k) ? /(?) (?)FA(k 1)
• max0, 1 (2k 1)/k2

62
Interpretation of Grammar

• axiom ? is start string
• Module F is rendered as a branch segment
• Module A(k) grows the tree
• integer k denotes number of rewriting rules
  applied
• Modules , denote rotation around the z-axis/
  denotes rotation around the y-axis angles in
  parentheses (? 32, ? 20, ? 90)
• Stochastic since choice when rewriting A(k)
• p1 produces two branches with probability min1,
  (2k 1)/k2
• p2 produces a single branch segment
• New segments are rotated with respect to their
  parent.
• Rewriting is in parallel every A is replaced in
  every step.

63
Trees Produced
Trees at 6, 8 and 12 rewrites
64
Complexity of Levels
65
Hierarchical Levels of Detail
66
LOD Generation

• LOD generated by replacing geometry with textured
  cross-polygons.
• The lowest level of detail, LOD0 replaces the
  entire tree with a cross-polygon.
• LOD-k replaces each k-subtree with a cross
  polygon.
• Textures are obtained by rendering several views
  of an arbitrary k-subtree.
• Since the silhouette of a tree is same from any
  two views at an angle of 180? to each other, it
  is not necessary to use distinct textures for the
  two sides of a polygon

67
Six LODs for a 12-level Tree
68
Rendering at Various LODs
69
Visibility Based Rendering
70
A Dense Forest Scene
71
Terrain Grid View Frustrum
72
Visibility Computation
The grid cells are processed at increasing
distances from the viewpoint. Thus, cells that
do not fall within viewing frustrum are
effectively culled without any processing.
Visibility can be computed at run-time since
trees at distance k are rendered before any at (k
1). Visibility Vis(t) of a given tree t is
73
LOD Selection

• LOD-12 (highest level of detail) 60 to 100
  visibility
• LOD-8 40 to 60 visibility
• LOD-4 20 to 40 visibility
• LOD-0 (single cross-polygon) 5 to 20
  visibility
• no rendering for under 5 visibility

74
Parallelization

• Tasks distributed among renderer nodes
• Each uses hardware assisted graphics algorithms
• Each renders a vertical "slice" of frame
• Two communications per frame
• the user node broadcasts the viewpoint's position
  and direction
• the user node collects and concatenates the
  subimages from the rendering servers.

75
Earth Echoes
76
Earth Echoes Motivation

• The history of Gettysburg has the most impact at
  the battle site
• The history and culture of Eatonville have the
  most impact in the town of Eatonville
• The stories of East Tennessee are best conveyed
  in the context of the Great Smoky Mountains

77
Earth Echoes Concept

• The Concept
• drop stories onto the earth
• echo stories to people who pass by
• echoes occur today, tomorrow, next week or even a
  hundred years from now
• Stories are preserved in relation to place
• a picture of ones grandparents might be
  associated with their family home
• Story genre and access rights are maintained
• give me only what I want and what I have a right
  to

78
Content Eatonville
79
Content Mt. LeConte
80
Content Leu Gardens
81
A Map Interface to Gardens
82
Content Seen from Map View
83
ITEC Network
Provides ADA-compliant route and stop
announcements. Enhances Guest experience with
daily-updated news, weather, games and other
information. Generates recurring/operating
revenue for transit authorities
finance. Network is maintained by the ITEC
Network.
          
84
By Ways Project
1. Take digitize multimedia segments based on
geographical location from cultural and
educational institutions
2. Use Global Positioning System (GPS) aboard
buses.
4. Insert media segments into existing ITEC
Network distribution system.
3. Download via wireless communications relevant
content.
85
Measure Me
86
Measure Me

UCF
Orlando Science Center
Measure Me

87
Our Customers


88
Sponsors

• National Science Foundation
• Canon MR Lab
• US ARMY
• US Navy
• Orlando Science Center
• ITEC

89
Media Convergence Lab
90
Contact Information

• ceh_at_cs.ucf.edu
• http//www.cs.ucf.edu/ceh
• SEECS
• http//www.seecs.ucf.edu
• MCL
• http//www.dart.ist.sucf.edu/MCL

91
Mixed Reality
Charles Hughes School of EE CS (SEECS) Media
Convergence Lab University of Central
Florida collaboration with IST (Stapleton) and
Digital Media (Moshell, Wirth)and CS
(Micikevicius, Pattanaik) and CREOL (Rolland) and
Psychology (Sims) And Film (Van Wagenen)