Tools for Edutainment

1
Sensory Interfaces, Augmented Reality and
Edutainment 

• Tools for Edutainment

2
Background

• Wu Pin Fong
• Virtual Reality Consulting company, Xtensory Inc,
  California, America
• Total 25 years industrial experience
• Done virtual reality a long time 14 years
• Emphasis mixed reality support for industry

3
Virtual Reality Background
4
Started

• Worked at Digital Equipment (now part of Hewlett
  Packard) 1990
• Start the virtual reality program at Digital
• First major US company to support virtual reality

5
Undocumented VR History

• Working for a major american computer company
• Able to work with all the key VR developers
• VR small community then

6
Relationships

• Jaron Lanier, VPL Research
• Scott Fisher, NASA
• Dr Tom Furness, HITL Labs, University of
  Washington
• Carolina Cruz , CAVE inventor, Iowa State
  University
• Rudy Darken, Naval Post Graduate School

7
Others

• William Bricken
• Autodesk
• HITL Labs
• Foundation for WorldToolKit
• Device manufacturers Ascension, Polhemius,
  Intersense, Immersion, etc

8
Jaron Lanier VPL Research

• Taught me
• Virtual Reality
• Worked on
• VR project
• For SIGGRAPH 1990

9
VPL Research in 1990

• Very exciting
• New technology
• Mark Bolas (later to start FakeSpace)

10
Tom Furness Director, HITL Labs

• Came from Aerospace Medical Research Laboratory
  (USAF), Wright-Patterson AFB, OH.
• Helped get funding from Digital Equipemt for HITL
  Labs

11
Electronic Visualization Lab - CAVE

• The CAVE is a projection based virtual reality
  system developed at the Electronic Visualization
  Lab created by Carolina Cruz-Neira

12
Naval Post Graduate School

• Rudy Darken, Director MOVES Institute, Monterey
  Post Graduate School
• Edutainment game - America's Army

13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Some Projects

• NASA
• American army, navy
• Wright patterson air force base, OH
• FAA
• Bank of America
• Mayo Clinic
• Toshiba
• HITL labs, University of Washington
• Hasbro
• Lockheed
• Schlumberger
• Monterey naval school
• Heitman
• Compaq

17
OverView

• Need make it easy to develop mixed reality games
• Provide Standard components

18
Why components

• Many mixed reality games involve people who are
  not virtual reality experts
• Most edutainment projects assume
• completely new software
• Control of actual real data
• Expert programmers
• Real world need interface with legacy software

19
Edutainment games

• Out of research into actual industrial use

20
Today

• Discuss one standard component - device support
• Example of its use
• Special device support Haptics
• A haptics example
• Component for building virtual cities for use in
  mixed reality games

21

• Device Support Component

22
XvsLink Standard device component for device
support

• Paper - A Component-based Sensor Architecture for
  CAVEs (CAVE Automated Virtual Environments),
  Proceedings of Immersive Projection Technology
  Workshop, Iowa State University, 2000.

23
Industry tested since 1996

• Not research
• Used in actual products
• Toshiba
• University of Pennsylvania virtual human Jack
• Mayo Clinic

24
Standard Device Interface

• Importance of components
• Example of components standardised device
  interfaces

25
XvsLink Purpose

• Instrument existing legacy software
• Linux, Windows, Sun Solaris, HP, SGI
• Easy to use VR devices
• Interfaces to non VR devices

26
Work Closely with Device Manufacturers

• So closely actually referenced in their manuals
  Example Intersense IS600 manual

27
Focused - Does not provide

• user interaction techniques
• scene graph (or transform tree)
• graphics techniques

28
Provides

• handle the device control layer
• provide communication between hosts
• hook to higher levels of an edutainment system

29
XvsLink Functional Diagram
Device support Session capture playback

• Legacy software

Virtual vehicle
Network support Users Developers administrators
User interaction analysis
30
One industrial Difference

• Need to monitor user interaction with the game
• Capture user data for later playback and analysis

31
Layered Architecture
Application
User interaction tools

• XvsLink

interface
device
transport
Legacy software
32
Component Device Interface

• Provide Easy Access
• Common Interface to different devices
• Access to VR peripherals from any application
• Storage and replay of device data

33
XvsLink

• XvsLink is a C class library with both a C and
  C interface that provides a standard device
  interface for integrating devices into simulator
  applications.

34
Make devices invisible to user

• Writing a driver for a new device usually a few
  days
• Special commands for specific device drivers
• Windows and linux kernel level drivers

35
Operation

• Open networked, serial, USB, shared memory
  connections
• Invisible to user
• Generic devices by functions
• connection automatically restored if the device
  server is shut down and restarted

36
Design

• generic devices
• polled or queued
• local or remote
• actual or virtual
• devices identified by logical name

37
Design

• polled or queued- application can specify
• polled
• reads data asynchronously
• typical example tracker, valuator
• queued
• circular
• typical example keyboard, button

38
Local and Remote devices

• remote devices treated as local devices
• Local client and server
• queries for device capabilities

39
Design

• local or remote device
• application unaware of source
• can use actual or virtual data
• Device data from hardware or calculations

40
Device Mapping

• interfaces specify a set of capabilities
• device is mapped onto one or more interfaces
• Client connects to the different interfaces
• run-time bindings

41
Design

• identified by logical name
• Device can map to any name
• new names can be created as needed

42
Extensible

• Generic Type devices
• Layered devices
• Multiple-behavior devices

43
Design

• generic devices
• locator
• valuator
• button

44
Device added dynamically

• dynamic device support
• examples
• Intersense tracker
• Fakespace wand, pinchGlove
• Phantom haptic

45
Support

• Linux, windows, SUN, HP, SGI
• hooks to other tools, packages
• Hooks for legacy systems

46

• Example of XvsLinks Use

47
Virtual Collaborative Clinic
48
NASA Virtual Collaborative Clinic

• 1998
• Dr. Muriel Ross, head of NASA's biometrics
  research at Ames
• Out of research into deployment usually no news

49
Virtual Collaborative clinic
50
Collaboration Partners
51
Objectives

• Medical support in remote locations
• surgical try-outs in mixed reality after which
  the most adequate procedure can be stored to be
  used in the actual operation.
• project computer images onto the patient in order
  to guide physicians during surgical
  interventions.

52

• Special device support Haptics

53
Haptic Rendering of Virtual Stimuli
54
Haptic
55
Haptic Information

• Haptic information is a form of sensory
  stimulation that is acquired through collisions
  with objects

56
Using the Haptic Sensory Channel

• Force, vibration, texture
• object shape, compliance, impact, contact,
  sliding, slipping, torque and kinematic
  constraint.
• encoding the feel and movement of real or virtual
  objects during manipulation or exploration

57
Haptic Exploratory Procedures
58
Human Sensing
59
Force Feedback
60
Force Feedback
61
Rutgers
62
Haptics
63
Example
64
Xtensorys Tactile Controller
65
Shape Memory Alloy (SMA)

• a system of long stroke vertical pin-rods, using
  a coil-form Shape Memory Alloy (SMA) as an
  actuator.
• This material can be stretched or deformed from
  its original shape but would spontaneously return
  to its original shape when heated.

66
Tactor Sensor
67
Tactor Array

• a 16 by 16 array of pin-rods
• high-speed update rates

68
Tactor Array in Action
69
Operation

• orchestrating the vertical motion of a dense
  array of pin-rods. The effect is similar to a
  children's "Pop up book", where a flat surface
  transforms into a 3D form.
• the driving method is scalable so that tactor
  arrays of any size can be constructed

70

• A Haptics Example

71
Call Options Game

• Done for major american bank to train its traders
• Conceptual architecture and framework for mixing
  real and virtual data
• Added haptically rendering virtual stimuli

72
(No Transcript)
73
(No Transcript)
74
The Game - Call Options

• The options portfolio consist of options with 3
  different times to expiration.
• The gamma value of an option is increasing
  dramatically just before expiration if the option
  is about at-the-money.
• Local peaks or valleys therefore occur for each
  strike, near to maturity . (Long positions give
  positive gamma and short positions give negative
  gamma).

75
Using Haptics

• Peaks and valleys shown visually rendered
  haptically
• Mix real and virtual data

76
In Operation
77

• Another Haptics Example

78
Enclosed Hazardous Spaces
79
Narrow field of View
80
Constrained Space

• Need sense of touch to move
• haptic orienting cues play a critical role

81
Inspection Game

• Mix real object containment vessel
• Virtual hazards
• Tactile sensing
• Training

82

• Virtual City Support Component

83
Why

• Many mixed reality games will use actual city
  buildings, streets, locations
• Need make it easy to mix real and virtual for
  actual cities
• Military urban operations

84
Urban Operations

• In the last 10 years, however, the U.S. has
  committed military forces to far-flung urban
  operations in Panama, Somalia, Liberia, Bosnia
  and now Kosovo.
• The Marine Corps alone has been involved in urban
  operations in 237 of its last 250 deployments.

85
Urban Operations

• Chracterised by extremely short firing ranges
  imposed by a city's long canyons of high-rise
  buildings and dense thickets of trackless housing
  and teeming alleys
• Need for local infantry autonomy
• No command and control capability

86
Intelligence Reconnaissance
87
Requirements

• Quick and easy way to build virtual cities for
  training games
• Acuisition of motion imagery from camera phones,
  UAVs
• Build virtual cities from analysis of motion
  imagery
• Time Frame - Late 1990s

88
(No Transcript)
89
Xtensory Contracts

• Enhanced Multiplatform, Integrated Virtual Urban
  Warfare Simulation. Navy SBIR N99-063.
• Soldier-Computer Interface for Enhanced
  Interaction and Communication. Army SBIR A95-035
• Immersive Visualization of Complex Situations for
  Mission Rehearsal. Army SBIR A95-089.

90
Component

• Motion Imagery analysis (acquired from another
  company)
• Polygon processing into low polygon count models,
  autocad, 3DS file formats
• Good way to build virtual cities

91
Conclusion

• Shown how use of component tools help in making
  industrial strength edutainment games

92
(No Transcript)
93
(No Transcript)
94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
(No Transcript)