You Are Player One User Interfaces in Interactive Entertainment

1
You Are Player One User Interfaces in
Interactive Entertainment

• Richard Marks
• Manager, Special Projects

2
An Interface Revolution is beginning

• Graphics have improved dramatically
• Interface is limiting the interactivity
• Interface is limiting the fun

3
Describe the ultimate interface

• Fun
• Intuitive
• Interactive
• Flexible
• Enabling
• Fun!

• PS9 commercial shows what people think is
  futuristic
• Blurring of distinction between real and virtual
• Sensory enhancement
• Gaming anywhere and everywhere
• Experiential
• Describe the system by describing its interface

4
Background Games are big busine

• For the last 2 years, video game sales revenue
  has exceeded box-office receipts
• FY 2004, Electronic Arts reports
• Revenue of 2.96 billion
• Net income of 577 million
• In less than 4 years worldwide.
• 70 million PlayStation2 hardware units
• 572 million PlayStation2 software units
• 1 in 4 American homes has PlayStation2

5
Accelerating adoption rate
Million units
10
PS2
9
8
7
32/64bit Game
CD Player
6
5
PC
4
3
Walkman
2
1
0
0
1
2
3
4
5
6
7
8
9
years
6
PlayStation RD (SCE America)

• Mission
• Catalyze new ideas for interactive entertainment
• Focus
• 0 to 5 years
• Groups
• Advanced Graphics
• Distributed Processing
• Developer Tools (compilers, optimizers)
• Special Projects
• Animation and physical simulation
• Man-machine interface

7
RD job description

• Improve existing experience
• Enable new experiences
• We dont make games, we make them better
• Give game developers more tools to use for
  creation
• Create software that game developers can use
• Prototype tech demos to inspire game developers

8
RD cycle

• Matched to PlayStation platform cycle (5-6 years)
• 2-3 years research
• Focused on about a 5 year look-ahead
• Intended to raise the bar of computer
  entertainment
• 2 years development
• Hardware specification, proving workloads
• Advanced libraries, technology demos, samples
• 1 year production
• Maintenance, support of our releases

9
Research models

• Push
• Research group pushes technology out of lab and
  into product groups
• Tough because product groups have schedules
• Example EyeToy
• Pull
• Tends to be evolutionary, not revolutionary
• Less freedom, but certainty of usefulness
• Example PlayStation Voice Recognition
• Stockpile
• What do we do with this? Dont know, save it for
  later.
• Most common model for large research groups

10
Level of interaction

• Games now.

L1 o ?
PS2
42
11
Interface Modalities

• Inertial (accelerometers, rate gyros)
• Touch (haptics, force-feedback)
• Audio
• Speech (speech recognition, speaker id)
• Waveform (frequency, amplitude, rhythm)
• Video
• Familiar
• Fast
• Physical
• Personal
• Multipurpose

12
Voice Input

• Voice recognition
• Command/control, user-independent, isolated word,
  text authored
• Used in NFL GameDay, NCAA Gamebreaker, NBA
  Shootout
• Online adaptation to users voice
• Far-microphone voice input in noisy environments
• Acoustic Echo Cancellation
• Multi-microphone techniques
• Speaker identification
• Voice fonts

13
Video interface categories

• Video As Input (VAI)
• joystick replacement
• user does not see video
• applicable to existing games
• Enhanced Reality (or augmented reality)
• live video mixed with computer graphics
• movie-like special effects
• new entertainment genre
• Real-time Motion Capture
• direct interaction
• enables physical interaction

14
Video As Input

• User does not see the video
• Video quality only important for processing
• Only need to capture/process useful info
• Frame rate and latency are critical
• Immediate and obvious visual feedback is
  important

15
Enhanced Reality

• Augmented reality, but with an entertainment
  focus
• Minimize encumbrance
• Utilize common hardware
• Simplify setup/calibration
• Create an enjoyable user experience

16
Video interface design goals

• Accessible
• Affordable
• Easy-to-use
• Unobtrusive
• Targeted
• Lens good for low light
• Compression specific to PS2 hardware
• Video size appropriate for PS2 processing power
• Multi-purpose
• Communication (e.g. EyeToy Chat)
• Snapshots

17
EyeToy research to product

• 1999-2000
• initial research and prototyping
• 2001
• join the London game team for 3 months
• learn about their world
• L transfer my technology, but more importantly,
  they absorb how I think, so they can extend the
  ideas
• 2002
• experienced game team creates games, not
  technologist
• 2003
• EyeToy launches globally
• 2004
• 5 million EyeToys shipped to date

18
EyeToy Play

• Easy to jump in and play
• Not too serious
• Very social
• Amazing demographics
• Young children, parents, girls, grandparents

19
Mature video technology

• Motion detection
• Color segmentation
• Simple-object pose recovery
• Sphere, position only
• Cylinder, positionorientation
• Gesture recognition
• Pattern matching
• Basic video/graphics compositing

20
Recent video technology

• Freeform mouse-like input (e.g. Minority Report)
• Retro-reflection
• The Clam deformable input device
• Face detection
• Face tracking
• Better color tracking
• Demos (clam, view tracking)

21
Just around the corner

• PSP camera
• PSP is your window to an enhanced world
• Mobile augmented reality
• Mobile video communication
• Face tracking for animation communication
• Next generation console camera
• Much more CPU
• Much higher input bandwidth
• Camera should be higher res, faster, more
  sensitive
• New interfaces an important design criterion

22
Future video technology

• Per-pixel distance measurement
• RGBZ video
• Allows true mix of graphics with video using Z
  buffer
• Easy per-pixel background/foreground (user)
  segmentation
• Video Enhanced Reality special effects play
  movie
• Video Real-time motion capture play movie

23
Concluding remarks

• Interface a key element of design
• Blend of technology and psychology
• Impacted by real-world considerations
• Entertainment is a great way to try new
  technology
• Especially the game industry
• A revolution is already underway
• Now, my 4-year-old waves his hand in front of the
  TV, expecting it to turn on
• Next year, hell be talking to it
• Soon, all this will fit in his pocket
• And he thinks nothing of it.

24
Thank you!

• Questions?