Digital Interactive Entertainment

1
Digital Interactive Entertainment
Dr. Yangsheng Wang Professor of Institute of
Automation Chinese Academy of Sciences Yangsheng.w
ang_at_mail.ia.ac.cn http//hci.ia.ac.cn/JointLab/onl
inegame-En.htm
2
Overview ofDigital Entertainment Industry
3
Digital Entertainment Industry

• Year 2008 the market size will be about 100B and
  major products are TV games?PC games and mobile
  games.
• Microsoft, Sony and Nintendo provide most digital
  entertainment products.
• China has a very big potential market for digital
  entertainment products.

4
Internet and Game in China

• The total amount of Chinese Internet users is up
  to 87 millions in 2004, about 25 are online
  game(OG) users.
• OGs are one of major parts of digital
  entertainment contents in China. Almost half of
  the OGs on the domestic market mainly come from
  Korea. Some Chinese game companiers are beginning
  to develop their own OGs.
• PC Stand-alone games are losing their market.
• Mobile games are developed rapidly, but the
  income is on the low level at the present stage.
  
• Overseas game companies enter Chinese market via
  Joint-venture.

5
Internet Users in China
sourceCGPAIDC 2004
6
Online Game Market in China
sourceCGPAIDC 2004
7
Stand-alone Game Market in China
sourceCGPAIDC 2004
8
New Research Applications of Natural
Interactive Technologies
9
Natural Interactive Technologies (NIT)

• Traditional games mainly depend on mouse,
  keyboard, joystick and special input device.
• Game players desire more natural and intelligent
  interaction methods, including voice, gesture,
  expression, gaze, and intelligent sensor, etc.
• User-oriented multi-model HCI (Human Computer
  Interaction) technologies has become the hotspot
  of research and application in many domains.

10
NIT Research and Application-I

• Computer Vision for Computer Games and Graphics
• Camera is used as a powerful interface device for
  games by sensing users body position, head
  orientation, direction of gaze, pointing
  commands, and gestures.
• Developer Mitsubishi Electric Research
  Laboratories.
• Applications Sega games

Vision device
11
NIT Research and Application-II

• Vision-based Human-Computer Interface for Games
• By a camera, the player's body movements are
  projected right into the game as an unencumbered
  input and control device.
• Developer Sony Computer Entertainment America
• Applications Sony PlayStation2 EyeToy games

Camera
12
NIT Research and Application-III

• Voice Control for Games and Simulations
• Many of today's games have a lot of keyboard
  commands that can be hard to remember. Speech
  recognition technologies are used in this area
  for game control.
• Product Game Commander (Mindmaker)
• Product Game Voice (Microsoft SideWinder)

13
Interactive Entertainment Research Group in CASIA
14
The Outline of Our Research

• Auto-Construction of a Game Object
• Player-Oriented Game
• Multi-model Human Computer Interaction
• Natural Interactive Entertainment Platform.

15
Photo-stereo Based Model Reconstruction
flashlight

• Images capture
• Normal calculation
• Surface reconstruction

bracket
camera
flashlight
flashlight
control card
16
Images capture
Images of the surface under different lighting
conditions is captured in a very short period,
that the register is avoid.
17
Normal calculation
A photo-stereo method is used to retrieve the
surface normal based on the reflect property of
the surface.
18
Surface reconstruction

• The relevant height between the points can be
  integrated from the derivative field.
• A pyramid integration method is used to optimize
  the results

19
Orthogonal images Based Model Reconstruction

• Generic Model and its feature points selection
• Two images feature points editing
• Generic model deformation
• Texture merging and mapping

20
Generic model and its feature points selection
79 feature points according MPEG4
21
Images feature points editing andtexture merging
frontal face
side face
texture merging result
22
Generic model deformation andtexture mapping
texture
generic model
deforming
model reconstruction result
23
Individual game characters
player A
Individual faces give players more fun and
immersion.
player B
24
Video-Based Model Reconstruction and Animation
Capture users frontal face with a camera, then
deform an existent face model and map the face
image to create an individual face model. Five
kind of users expressions are recognized
real-time to drive the relevant animations.

• Face detection
• Deformation Texture mapping
• Expression recognition
• Expression retargeting Animation

25
Auto-Extraction of key points of a face



An RealBoost-based statistical learning is
adopted to build a model for each facial point.
During extraction, the models are used to locate
facial points for each face image.
26
Deformation Texture mapping


generic model
reconstructed individual models
deforming
Origin image
filling image
merging texture

27
Expression recognition and retargeting
A novel approach to real time facial expression
recognition using an embedded HMM which uses
observation vectors that are composed of DCT
results.
Source expression models
Expression recognition


Individual expression results

surprise
anger
happy
fear
sad
28
Getting into a Game - Change face

• Players frontal face is captured by a camera.
• Players face feature points are detected to
  deform characters face model.
• Players face image is used change characters
  face texture.

29
Video-Based Motion Detection and Analysis

• Head detection and pose estimation
• Fist detection and motion analysis
• Other motion detection and analysis

30
Head detection and pose estimation






front
left
right
up
down
close
far

• locating the position of a head
• recognizing the pose of the head

The novel scheme based on Markov model fusing
appearance, color, and motion information.
31
Fist detection and motion analysis
training samples
prototype DEMO
32
Driving a car using your hands and voice

• Players hands control the cars direction.
• Players voice controls the cars speed and
  status.

33
Motion Detection and Analysis

• Project video into game
• Motion detection
• Parameters calculation
• Game objects control

34
Playing a game with your body

• Players body movement is projected right into a
  game
• The motion detection and analysis results are
  used to activate objects in a game.

35
Online-Game Strategic Research Group
36
Online-Game Strategic Research Group
The group includes major digital entertainment
companies, universities, research institutes, and
digital entertainment association in China.
A meeting of the group
37
The goal of the Strategic Research Group

• Write strategic report of digital entertainment
  for Chinese government.
• Establish China National Center for Digital
  Entertainment.

38
Thank you!