Gestures and Avatar Signing

1
Gestures and Avatar Signing
2
Gesture Research and Virtual Human Signing

• John Glauert
• Ralph Elliott
• Richard Kennaway
• Judy Tryggvason
• Vince Jennings
• Collaboration with Graphics, Linguistics,Vision,
  and Speech groups

3
Sign Language

• Purely visual
• Multimodal
• Manual and Facial gestures
• Natural language
• British Sign Language (BSL)
• 1 in 1000 prelingually deaf
• Grammar and phonetic structure not based on
  English

4
Challenges

• Capturing gestures
• Motion capture
• Gesture recognition
• Animation of gestures
• Notations for gesture
• Realistic synthetic animation
• Grammar and semantics of signing

5
Motion Capture

• Conventional Motion Capture
• Intrusive sensors
• Unstable
• Markerless Motion Capture
• Video-based
• Robust over long periods
• With BBC Barry Theobald, Vince Jennings, John
  Glauert, Andrew Bangham

6
Gesture Notation

• Hamburg Notation System
• Universal notation for sign language
• SiGML
• XML language
• Signing Gesture Markup Language
• EU eContent Richard Kennaway, Ralph Elliott,
  John Glauert

7
Realistic Synthetic Animation

• Signing demands precise animation
• Enhanced skeleton
• Collision detection
• Realistic motion
• Natural trajectory
• IK for aspects ignored by notation
• EU eContent Vince Jennings, Richard Kennaway,
  Judy Tryggvason, John Glauert, Ralph Elliott

8
Gestures and Avatar Signing
9
Virtual Human Signingat UEA

• John Glauert, Ian Marshall
• Andrew Bangham, Stephen Cox, Ralph Elliott
• School of Computing Sciences
• University of East Anglia, UK

10
ViSiCASTSign Language Using Virtual Humans

• John Glauert
• Andrew Bangham, StephenCox, Ralph Elliott,
  IanMarshall
• University of East Anglia,Norwich UK

11
UEA and Norwich
12
University of East AngliaNorwich UK

Norwich
Bristol
13
University of East AngliaSchool of Computing
Sciences

• Computing Science
• Electronic Engineering
• Leading Research and Education

14
science_at_uea

• Breadth of science
• Biological Sciences
• Chemical Sciences
• Environmental Sciences
• Information Systems
• Mathematics

15
science_at_uea

• Diversity of Research
• TESSA award winning technology translating
  speech into sign language
• effect of forest fires in the Amazon
• hazard management of volcanoes
• internationally renowned climate research
• ecology and conservation of endangered species
• disease processes, novel solutions

16
Norwich and Norfolk?
17
Norwich and Norfolk?
18
Norwich Research Park
19
Norwich Research Park
University of East Anglia
BUPA Hospital Norwich
Norfolk Norwich University Hospital
Institute of Food Research
Plant BioSciences Limited
John Innes Centre
British Sugar Technical Centre
The Sainsbury Laboratory
Zeneca Wheat Improvement Centre
20
Deafness in Britain
21
Deafness in Britain

• 1 in 7 deaf or hard of hearing
• Hearing aids, lip-reading English, Teletext
• 1 in 1000 pre-lingually deaf
• Signing is closest to natural language
• 50,000 pre-lingually deaf people in UK
• British Sign Language (BSL) has its own grammar,
  lexicon
• Reading age (for English) is usually low

22
Deafness in Britain

• Lose opportunities available to other members of
  society
• Access to Services is Limited
• Shops and the High Street
• Television
• Web and Electronic Information

23
Sign Supported vs.Authentic Sign Languages

• In UK
• SSE Sign-Supported English
• one sign per word (approx.)
• follows English word order
• BSL British Sign Language
• one sign per concept
• use of signing space around signers body
• has own word order, morphology
• SSE and BSL both utilize finger-spelling

24
Virtual Reality andVirtual Environments

25
Virtual Reality
26
Virtual Town Hall
27
Virtual HumansorAvatars

28
Annanova
29
Avatars Virtual humans

• Films
• Internet
• Games

30
Virtual Dancing
31
Simon the Signer
32
Simon the Signer, 1997-1999

• Simon-the-Signer Broadcast TV
• Generate signed accompaniment to broadcast
• Teletext stream as source
• SSE Sign Supported English

33
Simon the Signer , 1997-1999

• Developing transmission technology
• virtual signer in set-top boxes
• transmission of signing from text subtitles

Audio/Video Stream
TV Capture Card
Avatar
Software Mixer
Teletext Stream
Computer
34
Simon the Signer

• Winner of two Royal Television Society awards

35
TESSA
36
TESSA, 1998-2000

• TESSA Retail, PO
• Using speech recognizer
• Convert counter-clerks voice input to text
• Generate sign stream from text
• BSL limited repertoire

37
Post Office Counter Services

• Post Offices transact business with almost all
  Deaf people
• Counter Clerk asks questions using speech - No
  back channel yet
• Customer
• Listens or
• Reads or
• Watches Virtual Human Signer
• Real trials at 5 sites

38
TESSA
Tessa Winner of BCS Gold Medal and IT 2000 Award
39
ViSiCAST
40
ViSiCAST
Virtual Signing Capture, Animation, Storage and
Transmission

• Translation of text to sign
• Animation of signing
• Broadcast Transmission
• Web and Multimedia
• Counter Services

41
The ViSiCAST Project

• Virtual Signing Capture, Animation, Storage and
  Transmission
• Funded under EU Framework V Programme
• Additional Funding from ITC and Consignia
• pre-competitive research
• IST-1999-10500

42
ViSiCAST Aims

• Improved access for deaf citizens
• to information and services
• in their preferred medium of sign language
• Builds on SignAnim and Tessa

43
The ViSiCAST Consortium
44
ViSiCAST Partners
ITC, UK Project co-ordination IRT, Germany
Broadcast technology Televirtual, Norwich, UK
Avatar creation IDGS, Germany Sign language
notation UEA Norwich, UK Processing of
language, speech signing
45
ViSiCAST Partners
INT, France Broadcast imaging animation
standards IvD, Netherlands Multimedia content
creation Post Office, UK Face-to-face
transaction systems RNID, UK Monitoring of
signing and evaluation
46
ViSiCAST Background

• Simon-the-Signer (ITC) (1997-1999)
• ITC (UK Independent Television Commission),
  Televirtual, UEA Norwich
• Tessa (Consignia) (1998-2000)
• Post Office, Televirtual, UEA Norwich
• Both based on virtual human signing
• using Televirtuals motion-capture driven avatar
  technology

47
ViSiCAST Project

• Extend applications of virtual signing
• Target to natural sign languages
• BSL (British Sign Language) rather than
• SSE (Sign-Supported English)
• Improve animation technology
• increasingly natural avatars
• easier but more accurate sign capture

48
ViSiCAST Structure
Applications

Enabling Technologies
49
ViSiCAST Structure
WWW
Transactions
Broadcast
Language
Animation
50
Multimedia and the Internet

• Adding signing services to multimedia
• improves access to information for leisure,
  learning and communication
• Browser plug-in
• accurate signing of existing content on the
  internet
• translation of own text to generate signed
  content on own website

51
Face-to-Face Transactions

• Post Office, Advice Services, Shops
• Simple spoken phrases recognised and translated
  to sign language
• Aim for limited sign recognition for back
  channel

52
Television and Broadcast

• Developing transmission technology
• virtual signer in set-top boxes
• transmission of compressed signing data

53
Television and Broadcast

• Developing transmission technology
• virtual signer in set-top boxes
• transmission of signing from text subtitles

Audio/Video Stream
TV Capture Card
Avatar
Software Mixer
Teletext Stream
Computer
54
Television and Broadcast
55
Virtual Human Signing Contexts
56
Why useVirtual Human Signing?
57
When do we need Signing?

• Events
• TV
• High Street
• Web and Communications

58
Signing Interpreters

• Excellent for Events and TV
• Not enough to accompany all Deaf people
• Not practical for ephemeral information
• Newspapers
• Web

59
Video of Signing

• Excellent for Fixed information sources
• Need to blend video sequences
• Hard
• Inflexible
• Expensive for ephemeral information

60
Virtual Human Signing

• Can use realistic Captured motion
• Visual quality improving
• Possible to blend sequences
• Can be used to Synthesise signs
• Textual sign representation
• User freedom to create own content
• Much lower bandwidth than video

61
Virtual Human Signing Approaches
62
Virtual Human Signing

• Motion Capture and Playback
• Hand-Crafted Animation
• Blending
• Synthesis from Signing Notation

63
Motion Capture

• Very lifelike animation
• Time-consuming to set up
• Blending of signs
• Combining signs from different signers

64
Hand-Crafted Animation

• Define Key Frames
• Interpolate between Key Frames
• Can give good animation
• Time consuming (12 hours per sign)
• Blending of signs still required

65
Synthetic Signing

• Synthesis from Abstract Representation
• Quick to create lexicon
• a few minutes to transcribe a sign
• Instantly retargettable to any avatar with
  humanoid topology
• Automatic Blending
• Low Bandwidth

66
Virtual Human Signing Motion Capture
67
Motion-Capture for Virtual Human Signing

• Motion Capture Streams
• body
• magnetic tracking
• face
• reflective markers head-mounted camera
• hands
• gloves with bend-sensors

68
Data Capture Face Tracking
Face tracker 20 reflectors, helmet mounted
camera 60/2 Hz
69
Data Capture Cybergloves
Cybergloves 18 resistors modulated by bend sample
rate, lt50 Hz
70
Data Capture Magnetic Sensors
Magnetic sensors, Motion star Wrist, elbow,
head, body 86/2 Hz
71
Virtual Human Signing Animation
72
Virtual Humans Animation

• Good motion capture allied withFast real-time
  graphics
• Bones-Set
• Lengths and interconnection topology (joints)
• Specify joint angles and orientation
• Rendering
• attach mesh (wire-frame) to Bones-set
• apply texture-mapping to mesh
• Animation
• sequence of rendered frames
• each defined by a Bones-Set configuration

73
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data

74
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data
• Tracked by an enveloping mesh model
• Rendered with OpenGL

75
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data
• Tracked by an enveloping mesh model
• Rendered with OpenGL
• Texture map
• Some of the 5000 polygons

76
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data
• Tracked by an enveloping mesh model
• Rendered with OpenGL
• Texture map
• Some of the 5000 polygons
• 50 frames per sec
• nVidia GForce2

77
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data

78
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data
• Tracked by an enveloping mesh model
• Rendered with OpenGL

79
Virtual Humans Animation

• Three dimensional model
• Custom Skeleton driven by motion data
• Tracked by an enveloping mesh model
• Rendered with OpenGL
• Texture map
• Some of the 5000 polygons
• Animated in real time

80
Virtual Human Signing System
81
Motion Capture and Display System
Computer System
care for
82
Motion Capture
Computer System
Post-processing
83
Display System
Computer System
Weather Forecast
84
From Capture to Signing Simon Tessa

• Capture clips of signing
• based on vocabulary for chosen subject area
• requires calibration match signer to avatar
• Segment/Edit clips
• save as files, one per sign
• Generate Stream of Sign Names
• for required script
• Feed Sign Stream to Avatar
• acts as a Player for stream
• blending between signs

85
ViSiCAST Applications
86
Web Applications
87
Web ApplicationsWeather Forecasts

• Signed Weather Forecasts
• SLN (The Netherlands)
• DGS (Germany)
• BSL (Britain)
• Form Filling for Forecast
• Dull and misty in places at first but soon
  becoming warm, dry and sunny.
• Met Office Summary Midlands 24/04/2002

88
Web ApplicationsWeather Forecasts
89
Weather Forecasts
Friday
Tomorrow
Today
Dull and misty in places at first but soon
becoming warm, dry and sunny. Maximum temperature
23 deg C (73 deg F). Tonight Becoming cloudy
overnight with perhaps an odd spot of rain.
Minimum temperature 8 deg C (46 deg F).
Cloudy start, then dry with sunny spells but
cooler.
Met Office for Midlands 24/04/2002
90
Web ApplicationsWeather Forecasts

• Grammar for normal Weather Phrases
• Sign Language version for each Phrase
• Forecast is sequence of Phrases
• Generate Common XML Weather Model
• XSLT processing for each Sign Language
• XSLT processing for Spoken Languages

91
Web ApplicationWeather Forecasts

• Rather cloudy with patchy rain at times, but
  also some brighter intervals. Windy but mild.
  Maximum temperature 13 deg C (55 deg F).
• Tonight Patchy rain will clear during the
  night leaving clear spells. Still rather breezy.
  Mild. Minimum temperature 5 deg C (41 deg
  F). Met Office for Southeast 06/03/2002

92
Web Application Demo
93
Virtual Human Signing Synthesis from Notation
94
SiGML Notation for Signing

• Hamburg Notation System
• HamNoSys
• Code for hand shape and orientation, location,
  and movement
• Signing Gesture Markup Language
• XML Compliant (W3C standards)
• Builds on HamNoSys

95
HamNoSys

• General notation for signing
• originally defined primarily for purposes of
  recording, transcription, study of signing
• Intention
• capable of representing any sign language
• some enhancements in area of non-manual features
  needed
• Defines
• semantic model for signing gestures
• pictographic notation

96
HamNoSys Examples
DGS (German) Sign GOING-TO BSL Sign
NAME BSL Sign ME
97
SiGML Notation for Signing

• Gloss level
• GIVE_BOOK_I_YOU
• code for a complete sign
• similar to SignAnim and Tessa approach
• HamNoSys level
• encodes sign phonemes as in HamNoSys
• Articulation level
• represents captured or synthesised motion
• encodes arbitrary gestures

98
XML Format

• Use nested labelled bracket structure
• Similar to HTML
• represent brackets by element tags
• ltmyelement gt lt/myelementgt
• Element
• may contain sub-elements and/or text
• may have named attributes
• Document Type Definition
• Defines possible elements (tags)
• permitted attributes for elements

99
Current SiGML Definition

• Two XML Applications focussed on manual subset
  of HamNoSys
• HamNoSysML
• DTD as close as possible to HamNoSys
• SiGML
• Tuned for animation

100
SiGML Notation NAME-ME

• ltsigmlgt
• ltsigmlsigngt
• ltsign_manual both_hands"false"gt
• lthandconfig extfidir"ul" palmor"dl"
  handshape"point12" thumbpos"across"
  location"forehead_right"/gt
• ltdirectedmotion direction"or"gt
• lthandconfig palmor"r"/gt
• lt/directedmotiongt
• lt/sign_manualgt
• lt/sigmlsigngt
• ltsigmlsigngt
• ltsign_manual both_hands"false"gt
• lthandconfig extfidir"uil" palmor"l"
  handshape"point1" thumbpos"across"
  location"chest_near"/gt
• lt/sign_manualgt
• lt/sigmlsigngt
• lt/sigmlgt

101
NLP and Synthesis

102
English to Sign

• Translation via intermediate code Discourse
  Representation Structure (DRS)

103
Animation from Notation

104
Animation of HamNoSys

• Make explicit everything HamNoSys leaves implicit
  or fuzzy
• Position
• Elbows and Shoulders
• Speed
• Trajectories

105
Naturalistic Animation

• A hard problem in general
• e.g. walking
• Easier for signing
• No interaction with environment
• Ignore gravity

106
Controller Response
107
Inverse Kinematics

• Hand position and orientation given by HamNoSys
• From these, compute joint angles from clavicle to
  wrist
• Inverse Kinematics
• 3 degrees of freedom per arm left undetermined
• Respect the limits of the joints
• Avoid the arm passing through the body

108
Ambient Motion

• If only arms, hands, and face are animated, the
  result is stiff
• Mix synthetic animation with motion-captured
  ambient motion for the spine and head

109
Stick-figure Avatars

• Useful for developing animations
• Easier to render, so more frames per second
• Skeleton gives clearer view of motion
• Prototyping tool only, not intended for end user!

110
VRML for Prototyping

• Virtual Reality Modelling Language
• Textual description language for 3D animated
  scenes
• H-Anim standard for articulated humanoid figures
• H-Anim incorporated into MPEG-4

111
Conclusions

112
Role of Deaf Organisations

• Bridge between the project partners and the deaf
  people who could benefit from the technology
• Wide dissemination of project aims
• Collation of UK feedback by RNID through visits
  to deaf clubs and groups
• Evaluations of prototype systems by deaf people
  to influence how systems can be improved

113
ViSiCAST Conclusion

• Aims ambitious within 3 years
• Novel computational linguistics work to generate
  and represent signing
• Advanced avatar technology for signing virtual
  humans
• Input essential from deaf people so that the
  technology develops to maximise benefits

114
ViSiCAST
Virtual Signing Capture, Animation, Storage and
Transmission http//www.visicast.cmp.uea.ac.uk ht
tp//www.visicast.org
115
Blank

116
(No Transcript)
117
(No Transcript)