Ipke Wachsmuth

1
Embodied Communication

• Ipke Wachsmuth
• University of Bielefeld

2
Embodied Communication
Story

• Ipke Wachsmuth
• University of Bielefeld

3
Specifying terms
4
Specifying terms
EMBODY 1. to give concrete form to
personify or exemplify. 2. to provide with a
body incarnate. 3. to collect into a body
organize. embodiment, n.
5
Many many years back

• Cognition arose in living organisms, in nature it
  is inseparable from a body, and only makes sense
  in a body.

• Likewise, natural communication and human
  language developed in intimate connection with
  body.

 Embodied Cognition 
Embodied Communication
6
We dont know much about the origin of
communication...
7
PART 1 Embodied Communication What it is about
8
This talk is about how people communicate

• Man developed written language which allows to
  detach the transmission of information from body
• Spoken language also works without the body being
  visible (in the dark, on the phone)
• But in face-to-face communication a lot of other
  things than word symbols travel

                                   gt Example
Noams sentence
9
The general approach Im taking seems to me
rather simple-minded, and unsophisticated, but
nevertheless correct...
10
The general approach Im taking seems to me
rather simple-minded, and unsophisticated, but
nevertheless correct...
11
The general approach Im taking seems to me
rather simple-minded, and unsophisticated, but
nevertheless correct...
12
Body Motion  
When a person speaks, not only symbols
are transmitted, but the whole body is in
continuous motion.
13
   Face Expression
Emotional expression modifies what is
communicated in speaking.
14
Body, Flesh Muscles
The Expression of the Emotions in Man and Animals
Ch.Darwin
Diagram of the muscles of the face, from Sir C.
Bell.
15
Three basic types of sign        
here for gesture
C.S. Peirce

• Index pointing
• Icon characterizing
• depicting
• enacting
• Symbol designating in conventionalized form

16
Sign language conventional! (embodied, too)
Video clips courtesy R. Schulmeister, Hamburg
17
McNeill 1992/2003
Quite different Spontaneous gesture!  
Icon/enacting
David McNeill, 2003
The gestures I mean are the movements of the
hands and arms that we see when people talk...
These gestures are the spontaneous creations of
individual speakers, unique and personal. They
follow general principles... but in no sense are
they elements of a fixed repertoire.
David McNeill, 1992
18
Iconic gestures can serve to represent and
communicate mental images in an embodied form.
19
path in the middle
symbiosis
different interests
work together
20
go apart
go for different goals
this path
overlaps
21
Speaker-Hearer Communication Model
after T. Winograd, Language as a Cognitive
Process, Addison Wesley 1983
and Akmajians Information Transport Model
Emphasis on symbolic information transfer!
22
Spoken Language Processing
A lot has become known about early language
processing from brain research (e.g., MPI
Cognitive Neuroscience)
23
Gesture Processing (still harder to research)
It was in such a big box...
The gestural sign obtains meaning by iconicity,
i.e. a pictorial similarity between itself and
its imagined referent.
24
Gesture Space / Visuospatial Sketchpad
D. McNeill, Hand and Mind, 1992
A. Baddeley, Working Memory, 1986

• Visuospatial sketchpad (VSSP)
• one of two slave systems in Baddeley's model of
  working memory
• responsible for manipu-lation and temporary
  storage of visual and spatial information

?
25
Part 1 Conclusion

• Above symbolic communication (e.g., language),
  meanings are conveyed in a form which is not part
  of a conventionalized sign code but
  nevertheless understandable.
• This is what embodied communication is about!
• In face-to-face communication, meanings are
• multimodally encoded
• strongly situated in the context

26
Cognitive Modeling Challenge

• Devise operational models that
• are theoretically grounded and empirically guided
• specify how mental processes and embodiment work
  together in communication

27
PART 2 Artificial Humanoid Agents
28
Embodied communication, artificially
EMBODY 1. to give concrete form to
personify or exemplify. 2. to provide with a
body incarnate. 3. to collect into a body
organize. embodiment, n.
 If we want to embody communication in an
artificial system, we need to be concerned with
1., 2., 3.!
29
Face Robots, Embodied Conversational Agents
Gandalf (Thórisson)
REA (Cassell)
MEXI (Kleinjohann)
Kismet (Breazeal)
30
Situated communication  
Situated Artificial Communicators SFB 360
SFB 360 Research Scenario Bielefeld
User Mount it at the right. Agent You mean
here?
31
          Bielefeld Artificial Intelligence Lab
Research Mission

• AI methods used to establish an intuitive
  communication link between humans and multimedia
  systems
• Highly interactive Virtual Reality by way of
  multi-modal input and output systems (gesture,
  speech, gaze)
• Scientific enquiry and engineering of information
  systems closely interwoven (cognitive modeling
  approach)

32
New lab inaugurated in July 02
... and Max

• 3side cave-like projection (3dim)
• passive stereo (D-ILA), circular polarisation
  filters
• marker-based infrared-tracking (ART), wireless
  datagloves
• 8-channel spatial sound system
• Artabel Fleye Linux Cluster, 2GBit/s Myrinet

Thanks for s to DFG
33
Agent MAX
Situated Artificial Communicators SFB 360
An artificial communicator situated in virtual
reality

• Research into fundamentals of
• communicative intelligence
• PHYSIS the body system (especially gestures)
• COGNITION the knowledge system
• EMOTION the appraisal system

34
PHYSIS Articulated body  

• Hand animated by key framing
• Body animated by model-based animation
• Motion generators running concurrently and
  synchronized

Kinematic skeleton with 53 degrees of freedom
(DOF) in 25 joints for the body and 25 DOF for
each hand
Kopp Wachsmuth 2002, Proceedings of Computer
Animation (IEEE Press)
35
Measuring gestures

• Segmentation cues
• strong acceleration of hands, stopps, rapid
  changes in movement direction
• strong hand tension
• symmetries in two-hand gestures

36
Analyzing gestures
Symbolic classification of gesture shape
(HamNoSys)
37
Body Part Coding (HamNoSys)
(Hamburg Notation System Institut für
Deutsche Gebärdensprache, Hamburg)
38
Imitating Gestures
Imagery
STEP 2 Meaninglevel
Interpretation
Formation
STEP 1 Mimickinglevel
gesture description
Gesturerecognition
Gesturesynthesis
Real time animation
Sensor data
39
Gesture imitation game

• Human displays gestures, Max imitates them
• Parsing of gesture input HamNoSys
• HamNoSys for specification of gesture output

Real time!
40
Describing Shape
Sowa Wachsmuth 2002, in Gesture Sign Language
in HCI (Springer LNAI)
41
Fine-grained Hand Gestures
42
Iconic Mapping
43
Imagistic Description Tree (IDT)
Lang's (1989) Object Schemata as a basis for
structured shape representations
Sowa Kopp, EuroCogSci 2003
44
Analyzing language  
I Steck die gelbe Schraube    in die lange
Leiste. speech recognition
syntactic-semantic parsing reference to
perceived scene
Insert the yellow bolt into the long bar.
steck COMMAND CONNECT die DET gelbe
COLOR YELLOW Schraube OBJECTTYPE BOLT in
PREP IN die DET lange SIZE LARGE Leiste
OBJECTTYPE BAR
45
Multimodal Analysis tATN
Verbinde das gelbe Teil mit dem violetten
Teil... Connect the yellow part with the violet
part...
Integration of speech and gesture
Interpretation in application context
46
Lip-synchronous speech
Text-to-Speech TXT2PHO (IKP Uni Bonn),
MBROLA Phoneme transcription is the basis for
automatic generation of visemes. (Concept-to-Speec
h TO DO)
Historical Zemanek-Vocoder

• one viseme for M, P, B
• one viseme for N, L, T, D
• one viseme for F, V
• one viseme for K, G
• plus visemes for the vowels

47
Uttering speech and gesture
ltutterancegt ltspecificationgt Und jetzt
nimm lttime id"t1"/gt diese Leiste lttime
id"t2 chunkborder"true"/gt und mach sie
lttime id"t3/gt so gross. lttime id"t4"/gt
lt/specificationgt ltbehaviorspec
id"gesture_1"gt ltgesturegt ltaffiliate
onset"t1" end"t2"/gt ltconstraintsgt
ltparallelgt ltstatic slot"HandShape"
value"BSifinger"/gt ltstatic
slot"ExtFingerOrientation"
value"object_loc_1 mode"pointTo"/gt
ltstatic slot"GazeDirection" value"object_loc_1"
mode"pointTo"/gt
lt/parallelgt ...
 And now take this bar and make it this big.
MURML XML-based markup language for multimodal
utterance representations
Kranstedt, Kopp Wachsmuth 2002, AAMAS/ECA
Workshop Proceedings
48
Cognitively motivated architecture      
Situierte Künstliche Kommunikatoren SFB 360

• Perceive, Reason, Act running concurrently
• parallel processing by a reactive and a
  delibera-tive system
• information feedback in a cognitive loop
• BDI kernel with selfcon-tained dynamic planners
• account for embodiment (physis) of the agent,
  multimodality

Emotion system
49
Communicating with Agent Max
50
Simulated Muscle Effects
A Frontal muscle B Corrugator muscle C
Orbicular eye muscles D Eyelid E Levator
labii superioris alæque nasi F Zygomatic
and lifter of mouth corners G Depressor
anguli oris H Circular mouth muscle I
Unterlippenherabzieher J Jaw
Coordinated control of face muscles based on
Action Units (Ekman/Friesen) Student
Project (Körber Prize!) Emotion system under
development
51
Emotional Expression
52
Affect Dynamic Emotion Space
PREVAILING MOOD
SPONTANEOUS EMOTION

BOREDOM
PLEASURE
DOMINANCE
AROUSAL
53
Emocat Table
54
Wundt Emotion Dynamics
AROUSAL
RELAXATION
 PLEASURE
DISPLEASURE
TENSION
CALMING DOWN
55
Embodied Communication

• Intentionality
• knowledge / beliefs
• desires / motivations
• intentions
• commitments
• emotions...
• build on BDI architecture
• (Beliefs - Desires - Intentions)

• Anthropomorphic embodiment
• humanoid form
• personality
• facial expression
• gesture
• spoken language
• emotional features

Body Mind
56
Team
Bernhard Jung
Timo Sowa
Marc Latoschik
Ipke Wachsmuth
Ian Voß
Stefan Kopp
Alf Kranstedt
Peter Biermann
Nadine Leßmann
57
Max goes to Nixdorf Museum!     
58
CONCLUSION What is learned? What can be
expected?
59
Embodied Communication

• Have used examples to support research importance
  of embodied communication
• expect that the construction and test of an
  artificial communicator will help to reach more
  profound understanding
• Great impact in human interface research

60
A Multidisciplinary Endeavor

• More fundamental research is called for
• Fuller investigation will involve many more
  aspects, and many disciplines!
• WHAT ARE THE BIG QUESTIONS BEHIND?

be it body, be it mind
61
Bielefeld and the ZiF                     
Center for Interdisciplinary Research

• Bielefeld Universitys Institute for Advanced
  Study, founded 1968

62
Expectation                      
Center for Interdisciplinary Research

• Define a ZiF research year on Embodied
  Communication
• Invited workshops and fellowships (2005)
• If interested, get in touch with me!

63
Embodied Communication