Towards Robot Theatre

1
Towards Robot Theatre
Marek Perkowski Department of Electrical and
Computer Engineering, Portland State University,
Portland, Oregon, 97207-0751
2
Week 2

• Lectures 3 and 4

3

• Humanoid Robots and Robot Toys

4
Talking Robots

• Many talking robots exist, but they are still
  very primitive
• Work with elderly and disabled
• Actors for robot theatre, agents for
  advertisement, education and entertainment.
• Designing inexpensive natural size humanoid
  caricature and realistic robot heads

Dog.com from Japan
We concentrate on Machine Learning techniques
used to teach robots behaviors, natural language
dialogs and facial gestures.
Work in progress
5
Robot with a Personality?

• Future robots will interact closely with
  non-sophisticated users, children and elderly, so
  the question arises, how they should look like?
• If human face for a robot, then what kind of a
  face?
• Handsome or average, realistic or simplified,
  normal size or enlarged?

• The famous example of a robot head
• is Kismet from MIT.

• Why is Kismet so successful?
• We believe that a robot that will interact with
  humans should have some kind of personality and
  Kismet so far is the only robot with
  personality.

6
Robot face should be friendly and funny

• The Muppets of Jim Henson are hard to match
  examples of puppet artistry and animation
  perfection.

• We are interested in robots personality as
  expressed by its
• behavior,
• facial gestures,
• emotions,
• learned speech patterns.

7
Behavior, Dialog and Learning
Words communicate only about 35 of the
information transmitted from a sender to a
receiver in a human-to-human communication. The
remaining information is included in
para-language. Emotions, thoughts, decision and
intentions of a speaker can be recognized earlier
than they are verbalized. NASA

• Robot activity as a mapping of the sensed
  environment and internal states to behaviors and
  new internal states (emotions, energy levels,
  etc).
• Our goal is to uniformly integrate verbal and
  non-verbal robot behaviors.

8
(No Transcript)
9
Moritas Theory
10

• Robot Metaphors and Models

11
Animatronic Robot or device
brain
effectors
12
Perceiving Robot
brain
sensors
13
Reactive Robot is the simplest behavioral robot
Brain is a mapping
sensors
effectors
This is the simplest robot that satisfies the
definition of a robot
14
Reactive Robot in environment
ENVIRONMENT is a feedback
brain
sensors
effectors
This is the simplest robot that satisfies the
definition of a robot
15

• Braitenberg Vehicles and Quantum Automata Robots

16
Another Example Braitenberg Vehicles and Quantum
BV
17
Braitenberg Vehicles
18
Emotional Robot has a simple form of memory or
state
Brain is a Finite State Machine
sensors
effectors
This is the simplest robot that satisfies the
definition of a robot
19
Behavior as an interpretation of a string

• Newton, Einstein and Bohr.
• Hello Professor
• Hello Sir
• Turn Left . Turn right.

behavior
20
Behavior as an interpretation of a tree

• Newton, Einstein and Bohr.
• Hello Professor
• Hello Sir
• Turn Left . Turn right.

behavior
Grammar. Derivation. Alphabets.
21

• Our Base Model and Designs

22
Neck and upper body movement generation
23
Robot Head Construction, 1999
High school summer camps, hobby roboticists,
undergraduates
Furby head with new control
Jonas
We built and animated various kinds of humanoid
heads with from 4 to 20 DOF, looking for comical
and entertaining values.
24

• Mister Butcher

Latex skin from Hollywood
4 degree of freedom neck
25
Robot Head Construction, 2000
Skeleton
Alien
We use inexpensive servos from Hitec and Futaba,
plastic, playwood and aluminum. The robots are
either PC-interfaced, use simple
micro-controllers such as Basic Stamp, or are
radio controlled from a PC or by the user.
26
Technical Construction, 2001 Details
Marvin the Crazy Robot
Adam
27

• Virginia Woolf

2001
heads equipped with microphones, USB cameras,
sonars and CDS light sensors
28
2002
Max

• BUG (Big Ugly Robot)

Image processing and pattern recognition uses
software developed at PSU, CMU and Intel (public
domain software available on WWW). Software is
in Visual C, Visual Basic, Lisp and Prolog.
29
Visual Feedback and Learning based on
Constructive Induction
2002
Uland Wong, 17 years old
30
Professor Perky
2002, Japan
Professor Perky with automated speech recognition
(ASR) and text-to-speech (TTS) capabilities

• We compared several commercial speech systems
  from Microsoft, Sensory and Fonix.
• Based on experiences in highly noisy environments
  and with a variety of speakers, we selected Fonix
  for both ASR and TTS for Professor Perky and
  Maria robots.
• We use microphone array from Andrea
  Electronics.

1 dollar latex skin from China
31
Maria, 2002/2003
20 DOF
32
Construction details of Maria
location of head servos
skull
location of controlling rods
location of remote servos
Custom designed skin
33
Animation of eyes and eyelids
34
Cynthia, 2004, June
35
Currently the hands are not moveable.
We have a separate hand design project.
36
Software/Hardware Architecture

• Network- 10 processors, ultimately 100
  processors.
• Robotics Processors. ACS 16
• Speech cards on Intel grant
• More cameras
• Tracking in all robots.
• Robotic languages Alice and Cyc-like
  technologies.

37
Face detection localizes the person and is the
first step for feature and face recognition.
Acquiring information about the human face
detection and recognition, speech recognition and
sensors.
38
Face features recognition and visualization.
39
Use of Multiple-Valued (five-valued) variables
Smile, Mouth_Open and Eye_Brow_Raise for facial
feature and face recognition.
40
HAHOE KAIST ROBOT THEATRE, KOREA, SUMMER 2004
Czy znacie dobra sztuke dla teatru robotow?
Sonbi, the Confucian Scholar
Paekchong, the bad butcher
41
Editing movements
42
Yangban the Aristocrat and Pune his concubine
The Narrator
43
The Narrator
44
(No Transcript)
45
We base all our robots on inexpensive
radio-controlled servo technology.
46
We are familiar with latex and polyester
technologies for faces
Martin Lukac and Jeff Allen wait for your help,
whether you want to program, design behaviors,
add muscles, improve vision, etc.
47
New Silicone Skins
48
A simplified diagram of software explaining the
principle of using machine learning based on
constructive induction to create new interaction
modes of a human and a robot.
49

• Probabilistic and Finite State Machines

50
Probabilistic State Machines to describe emotions
you are beautiful / Thanks for a compliment
P1
Happy state
you are blonde! / I am not an idiot
P0.3
you are blonde! / Do you suggest I am an
idiot?
Unhappy state
P0.7
Ironic state
51
Facial Behaviors of Maria
Do I look like younger than twenty three?
Maria asks
Response

• no

• no

• yes

0.7
0.3
Maria smiles
Maria frowns
52
Probabilistic Grammars for performances
Speak Professor Perky, blinks eyes twice
P0.1
Speak Professor Perky
Where?
P0.3
Who?
P0.5
P0.5
P0.5
Speak in some location, smiles broadly
Speak In the classroom, shakes head
Speak Doctor Lee
What?
P0.1
P0.1
P0.1
Speak Was singing and dancing
P0.1
Speak Was drinking wine
.
53
Human-controlled modes of dialog/interaction
Human teaches
Thanks, I have a lesson
Hello Maria
Lesson finished
Robot performs
Robot asks
Question
Stop performance
Questioning finished
Command finished
Thanks, I have a question
Thanks, I have a command
Human commands
Human asks
54

• Dialog and Robots Knowledge

55
Robot-Receptionist Initiated Conversation
Human
Robot
What can I do for you?
Robot asks
This represents operation mode
56
Robot-Receptionist Initiated Conversation
Human
Robot
What can I do for you?
I would like to order a table for two
Robot asks
57
Robot-Receptionist Initiated Conversation
Human
Robot
Smoking or non-smoking?
Robot asks
58
Robot-Receptionist Initiated Conversation
Human
Robot
Smoking or non-smoking?
I do not understand
Robot asks
59
Robot-Receptionist Initiated Conversation
Human
Robot
Do you want a table in a smoking or non-smoking
section of the restaurant? Non-smoking section
is near the terrace.
Robot asks
60
Robot-Receptionist Initiated Conversation
Human
Robot
Do you want a table in a smoking or non-smoking
section of the restaurant? Non-smoking section
is near the terrace.
A table near the terrace, please
Robot asks
61
Human-Initiated Conversation
Human
Robot
Hello Maria
initialization
Robot asks
62
Human-Initiated Conversation
Human
Robot
Hello Maria
What can I do for you?
Robot asks
63
Human-Asking
Human
Robot
Question
Question
Robot asks
Human asks
64
Human-Asking
Human
Robot
Question
Yes, you ask a question.
Human asks
65
Human-Asking
Human
Robot
What book wrote Lee?
Yes, you ask a question.
Human asks
66
Human-Asking
Human
Robot
What book wrote Lee?
I have no sure information.
Human asks
67
Human-Asking
Human
Robot
Try to guess.
I have no sure information.
Human asks
68
Human-Asking
Human
Robot
Try to guess.
Lee wrote book Flowers.
Human asks
69
Human-Asking
Human
Robot
This is not true.
Lee wrote book Flowers.
Human asks
70
Human-Teaching
Human ends questioning
Human
Robot
Questioning finished
Human asks
Human teaches
Questioning finished
Robot asks
Thanks, I have a lesson
71
Human-Teaching
Robot enters asking mode
Human
Robot
Questioning finished
What can I do for you?
Human asks
Human teaches
Questioning finished
Robot asks
Thanks, I have a lesson
72
Human-Teaching
Human starts teaching
Human
Robot
Thanks, I have a lesson
What can I do for you?
Human asks
Human teaches
Questioning finished
Robot asks
Thanks, I have a lesson
73
Human-Teaching
Human
Robot
Thanks, I have a lesson
Yes
Human teaches
74
Human-Teaching
Human
Robot
I give you question-answer pattern
Yes
Human teaches
75
Human-Teaching
Human
Robot
Question pattern What book Smith wrote?
Yes
Human teaches
76
Human-Teaching
Human
Robot
Answer pattern Smith wrote book Automata
Theory
Yes
Human teaches
77
Human-Teaching
Human
Robot
Checking question What book wrote Smith?
Yes
Human teaches
78
Human-Teaching
Human
Robot
Checking question What book wrote Smith?
Smith wrote book Automata Theory
Human teaches
79
Human-Teaching
Human
Robot
I give you question-answer pattern
Yes
Human teaches
80
Human-Teaching
Human
Robot
Question pattern Where is room of Lee?
Yes
Human teaches
81
Human-Teaching
Human
Robot
Answer pattern Lee is in room 332
Yes
Human teaches
82
Human-Checking what robot learned
Human
Robot
Lesson finished
Human teaches
Lesson finished
Question
Robot asks
Human asks
83
Human-Checking what robot learned
Human
Robot
Lesson finished
What can I do for you?
Human teaches
Lesson finished
Question
Robot asks
Human asks
84
Human-Checking what robot learned
Human
Robot
Question
What can I do for you?
Human teaches
Lesson finished
Question
Robot asks
Human asks
85
Human-Asking
Human
Robot
Question
Yes, you ask a question.
Human teaches
Lesson finished
Question
Robot asks
Human asks
86
Human-Asking
Human
Robot
What book wrote Lee?
Yes, you ask a question.
Human asks
87
Human-Asking
Human
Robot
What book wrote Lee?
I have no sure information.
Human asks
88
Human-Asking
Human
Robot
Try to guess.
I have no sure information.
Human asks
89
Human-Asking
Human
Robot
Try to guess.
Lee wrote book Automata Theory
Observe that robot found similarity between Smith
and Lee and generalized (incorrectly)
Human asks
90
Behavior, Dialog and Learning

• The dialog/behavior has the following components
• (1) Eliza-like natural language dialogs based on
  pattern matching and limited parsing.
• Commercial products like Memoni, Dog.Com, Heart,
  Alice, and Doctor all use this technology, very
  successfully for instance Alice program won the
  2001 Turing competition.
• This is a conversational part of the robot
  brain, based on pattern-matching, parsing and
  black-board principles.
• It is also a kind of operating system of the
  robot, which supervises other subroutines.

91
Behavior, Dialog and Learning

• (2) Subroutines with logical data base and
  natural language parsing (CHAT).
• This is the logical part of the brain used to
  find connections between places, timings and all
  kind of logical and relational reasonings, such
  as answering questions about Japanese geography.

92
Behavior, Dialog and Learning

• (3) Use of generalization and analogy in dialog
  on many levels.
• Random and intentional linking of spoken
  language, sound effects and facial gestures.
• Use of Constructive Induction approach to help
  generalization, analogy reasoning and
  probabilistic generations in verbal and
  non-verbal dialog, like learning when to smile or
  turn the head off the partner.

93
Behavior, Dialog and Learning

• (4) Model of the robot, model of the user,
  scenario of the situation, history of the dialog,
  all used in the conversation.
• (5) Use of word spotting in speech recognition
  rather than single word or continuous speech
  recognition.
• (6) Continuous speech recognition (Microsoft)
• (7) Avoidance of I do not know, I do not
  understand answers from the robot.
• Our robot will have always something to say, in
  the worst case, over-generalized, with not valid
  analogies or even nonsensical and random.

94

• Constructive Induction

95
What is constructive induction?

• Constructive induction is a logic-based method of
  teaching a robot of new knowledge.
• It can be compared to neural networks.
• Teaching is constructing some structure of a
  logic function
• Decision tree
• Sum of Products
• Decomposed structue

96
(No Transcript)
97
Example Age Recognition
Name (examples) Age (output) d Smile Height Hair Color Hair Color
Joan Kid (0) a(3) b(0) c(0) c(0)
Mike Teenager (1) a(2) b(1) c(1) c(1)
Peter Mid-age (2) a(1) b(2) c(2)  
Frank Old (3) a(0) b(3) c(3) c(3)
Examples of data for learning, four people, given
to the system
98
Example Age Recognition
Smile - a Very often often moderately rarely
Values 3 2 1 0
Height - b Very Tall Tall Middle Short
Values 3 2 1 0
Color - c Grey Black Brown Blonde
Values 3 2 1 0
Encoding of features, values of multiple-valued
variables
99
Multi-valued Map for Data
Groups show a simple induction from the Data
ab\ c 0 1 2 3
00 - - - -
01 - - - 3
02 - - - -
03 - - - -
10 - - - -
11 - - - -
12 - - 2 -
13 - - - -
20 - - - -
21 - 1 - -
22 - - - -
23 - - - -
30 0 - - -
31 - - - -
32 - - - -
33 - - - -
ab\ c 0 1 2 3
00 - - - -
01 - - - 3
02 - - - -
03 - - - -
10 - - - -
11 - - - -
12 - - 2 -
13 - - - -
20 - - - -
21 - 1 - -
22 - - - -
23 - - - -
30 0 - - -
31 - - - -
32 - - - -
33 - - - -
d F( a, b, c )
100
Old people smile rarely
Groups show a simple induction from the Data
blonde hair
Grey hair
ab\ c 0 1 2 3
00 - - - -
01 - - - 3
02 - - - -
03 - - - -
10 - - - -
11 - - - -
12 - - 2 -
13 - - - -
20 - - - -
21 - 1 - -
22 - - - -
23 - - - -
30 0 - - -
31 - - - -
32 - - - -
33 - - - -
Middle-age people smile moderately
Teenagers smile often
Children smile very often
101
Another example teaching movements
Input variables
Output variables
102
Generalization of the Ashenhurst-Curtis
decomposition model
103
This kind of tables known from Rough Sets,
Decision Trees, etc Data Mining
104
Original table
Second variant
First variant of decomposition
At every step many decompositions exist
Decomposition is hierarchical
Which decomposition is better?
105
Constructive Induction Technical Details

• U. Wong and M. Perkowski, A New Approach to
  Robots Imitation of Behaviors by Decomposition
  of Multiple-Valued Relations, Proc. 5th Intern.
  Workshop on Boolean Problems, Freiberg, Germany,
  Sept. 19-20, 2002, pp. 265-270.
• A. Mishchenko, B. Steinbach and M. Perkowski, An
  Algorithm for Bi-Decomposition of Logic
  Functions, Proc. DAC 2001, June 18-22, Las Vegas,
  pp. 103-108.
• A. Mishchenko, B. Steinbach and M. Perkowski,
  Bi-Decomposition of Multi-Valued Relations, Proc.
  10th IWLS, pp. 35-40, Granlibakken, CA, June
  12-15, 2001. IEEE Computer Society and ACM SIGDA.
  

106
Constructive Induction

• Decision Trees, Ashenhurst/Curtis hierarchical
  decomposition and Bi-Decomposition algorithms are
  used in our software
• These methods create our subset of MVSIS system
  developed under Prof. Robert Brayton at
  University of California at Berkeley 2.
• The entire MVSIS system can be also used.
• The system generates robots behaviors (C program
  codes) from examples given by the users.
• This method is used for embedded system design,
  but we use it specifically for robot interaction.

107
Ashenhurst Functional Decomposition
Evaluates the data function and attempts to
decompose into simpler functions.
F(X) H( G(B), A ), X A ? B
X
B - bound set
if A ? B ?, it is disjoint decomposition if A ?
B ? ?, it is non-disjoint decomposition
108
A Standard Map of function z
Explain the concept of generalized dont cares
Bound Set
a b \ c
Columns 0 and 1 and columns 0 and 2 are
compatible column compatibility 2
Free Set
z
109
NEW Decomposition of Multi-Valued Relations
F(X) H( G(B), A ), X A ? B
A
X
Relation
Relation
B
if A ? B ?, it is disjoint decomposition if A ?
B ? ?, it is non-disjoint decomposition
110
Forming a CCG from a K-Map
Columns 0 and 1 and columns 0 and 2 are
compatible column compatibility index 2
Column Compatibility Graph
z
111
Forming a CIG from a K-Map
Columns 1 and 2 are incompatible chromatic number
2
Column Incompatibility Graph
112
Constructive Induction

• A unified internal language is used to describe
  behaviors in which text generation and facial
  gestures are unified.
• This language is for learned behaviors.
• Expressions (programs) in this language are
  either created by humans or induced automatically
  from examples given by trainers.

113
Conclusion. What did we learn

• (1) the more degrees of freedom the better the
  animation realism. Art and interesting behavior
  above certain threshold of complexity.
• (2) synchronization of spoken text and head
  (especially jaw) movements are important but
  difficult. Each robot is very different.
• (3) gestures and speech intonation of the head
  should be slightly exaggerated superrealism,
  not realism.

114
Conclusion. What did we learn(cont)

• (4) Noise of servos
• the sound should be laud to cover noises coming
  from motors and gears and for a better
  theatrical effect.
• noise of servos can be also reduced by
  appropriate animation and synchronization.
• (5) TTS should be enhanced with some new
  sound-generating system. What?
• (6) best available ATR and TTS packages should be
  applied.
• (7) OpenCV from Intel is excellent.
• (8) use puppet theatre experiences. We need
  artists. The weakness of technology can become
  the strength of the art in hands of an artist.

115
Conclusion. What did we learn(cont)

• (9) because of a too slow learning, improved
  parameterized learning methods should be
  developed, but also based on constructive
  induction.
• (10) open question funny versus beautiful.
• (11) either high quality voice recognition from
  headset or low quality in noisy room. YOU CANNOT
  HAVE BOTH WITH CURRENT ATR TOOLS.
• (12) low reliability of the latex skins and this
  entire technology is an issue.

116
We won an award in PDXBOT 2004. We showed our
robots to several audiences
Robot shows are exciting
Our Goal is to build toys for 21-st Century and
in this process, change the way how engineers are
educated.
International Intel Science Talent Competition
and PDXBOT 2004, 2005
117
What to remember?

• Robot as a mapping from inputs to outputs
• Braitenberg Vehicles
• State machines, grammars and probabilistic state
  machines
• Natural language conversation with a robot
• Image processing for a interactive robot.
• Constructive induction for behavior and language
  acquisition.

118
Projects

• Project 1
• Lego NXT . 2 people. Editor for state-machine and
  probabilistic state machine base robot behavior
  of mobile robots with sensors.
• Project 2
• Vision for KHR-1 robot Immitation. 2 people.
  Matthias Sunardi group leader.
• Project 3
• Head design for a humanoid robot

119
Projects

• Project 4
• Leg design for a humanoid robot
• Project 5
• Hand design for a humanoid robot
• Project 6
• EyeSim simulator no robot needed.
• Project 7
• Conversation with a humanoid robot (dialog and
  speech).

120
Projects

• Project 8
• Editor for an animatronic robot theatre
• Project 9
• Quantum-Computer Controlled Robot
• Project 10
• Project 11