On some decisional issues for humanrobot interaction

1
On some decisional issues for human-robot
interaction

• Rachid Alami
• LAAS-CNRS
• Toulouse - France

Munich August 14, 2007 --- PSSCR07
2
COGNIRON The Cognitive Robot Companion Project 1/
01/2004-31/12/2007 http//www.cogniron.org
Participating labs LAAS, Toulouse EPFL,
Lausanne IPA, Stuttgart KTH, Stockholm U.
Karlsruhe U. Bielefeld U. Hertfordshire U.
Amsterdam VU. Brussels GPS, Stuttgart
Funded by the EU FP6 -IST- FET Beyond Robotics
Program
3
Cogniron Research Areas
RA4 Skill and task learning
RA3 Social behavior and embodied interaction
RA5 Spatial cognition and multimodal situation
awareness
RA2 Detection and Understanding of human activity
RA1 Multimodal dialogue
RA6 Intentionality and initiative
Integration/Experimentation/Evaluation
4
Two recent IST projects

• URUS (Ubiquitous Networking Robotics in Urban
  Settings)
• A fleet of mobile robots in a pedestrian area
  (guides, object transfer, surveillance)
• PHRIENDS
• Design Hardware and Motion Planning and control
  algorithms for safe robot action

5
The Personal Robot Assistant

• the robot should be able to operate in an
  environment which has been essentially designed
  for humans
• the robot will have to perform its tasks in the
  presence of humans and even in interaction with
  them

6
The Personal Robot Assistant

• Capable to operate in an environment which has
  been essentially designed for humans
• - sensors, effectors
• - algorithms
• - architecture for autonomy

7
The Personal Robot Assistant

• the robot will have to perform its tasks in the
  presence of humans and even in interaction with
  them
• - interfaces for communication (hardware,
  software)
• - architectural and decisional issues

8
Human Robot Interaction (HRI)

• Task-Oriented
• How to perform a task, in presence or in
  interaction with humans, in the best possible way
• Efficiency, Safety, Acceptability, Legibility
• Target assisting humans

9
Human Robotic teamwork
10
MDRS Full Scale Field Tests
Sierhuis, M., J. M. Bradshaw, et al. (2003).
Human-agent teamwork and adjustable autonomy in
practice. Proceedings of the Seventh
International Symposium on Artificial
Intelligence, Robotics and Automation in Space
(i-SAIRAS), Nara, Japan.
11
Human-Robot Exploration

• Interactive
• Dialogue intensive
• Robots need to keep up with humans

• EVA Assistant Tasks
• Robots carrying equipment (mules)
• Robots in advance of humans (scouts)
• Robots maintaining services (data connectivity)

12
Robonaut
13
Concerning machines in close interaction with
humans
We have a precursor
14
(No Transcript)
15
An added parameter the human

• Besides design choices, it is necessary to endow
  the robot with the ability to take explicitly
  into account the presence of humans
• Re-visit the robot decisional capabilities

16
Robot Decisional Issues in HRI
17
Robot Decisional Issues

• Explicit reasoning on its abilities (reflexivity)
• Explicit reasoning on its interactors
• Explicit reasoning on interactive task
  performance

18
Human Robot Decisional Interaction (HRI)

• Task-Oriented
• How to perform a task, in presence or in
  interaction with humans, in the best possible way
• Efficiency, Safety, Acceptability, Intentionality
• Planning and On-Line Deliberation may help
• Anticipation, Reasoning
• In the search of
• (Workable) Models
• And (efficient) algorithms

19
HRI management

• Inspired from the Joint Intention theory
  (Cohen-Levesque), Teamwork (Tambe), Joint
  Activities (Clark)
• Refined into Joint Human-Robot goals
• Human robot interaction for task achievement
• Human and robot share the space
• Human perceived by the robot
• Robot perceived by the human

20
Human Robotic teamwork

• ( A set of slides borrowed from Jeff Bradshaw )

21
Cohen and LevesqueJoint Intentions

• Basic concepts
• Agents form teams by adopting joint persistent
  goals (JPGs) to achieve a team action
• JPGs hold if and only if all team members
  mutually believe
• the goal is not yet achieved
• they want the goal to be achieved
• until the goal is known to be achieved,
  unachievable, or no longer relevant, they should
  persist in holding the goal
• If a team member discovers the goal to be
  achieved, unachievable, or no longer relevant, it
  will tell its teammates
• Assumptions
• Teamwork involves more than simple coordination
• Teamwork knowledge should be explicitly modeled
  as a separate domain

Cohen, P. R. and H. J. Levesque (1991). Teamwork,
Menlo Park, CA SRI International.
22
From Joint Intentions to Joint Activity

• Importance of teamwork concepts
• Widely-accepted metaphor for agent-agent
  interaction
• Strong logic-based theoretical foundations
• High reusability of generic teamwork models
• Many approaches and applications
• New directions in teamwork
• Shift from agent-agent to human-agent interaction
• Consideration of a broader set of concerns
• Address choreography of joint activity
• Previous theoretical work incomplete
• Need to incorporate theory from linguistics and
  social sciences
• Need for study in practice

23
Aspects of Joint Activity
Klein, G., Feltovich, P., Bradshaw, J. M.,
Woods, D. D. (2005). Common ground and
coordination in joint activity. Organizational
Simulation. W. B. Rouse and K. R. Boff. New York
City, NY, John Wiley.
24
A. Criteria for Joint Activity

• Generalization of Herbert Clarks work in
  linguistics
• Extended set of behaviors that are carried out by
  an ensemble of actors who are coordinating with
  each other
• must intend to produce something that is a
  genuine joint product
• parties enter into a Basic Compact - an
  agreement that all parties will support the
  process of coordination
• a process, extended in space and time
• structure of embedded sets of actions
• synchronizing their entry and exit points is a
  major challenge to coordination

25
A. Criteria for Joint Activity The Basic Compact

• Highly coordinated joint activity assumes a
  basic compact, which is an agreement (often
  tacit) to facilitate coordination and prevent its
  breakdown.
• Includes a commitment to some degree of aligning
  multiple goals.
• All parties are expected to bear their portion
  of the responsibility to establish and sustain
  common ground and to repair it as needed.

26
B. Requirements for Joint Activity

• Interpredictability
• Need to be able to accurately predict what others
  will do
• Skilled teams - shared knowledge and
  idiosyncratic coordination devices developed by
  experience in working together
• Bureaucracies compensate for experience by
  substituting explicit predesigned structured
  procedures and expectations
• Common ground
• Pertinent mutual knowledge, beliefs, and
  assumptions about others skills and capabilities
• Directability
• Capacity for modifying the actions of the other
  parties as conditions and priorities change
• Responsiveness of each participant to the
  influence of the others

27
C. Choreography of Joint Activity

• Joint activity is guided by signaling and
  coordination devices
• Two levels of coordination
• in the more local joint acts
• of the phases of the overall joint activity

28
Ten Challenges

• 1. Forming and maintaining the Basic Contract
• 2. Forming and maintaining adequate models of
  others intentions and actions
• 3. Maintaining predictability without hobbling
  adaptivity
• 4. Maintaining adequate directability
• 5. Effective signaling of pertinent aspects of
  status and intentions

Klein, G., Woods, D. D., Bradshaw, J. M.,
Hoffman, R. R., Feltovich, P. (2004). "Ten
challenges for making automation a "team player"
in joint human-agent activity." IEEE Intelligent
Systems 19(6) 91-95.
29
Ten Challenges (continued)

• 6. Observing and interpreting signals of status
  and intentions
• 7. Engagement in goal negotiation
• 8. Autonomy and planning technologies that are
  incremental and collaborative
• 9. Attention management
• 10. Controlling the costs of coordinated activity

Klein, G., Woods, D. D., Bradshaw, J. M.,
Hoffman, R. R., Feltovich, P. (2004). "Ten
challenges for making automation a "team player"
in joint human-agent activity." IEEE Intelligent
Systems 19(6) 91-95.
30
(No Transcript)
31
Consequences - design issues

• On the robot control architecture
• On its decisional abilities

32
HRI Robot Decisional Abilities

• Planners and Interaction Schemes that will allow
  the robot
• to elaborate plans
• and to perform its tasks
• While taking into account explicitly the
  constraints imposed by
• the presence of humans,
• their needs
• and preferences.

33
A Cognitive Architecture
34
HRI Decisional Framework

• Detect Humans
• Instantiate IAAs
• Task-Oriented Interaction with IAAs
• A complete process of
• establishing a common goal,
• achieving it (in coordination)
• verifying and reacting to the commitment level
  of the human partner

the IAA (InterAction Agent) represents the human
state, abilities and preferences.
35
Task Supervision

• Task hierarchical dynamic structure due to
  incremental task refinement
• Individual or Joint Task
• Task plans (plan library ..)
• From Root to basic (non decomposable) activities
• Task state
• Agent Commitment to the task Committed,
  Uncommitted (level).
• Agent Belief concerning the task Unachieved,
  Achieved, Irrelevant, Impossible.
• A set of mechanisms to create / stop / retreive
  tasks from the task-structure
• A set of Conditions to be monitored Achieved,
  Impossible, Irrelevant

36
Task Supervision Incremental context-dependent
task refinement
Events Achieved Impossible Irrelevant
Propagation of events Create Retrieve (stop..)
Continual planning
37
Observation Dialogue
Explicit representation of the overall decisional
process and its link to the human Provides a
sound background to cooperation with multi-modal
dialog
Communicate (Goals / Facts /  plans ) Observe
Acticity Infer Intentions / Commitments
Produce legible / acceptable behabiour Perform
useful tasks
38
Supervision of H/R task achievement
Rackham at a museum  Cité de lEspace 
Searches for interaction when left
alone Establishes a common task Programming a H/R
task involving several perception and interaction
modalities Abandons mission if guided person
stops following
39
A museum guide
40
SHARY

• Builds an artificial language for task
  realization in an HRI context
• a set of communicative acts based on joint
  activity and oriented toward establishing common
  beliefs about the task and supporting its
  execution
• Inserts this language in a task refinement
  mechanism

41
Task refinement process
42
Communicative acts

• ask-task (proposing a task)
• suspend-task/resume-task
• give-up (unable to perform)
• cancel (not relevant)
• task-done
• realize-task (announce that task is starting)
• not implemented yet propose-plan (or a
  recipe), propose-plan-modification
• Depending on
• the context, on the person, on the task
• Can be implicit / explicit
• Speech .. Prompting Motion .. Start-Execution ..

43
(No Transcript)
44
(No Transcript)
45
Predictability, Common Ground, Responsiveness
TASKS
EVENTS
Jido says  Please, take it ! 
realize-task act Give
GetObject
realize-task act give-up/cancel/end act suspend
act
expected answers
Give
a) Thierry begins to take the bottle.
realize-task act Give
give-up/cancel/end act suspend act
expected answers
b) Thierry does not look at the robot (but is
still next to him)
suspend-task act_clear Give
c) Thierry does not take the bottle
give-up act not-clear Give
give-up act not-clear Give
give-up act clear Give
d) Thierry leaves.
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
Communication Scheme for Task achievement
50
(No Transcript)
51
Handing an object - TVB
52
Where is Thierry ?
53
Thierry does not take the bottle
54
 Disturbed  attention
55
Consequences

• On robot goals management
• On planning
• high-level (symbolic)
• and geometric level
• On task achievement
• monitoring and adapting to human commitment
• On interplay between Dialog and Decision

56
Task Planning

• From the robot point of view

57
Planning why?

• To assess the feasibility of the task (at a
  certain level) before performing it
• To share the load between the robot and the
  human
• To explain/illustrate a possible course of
  actions

58
Planning

• For the robot and for the human

Plan for the robot
Anticipation of Human activities
59
Planning what ?

• What actions ?
• Where they will take place ?
• How they will be performed and by whom?

60
Building a  good  plan

• Managing Joint task achievement
• Legibility of robot actions and intentions
  (intentionality)
• Acceptability of robot actions
• Compliance with conventions
• Coherent attitudes and behaviours
• Constraints on robot plans

61
Motion and Manipulation in Presence of Humans
62
Planning coordinated/cooperative motion
63
Adaptive anti-collision - Safety
64
Reasoning on human motion and sensing abilities

• Accessibility
• Vision field
• Shared motion

65
Motion Planning

• Classical Motion Planning methods do not take
  into account specifically the presence of humans
  obstacle free paths, coordination for
  non-collision or dead-lock avoidance

• Need to generate robot motion that is
  acceptable, legible and compliant with social
  rules

66
Parameters deduced from user trials
User trials performed at Univesity of
Hertfordshire
67
Visibility Criterion

• The robot must be as visible as possible to the
  human
• Invisible zones to the human have high costs.
• We can interpret it as the effort the human has
  to make to see the robot.
• Currently the cost of each cell is proportional
  to the angle made with the human's looking
  direction

68
Hidden Zones (an addition to the visibility)

• The robot must avoid hiding behind the obstacles.
• Higher costs behind an obstacle.
• The costs are inverse proportional to the
  distance
• The cost decreases if the robots appears
  sufficiently far from the human (surprise factor)
  and at a acceptable speed.

69
Real-time cost evaluation
70
An example
71
Classical motion planning

• No consideration of social distances to humans
• No consideration of humans field of view

72
(No Transcript)
73
Incremental path adaptation
74
First implemention of NHP
Standard obstacle avoidance
Human Aware Navigation
75
Crossing
Avoiding to loom too close
76
Manipulation

• In the close proximity of the human,
• the robot must not cause fear or surprise
• the motion of the robot must be predictable
• the robot must respect the humans preference
  zones
• Not only the robot motion and the speed but also
  robot postures have to be adapted to human needs
  and preferences

77
 Double-Grasp  for handing objects
78
Smooth motion
79
How to hand an object to a person?
Kinematic reachability Field of sight Trajectory
and Motion dynamics
80
How to hand an object to a person? (not yet
implemented)
Undesirable Placements /Motions
acceptable placements
81
(No Transcript)
82
(No Transcript)
83
One key robot capability reasoning about
placements and perspectives

• Relative Placement and Motion with respect to
  humans and objects in an environment
• Reasoning on the human (and the robot) perception
  and manipulation abilities
• In order to answer a number of questions such as
• Can the human see that object ? Can the human see
  the a given part of the robot ? (perspective)
• Can human reach an object (grasp)
• Where to place the robot in order to be able to
  see simultaneously an object, the hand and the
  face of a human partner (home tour, object
  handing)

84
Perspective Placement

• Robot (sensor) placement that satisfies
• task feasibility,
• sensor placement for task monitoring (servoing),
• visibility by the person.

Robot moves to see the pointed object
Pointed object not visible from the current Robot
configuation
85
Perspective planning
86
Choosing where to put an object
Good place to put an object
Inconvenient place to put an object
87
Building a  good  plan

• Managing Joint task achievement
• Legibility of robot actions and intentions
  (intentionality)
• Acceptability of robot actions
• Compliance with conventions
• Coherent attitudes and behaviours
• Constraints on robot plans

88
Task Planning

• The robot and the human are modelled as actors
  able to perfom actions
• Selection is performed
• Context dependent estimation of actions costs /
  Utilities (human preferences)
• Undesired states
• Undesired sequences of actions
• Social conventions
• Symbolic plan is limited to a short sequence.
• One key issue Interference between the main task
  and complementary monitoring actions

89
Humare Aware Task Planning (HATP)

• Providing plan related services required by robot
  supervision
• Plan Operations insertion / iterative refinement
  / restarting with partial plans and/or
  constraints
• Explicit management of two streams
• Link with geometric issues
• perspective taking
• providing context for socially acceptable motion
• Producing legible, acceptable behaviour

90
HATP features

• HTN formalism.
• Takes as input a partially decomposed task tree.
• An action is associated to one or more agent(s).
• A high-level task can have several decompositions

91
HATP plan construction

• A plan tree projection
• HTN (Hierarchical task Network)
• temporal plan projection on Directed Acyclinc
  Graph managed by IxTeT Library
• Maximising plan utility to help assist human /
  minimize human effort
• Agent abilities and preferences costs associated
  to each action he can perform.
• Social rules patterns to detect in the plan
  structure at different levels
• Undesired states
• Undesired sequences of actions
• Social conventions
• Maintaining the abstraction of the plan.
• Hierarchy of individual and common action
• for monitoring and plan presentation and
  negotiation

92
HATP results start situation

• Bob is on the sofa, Robot is at the door.
• Bob wants to drink something on his sofa. He
  needs a glass and the bottle.
• The glass is in the closed cupboard.
• The bottle is on the table.
• Bob must be at the sofa at the end of the plan.

93
Full HATP plan
GetObject(Bob,glass)
GetObject(Bob,bottle)
ReachFurniture(Bob, sofa)
GetObjectFromFurniture (Bob,glass,cupboard)
GetObjectFrom Furniture (robot,bottle,table)
TransmitObject (robot,bob,bottle)
PickupIn (Bob, glass, cupboard)
ReachPlace(Bob,sofa_place)
SomeoneOpen(cupboard)
PickupOn (robot, bottle, table)
ReachFurniture(Bob, sofa)
Reach Furniture (robot, table)
ReachAgent (robot, Bob)
SomeoneClose (cupboard)
Give (robot, Bob, bottle)
Move(Bob, cupboard_place, sofa_place)
ReachPlace (robot, table_place)
ReachPlace(Bob, sofa_place)
Close(Bob, cupboard)
Open(Bob, cupboard)
Move(robot, table_place, cupboard_place)
Move(Bob, sofa_place cupboard_place)
Move(robot, door table_place)
Task to decompose
Bob action
Robot action
Bob stream
Common action
Robot stream
Causal link
Hierarchical link
94
HATP

• Partially implemented / Not yet integrated
• Jido holds a bottle.
• Jido knows that Thierry wants the bottle
  (Joint Goal High level Common Task)
• Jido might decide to try to find Thierry or can
  wait for Thierry and then give the bottle to him.

95
GetObject(Thierry, bottle)
TransmitObject(Jido, Thierry, bottle)
WaitToGive(Jido, Thierry, bottle)
Give(Jido, Thierry, bottle)
WaitForAGivenperson (Jido, Thierry)
ReachAgent (Thierry, Jido)
ReachPlace (Thierry, doorPlace)
Move(Thierry, sofaPlace, doorPlace)
WaitForSomebody(Jido)
CheckID(Jido, Thierry)
96
Asymov a hybrid task planner

• Asymov is a hybrid task planner that is able to
  deal with symbolic as well as geometric
  constraints
• Robot models (actions, kinematics )
• Reasoning on motion and manipulation
• Extension Adding constraints on human-robot
  tasks (approach, coordination, eye contact)

97
(Ha)SyMov Planner Architecture
(Ha)SyMov
Task Library
Task refinement Library
(Ha)SyMov Library
HRI constraints and preferences
Executable Plans
HA Motion Plannning Library
98
A very first example

• Actions
• Human / Robot
• Pick, Place, Move, Eye-contact
• Robot
• Approach
• Attract attention
• Ask for authorization
• Human
• Accept / Refuse
• Geometric constraints
• Pick, Place,
• Eye contact
• Dialog constraints

99
Applicability / Validity ?

• The design choices and the results presented here
  are still preliminary.
• General scheme might be difficult to implement in
  a general sense
• We believe that it is a  reasonable 
  (motivating, fruitful) challenge to implement it
  in the case of a personal robot assistant
  essentially devoted to
• fetch-and-carry
• interactive manipulation tasks
• home tour
• associated activities.

100
Thank you