Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky

1
Dynamic Human-Robot Teams Engaged in Complex
Adversarial Tasks Using Language-Based
Communication

• Faculty Manuela Veloso, Anthony Stentz, Alex
  Rudnicky
• Brett Browning, M. Bernardine Dias
• Students Thomas Harris, Brenna Argall, Gil
  Jones
• Satanjeev Banerjee

Carnegie Mellon School of Computer
Science Boeing Human-Robot Teams Project
2
Project Goals

• Treasure hunt with two or more teams of humans
  and robots competing to locate target objects as
  they explore an unknown space
• Research Goals/Questions
• Specify team members roles and capabilities to
  perform tasks
• Rapidly form ad-hoc heterogeneous teams of robots
  and humans
• Robots and humans executing synchronized action
  as a team, while communicating via speech
• Improve team performance with experience

3
Alignment with Boeings Objectives

• Our project develops component technologies
  relevant to
• FCS Force multiplication for human-robot teams
• NASA Code T Emphasis on robots assisting and
  augmenting humans in complex tasks
• Space activities (Other than Code T)
• Automating launch sites
• Automating escape systems
• Automating maintenance inspections/repair
• Ice inspections
• Aircraft maintenance
• Automating maintenance inspections/repair
• Ice inspections
• Automated baggage handlers

In collaboration with Phillip Koons, Boeing
4
STP Overview

• Skills for low-level control, Tactics for
  single robot, Plays for teamwork

APPLICABLE offense DONE aborted !offense ROLE 1
pass 3 mark best_opponent ROLE 2 block ROLE
3 pos_for_pass R B 1000 0 receive_pass
shoot A ROLE 4 defend_lane
5
STP Implementation

• Play selection from playbook
• Dynamic role assignment
• Active tactic determines transition
• Execution monitoring
• Reward, and adaptation

Selection
Monitoring, Adaptation
Execution
Robot 1 Tactic
Robot 1 Tactic
Robot N Tactic
6
Relevant Research Interests

1. Methods for pickup teams
2. Extend play-based coordination to distributed
   execution
3. Effective human-robot teamwork with pickup teams

7
TraderBots Overview

• Robots are organized as an economy
• Team mission is to maximize production and
  minimize costs
• Robots exchange virtual money for tasks to
  maximize individual profit
• System is designed to align local and global
  profit maximization

8
TraderBots Implementation
9
Relevant Research Interests

1. Human-multi-robot interaction
2. Role assignment for highly heterogeneous teams
3. Improving robustness and adaptivity

10
Speech Overview

• Communication in mobile environments
• Natural spoken language-based interaction
• Supporting high semantic content
• Communication in multi-participant domains
• Human(s) and robots(s) interacting on the same
  channel

11
Speech Implementation

• Sphinx ASR engine
• Speaker-independent recognition
• Speaker-adaptive capability
• Phoenix semantic parser
• Concept extraction
• RavenClaw dialogue engine
• Task-based representations
• Full mixed-initiative dialogue
• Festival/Theta synthesis engine
• Rosetta generation engine
• Portable/Wearable platforms

Comms Channel
. . .
12
Relevant Research Interests

1. Developing conversational algorithms for
   human-robot polylogue
2. Communicating navigational information between
   humans and robots in unstructured environments
3. Cooperative grounding of objects, locations, and
   tasks in novel environments

13
Current Status

• Wireless network-based (UDP) communication
  protocol for human-robot interaction designed and
  implemented
• Closed loop integration between speech-based
  command system, a Segway robot, and a Pioneer
  robot accomplished
• 2-D kinematic simulation tool for testing
  interface to Pioneer robots
• Video demonstration of speech-based command of a
  Segway robot and a Pioneer robot
• Plan for year 1 scenario and relevant
  technology development and integration to
  accomplish this scenario

14
Treasure Hunt Scenario

• First Year Objectives

15
Basic Scenario

• 2 Segways, 2 Pioneers, and 1 human
• Discover and return to base as much treasure as
  possible within a 20 minute period
• Treasure will be identified by landmarks
  currently used by the Segway soccer team

Mapper (M) Seeker (S) Handler (H) Deliverer (D) Leader (L)
Humans X X X
Segways X X X
Pioneers X X X
Gators X X X
AIBOs X
16
Detailed Scenario

• Goal
• To combine TraderBots for negotiation and role
  assignment with plays for synchronization
• Low-level software remains independent
• Challenges
• How do we decide which Play to adopt?
• How do we handle sub-teams?
• How do we generate leaders?

17
Step 1 Form Sub-Teams
Lets form a sub-team
Fixed sub-teams for May 2005
18
Step 2 Command to Search
Team A, search area 1

1. Build map

19
Step 3 Explore and Search

1. Follow

1. Go to search area
2. Build map

20
Step 4 Search

1. Follow and search for treasure

1. Execute search pattern
2. Build map

21
Step 5 Found!
We found it!
We are at ltx,ygt
22
Step 6 Recovery
Human aided loading
23
Step 7 Return

1. Follow
2. Unload

1. Return home
2. Unload

Human aided unloading
24
Post Year 1 Goals

• Failure detection and recovery
• Outdoor environments
• Adversaries
• Larger more diverse teams
• Adaptation and learning