Collaborative Time Sensitive Targeting aka Technologies for Team Supervision

1
Collaborative Time Sensitive TargetingakaTechnol
ogies for Team Supervision
March 31, 2008 Missy Cummings Humans
Automation Lab MIT Aeronautics and
Astronautics http//halab.mit.edu
2
Review of Original Proposal

• Project Goal
• Develop technological support for futuristic
  unmanned aerial vehicle (UAV) teams engaged in
  TST operations (but really any time critical
  resource allocation task)

Technology to Support UAV Operators Activity
Awareness Displays Objective Performance Measures
Technology to Support the Team Supervisor Activit
y Awareness Displays Interruption Recovery Tools
3
Teamwork in UAV Operations
Intelligence Consumers (e.g., Ground Troops)
UAV Operators
Operations Center
Ground Crew
4
Generalizable to Other Domains
3 Confederatesas UAV Operators
5
Team Interface Issues in Time-Critical Operations

• Increasing reliance on collaboration tools
• e.g., email, instant messaging (chat)
• Increased communication overhead
• Need interfaces that intelligently share activity
  information to improve planning coordination in
  networked teams
• Activity awareness
• Research for technology aids to assist
  supervisors of teams virtually non-existent
• Measurement of technology interventions very
  difficult
• Objective team performance metrics are elusive

6
Boeing Team Test Facility
3 Confederatesas UAV Operators
7
Simulated Task Environment

• Surveillance support for a ground convoy through
  a hostile region
• UAV team consists of
• 1 Mission Commander
• 3 UAV operators, each controlling multiple UAVs
• UAVs have camera sensors only
• Team must coordinate with external strike team
  to destroy identified threats

Situation Map Display
Mission Status Display
Remote Assistance Display
8
Team Supervision Decision Aiding
UAV Operations Team
Interruption recovery tools for the team
supervisor
9
Interruption Recovery Assistance (IRA) tool

• Interactive event timeline, containing event
  bookmarks
• Increased recovery time but positive impact on
  decision accuracy, especially in complex task
  situations.
• IRA tool tended to provide greater benefits to
  participants without military experience
• Some experimental/interface issues so additional
  study needed
• Summer 2008 w/ Waterloo

10
Current Work Predictive Tool for Team Supervision
UAV Team Supervisor

• Technologies for assessing
• How well is the team doing?
• When who should received attention?

11
Current Work Team Performance Prediction

• The need for more objective team performance
  metrics
• Our focus is supervisory control in team settings
  with embedded layers of automation
• Predicting team states versus actual performance
• Artificial intelligence, behavioral pattern
  detection and performance correlations
• Alert team supervising agent of sub-optimal team
  behavior/cognitive strategies
• Relying on human knowledge-based reasoning to
  determine if an anomalous team state exists
• Two stage approach
• Predict an individual operators state transition
  as a proof of concept
• Adapt model to UAV team setting

12
Team Prediction Model Characteristics

• Evaluation and predictions of team behaviors
• Automatic, continuous, and in real-time
• Only relies on easily observable data
• User interaction with machine
• User communication
• Eye tracking?
• Use Bayesian pattern prediction methods applied
  to individual/team behaviors
• Hidden Markov Model
• Not predicting performance per se
• Anomalous conditions
• Value of human judgment

13
Hidden Markov Models

• Probabilistic state transitions
• Observables vs. Hidden States
• Hidden states are not directly visible, but
  variables influenced by them are.
• Cognitive states vs. behaviors
• HMM uses the observables to infer
• Most likely hidden state sequence
• Based on future sequences, the most likely
  observables
• Machine learns the clustering of behaviors (need
  lots of data!)

Target Monitoring
Modify Flight Parameters
HS Monitoring
Comms
Camera Control
Waypoint Update
Mission Replan
14
Predicting Individual Behavior
15
Human State Estimation - TRACS
16
Building the Model
17
4 State HMM for Mission Planning Task
Leveraging Automation
Manual Planning
Exploration

• Unsupervised vs. supervised model training for
  state recognition

18
Predicting an Operators Strategies
19
Live Demo!
20
Validating our HMM Approach

• Currently only behaviors that entail mouse
  clicks are observed
• A limit, particularly for monitoring interfaces
• Eye tracking arguably gives us additional data
  about the information operators access
• Noisy
• Cost-benefit analysis?
• Study will be conducted this summer comparing
  HMM results with and without eye tracking input

21
Extending our Approach to UV Teams
Sector A
Sector C
Visualization task
a) Replan Path b) Replan Target
Sector B
Evaluation/ Choice of UV engage
Extending to task to teams UV hand-off between
Sectors
Choice of UV
22
The Plan

• Develop HMM using experimental data from
  individual experiments
• Investigate supervised vs. unsupervised learning
• Develop a team version of the interface
• Update HMM with team experiments to be conducted
  this summer

23
Additional Future Work Designing for Team
Activity Awareness
UAV Operations Team
24
Preliminary UAV Operator Display Designs
CommunicationsDisplay
MapDisplay
Tasking DisplayTargetID rerouting,
reassigning UAVs
25
Deliverables

• 6 Conference papers
• ASNE, HSIS, ICCRTS, HFES
• 4 Technical reports
• 3 Undergraduate theses
• 2 pending
• 1 Masters thesis
• 1 PhD dissertation (pending)
• 3 Journal articles in various stages
• 1 workshop (CSCW 2006)
• IP disclosure pending HMM validation
• The Holy Grail of team research???

26
Future Plans

• Near Term
• Eye tracking study
• HMM-Multi UV experiment
• Extension for supervisor IRA tool
• Mid Term
• Activity awareness displays for operators
• Decision support display for HMM
• Multi-UV connection to predictive model system
  architecture research (ONR)
• Development of CE metric
• Project slated to end December 2008
• Requested no-cost extension to JUN 09
• Additional funding

27
Backup
28
Simulated Task Environment Display Detail
Map Display
Mission Status Display
29
Simulated Task Software Architecture
Large-Screen Wall Displays
Situation Map Display
Mission Status Display
Simulation Collaboration Server(Grouplab
SharedDictionary)
TabletPC Display
Mission Commander Display
30
Designing to Promote Activity Awareness

• UAV status tasking
• Current expected convoy safety
• Current expected operator task performance,
  relative to convoy safety

1
(reviewing ATR imagery)
4
(nominal)
Situation Map Display
8
(down)
Mission Status Display
31
Designing to Promote Activity Awareness

• UAV status tasking
• Current expected convoy safety
• Current expected operator task performance,
  relative to convoy safety

Situation Map Display
Potential threat envelope
Target strike indicators
Known threat envelope
Mission Status Display
32
Designing to Promote Activity Awareness

• UAV status tasking
• Current expected convoy safety
• Current expected operator task performance,
  relative to convoy safety

Situation Map Display
Mission Status Display
33
Developing Methodology for Deriving Collaborative
System Requirements

• Project Goal
• To develop techniques to identify dependencies in
  operator decision making to understand how to
  assist coordination of team member tasking