Understanding Human Behavior from Sensor Data

1
Understanding Human Behavior from Sensor Data

• Henry Kautz
• University of Washington

2
Trends
3
Growing Ubiquitous Sensing Infrastructure

• GPS
• Wi-Fi localization
• RFID tags
• Wearable sensors

4
Advances in Artificial Intelligence

• Graphical models
• Particle filtering
• Belief propagation
• Statistical relational learning

5
Crisis in Caring for the Cognitively Disabled

• Epidemic of Alzheimers
• Community integration of 7.5 million citizens
  with MR
• 100,000 _at_ year disabled by TBI
• Post-traumatic stress syndrome
• Caregiver burnout

6
...An Opportunity

• Create methods for modeling and interpreting
  human behavior from sensor data
• In order to develop assistive technologies to
  support independent living by people with
  cognitive disabilities
• Help people perform activities of daily living
• Monitor behavior to prevent health crises

7
Outline

• Learning and reasoning about transportation
  routines
• ACCESS personal guidance system
• Understanding activities of daily living
• CARE monitoring system
• Further directions

8
ACCESS
Assisted Cognition in Community, Employment,
Social Settings
9
Motivation Community Access for the Cognitively
Disabled
10
The Problem

• Using public transit is cognitively challenging
• Learning bus routes and numbers
• Transfers
• Recovering from mistakes
• Point to point shuttle service impractical
• Slow
• Expensive
• Current GPS units hard to use
• Require extensive user input
• Error-prone near buildings, inside buses
• No help with transfers, timing

11
Goal

• A personal guidance system that
• Requires no explicit programming by user or
  caregiver
• Proactively assists user in completing
  transportation plans
• Recognizes user errors, and helps user recover

12
Technical Problem

• Given a data stream from a wearable GPS unit...
• Infer the users location and mode of
  transportation (foot, car, bus, bike, ...)
• Predict the users destination
• Detect user errors

13
GPS Receivers

• GeoStats GPS logger
• Data capacity 3 weeks _at_ 0.5 second frequency
• Battery life 72 hours

• Nokia 6600 Cell Phone
• Java
• Bluetooth GPS unit
• Internet connectivity for off-board processing
• Battery life 8 hours

14
GIS Data

• Street map
• Census 2000
• Bus routes stops
• Seattle Metro
• Business / residential areas
• MapPoint

15
Challenges of GPS Data

• Many dead zones in urban areas
• Sparse measurements inside cars and buses
• Systematic error ? 10 meters
• Multi-path propagation
• Mapping errors

16
Representation

• Map is a directed graph G(V,E)
• Location
• Edge (block)
• Distance along edge
• Actual GPS reading
• Movement
• Mode foot, car, bus, indoors influences
  velocity
• Change edges at intersections
• Change mode at bus stops, parked car, buildings
• Tracking (filtering) Given some prior estimate,
• Update position mode according to motion model
• Correct according to next GPS reading

17
Motion Model for Mode
18
Dynamic Bayesian Network I
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
GPS reading
Time k
Time k-1
19
Rao-Blackwellised Particle Filtering

• Particle filtering
• Evolve approximation to state distribution using
  samples (particles)
• Supports multi-modal distributions and discrete
  variables (mode, edge)
• Rao-Blackwellisation
• Particles include distributions over variables
• Each particle is a Kalman filter (Gaussian along
  edge)
• Improved accuracy with fewer particles

20
Learning to Predict User

• Prior knowledge general constraints on
  transportation use
• Vehicle speed range
• Bus stops
• Learning specialize model to particular user
• 30 days GPS readings of one user, logged every
  second
• Unlabeled data
• Learn edge and mode transition parameters using
  expectation-maximization (EM)

21
Mode Location Tracking
Measurements Projections Bus mode Car mode Foot
mode
Green Red Blue
22
Predictive Accuracy
How to improve the models predictive power?
Probability of correctly predicting the future
5 blocks 50 accuracy
City Blocks
23
Transportation Routines
B
A
Workplace
Home

• Goal intended destination
• Workplace, home, friends, restaurants,
• Trip segments ltstart, end, modegt
• Home to Bus stop A on Foot
• Bus stop A to Bus stop B on Bus
• Bus stop B to workplace on Foot

24
Dynamic Bayesian Net II
gk-1
gk
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
GPS reading
Time k
Time k-1
25
Unsupervised Hierarchical Learning

• Use previous model to infer
• Goals - locations where user stays for
  longperiods of time
• Transition points - locations with high mode
  transition probability
• Trip segments paths connecting transition
  points or goals
• Perform EM learning on the hierarchical model
• Learn transition parameters
• Between goals
• Between trip segments, given the goal
• Between edges modes, given the trip segment

26
High Probability Trip SegmentsConditioned on Goal
Goal Workplace
Goal Home
27
Predicting Goal and Path
Predicted goal Predicted path
28
Improvement in Predictive Accuracy
45 blocks 50 accuracy
29
Detecting User Errors

• Learned model represents typical correct behavior
• Model is a poor fit to user errors
• We can use this fact to detect errors!
• Cognitive Mode
• Normal model functions as before
• Error switch in prior (untrained) parameters for
  mode and edge transition

30
Dynamic Bayesian Net III
Cognitive mode normal, error
Goal
Trip segment
Transportation mode
Edge, velocity, position
Data (edge) association
GPS reading
31
Goal Clamping

• The users goal may be explicitly known
• Ask the user to confirm highest-probability goal
• Appointment calendar
• Incorporating such information clamps the goal
• Distinguishes novelty from errors

32
Detecting User Errors
Untrained Trained
Clamped
33
ACCESS Prototype

• Cell phone with GPS, camera, high-speed internet
  access
• Prompts when it infers that user is

34
ACCESS Prototype

• Cell phone with GPS, camera, high-speed internet
  access
• Prompts when it infers that user is

• About to begin a transportation plan
• Confirm destination?
• Here is yourroute!

35
ACCESS Prototype

• Cell phone with GPS, camera, high-speed internet
  access
• Prompts when it infers that user is

• About to change mode
• This is your bus!
• Your stop is next!

36
ACCESS Prototype

• Cell phone with GPS, camera, high-speed internet
  access
• Prompts when it infers that user is

• Making an error
• You missed your stop!
• Here is how to getback on track

37
ACCESS Prototype

• Cell phone with GPS, camera, high-speed internet
  access
• Prompts when it infers that user is

• Visiting a new destination
• Please take a picture!

38
Status

• Medical partnerships
• Funding by National Institute of Disability
  Rehabilitation Research (NIDRR)
• Partnership with UW Center for Technology and
  Disability Studies
• User caregiver needs studies (TBI MR)
• Data collection by job coaches
• Extension to indoor navigation
• Hospitals, nursing homes, assisted care
  communities
• Wi-Fi localization
• Multi-modal interface
• Speech, graphics, text
• Guidance strategies
• Proactive / Just in time
• Coordinates / Landmarks
• WOZ design study

39
WOZ
40
Papers

• Patterson, Liao, Fox, Kautz, UBICOMP 2003
• Inferring High Level Behavior from Low Level
  Sensors
• Patterson et al, UBICOMP-2004
• Opportunity Knocks a System to Provide Cognitive
  Assistance with Transportation Services
• Liao, Fox, Kautz, AAAI 2004 (Best Paper)
• Learning and Inferring Transportation Routines

41
CARE
Cognitive Assistance in Real-world Environments
42
Goal

• A home monitoring system that
• Assists user in performing activities of daily
  living
• Tracks activities, and provides prompts and
  warnings as needed
• Can be deployed in an ordinary home
• Does not require the user to learn a different
  way to perform the activities the system
  adapts, not the user

43
Short-Term Application

• Accurate, automated ADL logs
• Changes in routine often precursor to illness,
  accidents
• Human monitoring intrusive inaccurate

Image Courtesy Intel Research
44
Technical Requirements

• Sensor hardware that can be practically deployed
  in a ordinary home
• Methods for activity tracking from sensor data
• Methods for automated prompting that consider
• Probability of user errors
• Probability of system errors
• Cost / benefit tradeoffs

45
Object-Based Activity Recognition

• Activities of daily living involve the
  manipulation of many physical objects
• Kitchen stove, pans, dishes,
• Bathroom toothbrush, shampoo, towel,
• Bedroom linen, dresser, clock, clothing,
• We can recognize activities from a time-sequence
  of object touches

46
Sensing Object Manipulation

• RFID Radio-frequency identification tags
• Small
• Semi-passive
• Durable
• Cheap
• Near future use products own tags

47
Wearable RFID Readers

• Designed by Intel Research Seattle
• Will be shared with other Intel partners later
  this year
• 13.56MHz reader, radio, power supply, antenna
• 12 inch range, 12-150 hour lifetime

48
Experiment Morning Activities

• 10 days of data from the morning routine in an
  experimenters home
• 61 tagged objects
• 11 activities
• Often interleaved and interrupted
• Many shared objects

49
Methodology

• Goal simplest model that can robustly track
  activities
• Comparison
• Hidden Markov Model
• Dynamic Bayesian Network with aggregate features
• DBN with aggregation and abstraction smoothing

50
Hidden Markov Model

• Trained on labeled data
• 10-fold cross validation
• 88 accuracy
• 9.4 errors per 20 minute episode

51
Cause of Errors

• Observations were types of objects
• Spoon, plate, fork
• Typical errors confusion between activities
• Using one object repeatedly
• Using different objects of same type
• Critical distinction in many ADLs
• Eating versus setting table
• Dressing versus putting away laundry

52
Aggregate Features

• HMM with individual object observations fails
• No generalization!
• Solution add aggregate variables
• Bit-vector maintains history of objects touched
• Aggregate distribution nodes sum number of
  distinct instances
• Aggregate nodes treated as pseudo-observations
  when an activity transitions
• DBN encoding avoids explosion of HMM

53
DBN with Aggregation

• Average number of errors reduced from 9.4 to 6.5
  (31)
• Deterministic nodes add minimal computational
  overhead

54
Improving Robustness

• Both HMM and DBN fail if novel (but reasonable)
  objects are used
• Solution smooth parameters over abstraction
  hierarchy of object types

55
(No Transcript)
56
Abstraction Smoothing

• Methodology
• Train on 10 days data
• Test where one activity substitutes one object
• Change in error rate
• Without smoothing 26 increase
• With smoothing 1 increase

57
Experiment ADL Form Filling

• Tagged real home with 108 tags
• 14 subjects each performed 12 of 14 ADLs in
  arbitrary order
• Used glove-based reader
• Given trace, recreate activities

58
Results Detecting ADLs
RFID
Inferring ADLs from Interactions with
Objects Philipose, Fishkin, Perkowitz, Patterson,
Hähnel, Fox, and Kautz IEEE Pervasive Computing,
4(3), 2004
59
Summary CARE

• Activities of daily living can be learned and
  robustly tracked using RFID tag data
• Simple, direct sensors can often replace (or
  augment) general machine vision
• Works for essentially all ADLs defined in
  healthcare literature

60
Next Steps
61
Key Next Problems

• Decision-theoretic control of user interfaces
• Prompts helpful or distracting?
• User error, or user model error?
• Context-dependent costs
• Decision-theoretic natural language processing

62
Key Next Steps

• Measures of effectiveness
• ACCESS user studies with potential clients
  (controlled conditions) this spring
• CARE Intel / UW / Wash Dept Social Services
  developing trial for caregiver evaluation
• Goal Long-term deployment in naturalistic
  settings
• Homes, nursing homes, assisted care facilities

63
Future Research

• Physiological sensors
• Heart rate, respiration, temperature
• Heterogeneous sensors
• Environmental wearables machine vision
• Smart homes
• Systems for improving self-awareness
• Emotional self-regulation
• Social pragmatics
• Target populations
• Autism spectrum disorders
• Traumatic brain injury

64
General Architecture
common-sense knowledge
decision making
user profile
physical behavior
userinterface
caregiveralerts
machinelearning
sensors
65
Conclusion Why Now?

• An early goal of AI was to create programs that
  could understand ordinary human experience
• This goal proved elusive
• Missing tools for probabilistic inference
• Systems not grounded in real world
• Lacked compelling purpose
• Today we have the mathematical tools, the
  sensors, and the motivation

66
Other Research
67
Building Social Network Models from Sensor
DataNSF Human Social Dynamics

• multi-modal
• sensor board

Coded Database
codeidentifier
real-time audio feature extraction
audiofeatures
WiFistrength
68
Modal Markov LogicApplied to Dialog
UnderstandingDARPA CALO
ASK_IF(S, H, P) ? B(S, B(H,P) v B(H,P))
69
Planning as SatisfiabilityNSF Intelligent Systems

• Unless PNP, no polynomial time algorithm for SAT
• But great practical progress in recent years
• 1980 100 variable problems
• 2005 100,000 variable problems
• Can we use SAT as a engine for planning?
• 1996 competitive with state of the art
• ICAPS 2004 Planning Competition 1st prize,
  optimal STRIPS planning
• Inspired research on bounded model-checking

70
Efficient Model CountingNSF Intelligent Systems

• SAT can a formula be satisfied?
• SAT how many ways can a formula be satisfied?
• Compact translation discrete Bayesian networks ?
  SAT
• Efficient model counting (Sang, Beame, Kautz
  2004, 2005)
• Formula caching
• Component analysis
• New branching heuristics
• Cachet fastest modelcounting algorithm

71
Credits

• Graduate students
• Don Patterson, Lin Liao, Ashish Sabharwal,
  Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu,
  Bill Pentney, Brian Ferris
• Colleagues
• Dieter Fox, Gaetano Borriello, Dan Weld, Matthai
  Philipose, Tanzeem Choudhury
• Funders
• NIDRR, Intel, NSF, DARPA