Understanding Human Behavior from Sensor Data

1
Understanding Human Behavior from Sensor Data

• Henry Kautz
• University of Washington

2
A Dream of AI

• Systems that can understand ordinary human
  experience
• Work in KR, NLP, vision, IUI, planning
• Plan recognition
• Behavior recognition
• Activity tracking
• Goals
• Intelligent user interfaces
• Step toward true AI

3
Plan Recognition, circa 1985
4
Behavior Recognition, circa 2005
5
Punch Line

• Resurgence of work in behavior understanding,
  fueled by
• Advances in probabilistic inference
• Graphical models
• Scalable inference
• KR U Bayes
• Ubiquitous sensing devices
• RFID, GPS, motes,
• Ground recognition in sensor data
• Healthcare applications
• Aging boomers fastest growing demographic

6
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Kodak Intelligent Systems Research Center
• Projects Plans

7
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Kodak Intelligent Systems Research Center
• Projects Plans

8
Object-Based Activity Recognition

• Activities of daily living involve the
  manipulation of many physical objects
• Kitchen stove, pans, dishes,
• Bathroom toothbrush, shampoo, towel,
• Bedroom linen, dresser, clock, clothing,
• We can recognize activities from a time-sequence
  of object touches

9
Application

• ADL (Activity of Daily Living) monitoring for the
  disabled and/or elderly
• Changes in routine often precursor to illness,
  accidents
• Human monitoring intrusive inaccurate

Image Courtesy Intel Research
10
Sensing Object Manipulation

• RFID Radio-frequency identification tags
• Small
• No batteries
• Durable
• Cheap
• Easy to tag objects
• Near future use products own tags

11
Wearable RFID Readers

• 13.56MHz reader, radio, power supply, antenna
• 12 inch range, 12-150 hour lifetime
• Objects tagged on grasping surfaces

12
Experiment ADL Form Filling

• Tagged real home with 108 tags
• 14 subjects each performed 12 of 65 activities
  (14 ADLs) in arbitrary order
• Used glove-based reader
• Given trace, recreate activities

13
Results Detecting ADLs
Inferring ADLs from Interactions with
Objects Philipose, Fishkin, Perkowitz, Patterson,
Hähnel, Fox, and Kautz IEEE Pervasive Computing,
4(3), 2004
14
Experiment Morning Activities

• 10 days of data from the morning routine in an
  experimenters home
• 61 tagged objects
• 11 activities
• Often interleaved and interrupted
• Many shared objects

15
Baseline Individual Hidden Markov Models
68 accuracy11.8 errors per episode
16
Baseline Single Hidden Markov Model
83 accuracy9.4 errors per episode
17
Cause of Errors

• Observations were types of objects
• Spoon, plate, fork
• Typical errors confusion between activities
• Using one object repeatedly
• Using different objects of same type
• Critical distinction in many ADLs
• Eating versus setting table
• Dressing versus putting away laundry

18
Aggregate Features

• HMM with individual object observations fails
• No generalization!
• Solution add aggregation features
• Number of objects of each type used
• Requires history of current activity performance
• DBN encoding avoids explosion of HMM

19
Dynamic Bayes Net with Aggregate Features
88 accuracy6.5 errors per episode
20
Improving Robustness

• DBN fails if novel objects are used
• Solution smooth parameters over abstraction
  hierarchy of object types

21
(No Transcript)
22
Abstraction Smoothing

• Methodology
• Train on 10 days data
• Test where one activity substitutes one object
• Change in error rate
• Without smoothing 26 increase
• With smoothing 1 increase

23
Summary

• Activities of daily living can be robustly
  tracked using RFID data
• Simple, direct sensors can often replace (or
  augment) general machine vision
• Accurate probabilistic inference requires
  sequencing, aggregation, and abstraction
• Works for essentially all ADLs defined in
  healthcare literature

D. Patterson, H. Kautz, D. Fox, ISWC 2005 best
paper award
24
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Kodak Intelligent Systems Research Center
• Projects Plans

25
Motivation Community Access for the Cognitively
Disabled
26
The Problem

• Using public transit cognitively challenging
• Learning bus routes and numbers
• Transfers
• Recovering from mistakes
• Point to point shuttle service impractical
• Slow
• Expensive
• Current GPS units hard to use
• Require extensive user input
• Error-prone near buildings, inside buses
• No help with transfers, timing

27
Solution Activity Compass

• User carries smart cell phone
• System infers transportation mode
• GPS position, velocity
• Mass transit information
• Over time system learns user model
• Important places
• Common transportation plans
• Mismatches possible mistakes
• System provides proactive help

28
Technical Challenge

• Given a data stream from a GPS unit...
• Infer the users mode of transportation, and
  places where the mode changes
• Foot, car, bus, bike,
• Bus stops, parking lots, enter buildings,
• Learn the users daily pattern of movement
• Predict the users future actions
• Detect user errors

29
Technical Approach

• Map Directed graph
• Location (Edge, Distance)
• Geographic features, e.g. bus stops
• Movement (Mode, Velocity)
• Probability of turning at each intersection
• Probability of changing mode at each location
• Tracking (filtering) Given some prior estimate,
• Update position mode according to motion model
• Correct according to next GPS reading

30
Dynamic Bayesian Network I
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
GPS reading
Time k
Time k-1
31
Mode Location Tracking
Measurements Projections Bus mode Car mode Foot
mode
Green Red Blue
32
Learning

• Prior knowledge general constraints on
  transportation use
• Vehicle speed range
• Bus stops
• Learning specialize model to particular user
• 30 days GPS readings
• Unlabeled data
• Learn edge transition parameters using
  expectation-maximization (EM)

33
Predictive Accuracy
How to improve predictive power?
Probability of correctly predicting the future
City Blocks
34
Transportation Routines
B
A
Workplace
Home

• Goal intended destination
• Workplace, home, friends, restaurants,
• Trip segments ltstart, end, modegt
• Home to Bus stop A on Foot
• Bus stop A to Bus stop B on Bus
• Bus stop B to workplace on Foot

35
Dynamic Bayesian Net II
gk-1
gk
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
GPS reading
Time k
Time k-1
36
Unsupervised Hierarchical Learning

• Use previous model to infer
• Goals
• locations where user stays for a long time
• Transition points
• locations with high mode transition probability
• Trip segments
• paths connecting transition points or goals
• Learn transition probabilities
• Lower levels conditioned on higher levels
• Expectation-Maximization

37
Predict Goal and Path
Predicted goal Predicted path
38
Improvement in Predictive Accuracy
39
Detecting User Errors

• Learned model represents typical correct behavior
• Model is a poor fit to user errors
• We can use this fact to detect errors!
• Cognitive Mode
• Normal model functions as before
• Error switch in prior (untrained) parameters for
  mode and edge transition

40
Dynamic Bayesian Net III
Cognitive mode normal, error
Goal
Trip segment
Transportation mode
Edge, velocity, position
Data (edge) association
GPS reading
41
Detecting User Errors
42
Status

• Major funding by NIDRR
• National Institute of Disability Rehabilitation
  Research
• Partnership with UW Dept of Rehabilitation
  Medicine Center for Technology and Disability
  Studies
• Extension to indoor navigation
• Hospitals, nursing homes, assisted care
  communities
• Wi-Fi localization
• Multi-modal interface studies
• Speech, graphics, text
• Adaptive guidance strategies

43
Papers

• Liu et al, ASSETS 2006
• Indoor Wayfinding Developing a Functional
  Interface for Individuals with Cognitive
  Impairments
• Liao, Fox, Kautz, AAAI 2004 (Best Paper)
• Learning and Inferring Transportation Routines
• Patterson et al, UBICOMP 2004
• Opportunity Knocks a System to Provide Cognitive
  Assistance with Transportation Services

44
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Kodak Intelligent Systems Research Center
• Projects Plans

45
Task

• Learn to label a persons
• Daily activities
• working, visiting friends, traveling,
• Significant places
• work place, friends house, usual bus stop,
• Given
• Training set of labeled examples
• Wearable sensor data stream
• GPS, acceleration, ambient noise level,

46
Learning to Label Places Activities

• Learned general rules for labeling meaningful
  places based on time of day, type of
  neighborhood, speed of movement, places visited
  before and after, etc.

47
Applications

• Activity Compass
• Automatically assign names to places
• Life Capture
• Automated diary

48
Conditional Models

• HMMs and DBNs are generative models
• Describe complete joint probability space
• For labeling tasks, conditional models are often
  simpler and more accurate
• Learn only P( label observations )
• Fewer parameters than corresponding generative
  model

49
Things to be Modeled

• Raw GPS reading (observed)
• Actual user location
• Activities (time dependent)
• Significant places (time independent)
• Soft constraints between all of the above
  (learned)

50
Conditional Random Field

• Undirected graphical model
• Feature functions defined on cliques
• Conditional probability proportional to exp(
  weighted sum of features )
• Weights learned by maximizing (pseudo) likelihood
  of training data

51
Relational Markov Network

• First-order version of conditional random field
• Features defined by feature templates
• All instances of a template have same weight
• Examples
• Time of day an activity occurs
• Place an activity occurs
• Number of places labeled Home
• Distance between adjacent user locations
• Distance between GPS reading nearest street

52
RMN Model
s1
Global soft constraints
p1
p2
Significant places
a1
a2
a3
a4
a5
Activities
g1
g2
g3
g4
g5
g6
g7
g8
g9
GPS, location
time ?
53
Significant Places

• Previous work decoupled identifying significant
  places from rest of inference
• Simple temporal threshold
• Misses places with brief activities
• RMN model integrates
• Identifying significant place
• Labeling significant places
• Labeling activities

54
Efficient Inference

• Some features are expensive to handle by general
  inference algorithms
• E.g. belief propagation, MCMC
• Can dramatically speed up inference by
  associating inference procedures with feature
  templates
• Fast Fourier transform (FFT) to compute
  counting features
• O(n log2n) versus O(2n)

55
Results Labeling

• One user, 1 week training data, 1 week testing
  data
• Number of (new) significant places correctly
  labeled 18 out of 19
• Number of activities correctly labeled 53 out
  of 61
• Number of activities correctly labeled, if
  counting features not used 44 out of 61

56
Results Finding Significant Places
57
Results Efficiency
58
Summary

• We can learn to label a users activities and
  meaningful locations using sensor data state of
  the art relational statistical models
• (Liao, Fox, Kautz, IJCAI 2005, NIPS 2006)
• Many avenues to explore
• Transfer learning
• Finer grained activities
• Structured activities
• Social groups

59
This Talk

• Activity tracking from RFID tag data
• ADL Monitoring
• Learning patterns of transportation use from GPS
  data
• Activity Compass
• Learning to label activities and places
• Life Capture
• Kodak Intelligent Systems Research Center
• Projects Plans

60
Intelligent Systems Research

• Henry Kautz
• Kodak Intelligent Systems Research Center

61
Intelligent Systems Research Center

• New AI / CS center at Kodak Research Labs
• Directed by Henry Kautz, 2006-2007
• Initial group of 4 researchers, expertise in
  machine vision
• Hiring 6 new PhD level researchers in 2007
• Mission providing KRL with access to expertise
  in
• Machine learning
• Automated reasoning planning
• Computer vision computational photography
• Pervasive computing

62
University Partnerships

• Goal access to external expertise technology
• gt 3M _at_ year across KRL
• Unrestricted grants 20 - 50K
• Sponsored research grants 75 - 150K
• CEIS (NYSTAR) eligible
• KEA Fellowships (3 year)
• Summer internships
• Individual consulting agreements
• Master IP Agreement Partners
• University of Rochester
• University of Massachusetts at Amherst
• Cornell
• Many other partners
• UIUC, Columbia, U Washington, MIT,

63
New ISRC Project

• Extract meaning from combination of image
  content and sensor data
• First step
• GPS location data
• 10 hand-tuned visual concept detectors
• People and faces
• Environments, e.g. forest, water,
• 100-way classifier built by machine learning
• High-level activities, e.g. swimming
• Specific objects, e.g. boats
• Learned probabilistic model integrates location
  and image features
• E.g. Locations near water raiseprobability of
  image contains boats

64
Multimedia Gallery/Blog (with automatic
annotation)
Swimming in the Caymans
65
Enhancing Video

• Synthesizing 3-D video from ordinary video
• Surprisingly simple and fast techniques use
  motion information to create compelling 3-D
  experience
• Practical resolution enhancement
• New project for 2007
• Goal convert sub-VGA quality video (cellphone
  video) to high-quality 640x480 resolution video
• Idea sequence of adjacent frames contain
  implicit information about missing pixels

66
Workflow Planning Research Opportunities
Problem Assigning workflow(s) to particular jobs
requires extensive expertise Solution System
learns general job planning preferences from
expert users
Problem Sets of jobs often not scheduled
optimally, wasting time raising costs Solution
Improved scheduling algorithms that consider
interactions between jobs (e.g. ganging)
Problem Users find it hard to define new
workflows correctly (even with good
tools) Solution Create and verify new workflows
by automated planning
67
Planning as Satisfiability

• Unless PNP, no polynomial time algorithm for SAT
• But great practical progress in recent years
• 1980 100 variable problems
• 2005 100,000 variable problems
• Can we use SAT as a engine for planning?
• 1996 competitive with state of the art
• ICAPS 2004 Planning Competition 1st prize,
  optimal STRIPS planning
• Inspired research on bounded model-checking

68
Understanding Clause Learning

• Modern SAT solvers extend Davis-Putnam backtrack
  search with clause learning
• Cache reason for each backtrack as an inferred
  clause
• Works in practice, but not clear why
• Understanding clause learning using proof
  complexity
• (Sabharwal, Beame, Kautz 2003, 2004, 2005)
• More powerful than tree-like resolution
• Separation from regular resolution
• Increase practical benefit bydomain-specific
  variable-selection heuristics
• Linear-size proofs of pebbling graphs

69
Efficient Model Counting

• SAT can a formula be satisfied?
• SAT how many ways can a formula be satisfied?
• Compact translation discrete Bayesian networks ?
  SAT
• Efficient model counting (Sang, Beame, Kautz
  2004, 2005)
• Formula caching
• Component analysis
• New branching heuristics
• Cachet fastest modelcounting algorithm

70
Learning Control Policies for Satisfiability
Solvers

• High variance in runtime distributions of SAT
  solvers
• Over different random seeds for tie-breaking in
  branching heuristics
• Sometimes infinite variance (heavy-tailed)
• Restarts can yield super-linear speedup
• Can learn optimal restart policies using features
  of instance solver trace (Ruan, Kautz,
  Horvitz 2001, 2002, 2003)

71
Other Projects

• Continuous time Bayesian networks
• (Gopalratnam, Weld, Kautz, AAAI 2005)
• Building social network models from sensor data
• Probabilistic speech act theory
• Modal Markov Logic

72
Conclusion Why Now?

• An early goal of AI was to create programs that
  could understand ordinary human experience
• This goal proved elusive
• Missing probabilistic tools
• Systems not grounded in real world
• Lacked compelling purpose
• Today we have the mathematical tools, the
  sensors, and the motivation

73
Credits

• Graduate students
• Don Patterson, Lin Liao, Ashish Sabharwal,
  Yongshao Ruan, Tian Sang, Harlan Hile, Alan Liu,
  Bill Pentney, Brian Ferris
• Colleagues
• Dieter Fox, Gaetano Borriello, Dan Weld, Matthai
  Philipose
• Funders
• NIDRR, Intel, NSF, DARPA