Activity Recognition: Linking Low-level Sensors to High-level Intelligence

1
Activity Recognition Linking Low-level Sensors
to High-level Intelligence

• Qiang Yang
• Hong Kong University of Science and Technology
• http//www.cse.ust.hk/qyang/

2
Whats Happening Outside AI?

• Pervasive Computing
• Sensor Networks
• Health Informatics
• Logistics
• Military/security
• WWW
• Computer Human
• Interaction (CHI)
• GIS

3
Whats Happening Outside AI?
Apple iPhone
Wii
Ekahau WiFi Location Estimation
4
Theme of The Talk

• Activity Recognition
• What it is
• Linking low level sensors to high level
  intelligence
• Activity recognition research Embedded AI
• Empirical in nature
• Research on a very limited budget

5
A Closed Loop
Cooking Preconditions (), Postconditions
(), Duration ()
Eating, Resting, Cooking, Doing Laundry, Meeting,
Using the telephone, Shopping, Playing Games,
Watching TV, Driving
6
Activity Recognition A Knowledge Food Chain

• Action Model Learning
• How to model users actions?
• Activity Recognition
• What is the user doing / will do next?
• Localization Context
• Where is the user?
• Whats around her?

• Knowledge Food Chain
• Output of each level acts as input to an upper
  level in a closed feedback loop

7
Basic Knowing Your Context

• Locations and Context
• Where are you?
• Whats around you?
• Whos around you?
• How long are you there?
• Where were you before?
• Status of objects (door open?)
• What is the temperature like?

8
Knowing Your Context

• Locations and Context
• Where are you?
• Whats around you?
• Whos around you?
• How long are you there?
• Where were you before?
• Status of objects (door open?)
• What is the temperature like?

9
Focusing on locations
Dr. Yin, Jie _at_ work (HKUST)

• Input
• Sensor Readings
• Wifi, RFID, Audio, Visual, Temperature
• Infrared, Ultrasound, magnetic fields
• Power lines Stuntebeck, Patel, Abowd et al.,
  Ubicomp2008
• Localization Models
• Output predicted locations

10
Location-based Applications Indoor

• Healthcare at home and in hospitals
• Logistics Cargo Control
• Shopping, Security
• Digital Wall
• Collaboration with NEC China Lab

11
How to obtain a localization model?

• Propagation-model based
• Modeling the signal attenuation
• Advantages Less data collection effort
• Disadvantages
• Need to know emitter locations
• Uncertainty

• Machine Learning based
• Advantages
• Modeling Uncertainty Better
• Benefit from sequential info
• Disadvantages
• May require a lot of labeled data

RADAR Bahl and Padmanabhan, CCC2000
12
Using both labeled and unlabeled data in subspace
learning

• LeMan Location-estimation
• w/ Manifolds J. J. Pan and Yang et al.,
  AAAI2006
• Manifold assumption similar signals have similar
  labels
• Objective Minimize the loss over labeled data,
  while propagating labels to unlabeled data

13
LeMan J.J. Pan and Yang et al., AAAI2006

• Supervised vs. Semi-Supervised in a 4m x 5m
  testbed
• To achieve the same accuracy under 80cm error
  distance

Supervised Semi-supervised
RADAR LeMan
Percentage of labeled data used 100 23
14
Adding sequences Graphical Model
CRF based localization R. Pan, Zheng, Yang et
al., KDD2007

• Conditional Random Fields Lafferty, McCallum,
  Pereira, ICML2001
• Undirected graph, a generalization to HMM

Not using sequential information Using sequential information Using sequential information
Support vector regression(supervised learning) CRF(supervised learning) SemiCRF(semi-supervised learning)
Accuracy 67.33 83.67 85.67
15
What if the signal data distribution changes?

• Signal may vary over devices, time, spaces
• A -gt B the localization error may increase

Transfer Learning!
16
Our work to address the signal variation problems

• Transfer Learning
• Problem 1 Transfer Across Devices
• Zheng and Yang et al., AAAI2008a
• Problem 2 Transfer Across Time
• Zheng and Yang et al., AAAI2008b
• Problem 3 Transfer Across Spaces
• S. J. Pan and Yang et al., AAAI2008

17
Transferring Localization Models Across Devices
Zheng and Yang et al., AAAI2008a

• Input
• Output
• The localization model on the target device

18
Transferring Localization Models Across Devices
Zheng and Yang et al., AAAI2008a

• Model
• Latent Multi-Task Learning Caruana, MLJ1997
• Each device a learning task
• minimize its localization error, and
• devices share some common constraints
• in a latent space
• Regression with signals x to locations y

shared
19
Transferring Localization Models Across Devices
Zheng and Yang et al., AAAI2008a

• Latent Multi-Task Learning
• Experiments in a 64m x 50m wireless environment

Transfer No transfer
Accuracy under 5m error distance 80 67
20
Transferring Localization Models Over Time Zheng
and Yang et al., AAAI2008b
21
Transferring Localization Models Over Time Zheng
and Yang et al., AAAI2008b

• Model
• Transferred Hidden Markov Model

Reference points (RPs)
Radio map
Transition matrix of user moves
Prior knowledge on the likelihood of where the
user is
Transfer No-transfer
Accuracy under 3m error distance 85 73
22
Transferring Localization Models Across Space S.
J. Pan and Yang et al., AAAI2008

• Input
• Output
• Localization model for Area B

Area B Few labeled data Some unlabeled data
Area A Plentiful labeled data (red dots in the
picture)
23
Transferring Localization Models Across Space S.
J. Pan and Yang et al., AAAI2008

• Transfer Learning Model
• Extended Co-Localization
• Co-Localization J.J. Pan and Yang IJCAI 2007
  Localize both mobile devices and access points
  (APs)
• Video (from www.pancube.com )
• Solved by dimensionality reduction (e.g. SVD)
• Co-Localization over area A and B
• The estimated locations for APs are shared by
  both areas
• Formulated as a quadratically constrained
  quadratic program (QCQP)

Transfer Non-transfer
Accuracy under 3m error distance 70 65
24
Summary Localization using Sensors

• Research Issues
• Optimal Sensor Placement Krause, Guestrin,
  Gupta, Kleinberg, IPSN2006
• Integrated Propagation and learning models
• Sensor Fusion
• Transfer Learning
• Location-based social networks

• Locations
• 2D / 3D Physical Positions
• Locations are a type of context
• Other contextual Information
• Object Context Nearby objects usage status
• Locations and Context
• Where you are
• Whos around you
• How long you are there
• Status of objects (door open?)
• What is the temperature like?

25
Activity Recognition

• Action Model Learning
• How do we explicitly model the users possible
  actions?
• Activity Recognition
• What is the user doing / trying to do?
• Localization and context
• Where is the user?
• Whats around her?
• How long/duration?
• What time/day?

Events
26
Steps in activity recognition
Action Recognition
Loc/Context Recognition
Goal Recognition
sensor
sensor
sensor
sensor

• Also,
• Plan, Behavior, Intent, Project

27
Activity Recognition Input Output

• Input
• Context and locations
• Time, history, current/previous locations,
  duration, speed,
• Object Usage Information
• Trained AR Model
• Training data from calibration
• Calibration Tool VTrack
• Output
• Predicted Activity Labels
• Running?
• Walking?
• Tooth brushing?
• Having lunch?

http//www.cse.ust.hk/vincentz/Vtrack.html
28
Activity Recognition Applications

• GPS based Location-based services
• Inferring Transportation Modes/Routines
• Liao, Fox, Kautz, AAAI2004
• Unsupervised, bridges the gap between raw GPS and
  users mode of transportation
• Can detect when user missed bus stops ? offer
  help
• Healthcare for elders
• Example The Autominder System
• Pollack, et al. Robotics and Autonomous Systems,
  2003.
• Provide users w/ reminders when they need them
• Recognizing Activities with Cell Phones (Video)
• Chinese Academy of Sciences (Prof Yiqiang Chen
  and Dr. Junfa Liu)

29
Microsoft Research Asia GeoLife Project Zheng,
Xie, WWW2008

• Inferring Transportation Modes, and
• Compute similarity based on itineraries and link
  people in a social net GeoLife Video

Segmenti.P(Bike) Segmenti.P(Bike)
P(BikeCar)
Segmenti.P(Walk) Segmenti.P(Walk)
P(WalkCar)
30
Activity Recognition (AR) ADL

• ADL Activities of daily living (ADLs)
• From sound to events, in everyday life
• Lu and Choudhury et al., MobiSys2009
• iCare (NTU) Digital home support, early
  diagnosis of behavior changes
• iCare Project at NTU (Hao-hua Chu, Jane Hsu, et
  al.) http//mll.csie.ntu.edu.tw/icare/index.php
• Duration patterns and inherent hierarchical
  structures
• Duong, Bui et al., AI Journal 2008

31
Early Work Plan Recognition

• Objective Kautz 1987
• Inferring plans of an agent from (partial)
  observations of his actions
• Input
• Observed Actions (K,L)
• Plan Library
• Output
• Recognized Goals/Plans

32
Review Event Hierarchy in Plan Recognition
Abstraction relationship
Actions

• The Cooking Event Hierarchy Kautz 1987
• Some works
• Kautz 1987 graph inference
• Pynadath and Wellman, UAI2000 probabilistic
  CFG
• Geib and Steedman, IJCAI2007 NLP and PR
• Geib, ICAPS2008 string rewriting techniques

Step 2 of Make Pasta Dish
33
A Gap?
34
AR Sequential Methods

• Dynamic Bayesian Networks
• Liao, Fox, Kautz, AAAI2004 Yin, Chai, Yang,
  AAAI2004
• Conditional Random Field Vail and Veloso,
  AAAI2008
• Relational Markov Network Liao, Fox, Kautz,
  NIPS2005

35
Intel Wyatt, Philipose, Choudhury, AAAI2005
Incorporating Commonsense

• Model Commonsense Knowledge
• Work at Intel Seattle Lab / UW
• Calculate Object Usage Information from Web Data
  P(Obj Action)
• Train a customized model
• HMM parameter learning Wyatt et al. AAAI2005
• Mine model from Web Perkowitz, Philipose et al.
  WWW2004

36
Datasets MIT PlaceLab http//architecture.mit.edu
/house_n/placelab.html

• MIT PlaceLab Dataset (PLIA2) Intille et al.
  Pervasive 2005
• Activities Common household activities

37
Datasets Intel Research Lab

• Intel Research Lab Patterson, Fox, Kautz,
  Philipose, ISWC2005
• Activities Performed 11 activities
• Sensors
• RFID Readers Tags
• Length
• 10 mornings

38
Complex Actions? Reduce Labels?

• Complex Actions
• For multiple activities with complex
  relationshipsHu and Yang, AAAI2008
• concurrent and interleaving activities
• Label Reduction
• What if we are short of labeled data in a new
  domain? Zheng, Hu, Yang, et al. Ubicomp 2009
• Use transfer learning to borrow knowledge from a
  source domain (where labeled data are abundant)
• For recognizing activities
• where labeled data are scarce

39
Concurrent and Interleaving Goals Hu, Yang,
AAAI2008
40
Concurrent and Interleaving Goal and Activity
Recognition Hu, Yang, AAAI2008
Use the long-distance dependencies in Skip-Chain
Conditional Random Fields to capture the
relatedness between interleaving activities.
Factors for linear chain edges
41
Concurrent and Interleaving Goal and Activity
Recognition Hu, Yang, AAAI2008

• Concurrent Goals
• correlation matrix between different goals
  learned from training data

Example attending invited talk and browsing
WWW.
Our Approach Only Concurrent Only Interleaving
MIT PlaceLab Dataset 86 73 80
42
Cross Domain Activity Recognition Zheng, Hu,
Yang, Ubicomp 2009

• Challenges
• A new domain of activities without labeled data
• Cross-domain activity recognition
• Transfer some available labeled data from source
  activities to help training the recognizer for
  the target activities.

CleaningIndoor
Laundry
Dishwashing
43
Calculating Activity Similarities

• How similar are two activities?
• Use Web search results
• TFIDF Traditional IR similarity metrics (cosine
  similarity)
• Example
• Mined similarity between the activity sweeping
  and vacuuming, making the bed, gardening

44
How to use the similarities?
Example sim(Make Coffee, Make Tea) 0.6
ltSensor Reading, Activity Namegt Example ltSS,
Make Coffeegt
Similarity Measure
THE WEB
Target Domain Pseudo Labeled Data
Source Domain Labeled Data
Weighted SVM Classifier
45
Cross-Domain AR Performance
Mean Accuracy with Cross Domain Transfer Activities (Source Domain) Activities (Target Domain) Baseline (Random Guess)
MIT Dataset (Cleaning to Laundry) 58.9 13 8 12.5
MIT Dataset (Cleaning to Dishwashing) 53.2 13 7 14.3
Intel Research Lab Dataset 63.2 5 6 16.7

• Activities in the source domain and the target
  domain are generated from ten random trials, mean
  accuracies are reported.

46
How Does AR Impact AI?

• Action Model Learning
• How do we explicitly model the users possible
  actions?
• Activity Recognition
• What is the user doing / trying to do?
• Localization
• Where is the user?

47
Relationship to Localization and AR

• Learning action models
• Motivation
• solve new planning problems
• knowledge-engineering effort
• for Planning
• Can even recognize goals using planning
• Ramirez and Geffner, IJCAI2009

• From context
• ? state description from sensors
• From activity recognition
• ? activity sequences

48
What is action model learning?

• Input activity sequences
• Sequences of labels/objects
• Example pick-up(b1) , stack(b1,b2)etc
• Initial state, goal, and partial intermediate
  states
• Example ontable(b1),clear(b1), etc
• Output Action models
• preconditions of actions
• Example preconditions of pick-up ontable(?x)
  , handempty, etc.
• effects of actions
• Example effects of pick-up holding(?x), etc

• TRAIL Benson, ICML1994 learns Teleo-operator
  models (TOP) with domain experts help.
• EXPO Gil, ICML1994 learns action models
  incrementally by assuming partial action models
  known.
• Probabilistic STRIPS-like models Pasula et al.
  ICAPS2004 learns probabilistic STRIPS-like
  operators from examples.
• SLAF Amir, AAAI2005 learns exact action
  models in partially observable domains.

49
ARMS Yang et al. AIJ2007An overview

• what can be in the preconditions/Postcond

Activity Sequences
Sensor states, object usage
Information constraints
Build constraints
Plan constraints
Solved w/ Weighted MAXSAT/MLN
Each relation has a weight that can be learned
Action models
50
Evaluation by Students _at_ HKUSTexecute learned
actions Lego-Learning-Planning (LLP) System
Design
Notebook
Activity recognition planning
Bluetooth
Robot
PDA
Web Server
Robot Status/ Data
Internet
Control Command
51
A Lego Planning Domain

• Initial state
• (empty ) (face grid0)
• Goal
• (holding Ball)
• Collection of Activity Sequences (Video 1 robot)
  (video 2 human)

• Relations
• given by sensors/phy. map
• (motor_speed )
• (empty )
• (across x-loc y-loc z-loc)
• Actions
• Known to the robot
• (Move_forw x-loc y-loc z-loc)
• (Turn_left x-loc y-loc z-loc)

52
Activity Sequences

• Human manually achieves goal

• 0 (MOVE_FORW A B C)
• 4 (MOVE_FORW D E F)
• 5 (MOVE_FORW E F W)
• 6 (STOP F)
• 7 (PICK_UP F BALL)
• 10 (STOP D)
• 11 (TURN_LEFT D W E)
• 12 (PUT_DOWN BALL D)
• 13 (PICK_UP D BALL)

Activity Recognizer
ARMS Action Model Learning
53
Learned Action Models
Dr. Hankz Hankui Zhuo _at_ HKUST

• (action Stop
• parameters (?x - loc)
• precondition (and
• (motor_speed)
• (is_at ?x) )
• effect (and (empty)
• (face ?x)
• (not (motor_speed)) ))

This is an error
54
LLP Solves a new Lego Planning Problem
F

• (init (empty) (face B) (is_at A) (at ball F)
  (across C D W) (across D E F) (across E F W)
  )
• (goal
• (and (is_at F) (holding Ball)
• )

Goal
Ball
E
A
B
C
D
W
Init
Lego
Ball
F
55
Solving the Planning Problem

• Generate a plan using a planner
• 0 (MOVE_FORW A B C)
• 4 (MOVE_FORW D E F)
• 5 (MOVE_FORW E F W)
• 6 (STOP F)
• 7 (PICK_UP F BALL)
• 10 (STOP D)
• 11 (TURN_LEFT D W E)
• 11 (PUT_DOWN BALL D)
• 12 (PICK_UP D BALL)
• 13 (MOVE_FORW D E F)
• 14 (MOVE_FORW E F W)

• Execution
• sometimes it will
• Succeed!!!?
• But, sometimes it will
• Fail?
• More feedback on learning required

56
Closing the Loop in the Knowledge Food Chain

• Close the feedback loop
• Loop
• Signal traces collected
• Location, Context, Activities predicted
• Action models learned
• New plan generated and executed
• Errors found
• Human intervention to correct plans
• New Plans
• End loop

57
Activity Recognition Truly Multidisciplinary
58
Open issues in Activity Recognition

• User privacy
• Klasjna, Consolvo, Choudhury, et al.,
  Pervasive2009
• False Positives
• Cost-sensitive appl.
• Market study
• Business Models?

• Many users
• mobile social networks
• Multi-person AR (cooperation? Competition?)
• Transfer Learning
• Between users
• Between activities
• Between different types of sensors

59
Conclusions

• Future
• Cheaper and more ubiquitous sensors will bring a
  new era for AI through activity recognition
  research and applications
• Acknowledgement

• Theme of the Talk
• Activity Recognition
• What it is
• Linking low level sensors to high level
  intelligence
• Closed loop
• Activity recognition as Embedded AI
• Empirical in nature
• Research on a very limited budget

60
Theme of The Talk
61
Acknowledgement

• Students (Former) _at_ HKUST
• Vincent W. Zheng, Derek H. Hu, Hankz H. Zhuo,
  Sinno J. Pan
• Jie Yin, Dou Shen, Jeffrey J. Pan, Rong Pan
• Collaborators
• Drs. Junhui Zhao and Yongcai Wang (NEC China Lab)
• Drs. Yiqiang Chen and Junfa Liu (CAS)
• Drs. Xing Xie and Yu Zheng (MSRA)

62
References (Localization)

• Bahl and Padmanabhan, CCC2000 RADAR An
  in-building RF-based user location and tracking
  system.
• Caruna, MLJ1997 Multi-task Learning.
• Ferris, Fox and Lawrence, IJCAI2007 WiFi-slam
  using Gaussian process latent variable models.
• Fox and Hightower et al., PervasiveComputing2003
  Bayesian filtering for location estimation.
• Krause, Guestrin, Gupta, Kleinberg, IPSN2006
  Near-optimal sensor placements maximizing
  information while minimizing communication cost.
• Ladd et al., MobiCom2002 Robotics-based
  Location Sensing using Wireless Ethernet.
• Ni et al., PerCom 2003 LANDMARC indoor
  location sensing using active RFID.
• J. J. Pan and Yang et al., IJCAI2005 Accurate
  and low-cost location estimation using kernels.
• J. J. Pan and Yang et al., AAAI2006 A Manifold
  Regularization Approach to Calibration Reduction
  for Sensor-Network Based Tracking.
• R. Pan, Zheng, Yang et al., KDD2007
  Domain-Constrained Semi-Supervised Mining of
  Tracking Models in Sensor Networks.
• S. J. Pan and Yang et al., AAAI2008
  Transferring Localization Models Across Space.

63
References (Localization)

• Stunteback and Patel et al., Ubicomp2008
  Wideband powerline positioning for indoor
  localization.
• Zheng and Yang et al., AAAI2008a Transferring
  Multi-device Localization Models using Latent
  Multi-task Learning.
• Zheng and Yang et al., AAAI2008b Transferring
  Localization Models Over Time.

64
References (Activity Recognition)

• Bao and Intille, Pervasive 2004 Activity
  Recognition from User-Annotated Acceleration
  Data.
• Bui et al., AAAI 2008 The Hidden Permutation
  Model and Location-Based Activity Recognition.
• Chian and Hsu, IJCAI2009 Probabilistic Models
  for Concurrent Chatting Activity Recognition
• Choudhury and Basu, NIPS 2004 Modeling
  Conversational Dynamics as a Mixed-Memory Markov
  Process.
• Kautz 1987 A Formal Theory of Plan Recognition.
• Geib and Steedman, IJCAI 2007 On Natural
  Language Processing and Plan Recognition.
• Geib et al., ICAPS 2008 A New Probabilistic
  Plan Recognition Algorithm Based on String
  Rewriting.
• Hu, Yang, AAAI2008 CIGAR Concurrent and
  Interleaving Goal and Activity Recognition.
• Klasnja, Consolvo, Choudhury et al.
  Pervasive2009 Exploring Privacy Concerns about
  Personal Sensing.
• Liao, Fox, Kautz, AAAI2004 Learning and
  Inferring Transportation Routines.

65
References (Activity Recognition)

• Liao, Fox, Kautz, NIPS2005 Location-Based
  Activity Recognition.
• Lu and Choudhury et al., MobiSys2009
  SoundSense scalable sound sensing for
  people-centric applications on mobile phones.
• Patterson, Fox, Kautz, Philipose, ISWC 2005
  Fine-Grained Activity Recognition by Aggregating
  Abstract Object Usage.
• Pollack, 2003 Autominder an intelligent
  cognitive orthotic system for people with memory
  impairment.
• Pynadath and Wellman, UAI 2000 Probabilistic
  State-Dependent Grammars for Plan Recognition.
• Vail and Veloso, AAAI 2008 Feature Selection
  for Activity Recognition in Multi-Robot Domains.
• Wyatt, Philipose and Choudhury, AAAI 2005
  Unsupervised Activity Recognition Using
  Automatically Mined Common Sense.
• Yin, Chai, Yang, AAAI2004 High-level Goal
  Recognition in a Wireless LAN.
• Zheng, Hu, Yang, Ubicomp 2009 Cross-Domain
  Activity Recognition.
• Zheng, Xie, WWW 2008 Learning transportation
  mode from raw GPS data for geographic
  applications on the web.

66
References (Action Model Learning)

• Amir, IJCAI2005 Learning Partially Observable
  Deterministic Action Models.
• Benson, ICML1994 Inductive Learning of Reactive
  Action Models.
• Gerevini, AIPS2002 A Planner Based on Local
  Search for Planning Graphs with Action Costs.
• Gil, ICML1994 Learning by Experimentation
  Incremental Refinement of Incomplete Planning
  Domains.
• Pasula et al., ICAPS2004 Learning Probabilistic
  Planning Rules.
• Yang et al., AIJ2007 Learning action models
  from plan examples using weighted MAX-SAT.
• Zhuo et al. PAKDD-09 Transfer Learning Action
  Models by Measuring the Similarity of Different
  Domains.