Spatio-Temporal and Context Reasoning in Smart Homes

1
Spatio-Temporal and Context Reasoningin Smart
Homes

• Sook-Ling (Linda) Chua
• Stephen Marsland, Hans W. Guesgen

COSIT 2009
School of Engineering and Advanced
Technology Massey University, New Zealand
2
T
h e S i t u a t i o n . . .
?
?
?
?
The world is aging
- have we noticed?
Source United Nations (2007)
3
T
h e S i t u a t i o n . . .
?
?
?
?

• The populations of the world are aging

Source United Nations (2007)
4
T
h e S i t u a t i o n . . .
?
?
?
?

• The populations of the world are aging

Source United Nations (2007)
5
T
h e S i t u a t i o n . . .
?
?
?
?

• The populations of the world are aging

gt 25
lt 10
Source United Nations (2007)
6
T
h e S i t u a t i o n . . .
?
?
?
?

• People choose to stay in their own homes
• as long as possible and remain independent

7
T
h e S i t u a t i o n . . .
?
?
?
?

• People choose to stay in their own homes
• as long as possible and remain independent

Aging
leads to
Physical disability
Cognitive impairment

• diminished sense and touch
• slower ability to react
• poor vision, hearing problems
• memory problems

8
T
h e S i t u a t i o n . . .
?
?
?
?

• Supporting inhabitants daily activities

• Smart Homes

Figure extracted from http//www.dreamhomesmagazi
ne.com/
9
T
h e S i t u a t i o n . . .
?
?
?
?

• To react intelligently, the smart home needs to

(1) recognise inhabitants behaviour
(2) perform reasoning

• spatio-temporal information

• contextual information

10
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
sensor output
The Smart Home
Figure extracted from http//www.dreamhomesmagazi
ne.com/
11
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
The direct representation of current sensor
states being triggered
E.g. of a sequence of tokens from the sensors
Date Activation Time Activation Room Object Type Sensor State
16/6/2008 180523 Living room Television Off
16/6/2008 180819 Living room Curtain Closed
16/6/2008 180948 Kitchen Light On
16/6/2008 181035 Kitchen Cupboard Open
16/6/2008 182506 Kitchen Fridge Open
16/6/2008 190002 Laundry Washing Machine On
12
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
The direct representation of current sensor
states being triggered
E.g. of a sequence of tokens from the sensors
Date Activation Time Activation Room Object Type Sensor State
16/6/2008 180523 Living room Television Off
16/6/2008 180819 Living room Curtain Closed
16/6/2008 180948 Kitchen Light On
16/6/2008 181035 Kitchen Cupboard Open
16/6/2008 182506 Kitchen Fridge Open
16/6/2008 190002 Laundry Washing Machine On
13
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Office/Study
Bedroom
Laundry
Dining/Living Room
Kitchen
Figure extracted from The Aware Home, 2002
14
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Office/Study
Bedroom
Laundry
Dining/Living Room
Kitchen
Figure extracted from The Aware Home, 2002
15
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Office/Study
Bedroom
Q How do we recognise behaviours?
Laundry
Dining/Living Room
Kitchen
Figure extracted from The Aware Home, 2002
16
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(a) Exact activities are not directly observed,
only the sensor observations
19th Jan 2009 180316
. . .
19th Jan 2009 180756
19th Jan 2009 182027
19th Jan 2009 183344
19th Jan 2009 185012
19th Jan 2009 190108
19th Jan 2009 193721
. . .
. . .
Figure extracted from The Aware Home, 2002
17
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(a) Exact activities are not directly observed,
only the sensor observations
19th Jan 2009 180316
. . .
19th Jan 2009 180756
19th Jan 2009 182027
19th Jan 2009 183344
19th Jan 2009 185012
19th Jan 2009 190108
19th Jan 2009 193721
. . .
. . .
Figure extracted from The Aware Home, 2002
18
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(a) Exact activities are not directly observed,
only the sensor observations
19th Jan 2009 180316
. . .
19th Jan 2009 180756
19th Jan 2009 182027
19th Jan 2009 183344
19th Jan 2009 185012
19th Jan 2009 190108
19th Jan 2009 193721
. . .
. . .
Figure extracted from The Aware Home, 2002
19
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(b) Same sensor activations will be involved in
multiple behaviours
20
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(b) Same sensor activations will be involved in
multiple behaviours
Figure extracted from www.rebecca-waring.com,
www.cyh.com, www.chow.com
21
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(c) No. of observations can vary between
activities
Making breakfast
Making dinner
Microwave Oven
Fridge
Toaster
Stove
Cupboard
Cupboard
Drawer
Tap
22
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(d) Behaviours are rarely identical on each use

• components can be present/absent
• the order of individual components happen can
  change
• length of time each piece takes can change

E.g. Making a cup of tea
With / without
Milk / water first?
How long?
23
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Challenges

(d) Behaviours are rarely identical on each use

• components can be present/absent
• the order of individual components happen can
  change
• length of time each piece takes can change

Stochastic Approach
E.g. Making a cup of tea
With / without
Milk / water first?
How long?
24
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
The Hidden Markov Model (HMM)

• probabilistic graphical model

• uses probability distributions to determine the
  states for a sequence of observations over
  time

Source Rabiner, L. (1989)
25
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
The Hidden Markov Model (HMM)

• probabilistic graphical model

• uses probability distributions to determine the
  states for a sequence of observations over
  time

Observations
We know this..
Source Rabiner, L. (1989)
26
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
The Hidden Markov Model (HMM)

• probabilistic graphical model

• uses probability distributions to determine the
  states for a sequence of observations over
  time

Observations
We know this..
Source Rabiner, L. (1989)
27
B
e h a v i o u r R e c o g n i t i o n

?
?
?
?
The Hidden Markov Model (HMM)


States
Observations

• Markov property

• The probability of transition to a state (St1)
  depends only
• on the current state (St) represented by
  solid line
• The observation at Ot depends only on the state
  St
• at that time slice represented by dashed
  line

Source Rabiner, L. (1989)
28
B
e h a v i o u r R e c o g n i t i o n

?
?
?
?
The Hidden Markov Model (HMM)


States
Observations

• Markov property

• The probability of transition to a state (St1)
  depends only
• on the current state (St) represented by
  solid line
• The observation at Ot depends only on the state
  St
• at that time slice represented by dashed
  line

Source Rabiner, L. (1989)
29
B
e h a v i o u r R e c o g n i t i o n

?
?
?
?
The Hidden Markov Model (HMM)


States
Observations

• Markov property

• The probability of transition to a state (St1)
  depends only
• on the current state (St) represented by
  solid line
• The observation at Ot depends only on the state
  St
• at that time slice represented by dashed
  line

Source Rabiner, L. (1989)
30
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
19th Jan 2009 180316
19th Jan 2009 180756
19th Jan 2009 182027
19th Jan 2009 183344
19th Jan 2009 185012
19th Jan 2009 190108
. . .
. . .
31
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
19th Jan 2009 180316
19th Jan 2009 180756
19th Jan 2009 182027
19th Jan 2009 183344
19th Jan 2009 185012
19th Jan 2009 190108
. . .
. . .
32
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
19th Jan 2009 180316
19th Jan 2009 180756
19th Jan 2009 182027
19th Jan 2009 183344
19th Jan 2009 185012
19th Jan 2009 190108
. . .
. . .
33
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• To use HMM to recognise behaviours

(1) Segmentation ? break the token sequence into
appropriate pieces that represent
individual behaviours
start
start
end
end
Observations
. . .
34
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• To use HMM to recognise behaviours

(2) Classification ? identify the behaviours
using the HMM
. . .
Observations
Behaviour A
Behaviour B
35
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Behaviour Recognition using HMM

• Our approach

• Use a set of HMMs that each recognise different
  behaviours

. . .
Making lunch
Making coffee
Showering

• These HMMs will compete to explain the current
  observations
• Model selection is based on maximum likelihood

Source Chua, Marsland and Guesgen (2009)
36
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Experiment Competition between HMMs

• Datasets

• MIT PlaceLab
• Designed a set of simply installed state-change
  sensors that were placed in two different
  apartments with real people living in them

Source Tapia (2004)
37
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Experiment Competition between HMMs

• Datasets

• The subjects kept a record of their activities
  that form a set of annotations for the data

? Ground-truth segmentation of the dataset

• We used the dataset from the first subject

• 77 sensors
• collected for 16 consecutive days

38
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Experiment Competition between HMMs

• Datasets

• Activities take place in one room (kitchen)
• Location of the sensors is known a priori
• Behaviours

• Prepare breakfast (toaster)
• Prepare breakfast (cereal)
• Prepare beverage
• Prepare lunch
• Do the laundry

39
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Based on 727 observations (using 11 days testing
and 5 days training set)
40
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Based on 727 observations (using 11 days testing
and 5 days training set)
41
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Based on 727 observations (using 11 days testing
and 5 days training set)
42
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Experimental Results

• Method works effectively

• performs segmentation and detects changes of
  activities

. . .
observation
Coffee Machine
Drawer
Microwave
Fridge
Drawer
Fridge
Cupboard
43
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?
Experimental Results

• Method works effectively

• performs segmentation and detects changes of
  activities

Preparing lunch
Preparing a beverage
. . .
observation
Coffee Machine
Drawer
Microwave
Fridge
Drawer
Fridge
Cupboard
44
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Discussion

Lack of spatio-temporal information

• Misclassification

? The end of one behaviour contains observations
that should be the start of the next
Preparing lunch
Preparing a beverage

Drawer
Microwave
Cupboard
Fridge
Fridge
Coffee Machine
Drawer
observation
45
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Discussion

Lack of spatio-temporal information

• Misclassification

? The end of one behaviour contains observations
that should be the start of the next
Preparing lunch
Preparing a beverage

Drawer
Microwave
Cupboard
Fridge
Fridge
Coffee Machine
Drawer
observation
46
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Discussion

Lack of spatio-temporal information

• Misclassification

? The end of one behaviour contains observations
that should be the start of the next
Preparing lunch
Preparing a beverage

Drawer
Microwave
Cupboard
Fridge
Fridge
Coffee Machine
Drawer
observation
47
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Discussion

Lack of spatio-temporal information

• Misclassification

? The end of one behaviour contains observations
that should be the start of the next
Preparing lunch
Preparing a beverage
. . .
Drawer
Microwave
Cupboard
Fridge
Fridge
Coffee Machine
Drawer
observation
48
B
e h a v i o u r R e c o g n i t i o n
?
?
?
?

• Discussion

Lack of spatio-temporal information
Q How to reduce misclassification?
A Augment current algorithm to include
spatio-temporal information
49
S
p a t i o - t e m p o r a l
?
?
?
?

• Spatial information (Where?)

• NOT directly interested in the exact coordinates

• So, what are we interested in?

? Room location
? e.g.
Figures extracted from www.istockphoto.com,
www.clubjam.jammag.com, www.nancilea.blogspot.com
50
S
p a t i o - t e m p o r a l
?
?
?
?

• Spatial information (Where?)

• Current study used very basic spatial
  information
• (just the kitchen!)

• In the future, . . .

51
S
p a t i o - t e m p o r a l
?
?
?
?
B e d r o o m
K i t c h e n
Cooking
Sleeping
Preparing a beverage
Grooming
Washing dishes
Computing
D i n i n g R o o m
B a t h r o o m
Eating
Showering
Reading
L i v i n g R o o m
Sitting around fireplace
Watching TV
Resting
Exercising
52
S
p a t i o - t e m p o r a l
?
?
?
?
B e d r o o m
K i t c h e n
Cooking
Sleeping
Preparing a beverage
Grooming
Washing dishes
Computing
D i n i n g R o o m
B a t h r o o m
Eating
Showering
Reading
L i v i n g R o o m
Sitting around fireplace
Watching TV
Resting
Exercising
53
S
p a t i o - t e m p o r a l
?
?
?
?

• Spatial information (Where?)

. . .
observation
Kitchen
Bathroom
Kitchen
54
S
p a t i o - t e m p o r a l
?
?
?
?

• Spatial information (Where?)

. . . is this sufficient for reasoning?
WITHOUT temporal, the system cannot
differentiate
3 am
Vs.
8 am
Bathroom
Figure extracted from http//hazard.com/graphics
55
S
p a t i o - t e m p o r a l
?
?
?
?

• Temporal information

• When does a behaviour occur?

? Mapping to time scale
e.g. Mary vacuums every Sunday

• How long does behaviour take?

? Duration
e.g. Microwave used for a dangerously long time

• How often does behaviour occur?

? Frequency
e.g. Peter showers 3 times a day
Source Guesgen and Marsland (2009)
56
S
p a t i o - t e m p o r a l
?
?
?
?

• Temporal information (When)

Absolute time
Relative time

• 3.03 pm

• ½ hour after shower

• having breakfast 2 hours before meeting

• am vs. pm

• weekends vs. weekdays

. .

• winter vs. summer

. .
57
S
p a t i o - t e m p o r a l
?
?
?
?

• Time scales

• Yearly (e.g. Christmas, New Year, Easter, etc.)
• Weekly (e.g. vacuuming, visit from health
  worker, etc.)
• Daily (e.g. showering, eating, etc.)

58
S
p a t i o - t e m p o r a l
?
?
?
?

• Temporal information

• tells us when, for how long and how frequent
  behaviour occurs

(a) segment the behaviours
(b) generate a sequence of behavioural patterns
Source Guesgen and Marsland (2009)
59
S
p a t i o - t e m p o r a l
?
?
?
?
a m
p m
Reading
Exercising
Cooking
Exercising
Cooking
Watching TV
Grooming
Computing
Preparing a beverage
Computing
Washing dishes
Eating
Resting
Showering
Preparing a beverage
Reading
Eating
Washing dishes
E v e n i n g
Showering
Watching TV
Computing
Cooking
Washing dishes
Reading
Preparing a beverage
Sitting around fireplace
Eating
N i g h t
Sleeping
60
S
p a t i o - t e m p o r a l
?
?
?
?
a m
p m
Reading
Exercising
Cooking
Exercising
Cooking
Watching TV
Grooming
Computing
Preparing a beverage
Computing
Washing dishes
Eating
Resting
Showering
Preparing a beverage
Reading
Eating
Washing dishes
E v e n i n g
Showering
Watching TV
Computing
Cooking
Washing dishes
Reading
Preparing a beverage
Sitting around fireplace
Eating
N i g h t
Sleeping
61
S
p a t i o - t e m p o r a l
?
?
?
?
a m
p m
Reading
Exercising
Cooking
Exercising
Cooking
Watching TV
Grooming
Computing
Preparing a beverage
Computing
Washing dishes
Eating
Resting
Showering
Preparing a beverage
Reading
Eating
Washing dishes
E v e n i n g
Showering
Watching TV
Computing
Cooking
Washing dishes
Reading
Preparing a beverage
Sitting around fireplace
Eating
N i g h t
Sleeping
62
S
p a t i o - t e m p o r a l
?
?
?
?

• Temporal information

• tells us when, for how long and how frequent
  behaviour occurs

(a) segment the behaviours
(b) generate a sequence of behavioural patterns
63
S
p a t i o - t e m p o r a l
?
?
?
?
Behaviour
Cooking
Eating
Watching TV
Computing
Resting
Time
Preparing a beverage
Washing dishes
Reading
Exercising
Watching TV
Cooking
64
S
p a t i o - t e m p o r a l
?
?
?
?
Kitchen
Competition among HMMs
Behaviour
Cooking
Cooking
Preparing a beverage
Eating
Washing dishes
Watching TV
Bedroom
Computing
Sleeping
Grooming
Resting
Computing
Time
Space
Preparing a beverage
Dining Room
Washing dishes
Eating
Reading
Reading
Living Room
Exercising
Watching TV
Watching TV
Exercising
Cooking
Sitting around fireplace
. . .
Resting
65
S
p a t i o - t e m p o r a l
?
?
?
?
Kitchen
Competition among HMMs
Behaviour
Cooking
Cooking
Preparing a beverage
Eating
Washing dishes
Watching TV
Bedroom
Computing
Sleeping
Grooming
Resting
Computing
Time
Space
Preparing a beverage
Dining Room
Washing dishes
Eating
Reading
Reading
Living Room
Exercising
Watching TV
Watching TV
Exercising
Cooking
Sitting around fireplace
. . .
Resting
66
S
p a t i o - t e m p o r a l
?
?
?
?
Competition among HMMs
Kitchen
Behaviour
Cooking
Cooking
Preparing a beverage
Eating
Washing dishes
Watching TV
Bedroom
Computing
Sleeping
Resting
Grooming
Computing
Time
Space
Preparing a beverage
Dining Room
Washing dishes
Eating
Reading
Reading
Living Room
Exercising
Watching TV
Watching TV
Exercising
Cooking
Sitting around fireplace
. . .
Resting
67
S
p a t i o - t e m p o r a l
?
?
?
?
Competition among HMMs
Kitchen
Behaviour
Cooking
Cooking
Preparing a beverage
Eating
Washing dishes
Watching TV
Bedroom
Computing
Sleeping
Resting
Grooming
Computing
Time
Space
Preparing a beverage
Dining Room
Washing dishes
Eating
Reading
Reading
Living Room
Exercising
Watching TV
Watching TV
Exercising
Resting
Sitting around fireplace
. . .
Resting
68
S
p a t i o - t e m p o r a l
?
?
?
?

• The system may make mistakes, particularly
  with time!

• What happens if the person is late one day
  and makes lunch at 3 pm?

? Fuzzy logic system
69
C
o n t e x t u a l R e a s o n i n g
?
?
?
?

• Contextual information

• How was the current situation is reached?

• What else is happening?

• What is the state of the environment?

. . . needs to be considered!
70
C
o n t e x t u a l R e a s o n i n g
?
?
?
?
John is boiling water in the middle of the
night
. . . is this normal?
71
C
o n t e x t u a l R e a s o n i n g
?
?
?
?
John is boiling water in the middle of the
night

• Spatial Kitchen
• Temporal Middle of the night

Is the information sufficient for reasoning?
72
C
o n t e x t u a l R e a s o n i n g
?
?
?
?
John is boiling water in the middle of the
night after watching late night movie
Contextual information

• Spatial Living room ? Kitchen
• Temporal Middle of the night and is Saturday

. . . he stays up longer !!!
73
C
o n c l u s i o n
?
?
?
?

• Competition between HMMs ? a possible
  mechanism for behaviour recognition and
  segmentation

• Spatio-temporal and context awareness ?
  play an important role in interpreting behaviour

74
A
c k n o w l e d g e m e n t s
?
?
?
?

• Stephen Marsland, Hans Guesgen

• Massey University Smart Environment (MUSE)
  members

• School of Engineering and Advanced
  Technology (SEAT)

• Massey University

75
F
i n a l l y . . .
?
?
?
?
Thank you!(Merçi!)