Artificial Intelligence in the Design of Assistive Technology

1
Artificial Intelligence in the Design of
Assistive Technology

• Martha E. Pollack
• Computer Science Engineering
• University of Michigan

2
An Example of Use-Inspired Research

• The research is motivated by an important
  societal problem
• But is not overly constrained by the problem
• The research should have fundamental interest
• And thus be potentially useful for multiple
  applications

3
The Population is Aging
gt Age 60 2000 2050
World 10.0 21.4

Belarus 19.3 37.6
Germany 23.2 34.5
Italy 24.1 40.6
Netherlands 18.2 30.7
Slovenia 19.2 41.5

United States 16.1 25.5
Mexico 6.9 26.2
Brazil 7.8 25.9
Columbia 6.9 22.7
China 6.9 22.7
India 7.5 20.1
Japan 23.3 42.4
Myanmar 6.8 20.5

Australia 16.4 29.9
Fiji 5.7 22.7

Egypt 6.8 18.7
Iran 6.4 24.8
Jordan 4.6 19.0

Botswana 4.2 6.0
Ethiopia 4.6 7.7
Mali 3.9 5.3
Source United Nations Population Division
http//esa.un.org/unpp/
4
The Oldest Old
gt Age 80 2000 2050
World 1.1 4.2
United States 3.2 7.2
China 0.9 7.2
Source United Nations Population Division
http//esa.un.org/unpp/
5
Intelligent Technology

• Can support people with
• Mobility impairment
• Declines in sensory function
• Social and emotional challenges
• Cognitive decline

Obstacle-avoiding wheelchairs
Adaptive digital hearing aids
Elder-friendly Internet and email systems
6
The Challenge

• Cognitive impairment can impact performance of
  daily activities necessary for health and
  wellbeing
• ADLs eating, dressing, bathing, toileting, . .
  .
• IADLs managing medicines, housekeeping,
  arranging transportation, preparing meals, . .
• Can become difficult to follow a daily plan

7
Autominder An Adaptive Reminder System

• Uses Artificial Intelligence techniques to
• Model, update, and maintain the clients plan
• Including complex temporal and causal constraints
• Monitor the clients performance
• Updating the plan as execution proceeds
• Reason about what reminders to issue, and when
• Ensuring compliance, without sacrificing client
  independence

8
Autominder Interaction
Req/Opt Activity Allowed Expected Observed
R check blood glucose 1150-1210
R lunch 1215- 100
O TV 1700-1730
1155
REMIND 1240
1242
REMIND 1655
9
Autominder Architecture
What should the client do?
Activity Info
Client Modeler
Plan Manager
Plan Updates
Sensor Data
Inferred Activity
Client Plan
smart home
Client Model Info
Activity Info
Intelligent Reminder Generator
Reminders
Preferences
10
Plan Manager

• Maintains up-to-date record of clients planned
  activities
• Techniques
• AI Planning
• Temporal Constraint Satisfaction

11
Plan Representation

• Plans are structured sets of activities
• causal connections
• temporal constraints
• qualitative, quantitative, disjunctive
• conditional constraints
• Temporal constraints modeled as Disjunctive
  Temporal Problems (DTPs)

12
Disjunctive Temporal Problems

• A set of time points (variables) V and a set of
  constraints C of the form
• lbji ? Xi Xj ? ubji ? ? lbmk ? Xk Xm ? ubmk
  
• Solution assignment of times to all variables,
  so that all constraints in C are satisfied
• Generalization of the Simple Temporal Problems
• Disjunctions critical to model non-overlap
  requirements

13
Example Temporal Constraints

• Blood glucose should be checked between 1150 and
  1210
• C1 1150 ? BS TR ? 1210
• Lunch takes between 15 and 30 minutes
• C2 15 ? LE LS ? 30
• The TV show takes 30 minutes
• C3 30 ? TVE TVS ? 30
• Watching TV can begin at 1700 or at 1800
• C4 1700 ? TVS TR ? 1700 ?
• 1800 ? TVS TR ? 1800
• Checking glucose and watching TV should not
  overlap
• C5 0 ? TVS BE ? 0 ? BS TVE
• Dishes should be taken within 1 hour of
  finishing lunch.
• C6 0 ? DS LE ? 60

14
DTP Solving as Meta-Level Propagation

• C1 c11 y x ? 5
• C2 c21 w y ? 5 ? c22 x y ? -10 ?
  c23 z y ? 5
• C3 c31 y w ? -10

NOT c21, c31
NOT c11, c22
Component STP C1 ? c11, C2 ? c23 , C3 ? c31
One exact solution x 0, y 1, z 2, w
12
15
Temporal Constraint Satisfaction

• Efficiently solving DTPs Tsamardinos
  Pollack, AIJ 2003
• Add temporal uncertainty Morris, Muscettola,
  Vidal, IJCAI 2001, Venable Yorke-Smith, IJCAI
  2005, Rossi, Venable, Smith, CP 2004
• causal uncertainty Tsamardinos Pollack,
  Constraints, 2003
• preference functions Khatib et al., IJCAI
  2001, 2003 Peintner Pollack, AAAI 2004, 2205,
  Morris, Morris, Khatib Yorke-Smith, ICAPS
  Wkshp. 2005, Sheini, Peintner, Sakallah
  Pollack, CP 2005, Moffitt Pollack, AAAI 2006
• partial satisfaction Moffitt Pollack,
  FLAIRS 2005, Peintner, Moffitt, Pollack, ICAPS
  2005, Moffitt Pollack, IJCAI 2005, Liffiton,
  Moffitt, Pollack, Sakallah, IJCAI 2005
• dynamic structure Schwartz Pollack, ICAPS
  Wkshp. 2005
• hybrid constraints Moffitt, Peintner,
  Pollack, AAAI 2005

16
Efficiently Solving DTPs
Example Removal of Subsumed Variables
If this assignment to Ci is implied by the
partial assignment above it, prune the other
values for Ci
Ci ? cij
Ci ? cik
Ci ? cil
17
DTPs with Preferences

• Plan includes exercising and a visit from a
  friend
• Should finish exercise before visit or start
  after visit
• VS EE ? 5,? v ES VE ? 0,?
• Now an optimization problem

18
Solution Technique AAAI 2006

• Convert each constraint into a set of valued
  constraints, one per preference level
• Apply branch-and-bound with partial satisfaction
• Performs very well in comparison with previous
  methods

3
2
V
1
not allowed
0 5 10 15 ?
VS EE
V1
V2
V3
V
0 5 10 15 ?
0 5 10 15 ?
0 5 10 15 ?
VS EE
ES VE
VS EE
19
Same Techniques Apply Elsewhere

• Rectangle Packing
• Rectangle i must be contained within the
  enclosing space of dimensions W x H
• xi ? 0, yi ? 0, xi wi ? W, yi hi ? H
• Rectangle i and j must not overlap
• xi wi ? xj ? xj wj ? xi ? yi hi ? yj ?
  yj hj ? yi
• Orientation of rectangles adds hybrid
  constraints
• xi B(oi) ? xj ? xj B(oj) ? xi ?

20
Comparison of Approaches ICAPS 2006
For the case of optimal packings fixed
orientations
Number of squares Number of squares Number of squares
State-of-the-art in
VLSI / Floorplanning1 74549 TIMES OUT TIMES OUT
Operations Research2 00013 01005 3005318
Artificial Intelligence 00000 00003 0000409
16
19
12
1 (Chan and Markov, 2004)
10 day time-out limit
2 (Clautiaux, Carlier, Moukrim, 2004)
21
Autominder Architecture
What has the client done?
Activity Info
Client Modeler
Plan Manager
Plan Updates
Sensor Data
Inferred Activity
Client Plan
smart home
Client Model Info
Activity Info
Intelligent Reminder Generator
Reminders
Preferences
22
Client Modeler

• Given what can be observed (sensor input, clock
  time, stored plan, reminder information, etc.),
  infer probabilities that various actions were
  performed
• Techniques
• Wireless sensor networks
• Reasoning under uncertainty
• Ensuring privacy and security of data collected
  is paramount!

23
Probabilistic Reasoning for Activity Recognition

• Use Hidden Markov Models (HMMs), Dynamic Bayes
  Nets (DBNs), etc.
• Example Train one HMM per activity type
  observed variables sensors firings
• For discrete activities, compute probability of
  each type
• Key question segmentation

Privacy Matters!
Make Tea
Courtesy M. Philipose et al., Intel Research
Pr(faucet) .6 Pr(cup) .4 . . .
Pr(faucet) .5 Pr(tea) .45 . . .
24
Same Techniques Apply Elsewhere

• Recent work at Intel Seattle on using these
  techniques to train anesthesiologists

25
Autominder Architecture
Activity Info
Client Modeler
Plan Manager
Plan Updates
Sensor Data
Inferred Activity
What action should the system take?
Client Plan
smart home
Client Model Info
Activity Info
Intelligent Reminder Generator
Reminders
Preferences
26
Intelligent Reminder Generation

• Given a clients plan and its execution status
• Easy to generate reminders at earliest possible
  time of each action
• Harder to remind well
• Maximize likelihood of appropriate performance of
  activities
• Allow flexibility
• Facilitate efficient performance
• Avoid annoying client and/or making the client
  overly reliant
• Techniques
• iterative refinement (local search)
• machine learning

27
One Approach Iterative Refinement
AIPS 2002
28
Alternative Approach Learning
ICML 2004

• Use reinforcement learning to induce the best
  interaction strategy
• Decide whether, when, and how to remind, given
  information about the clients state
• Add a dynamic action proposerthe plan managerto
  a standard RL architecture

29
Augmented RL Architecture
Environment Client
Sensors Sensors
Actuators Reminder Production
Payoff
Percepts
State Estimator Client Modeler
Agent Adaptive Reminder Generator
Selected Actions
Estimated State
Action Proposer Plan Manager
Plan
30
Sample Experiment (with a client simulator)
31
Autominder Platforms
32
Key Research Challenges

• Integrated activity recognitioninside and
  outside the home
• Enhanced privacy assurance
• Elaboration of machine learning techniques for
  interaction strategies
• Platform and interface design
• Use of adaptive systems for additional purposes
• Social integration
• Cueing for behavioral adaptation
• Incoporation of other assistance mechanisms
  (e.g., face recognition)
• Field testing and iterative design!