422: Unexpected Hanging and other sadistic pleasures of teaching

1
4/22 Unexpected Hanging and other sadistic
pleasures of teaching

• ?Today Probabilistic Plan Recognition
• ?Tomorrow Web Service Composition (BY 510 11AM)
• ?Thursday Continual Planning for Printers
  (in-class)
• ?Tuesday 4/29 (Interactive) Review

2
Approaches to plan recognition

• Consistency-based
• Hypothesize revise
• Closed-world reasoning
• Version spaces
• Probabilistic
• Stochastic grammars
• Pending sets
• Dynamic Bayes nets
• Layered hidden Markov models
• Policy recognition
• Hierarchical hidden semi-Markov models
• Dynamic probabilistic relational models
• Example application Assisted Cognition

Can be complementary.. First pick the
consistent plans, and check which of them
is most likely (tricky if the agent can make
errors)
3
Agenda (as actually realized in class)

• Plan recognition as probabilistic (max weight)
  parsing
• On the connection between dynamic bayes nets and
  plan recognition with a detour on the special
  inference tasks for DBN
• Examples of plan recognition techniques based on
  setting up DBNs and doing MPE inference on them
• Discussion of Decision Theoretic Assistance paper

4
Stochastic grammars
CF grammar w/ probabilistic rules Chart parsing
Viterbi Successful for highly structured tasks
(e.g. playing cards) Problems errors, context

• Huber, Durfee, Wellman, "The Automated Mapping
  of Plans for Plan Recognition", 1994
• Darnell Moore and Irfan Essa, "Recognizing
  Multitasked Activities from Video using
  Stochastic Context-Free Grammar", AAAI-02, 2002.

5
Probabilistic State-dependent grammars
6
Connection with DBNs
7
Time and Change in Probabilistic Reasoning
8
Temporal (Sequential) Process

• A temporal process is the evolution of system
  state over time
• Often the system state is hidden, and we need to
  reconstruct the state from the observations
• Relation to Planning
• When you are observing a temporal process, you
  are observing the execution trace of someone
  elses plan

Review
9
Review
10
Review
11
Review
12
Dynamic Bayes Networks are templates for
specifying the relation between the values of
a random variable across time-slices ?e.g.
How is Rain at time t related to Rain at time
t1? We call them templates because they need
to be expanded (unfolded) to the required
number of time steps to reason about the
connection between variables at different
time points
13
Normal LW doesnt do well when the evidence
is downstream (the sample weight
will be too small) In DBN, none of the
evidence is affecting the sampling! EVEN
MORE of an issue
Normal LW takes each sample through the network
one by one Idea 1 Take them all from t to
t1 lock-step ?the samples are the
distribution
14
Special Cases of DBNs are well known in the
literature

• Restrict number of variables per state
• Markov Chain DBN with one variable that is fully
  observable
• Hidden Markov Model DBN with only one state
  variable that is hidden and can be estimated
  through evidence variable(s)

• Restrict the type of CPD
• Kalman Filters DBN where the system transition
  function as well as the observation variable are
  linear gaussian
• The advantage of Gaussians is that the posterior
  distribution remains Gaussian

15
(No Transcript)
16
(No Transcript)
17
Plan Recognition Approaches based on setting up
DBNs
18
Dynamic Bayes nets (I)
Models relationship between users recent actions
and goals (help needs) Probabilistic goal
persistence Programming in machine language?

• E. Horvitz, J. Breese, D. Heckerman, D. Hovel,
  and K. Rommelse. The Lumiere Project Bayesian
  User Modeling for Inferring the Goals and Needs
  of Software Users. Proceedings of the Fourteenth
  Conference on Uncertainty in Artificial
  Intelligence, July 1998.
• Towards a Bayesian model for keyhole plan
  recognition in large domains Albrecht, Zukermann,
  Nicholson, Bud

19
Excel help (partial)
20
Dynamic Bayesian Nets
Cognitive mode normal, error
Learning and Inferring Transportation Routines
Lin Liao, Dieter Fox, and Henry Kautz, Nineteenth
National Conference on Artificial Intelligence,
San Jose, CA, 2004.
21
Pending sets
Explicitly models the agents plan agenda using
Pooles probabilistic Horn abduction
rules Handles multiple concurrent interleaved
plans negative evidence Number of different
possible pending sets can grow exponentially Conte
xt problematic? Metric time?
Pending(P,T1)? Pending(P,T),
Leaves(L), Progress(L, P, P, T1).
Happen(X,T1) ? Pending(P,T), X in P,
Pick(X,P,T1).

• A new model of plan recognition. Goldman, Geib,
  and Miller
• Probabilistic plan recognition for hostile
  agents. Geib, Goldman

22
Layered hidden Markov models
Cascade of HMMs, operating at different temporal
granularities Inferential output at layer K is
evidence for layer K1

• N. Oliver, E. Horvitz, and A. Garg. Layered
  Representations for Recognizing Office Activity,
  Proceedings of the Fourth IEEE International
  Conference on Multimodal Interaction (ICMI 2002)

23
Policy recognition
Model agent using hierarchy of abstract policies
(e.g. abstract by spatial decomposition) Compute
the conditional probability of top-level policy
given observations Compiled into DBN

• Tracking and Surveillance in Wide-Area Spatial
  Environments Using the Hidden Markov Model. Hung
  H. Bui, Svetha Venkatesh and West.
• Bui, H. H., Venkatesh, S., and West, G. (2000) On
  the recognition of abstract Markov policies.
  Seventeenth National Conference on Artificial
  Intelligence (AAAI-2000), Austin, Texas

24
Hierarchical hidden semi-Markov models

• Combine hierarchy (function call semantics) with
  metric time
• Compile to DBN
• Time nodes represent a distribution over the time
  of the next state switch
• Linear time smoothing
• Research issues parametric time nodes, varying
  granularity

• Hidden semi-Markov models (segment models) Kevin
  Murphy. November 2002.
• HSSM Theory into Practice, Deibel Kautz,
  forthcoming.

25
Dynamic probabilistic relational models
PRM - reasons about classes of objects and
relations Lattice of classes can capture plan
abstraction DPRM efficient approximate
inference by Rao-Blackwellized particle
filtering Open approximate smoothing?

• Friedman, N., L. Getoor, D. Koller, A. Pfeffer.
  Learning Probabilistic Relational Models. 
  IJCAI-99, Stockholm, Sweden (July 1999).
• Relational Markov Models and their Application to
  Adaptive Web Navigation, Anderson, Domingos, Weld
  2002.
• Dynamic probabilistic relational models,
  Anderson, Domingos, Weld, forthcoming.

26
Assisted cognition
Computer systems that improve the independence
and safety of people suffering from cognitive
limitations by

• Understanding human behavior from low-level
  sensory data
• Using commonsense knowledge
• Learning individual user models
• Actively offering prompts and other forms of help
  as needed
• Alerting human caregivers when necessary
• http//www.cs.washingt
  on.edu/assistcog/

27
Activity Compass

• Zero-configuration personal guidance system
• Learns model of users travel on foot, by public
  transit, by bike, by car
• Predicts users next destination, offers
  proactive help if lost or late
• Integrates user data with external constraints
• Maps, bus schedules, calendars,
• EM approach to clustering segmenting data

The Activity Compass Don Patterson, Oren
Etzioni, and Henry Kautz (2003)
28
Activity of daily living monitor prompter
Foundations of Assisted Cognition Systems. Kautz,
Etzioni, Fox, Weld, and Shastri, 2003
29
Recognizing unexpected events using online model
selection

• User errors, abnormal behavior
• Select model that maximizes likelihood of data
• Generic model
• User-specific model
• Corrupt (impaired) user model
• Neurologically-plausible corruptions
• Repetition
• Substitution
• Stalling

fill kettle
put kettleon stove
fill kettle
put kettleon stove
put kettlein closet
Fox, Kautz, Shastri (forthcoming)
30
Decision-Theoretic Assistance

• Dont just recognize!
• Jump in and help..
• Allows us to also talk about POMDPs

31
Intelligent Assistants

• Many examples of AI techniques being applied to
  assistive technologies
• Intelligent Desktop Assistants
• Calendar Apprentice (CAP) (Mitchell et al. 1994)
• Travel Assistant (Ambite et al. 2002)
• CALO Project
• Tasktracer
• Electric Elves (Hans Chalupsky et al. 2001)
• Assistive Technologies for the Disabled
• COACH System (Boger et al. 2005)

32
Not So Intelligent

• Most previous work uses problem-specific,
  hand-crafted solutions
• Lack ability to offer assistance in ways not
  planned for by designer
• Our goal provide a general, formal framework for
  intelligent-assistant design
• Desirable properties
• Explicitly reason about models of the world and
  user to provide flexible assistance
• Handle uncertainty about the world and user
• Handle variable costs of user and assistive
  actions
• We describe a model-based decision-theoretic
  framework that captures these properties

33
An Episodic Interaction Model
Action set A
Action set U
Each user and assistant action has a cost
Objective minimize expected cost of episodes
34
Example Grid World Domain
World states (x,y) location and door
status Possible goals Get wood, gold, or
food User actions Up, Down, Left, Right,
noop Open a door in current room (all actions
have cost 1) Assistant actions Open a door,
noop (all actions have cost 0)
35
World and User Models

• Model world dynamics as a Markov decision
  process (MDP)
• Model user as a stochastic policy

P(Wt1 Wt, Ut, At)
Given model,action sequence Output assistant
action
36
Optimal Solution Assistant POMDP
P(Wt1 Wt, Ut, At)

• Can view as a POMDP called the assistant POMDP
• Hidden State user goal
• Observations user actions and world states
• Optimal policy gives mapping from observation
  sequences to assistant actions
• Represents optimal assistant
• Typically intractable to solve exactly

37
Approximate Solution Approach

• Online actions selection cycle
• 1) Estimate posterior goal distribution given
  observation
• 2) Action selection via myopic heuristics

Goal Recognizer
Action Selection
P(G)
Assistant
Wt
Ot
At
Environment
User
Ut
38
Approximate Solution Approach

• Online actions selection cycle
• 1) Estimate posterior goal distribution given
  observation
• 2) Action selection via myopic heuristics

Goal Recognizer
Action Selection
P(G)
Assistant
Wt
Ot
At
Environment
User
Ut
39
Goal Estimation

• Given
• P(G Ot) goal posterior at time t initally
  equal to prior P(G)
• P(Ut G, Wt) stochastic user policy
• Ot1 new observation of user action and world
  state
• it is straightforward to update goal
  posterior at time t1

Goal posterior given observations up to time t
Wt
P(G Ot)
Current State
40
Learning User Policy

• Use Bayesian updates to update user policy P(UG,
  W) after each episode
• Problem can converge slowing, leading to poor
  goal estimation
• Solution use strong prior on user policy derived
  via planning
• Assume that user behaves nearly rational
• Take prior distribution on P(UG, W) to be bias
  toward optimal user actions
• Let Q(U,W,G) be value of user taking action U in
  state W given goal G
• Can compute via MDP planning
• Use prior P(U G, W) a exp(Q(U,W,G))

41
Q(U,W,G) for Grid World
42
Approximate Solution Approach

• Online actions selection cycle
• 1) Estimate posterior goal distribution given
  observation
• 2) Action selection via myopic heuristics

Goal Recognizer
Action Selection
P(G)
Assistant
Wt
Ot
At
Environment
User
Ut
43
Action Selection Assistant POMDP

• Assume we know the user goal G and policy
• Can create a corresponding assistant MDP over
  assistant actions
• Can compute Q(A, W, G) giving value of taking
  assistive action A when users goal is G
• Select action that maximizes expected (myopic)
  value

Assistant MDP
G
At
U
Wt
Wt1
Wt2
If you just want to recognize, you only need
P(GOt) If you just want to help (and know the
goal), you just need Q(A,W,G)
44
Experiments Grid World Domain
45
Experiments Kitchen Domain
46
Experimental Results

• Experiment 12 human subjects, two domains
• Subjects were asked to achieve a sequence of
  goals
• Compared average cost of performing tasks with
  assistant to optimal cost without assistant
• Assistant reduced cost by over 50

47
Summary of Assumptions

• Model Assumptions
• World can be approximately modeled as MDP
• User and assistant interleave actions (no
  parallel activity)
• User can be modeled as a stationary, stochastic
  policy
• Finite set of known goals
• Assumptions Made by Solution Approach
• Access to practical algorithm for solving the
  world MDP
• User does no reason about the existence of the
  assistance
• Goal set is relatively small and known to
  assistant
• User is close to rational

48
While DBNs are special cases of B.N.s there are
a certain inference tasks that are
particularly frequently useful for them (Notice
that all of them involve estimating posterior
probability distributionsas is done in any B.N.
inference)
49
Can do much better if we exploit the repetitive
structure Both Exact and Approximate B.N.
Inference methods can be made to take the
temporal structure into account.
?Specialized variable-elimination method
?Unfold t1th level, and roll-up tth
level by variable elimination ?Specialized
Likelihood-weighting methods that take evidence
into account ?Particle
Filtering Techniques
50
(No Transcript)
51
Can do much better if we exploit the repetitive
structure Both Exact and Approximate B.N.
Inference methods can be made to take the
temporal structure into account.
?Specialized variable-elimination method
?Unfold t1th level, and roll-up tth
level by variable elimination ?Specialized
Likelihood-weighting methods that take evidence
into account ?Particle
Filtering Techniques
52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
Class Ended here..Slides beyond this not
discussed
61
Belief States

• If we have k state variables, 2k states
• A belief state is a probability distribution
  over states
• Non-deterministic
• We just know the states for which the probability
  is non-zero
• 22k belief states
• Stochastic
• We know the probability distribution over the
  states
• Infinite number of probability distributions
• A complete state is a special case of belief
  state where the distribution is dirac-delta
• i.e., non-zero only for one state

In blocks world, Suppose we have blocks A and
B and they can be clear, on-table On
each other -A state A is on table, B is on
table, both are clear, hand is empty -A
belief state A is either on B or on
Table B is on table. Hand is empty ? 2
states in the belief state
62
Actions and Belief States
A belief state A is either on B or on
Table B is on table. Hand is empty

• Two types of actions
• Standard actions Modify the distribution of
  belief states
• Doing C on A action in the belief state gives
  us a new belief state (with C on A on B OR C on
  A B clear)
• Doing Shake-the-Table action converts the
  previous belief state to (A on table B on Table
  A clear B clear)
• Notice that actions reduce the uncertainty!

• Sensing actions
• Sensing actions observe some aspect of the belief
  state
• The observations modify the belief state
  distribution
• In the belief state above, if we observed that
  two blocks are clear, then the belief state
  changes to A on table B on table both clear
• If the observation above is noisy (i.e, we are
  not completely certain), then the probability
  distribution just changes so more probability
  mass is centered on the A on table B on Table
  state.

63
Actions and Belief States
A belief state A is either on B or on
Table B is on table. Hand is empty

• Two types of actions
• Standard actions Modify the distribution of
  belief states
• Doing C on A action in the belief state gives
  us a new belief state (with C on A on B OR C on
  A B clear)
• Doing Shake-the-Table action converts the
  previous belief state to (A on table B on Table
  A clear B clear)
• Notice that actions reduce the uncertainty!

• Sensing actions
• Sensing actions observe some aspect of the belief
  state
• The observations modify the belief state
  distribution
• In the belief state above, if we observed that
  two blocks are clear, then the belief state
  changes to A on table B on table both clear
• If the observation above is noisy (i.e, we are
  not completely certain), then the probability
  distribution just changes so more probability
  mass is centered on the A on table B on Table
  state.

64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
(No Transcript)