Jihie Kim

1
Metacognitive Capabilities for Advice Taking

• Jihie Kim
• Yolanda Gil
• Jim Blythe
• Intelligent Systems Division
• USC/Information Sciences Institute
• www.isi.edu/ikcap/

2
Challenges

• Systems must understand how users
  advice/guidance/request fits before accepting it.
• Self-assessment of competence and confidence

3
Key Idea

• Apply metacognitive skills in human learning
  White and Frederickson 98,Harris 1990
• Awareness of its own knowledge and the
  ability to understand, control and manipulate its
  own cognitive process

4
Approach

• Explicit representation of meta-level goals to
  understand, control, and manipulate systems
  cognitive process of KA and develop strategies
  for them.
• Develop annotations to the dialogue history based
  on the goals and strategies (episodic memory)
• Apply our techniques in proactive KA dialogue
  Kim Gil AIEd 2002, Gil Kim CogSci-2001 Kim
  Gil ITS-2001

5
Meta-Level Goals

• Establish confidence on KB items
• Ensure that KB items are useful and run
  efficiently
• Self-evaluate successful/unsuccessful procedures
• procedures that are often used vs rarely used
  ones
• procedures that are efficient/inefficient
• evaluate other KB items in a similar way
• Be aware of the goals set (i.e. understand the
  purpose)
• Achieve goals or Ensure progress towards the
  goals
• Achieve proficiency
• goals (intentions) that are often set
• errors/gaps found
• progress towards a goal (history)
• answer changes (history)
• redundant calls to a procedure

6
Meta-Level Goals (cont)

• Ensure that advices were useful in making the
  problem solving successful
• Ask for more advices when there are conflicts,
  changes in answers, redundancies, etc.
• Ensure that advices are consistent with existing
  knowledge
• advices that were useful for making PS successful
• advices that didnt affect the results
• frequent rejects or dislikes
• contradicting advices
• redundant advices
• Establish competence assess own problem solving
  capability
• kinds of problems that can be solved
• kinds of problems that cant be solved yet (based
  on attempts)

7
Developing Strategies for Meta-Level Goals

• Goals on KB items
• Establish confidence on KB items
• Ensure that KB items are useful and run
  efficiently
• self evaluate successful/unsuccessful procedures
• ? check the involved PS episodes, whether any
  of them that use the procedure succeeded
• Warning find_laptop still doesnt return
  any results
• procedures that are often used vs rarely used
  ones
• ? check how used, how many times
• Warning the procedure of find_midrange_syste
  ms hasnt been used for a while.
• procedures that are efficient/inefficient
• ? check how used, how long it took to run
• Warning calls to find_all_laptops took more
  than 5 min.
• evaluate other KB items in a similar way

8
Developing Strategies for Meta-Level Goals (cont)

• Goals on Problem Solving Goals
• Be aware of the goals set (i.e. understand the
  purpose)
• Achieve goals or Ensure progress towards the
  goals
• Achieve proficiency
• monitor goals that are often set
• ? check the goals set in the history
• goals set but not visited for a while
• ? check the goals PS episodes
• Warning are you still interested in send_email
  to Susie?
• errors/gaps found
• Warning (if error message not sent from other
  modules) there are still some errors
• progress towards a goal (history)
• ? number of errors/ number of steps taken in PS
  episode
• Feedback Q now we can perform find-laptops, but
  still dont know how to do get_autorization
• answer changes (history)
• ? number of errors/ number of steps taken in PS
  episode
• Feedback now we have IBM think pad as well as
  SONY VAIO as the candidates
• redundant calls to a procedure
• Warning in solving buy laptop the same calls
  to find-laptop have been made multiple times

9
Developing Strategies for Meta-Level Goals (cont)

• Goals on advices taken
• Ensure that advices were useful in making the
  problem solving successful
• Ask for more advices when there are conflicts,
  changes in answers, redundancies, etc.
• Ensure that advices are consistent with existing
  knowledge
• advices that were useful for making PS successful
• ? check changes in the PS episodes
• Feedback since you raised the max price, now we
  can find candidate laptops
• advices that didnt affect the results
• ? check changes in the PS episodes
• Warning you advice to reduce the min speed
  didnt change the answer
• frequent rejects or dislikes
• contradicting advices
• ? check inconsistent results
• redundant advices

10
Developing Strategies for Meta-Level Goals (cont)

• Goals on Overall problem solving capability
• Establish competence assess own problem solving
  capability
• kinds of problems that can be solved
• ? monitor PSs completed and how well they are
  done
• kinds of problems that cant be solved yet (based
  on attempts)
• ? check the answers
• Warning we cant solve find_autorization yet

11
SummaryMetacognitive Capabilities for Advice
Taking
Advice
User Advice
KB changes KB item usage
Follow-up dialogue
KB-1
Episodic Memory with Annotations

• Meta reasoning
• Self assessment of competence
• Strategies for resolving conflicts
• Relevance (usage) and gaps

• 3/28/03 Jihie buys laptop
• Initial task goals
• solution proposed
• advice new cost limit
• 2nd solution
• . . .
• 9/1/03 Jim buys laptop
• . . .
• 10/28/03 Jihie finds laptop

KB-2
. . .
KB annotations
Problem Solving
KB