1
Evaluating Dialogue Systemsthat Reason and
LearnAist, Michalak, Ferguson, Allen

• Dialogue behaviors change over time
• Between users (aka user modeling)
• Between sessions (e.g. dialogue policy)
• Within a session
• In what ways can such behaviors change?Write
  dialogue as D(Input) Output
• Parameter learning change internal values of
  De.g. reinforcement learning of dialogue
  policies
• Input learning change Input (of D)e.g acoustic
  adaptation
• Output learning change Output (of D)e.g.
  learning a new web-based procedure

2
Evaluation criteriafor dialogue systems that
learnAfter Learning to Handle X

• Can the system now do X? performance
• How human-like is the systems X? naturalness
• How much better than before? improvement
• How does it compare to other options?
  effectiveness
• How much more easily or more effectively can the
  system learn a new task X? transfer
• Good news there are techniques from dialogue
  systems, machine learning, educational
  psychology, etc.
• More good news Many challenges remain. For
  examplevisualization of aggregate dialogue
  behaviorevaluation of novel (non-humanlike?)
  behaviors.