Learning Science by Conversing with Animated Agents

1
Learning Science by Conversing with Animated
Agents
Keith Millisa, Art Graesserb, Diane Halpernc,
Anne Britta aDept. of Psychology, Northern
Illinois University bDept. of Psychology,
University of Memphis cDept. of Psychology,
Claremont McKinna College
2
Serious Issues with Science Education

• Students have difficulty learning scientific
  inquiry
• Only 18 of 12th graders scored at the proficient
  level (NAEP, 2006)
• Scientific literacy is emphasized
• Inquiry involves critical thinking about every
  day events
• Solutions will need buy in from administrators,
  policy-makers, teachers and parents

3
Computerized Intelligent Tutors

• Tutor interacts with the student, often in
  natural language
• Tutor gives hints, prompts and feedback
• Scaffolds content and feedback
• Keeps track of the students understanding
• Uses state of the art computational linguistic
  techniques
• Often given by animated pedagogical agents

4
Exploring a Sea of Animated Conversational Agents
SI Agent
Laura
SI Agent
Adele
STEVE
Carmen
AutoTutor
Leonardo
PKD Android
iMAP
BEAT
Casey
iSTART
TLTS
Spark
MRE
5
Our Tutors in Science Education

• Critical Thinking (CT) Tutor
• Project ARIES!
• Both use AutoTutor, a computer tutoring
  environment developed by Art Graesser and
  colleagues

6
(No Transcript)
7
LEARNING GAINS OF TUTORS(effect sizes)

• .42 Unskilled human tutors
• (Cohen, Kulik, Kulik, 1982)
• .80 AutoTutor (14 experiments)
• (Graesser and colleagues)
• 1.00 Intelligent tutoring systems
  
• PACT (Anderson, Corbett, Aleven, Koedinger)
• Andes, Atlas (VanLehn)
• Diagnoser (Hunt, Minstrell)
• Sherlock (Lesgold)
• (?) Skilled human tutors
• (Bloom, 1987)

8
(No Transcript)
9
Possible advantages for using multiple
pedagogical agents

• Each agent has single role (e.g., teacher or
  student) versus multiple roles (e.g., teacher and
  student).
• Student agent (Joe) can be taught (teachable
  agents)
• Stage for interesting trialogs
• Provide modeling for the human
• Increase engagement and motivation

10
What our Agents can do, physically

• Look
• Right, left, up, down
• Gestures
• Headshake, nod, yawn, wink, blink, kiss, roll
  eyes, smirk
• Emotions
• Happy, sad, angry, broken heart, shy, pondering
• Talk

11
Three Domains Of Critical Thinking Tutor

• Scientific Reasoning evaluating the validity of
  studies
• Everyday Arguments evaluation and learn
  components of arguments
• Fallacies Unsound reasoning techniques used for
  the purposes of persuasion.

12
(No Transcript)
13
(No Transcript)
14
AutoTutor

• Curriculum script
• Problem (e.g., a faulty study)
• Expectations ideas that the tutor wants the
  student to say (e.g., why a study is faulty)
• Tutor uses hints and prompts to get the student
  to say the expectations
• Tutor gives summary
• Uses sophisticated computational linguistic
  algorithms to assess the match between the
  students response and intended answer

15
Sample dialog

• Student I dont see any problems.
• Crystal Uh uh. I see. What can you say about
  the design?
• Student There was a group of participants who
  took the pills.
• Crystal Ok. What can you say about establishing
  control?
• Student I dont know.
• Crystal I know you can get this. This study has
  no control what?
• Student group.
• Crystal Great. This study has no control group.
  Lets move on

16
Sample dialog
Hints

• Student I dont see any problems.
• Crystal Uh uh. I see. What can you say about
  the design?
• Student There was a group of participants who
  took the pills.
• Crystal Ok. What can you say about establishing
  control?
• Student I dont know.
• Crystal I know you can get this. This study has
  no control what?
• Student group.
• Crystal Great. This study has no control group.
  Lets move on

Prompt
Assertion ( Expectation)
17
Joe gives summary if the student had learned
18
Classifying Student Input

• First attempt to solve problem
• All student input is compared to expectations
• A matching score is computed for each
  expectation using Latent Semantic Analysis (LSA)
  and word-matching
• If matching score gt threshold, then expectation
  is met
• If matching score lt threshold, then expectation
  is not met, and Tutor will select these for
  dialog exchanges

19
How effective is the Critical Thinking Tutor?

• How well does the tutor do in matching students
  input to expectations?
• How well does the tutor do in increasing learning?

20
How well does the tutor do in matching students
input to expectations?

• Sampled one problem jogging problem
• Listed all expectations and student responses
• Tallied whether the CT tutor classified the input
  as matching expectation human rater did the same
• Computed ds as a measure of discrimination

21
How well does the tutor do in matching students
input to expectations?

• I'm not ever going to jog because I read in a
  health magazine that running increases stress
  levels. It described an experiment that showed
  just that. In this experiment, participants were
  placed into either a low stress group or a high
  stress level group, based on their pre-existing
  stress levels. There were five participants in
  each group. Both groups were instructed to run
  three times a week. The result was that the
  stress levels in the high stress level group
  increased even more. They were really stressed.
  The stress level in the low stress group stayed
  about the same. So you see, doing exercise leads
  to higher stress, at least for people with some
  stress in their lives
• Expectations
• There could be confounding variables that are
  responsible for the differences on stress between
  the high and low stress groups
• It is a correlational study and not an experiment
  
• The sample size is too small to generalize the
  results
• There is no control group in which individuals
  did not jog
• There was no operational definition of stress

22
(No Transcript)
23
(No Transcript)
24
How well does the tutor do in increasing learning?

• Undergraduate psychology students
• 3 conditions
• Critical thinking tutor 6 problems
• Textbook only (matched for content)
• Do nothing
• Pre- and posttests multiple choice, evaluate
  experiments

25
How well does the tutor do in increasing learning?
26
Project ARIES!

• ARIES Acquiring Research Investigative and
  Evaluative Skills
• Cover Story
• Aliens from the Aries constellation are
  attempting to colonize Earth
• Alien spies are teaching poor science and selling
  products based on faulty research
• Goal recruiting training new FBS (Federal
  Bureau of Science) agents to help identify the
  alien spies, and thus prevent being colonized.

27
What will ARIES teach?

• Developing Research Ideas
• Theories, hypotheses, pseudoscience,
  falsifiability
• The Independent and Dependent Variables
• Operational definitions, reliability, accuracy,
  precision, validity, objectivity of scoring
• Experimental Control
• Comparison groups, random assignment, subject
  bias, attrition/mortality
• The Sample Experimenter
• Representative, sample size, experimenter bias,
  conflict of interest
• Drawing conclusion
• Alternative interpretations, limits of
  correlation research, quasi-experimental designs,
  replication of results

28
(No Transcript)
29
Project ARIES!

• Three training levels
• Level 1 Learn about science by reading a Big
  Book of Science written by the aliens
• Level 2 Help identify potential aliens by
  evaluating case studies (i.e., faulty studies)
• Level 3 Interrogate potential aliens

30
FBS Handler
31
Support Tools

• Searchable Thought and Knowledge by Halpern
• Student information
• List of covered concepts
• Current level
• Time to invasion
• Student score
• Money saved number of aliens arrested because
  of students performance
• Student notebook

32
ARIES uses the following learning principles

• Self-explanation (Chi et al., 1994 McNamara,
  2004)
• Generate reasons why a study is faulty or not
  faulty
• Reciprocal teaching (Biswas, et al., 2005
  Palincsar Brown, 1984)
• Students teach the fellow student
• Spacing, testing effects (Bahrick Hall, 2005
  Roediger Karpicke, 2006)
• Students must recognize concepts across many
  examples
• Variable encoding (Benjamin, et al, 1998)
• Psychology, biology and chemistry problems
• Authentic learning (Bransford et al., 2002)
• Case studies are magazine, news articles,
  advertisements
• Motivation, engagement (Lepper Malone, 1987)
• Consequences for their performance

33
Summary

• Work still in progress
• Traditional inquiry learning vs. more cased-based
  reasoning
• AutoTutor would be suited to teaching engineering
• Good at explanation-based content (causal
  mechanisms)
• Auto-tutor authoring tool
• Retrofitting a learning environment like
  AutoTutor to be game-like poses new questions
• Impact of role-playing, story lines (etc.) on
  deep learning, transfer, etc.