Author: Fang Wei

1
A Student Model for an Intelligent Tutoring
System Helping Novices LearnObject Oriented
Design

• Author Fang Wei
• Advisor Prof. Blank
• Department of Computer Science
• Lehigh University
• May 10, 2007

2
Presentation Outline

• Publications
• Background
• Research questions
• Methodology
• Evaluation
• Conclusions and future work

3
Publications

• Wei, F. Blank, G.D. (2007) Atomic Dynamic
  Bayesian Networks for a Responsive Student Model,
  Proceedings of the 13th International Conference
  on Artificial Intelligence in Education, AIED
  2007
• Wei, F. Blank, G.D. (2006) Student Modeling
  with Atomic Bayesian Networks, Proceedings of the
  8th International Conference on Intelligent
  Tutoring Systems, ITS 2006, pp. 491-502
• Wei, F., Moritz, S., Parvez, S., Blank, G.D.
  (2005) A Student Model for Object-Oriented Design
  and Programming. The Journal of Computing
  Sciences in Colleges (CCSC), Vol. 20, pp.
  260-273. "Best Paper" Award
• Blank, G. D., Parvez, S., Wei, F., Moritz, S
  (2005) A web-based ITS for object-oriented
  design. Poster for 12th International Conference
  on Artificial Intelligence in Education, Workshop
  of Adaptive Systems for Web-Based Education
  Tools and reusability, Amsterdam, The
  Netherlands, June
• Moritz, S., Wei, F., Parvez, S., Blank, G. D.
  (2005), From objects-first to design-first with
  multimedia and intelligent tutoring. The Tenth
  Annual Conference on Innovation and Technology in
  Computer Science Education (ITiCSE), Monte da
  Caparica, Portugal, June.

4
Presentation Outline

• Publications
• Background
• Research questions
• Methodology
• Evaluation
• Conclusions and future work

5
Intelligent Tutoring System (ITS)

• A computer-based instructional system
• has knowledge bases for instructional content and
  teaching strategies
• uses a students level of mastery of topics to
  adapt instruction dynamically
• A cost-effective means of one-on-one tutoring to
  provide novices with individualized attention
• Computer Assisted Instruction (CAI) system does
  not model what a student is learning and cannot
  adapt to student
• CAI provides same instruction, problems and
  feedback to every student

6
Intelligent Tutoring System

• Typically contains three main components
• An expert evaluator that observes a students
  work and identifies errors in his/her solution
• A student model that diagnoses gap in students
  knowledge
• A pedagogical advisor that provides feedback to
  student

7
Student Model

• Maintains a model of students current knowledge
  state by representing and updating
• Provides information for intelligent pedagogical
  decisions and actions including
• curriculum sequencing
• interactive problem solving support
• pedagogical tutoring customized to each
  individual students learning state

8
Motivation

• Novices in high school and college have much
  difficulty in object oriented design
• ITSs can aid learners with complex
  problem-solving
• DesignFirst-ITS developed to help novices learn
  object-oriented design (Blank et al. 2005, Moritz
  Blank 2005)
• Research has shown that an ITS that adapts to
  accurate student knowledge enhances learning
  (Corbett et al. 2000)
• Corbetts ITS focuses on adaptive problem
  sequencing rather than adaptive feedback
• Our ITS focuses on adaptive feedback

9
Student Model in Wei Blank (2006,2007) compared
with other BN Student Models
Authors System Context Consider history Diagnose Concept Pre- requisites Real Time
Murray (1998) Desktop Associate skills v
VanLehn et al.(2001, 2005) Solve physics problems rules, not concepts v
Butz et al. (2004) C programming v No evaluation
Millan et al.(2002, 2005) CAT for math v v Post-process
Reye(1996, 1998, 2004) Theoretical analysis v v
WeiBlank (2006,2007) OO Design (UML) v v v v
10
Layers of Student Knowledge(Self 1994)

• Domain knowledge layer
• explain all vocabulary for discussing or solving
  problems
• Reasoning knowledge layer
• contain reasoning relationships between
  propositions in domain knowledge
• Monitoring knowledge layer
• specify how to solve a problem using reasoning
  knowledge and domain knowledge
• Reflective knowledge layer
• specify appropriate strategies students should
  have in a learning environment

11
Common Problems with Student Models

• Do not consider relationship between individual
  concepts
• Do not represent layered knowledge (Self 1994)
• Do not simulate students knowledge history
• Separate the inferred students knowledge from
  closed- and open-ended exercises
• Do not consider students cognitive strategies
  including general and domain-specific
• Bayesian student models require exponential
  updating time and hence cannot provide real-time
  tutoring adaptive to individual students
  learning state

12
Presentation Outline

• Publication
• Background
• Research questions
• Methodology
• Evaluation
• Conclusions and future work

13
Research Questions (1 of 2)

• Can this student model provide information for
  pedagogical decisions?
• How should this student model represent a
  students current knowledge state and the
  students knowledge structure?
• How will the student model track students
  knowledge state over time? Under this research
  question there are two sub questions
• Would tracking a history of students knowledge
  state be useful for pedagogical decisions?
• Can a history be maintained efficiently enough to
  be responsive in real-time?

14
Research Questions (2 of 2)

• How to synthesize information from two different
  sources, open-ended problem solving
  (object-oriented class diagram design) and
  closed-ended exercises (multiple choice quizzes
  or drag-and-drop exercises)?
• What cognitive strategies should the student
  model consider and how to consider them?

15
Presentation Outline

• Publication
• Background
• Research questions
• Methodology
• Evaluation
• Conclusions and future work

16
Three Layered Architecture

• CM recognizes cognitive strategies that a
  student is using
• HM simulates students hierarchical knowledge in
  a history
• PDM simulates current students hierarchical
  knowledge

17
Curriculum Information Network
int
double
int_string
actor_method
string
double_string
datatype_returntype
object_method
datatype_variable
object_attribute
class
class_constructor
attribute
method
returntype
parameter
method_returntype
A
B
A is prerequisite of B
attribute_parameter
18
Two kinds of concepts

• Unique concept, such as attribute or parameter
• Relationship concepts, such as attribute_parameter
  
• Relationships emerge because of students
  confusions between concepts
• E.g., student defines movieTitle as a parameter
  when he has already defined movieTitle as an
  attribute

19
Prerequisite relationships

• Prerequisite is relationship between concepts
• The concepts a learner needs to understand before
  understanding a concept
• E.g., one needs to understand int and double in
  order to understand numericDatatype
• Relationship concepts are prerequisites of unique
  concepts and vice versa
• E.g., class_constructor -gt constructor
• Understanding constructor doesnt imply
  understanding of class, just how to define a
  constructor for a class

20
Connecting Knowledge with Performance

• Student action unit and knowledge unit make a
  pair(KU,AU)
• Infer understanding of a concept (KU) from a
  student solution step (AU)
• Action unit (AU)
• A single action or step in a students solution
• E.g., add an attribute to a class
• Knowledge unit (KU) concept a student need to
  learn
• KU directly causes a student action unit
• KU is a concept in Curriculum Information Network
  (CIN)

21
Atomic Bayesian Network (ABN)


d-prereq(ku)N

d-prereq(ku)1
d-prereq(ku)2


Noisy-and generalizes logical-and
ku
Students must understand all direct
prerequisites of the concept ku in order to
understand ku

au

22
How to generate an ABN

• Student model generates an ABN in response to a
  student solution step
• First, define the structure of an ABN, i.e., the
  causal relationship between KU and AU, and the
  direct-prerequisites of KU
• Second, determine conditional probability tables
  for this ABN

23
Atomic Dynamic Bayesian Network (ADBN) for HM
layer
24
How to generate an ADBN

• Student model generates an ADBN in response to a
  student solution step
• First, look for the ABN in response to previous
  student solution step
• Second, generate an ABN in response to current
  student solution step
• Third, determine conditional probability tables
  for the ADBN

25
Concrete Example

• Student defined movieTitle as a parameter for
  method displayMovieTitle after she has already
  defined movieTitle as an attribute to a class
  TicketMachine
• EE determines that movieTitle should not be a
  parameter
• SM determines that the center concept of an ABN
  is attribute_parameter, and finds all direct
  prerequisites, attribute and parameter, from CIN

26
Concrete Example

• attributes prior can be found from the database
• parameters prior is 0.5, students knowledge
  state is assessed based on the difference between
  prior and posterior probabilities (VanLehn et al.
  1998, Millán Pérez-de-la-Cruz 2002)
• SM determines
• student has good understanding of class,
  attribute, methods, and parameter but low
  understanding of attribute_parameter
• the tutoring need is explanation of
  attribute_parameter

27
Concrete Examplefeedback

• Since you have added movieTitle as an attribute
  to the class TicketMachine, you shouldnt also
  make it a parameter to the method
  displayMovieTitle. To decide whether movieTitle
  should be an attribute or a parameter, remember
  attributes are accessible anywhere within the
  scope of a class, while parameters are accessible
  only within the scope of a method

28
Presentation Outline

• Publication
• Background
• Research questions
• Methodology
• Evaluation
• Conclusions and future work

29
Evaluation of ABNs with simulated students

• Hypotheses
• Pre-setting slip and guess values will lead to a
  reliable student model
• Varying slip and guess values will affect the
  accuracy of the student model

30

• Pre-setting slip and guess values to same
  (relatively small e.g. lt0.1) values produces
  accuracy of at least 93, confirming the first
  hypothesis
• Changing of presetting slip and guess causes
  accuracy to change from 79.1 to 94.3,
  confirming the second hypothesis
• Correct diagnostic rates are higher when slipp,
  guessp and slipe, guesse take same value
• No significant difference when slip and guess
  take a same small value (lt0.1)

31
Evaluation of ADBNs with simulated students

• Hypotheses
• Pre-setting slip and guess values can lead to a
  reliable student model
• Modeling learning history with ADBNs will enhance
  the accuracy of the student model

32

• The significant difference between correct
  diagnostic rates using ABNs versus using ADBNs
  demonstrates that ADBNs enhance the accuracy of
  the student model, confirming the second
  hypothesis

33

• Pre-setting slipp/e, guessp/e, to relatively
  small (e.g. lt0.1) values produces accuracy of at
  least 97, confirming the first hypothesis.
• The accuracy is not sensitive to change of slip
  and guess values so along as the values are
  relatively small (lt0.1)

34
Evaluation of student model for CIMEL multimedia
with real students

• Hypotheses
• Pre-setting slip, guess will lead to a reliable
  student model
• Pre-setting slip and guess to various values will
  affect the accuracy of the student model
• The student model will perform in real-time,
  i.e., it will support responses as students are
  working on exercises
• Results
• The average response time of the student model
  after a students enters the solution step is 0.24
  seconds

35

• Presetting slip and guess with relatively small
  values (lt0.1) can produce accuracy of up to
  80.1.
• Changing the presetting slip and guess causes the
  accuracy to change from 71.7 to 80.1

36
(No Transcript)
37

• Has a higher value of r than student model by
  Corbett et al. (2000)

38
Evaluation of student model for DesignFirst-ITS
with real students

• Hypotheses
• Pre-setting slip, guess will lead to a reliable
  student model
• The student model will perform in real-time,
  i.e., it will support responses as students are
  working on problem-solving steps
• Results
• The average response time of the student model
  after a student enters a solution step is 0.63
  seconds

39

• Presetting slip and guess with relatively small
  values (lt0.1) can produce accuracy of up to
  81.8
• Varying the slip and guess value does not affect
  the accuracy of the student model, so long as
  slip and guess values are relatively small (lt0.1)

40

• Has a higher value of r than student model by
  Corbett et al. (2000)

41
Evaluation of diagnoses integration(multimedia
and DesignFirst-ITS) with real students

• Hypothesis
• Integrating diagnoses from closed-ended questions
  will enhance the accuracy of diagnoses of student
  model for open-ended questions

42

• Accuracy increases 7.7 when adding diagnoses
  from closed-ended exercises in multimedia,
  confirming hypothesis

43
Comparing with non-advanced-numerical student
models

• Non-advanced-numerical techniques include match,
  summation and subtraction
• Advanced-numerical techniques include Bayesian
  networks
• Hypotheses
• Straightforward algorithm will not lead to a
  reliable student model
• ADBNs will perform better than match student
  model

44

• From same set of evidences, ADBNs perform more
  than two times better than match student models

45
Presentation Outline

• Publications
• Background
• Related Research
• Research questions
• Methodology
• Evaluation
• Conclusions and future work

46
Conclusions

• Student models with ADBNs can diagnose student
  knowledge states accurately in real-time
• Accuracy of ADBN-based student model is
  significantly higher than ABN student model
• Integrating diagnoses from closed- and open-ended
  exercises is an effective way to increase
  accuracy of student models
• Student models using ADBNs perform much better
  than the student models that use straightforward
  algorithm

47
Future work

• Implement cognitive model to simulate monitoring
  knowledge and reflective knowledge
• Consider students learning gain from reviewing
  feedback
• how do we determine the conditional probability
  table for the ADBN so as to simulate the real
  student learning?
• how do we update the new ADBN?
• how do we convey empirical studies with simulated
  students and human subjects?
• Diagnose students learning state in other
  domains, such as object-oriented programming

48
An ADBN considers feedback
49
Contributions (1 of 2)

• A novel way to represent students knowledge
  structure, where both concepts and relationship
  between concepts are knowledge units that
  students need to learn
• A novel three-layered architecture which can be
  standardized in modeling various stratums of
  students knowledge
• ABN a novel Atomic Bayesian network that
  provides a refined representation of prerequisite
  relationships, diagnoses students knowledge
  structure, and guarantees real-time
  responsiveness

50
Contributions (2 of 2)

• ADBN an innovative dynamic Bayesian network
  that represents refined representation of
  prerequisite relationships and diagnoses
  students knowledge structure in real-time
  considering learning history
• A unique student model that integrates knowledge
  from open-ended problem solving (object-oriented
  class diagram design) and closed-ended exercises
• A general approach for student models that help
  students learn complex problem solving in real
  time

51
Questions?