Zdeslav Hrepic Dean A' Zollman N' Sanjay Rebello

1
Issues in Addressing and Representing Hybrid
Mental Models

• Zdeslav HrepicDean A. ZollmanN. Sanjay Rebello

AAPT, Sacramento
Fort Hays State UniversityKansas State University
Supported by NSF ROLE Grant REC-0087788
2
Outline

• Rationale Why use in-class, real-time
  assessment?
• Previous research
• Mental models of sound propagation.
• Hybrid mental models and their role.
• Test construction and validation
• Results
• Using the test
• Further study

3
Goal of the study

• To create a multiple choice test
• that can elicit students mental models of sound
  propagation
• during the lecture
• using a class response system and appropriate
  software.

4
Real time, in class assessment
Enables quick collection and immediate analysis
of students responses in the classroom.
Uses some form of Class Response System
5
Benefits of class assessment

• Engages students.
• Facilitates interactive learning and peer
  instruction (especially in large enrolment
  classes).
• Gives immediate feedback to the teacher.
• Enables the teacher to adjust the teaching before
  the exam rather than after it and according to
  specific needs of his/her students.
• Allows a post lecture detailed analysis.

6
Research questions

• Main question
• What is the optimal multiple choice test that can
  elicit students mental models of sound
  propagation in a real time, during the
  instruction?
• Sub questions (Addressed in the presentation)
• What is the optimal analytical tool for analysis
  of students responses in this test?
• How do we represent data so the display provides
  a variety of instruction guiding information?

7
Starting point in test creationIdentifying
mental models of sound propagation

• Wave Model - Scientifically accepted model
• Independent Entity Model - Dominant alternative
  model Sound is a self-standing, independent
  entity different from the medium through which it
  propagates.
• Hybrid models - Composed of entity and wave model
  features and at the same time they are
  incompatible with both the entity and the wave
  models. (E.g.)

Hrepic, Z., Zollman, D., Rebello, S. (2002).
Identifying students' models of sound
propagation. Paper presented at the 2002 Physics
Education Research Conference, Boise ID.
8
Mental models of Earth
Mixed Model State
Hybrid Models
Target model
Initial model
9
MetaphorMental Models and Model states
Horse
Hybrid Mule
Donkey
A mule hybrid of a donkey a horse. A horse
64 chromosomesA donkey 62 chromosomesA
mule 63 chromosomes http//www.luckythreera
nch.com/muletrainer/mulefact.asp
10
Model States(In terms of childrens mental
models of Earth Vosniadou, 1994)
Mixed Model State
Hybrid Model State
Pure Model 2 State
Pure Model 1 State
Instance1
Instance2
11
Hybrid mental models identified in domains of

• Earth science(Vosniadou, 1994)
• Electrostatics(Otero, 2001)
• Newtonian mechanics(Hrepic, 2002 Itza-Ortiz,
  Rebello, Zollman, 2004)
• Sound(Hrepic, 2002 Hrepic, Zollman, Rebello,
  2002).
• Optics(Galili, Bendall, Goldberg, 1993)
  (hybridized knowledge)
• Inertia and gravity(Brown Clement, 1992)
  (intermediate concepts)

12
Implications of hybrid mental models

• Implications for teaching
• A student can give a variety of correct answers
  on standard questions using a hybrid (wrong)
  model.
• Implications for analysis of our test
• Hybrid models cause overlaps in multiple choice
  questionnaires more than one model corresponds
  to the same choice.
• Complexity 3 questions define a model
• Model analysis requires that each answer choice
  is uniquely associated with a model.

13
Model States
x
Knowledge elements related to Model 1 only
Knowledge elements related to both models or
neither one
Knowledge elements related to Model 2 only
NoModelState
Mixed Model State
Hybrid Model State
Pure Model 2 State
Pure Model 1 State
x
x
x
x
x
Instance1
x
x
x
x
x
x
x
x
x
x
x
Instance2
x
x
x
x
x
x
x
14
4 basic models - mechanisms of propagation
15
4 basic models - mechanisms of propagation
Wave ModelScientifically Accepted Model
() Ear Born Sound
Propagating Air
Hybrid Models
Dependent Entity
Independent Entity Dominant Alternative Model
16
Pilot testing

• Did we miss anything in terms of mental models?
• Open-ended questionnaire on a large sample
• Did we miss anything in terms of productive
  questions to determine students mental models?
• Battery of semi-structured conceptual questions
  related to sound as a wave phenomena in variety
  of situations

17
Test Contexts1. Air
How does sound propagate in this situation?
18
Test Contexts2. Wall
How does sound propagate in this situation?
19
Test Contexts1a, 2a - Vacuum
What happens without the medium (air or wall)?
20
Test questions - paraphrased

• What is the mechanism of sound propagation in the
  air/wall?
• How do particles of the medium vibrate, if at
  all, while the sound propagates?
• How do particles of the medium travel, if at all,
  while the sound propagates?
• What does this motion have to do with sound
  propagation cause and effect relationship?
• What does this motion have to do with sound
  propagation time relationship?
• What happens with sound propagation in the vacuum?

21
Displaying the test results

• Several representations of students state of
  understanding
• Available in real time and in post instruction
  analysis
• Consistency
• Consistent a student uses one model(Pure model
  state)
• Inconsistent a student uses more than one
  model(Mixed model state)

22
Using a particular model Pre Instruction
Calculus based University NY
Inconsistently
Consistently
N 100
23
Using a particular model at least once Pre
Instruction Calculus based University NY
Inconsistently
Consistently
N 100
24
Movements of particles of the medium Pre
Instruction Calculus based University NY
() Random Travel
() Travel Away From The source
Vibration on the Spot
N 100
25
Model states Pre Instruction Calculus based
University NY
Mixed Any
Pure Other
Mixed Entity
Pure Wave
Mixed Ear-Wave
N 100
26
Correctness Pre Instruction Calculus based
University NY
N 100
27
Using a particular model Pre Instruction
Calculus based University NY
Inconsistently
Consistently
N 100
28
Using a particular model Post Instruction
Calculus based University NY
Inconsistently
Consistently
N 95
29
Movements of particles of the medium Pre
Instruction Calculus based University NY
() Random Travel
() Travel Away From The source
Vibration on the Spot
N 100
30
Movements of particles of the medium Post
Instruction Calculus based University NY
() Random Travel
() Travel Away From The source
Vibration on the Spot
N 95
31
Correctness Pre Instruction Calculus based
University NY
N 100
32
Correctness Post Instruction Calculus based
University NY
N 95
33
Test validityBuilt and shown through

• Interviews with students
• Expert reviews
• Role playing validation with experts
• Validity strengthening test development
  procedures
• Tables of content and construct specifications
• Meaningful correlations between all answer
  choices
• Instructional sensitivity of the test
• Stability (reliability) of results obtained in
  the large scale survey
• across different educational levels
• across different institutions at equivalent
  educational levels
• across different course levels at same
  institutions

34
Constructing the test

• Four steps of test construction and validation
• Pilot testing
• large open ended survey settling on models,
  choosing contexts
• Pre-survey testing
• expert validation, 7 choice survey (with none of
  the above, more than one of the above),
  correlation analysis of answer choices,
  refinement through interviews
• Survey testing
• large scale survey correlation analysis,
  comparisons between levels, pre-post results
  interview validation
• Post Survey testing
• moderately large scale survey, role playing,
  expert validation

35
Survey participants
36
Survey phase - Validity interviews

• 17 x 4 probes in the interviewed sample.
• The invalid display of a model would have
  occurred in 6 instances (out of 68).
• 8.8 of the probes
• 3 instances because of 5a
• ( another 3 that did not cause invalid probe)

37
Correlation analysis of answer choices
38
Post-Survey Testing

• Expert review
• To validate post survey version
• Few minor items improved
• Surveying
• To determine correlations between response items
  and see if changes made the desired effect.
• Problems fixed
• Role playing validation
• To validate new test version in an additional way
• Perfect score

39
Comparing model distributionDifferent
educational levels
40
Comparing model distribution Grouped models
Different Educational Levels
41
Comparing model distributionDifferent course
levels
42
Comparing differences in model distributionVariab
ility within different educational levels
43
Pre-Post instruction difference
Gain (G) (post-test) (pre-test) Normalized
gain (h) gain / (maximum possible gain) (Hake,
1997).
44
Test packageProspective uses of test, test
questions

• Online package related to test and analysis of
  data available at http//web.phys.ksu.edu/role/so
  und/
• Formative assessment combined with any
  instructional method/approach
• traditional
• progressive
• misconception oriented
• Model cause
• Misconception symptom
• As peer instruction questions (not model
  defining)
• Not recommended as a summative assessment

45
Limitations

• Common to multiple choice tests
• Answer options do affect students understanding /
  models
• Test taking strategies may obscure results
• Test projects no model state as mixed model state
  and possibly pure model state.

46
Future researchUnique approach - Wide themes
opened

• Applicability of the approach in other domains of
  physics
• Is the approach hybrid model-(in)dependent?
• Applicability in domains of other natural
  sciences?
• How effectively teachers can implement the
  real-time aspect of this testing approach?
• Instructional utility of this type of testing
  Will addressing of the underlying models in real
  time help students learn?
• Possibility of individualized addressing of
  students models in real time?
• Applicability of the testing approach in
  eliciting non-cognitive psychological constructs
• Personality tests Would it provide information
  that current tests in that field do not?
• Reduction of items when compared to Likert scale

47
Future researchSpecific issues opened

• Optimal using of the test in combination with
  online homework
• Saving of time
• Any classroom benefit counterbalance?
• How applicable is this test at the middle school
  level?
• How would a branched version of the test look,
  and would it have any advantages with respect to
  this one?
• Improved simplicity and validity of the test

48
More Information / Feedback
zhrepic_at_fhsu.edu www.fhsu.edu/zhrepic(www.hrepi
c.com)
Thank You!
49
Literature

• Brown, D., Clement, J. (1992). Clasroom
  teaching experiments in mechanics. In R. Duit,
  Goldberg, F. , Niedderer, H. (Ed.), Research in
  physics learning Theoretical issues and
  empirical studies (pp. 380-389). Kiel IPN.
• Galili, I., Bendall, S., Goldberg, F. M.
  (1993). The effects of prior knowledge and
  instruction on understanding image formation.
  Journal of Research in Science Teaching, 30(3),
  271-301.
• Hrepic, Z. (2002). Identifying students' mental
  models of sound propagation. Unpublished Master's
  thesis, Kansas State University, Manhattan.
• Hrepic, Z., Zollman, D., Rebello, S. (2002).
  Identifying students' models of sound
  propagation. Paper presented at the 2002 Physics
  Education Research Conference, Boise ID.
• Itza-Ortiz, S. F., Rebello, S., Zollman, D. A.
  (2004). Students models of Newtons second law
  in mechanics and electromagnetism. European
  Journal of Physics, 25, 8189.
• Otero, V. K. (2001). The process of learning
  about static electricity and the role of the
  computer simulator. Unpublished Ph.D.
  Dissertation, University of California, San
  Diego, CA.
• Vosniadou, S. (1994). Capturing and modeling the
  process of conceptual change. Learning
  Instruction, 4, 45-69.
• Greca, I. M., Moreira, M. A. (2002). Mental,
  physical, and mathematical models in the teaching
  and learning of physics. Science Education,
  86(1), 106-121.
• diSessa, A. A. (2002). Why "conceptual ecology"
  is a good idea. In M. Limon L. Mason (Eds.),
  Reconsidering conceptual change Issues in theory
  and practice (pp. 29-60). Dordrecht, Netherlands
  Kluwer Academic Publishers.
• Hrepic, Z., Zollman, D., Rebello, S. (2002).
  Identifying students' models of sound
  propagation. Paper presented at the 2002 Physics
  Education Research Conference, Boise ID.
• Vosniadou, S. (1994). Capturing and modeling the
  process of conceptual change. Learning
  Instruction, 4, 45-69.
• Physics Education Group at Arizona State
  University. (2000). Modeling Instruction Program
  www. Arizona State University. Retrieved 24.
  Aug. 2003, 2003, from the World Wide Web
  http//modeling.la.asu.edu/
• Clement, J. M. (2003). Re testing to
  discriminate between students vs other
  approaches PhysLrnR post of 18 Apr 2003 103546
  -0500 online at lthttp//listserv.boisestate.edu/c
  gi-bin/wa?A2ind0304LphyslrnrFSX412D9A0AB9
  B02985B9Yzhrepic_at_phys.ksu.eduP6582gt.
• Hanna, G. S. (1993). Better teaching through
  better measurement. Orlando, Florida Harcourt
  Brace Jovanovic, Inc.
• Oosterhof, A. (2001). Classroom applications of
  educational measurement. Upper Saddle River, New
  Jersey Prentice Hall, Inc.