Assessing Who Is Learning and How

1
Assessing Who Is Learning and How

• J. Michael Spector
• Learning Systems Institute, Florida State
  University
• Tallahassee, FL USA mspector_at_lsi.fsu.edu
• ASCILITE 2006
• Sydney, Australia
• 4 December 2006

2
Overview

• Adventures and advances
• Simplicity and complexity
• Who is learning what and how
• Internal and external representations
• Assessment using external representations
• Predictive power of current measures
• Assessing to support learning
• Issues and further research

3
Changes in Use of Technology
Percentage of public school classrooms in the USA
with Internet access 1994 1995 1996
1997 1998 1999 2000 2001
2002 2003 2005 3 8
14 27 51 64 77
87 92 93 94
According to TIMSS 2003, average math scores of
US 8th graders rose slightly from 492 in 1995 to
502 on 2001 to 504 in 2003 a similar slight
upward trend in science scores in 8th grade US
students was reported. However, when compared
with trends in other developed countries, these
slight gains are even less significant.
Meanwhile, according to IES (2006), the
percentage of US schools restricting and
filtering access to the Internet rose fro 91 in
2001 to 99 in 2005.
4
Learning

• Stable changes in abilities, attitudes,
  behavior, beliefs, mental models, skills

5
Adventures and Advances
6
Adventures
7
Advances

• Progress (?)
• Improved learning and performance
• Replicable results that scale up
• Relevant and reliable assessments
• Significant findings
• Time on task
• Practice with informative feedback
• Limited short-term memory
• Limited time
• Limited resources
• Excessive promises

8
Promises and Expectations Educational
Technologies

• Promises
• Two sigma improvements in learning outcomes
• No significant difference in many studies
• Expectations
• Technology will significantly improve learning
  and instruction
• Little foundation for this based on educational
  technology developments and applications in the
  last 50 years

9
Starting Over

• I am waiting for the great divide to be crossed
  and I am anxiously waiting For the secret of
  eternal life to be discovered by an obscure
  practitioner
• (Lawrence Ferlinghettis I am Waiting)
• Its never been my duty to remake the world at
  large
• (Bob Dylans Wedding Song)

10
Simplicity and Complexity (1/2)

• Avoiding simplicity, confronting complexity
• A state of affairs or the goal ?
• It is human nature to simplify
• We picture facts to ourselves (Wittgenstein,
  Tractatus, 2.1, 1922)
• Pictures are models of reality (2.12)

11
Simplicity and Complexity (2/2)

• To simplify or not to simplify
• that is not the question we cannot help
  ourselves mental models are simplifications
• Things are generally more complex than we are
  inclined to believe
• The principle of humility once accepted, this
  creates a conflict with our natural inclinations
• An approach-avoidance conflict
• Kurt Lewins Resolving Social Conflicts Field
  Theory in Social Science (1948)

12
Mental Model Development
13
A Model of Understanding
External Representations
Effective Action
Events
Mental Models
Meaning Making
Schema
14
Concept Maps

• External representations
• Involve language
• May involve pictures and graphics
• Used to develop individual and group
  understanding
• Many types (with many subtypes variants)
• Association networks (nearness in meaning)
• Petri networks (effects on other things)
• Semantic networks (meaning relationships)
• Causal influence diagrams (systemic relationships)

15
Complex System Dynamics Model
16
(No Transcript)
17
Complex Problem Domains

• Involve many components
• Internal relationships and feedback
• Non-linear relationships
• Delayed effects
• Multiple solutions and approaches
• Examples
• Curriculum planning
• Engineering design
• Medical care
• Social policy development
• Teams of people with different backgrounds

18
DEEP An Assessment Tool http//deep.lsi.fsu.edu
/DMVS/jsp/index.htm

• Problem determine progress of learning in
  complex domains
• Approach identify and annotate key influence
  factors
• Strategy compare responses to those of known
  experts and track development
• Tactic minimize extraneous cognitive load on
  respondents

19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
(No Transcript)
23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
(No Transcript)
27
Analysis and Findings

• Three levels of analysis
• Surface similarities
• Structural similarities
• Semantic similarities
• Expert and non-expert differences
• General differences
• Domain-specific differences
• Individual differences

28
Level 1 Analysis Medical Domain
29
Structural Semantic Analysis

• Intertwined due to resolving names of nodes and
  links
• Protocols based on expert analysis of the problem
  scenarios and a selected set of responses
• Individual responses coded based on protocols
• Similarities and differences among and between
  experts and novices

30
Medical Summary Data
5 Experts 14 Novices
Scenario 1
5 Experts 14 Novices
Scenario 2
31
Node-Link Clusters
32
Comparing Experts Novices
Biology Experts Scenario 2 (N 5 Links 128).
Biology Novices Scenario 2 (N 16 Links
147).
33
Additional Issues Measures

• Separating structural and semantic analysis
• Structural analysis
• Central nodes
• Terminal nodes (all links in same direction)
• Feedback and systemic measures
• Similarity metrics
• Graph theory diameter, density, path analysis
• Tversky similarity metric

34
A Metric for Systematicity

• Hypothesis experts will tend to think more
  systemically than non-experts
• Indicators of systemic thinking
• Internal feedback (two-way links and links to
  other parts of the system)
• Global effects (rather than localized effects)
• A measure ratio of unreachable pairs to all
  possible ordered pairs of nodes in the problem
  conceptualization

35
7 nodes possible ordered pairs 2,520 lots of
internal feedback depicted no unreachable
pairs No terminal nodes
36
7 nodes possible pairs much internal
feedback 6 unreachable pairs 1 terminal node
37
7 nodes little internal feedback 6 terminal
nodes 38 unreachable pairs
38
Research Summary

• Surface and structural analysis are promising
  indicators of relative levels of expertise
  measure of systemic thinking is most promising
• Confirmatory analysis using semantic analysis and
  verbal protocols should be continued
• Exploring task types and different domains should
  also be pursued

39
Model-Supported Learning and Instruction

• Models can support
• Illustration
• Demonstration
• Envisioning
• Simplification
• Reasoning
• Hypothesis Testing
• Policy Development
• Interactive Simulations
• Planning and Management
• Assessment and Evaluation
• Metacognitive Development

40
A Model ALT CurriculumIEEE Learning Technology
Technical Committee

• Introduction to advanced learning technologies
• Introduction to human learning
• Foundations, evolution and recent developments
• Typologies and taxonomies
• User perspectives
• Learner perspectives
• System perspectives
• Design requirements
• Design processes, architectures and learning
  objects
• Evaluation models and practices
• Social perspectives
• Emerging issues

41
Model Facilitated Learning
42
Story-Based Instruction

• Case-based instruction
• management, law, medicine, social sciences
• Scenario-based instruction
• technical training, tactics, sports
• Story-based instruction
• some aspects of cases and scenarios
• personal elements and authenticity
• active role for a story teller
• motivational, engaging, thought provoking

43
Dynamic StoriesStefanie Hillens Postodoctoral
Study

• Generating stories
• Personalized
• Adaptive
• Just-when-needed
• With existing simulation models
• Prior to a decision making point, use annotated
  aspects of the model to generate a situation that
  requires a user response based on the current
  condition of the model
• Then engage the learner in hypothesis formation
  and testing
• Sequence stories so they fit the learners
  progression from an inexperienced newcomer to a
  more experienced problem solver

44
Concluding Remarks

• What can we expect of new and improved
  educational technologies?
• Where should we be focusing our efforts?
• Why believe that the future will resemble the
  past?
• Surely it would be a remarkable coincidence if
  the world happened to conform to the limits of my
  imagination.

45
Links

• DEEP Data Collection Software
• http//deep.lsi.fsu.edu/DMVS/jsp/index.htm
• MOT Knowledge Modeling Tool
• http//www.licef.teluq.uquebec.ca/eng/index.htm
• CMAP Knowledge Models and Concept Maps
• http//cmap.ihmc.us/
• Enovate AS - Adapt-IT Blueprint Designer
• http//merdan.intermedia.uib.no/
• Powersim System Dynamics Modeling Software
• http//www.powersim.com/default.asp
• Hylighter Collaborative Annotation Tool
• http//www.hylighter.org/index.htm

46
References (1/4)

• Anderson, J. R., Bothell, D., Byrne, M. M.,
  Douglass, S., Leviere, C., Qin, Y. (2004). An
  integrated theory of the mind. Psychological
  Review, 111(4), 1036-1060.
• Andre, T. (1986). Problem-solving in education.
  In G. D. Phye T. Andre (Eds.). Cognitive
  classroom learning (pp. 169204). New York
  Academic Press.
• Andrews, G. Halford, G. S. (2002). A cognitive
  complexity metric applied to cognitive
  development. Cognitive Psychology, 45(2),
  153-219.
• Bransford, J. D., Brown, A. L., Cocking, R. R.
  (Eds.) (2000). How people learn Brain mind
  experience and school. Washington, DC National
  Academy Press.
• Chi, M.T.H., Feltovich, P.J., Glaser, R.
  (1981). Categorization and representation of
  physics problems by experts and novices.
  Cognitive Science, 5, 121-152.
• Christensen, D. L., Spector, J. M., Sioutine, A.,
  McCormack, D. (2000, August). Evaluating the
  impact of system dynamics based learning
  environments A preliminary study. Paper
  presented at the International System Dynamics
  Society Conference, Bergen, Norway, August, 2000.
• Collins, A., Brown, J. S., Newman, S. E.
  (1989). Cognitive apprenticeship Teaching the
  crafts of reading, writing, and mathematics. In
  L. B. Resnick (Ed.), Knowing, learning, and
  instruction Essays in honor of Robert Glaser
  (pp. 453-494). Hillsdale, NJ Lawrence Erlbaum.
• Dijkstra, E. (1972). The humble programmer.
  Communications of the ACM, 15(10), 859-866.
• Dörner, D. (1996). The logic of failure Why
  things go wrong and what we can do to make them
  right (R. Kimber R. Kimber, Trans.). New York
  Metropolitan Books. (Original work published in
  1989)
• Dummer, P., Ifenthaler, D. (2005, March).
  Planning and assessing navigation in
  model-centered learning environments Why
  learners often do not follow the path laid out
  for them. Presentation at the International
  Conference for Methods and Technologies for
  Teaching, Palermo, Italy, March 10, 2005.
• Ericsson, K. A, (1993). The role of deliberate
  practice in the acquisition of expert
  performance. Psychological Review, 100(3),
  363-406.
• Ericsson, K. A. (2001). Attaining excellence
  through deliberate practice Insights from the
  study of expert performance. In M. Ferrari (Ed.),
  The pursuit of excellence in education (pp.
  21-55). Mahwah, NJ Erlbaum.

47
References (2/4)

• Gardner, H. (1999). Multiple approaches to
  understanding. In C. M. Reigeluth (Ed.),
  Instructional design theories and models A new
  paradigm of instructional theory (Vol. II) (pp.
  6989). Mahwah, NJ Lawrence Erlbaum Associates.
• Herl, H. E., ONeil, H. F., Jr., Chung, G. L. W.
  K., Bianchi, C., Wang, S., Mayer, R., Lee, C. Y.,
  Choi, A., Suen, T., Tu, A.(1999). Final report
  for validation of problem solving measures. CSE
  Technical Report 501. Los Angeles CRESST.
• Johnson-Laird, P. (1983). Mental models.
  Cambridge Cambridge University Press.
• Jonassen, D. (1999) Designing constructivist
  learning environments. In C. M. Reigeluth (Ed.),
  Instructional design theories and models A new
  paradigm of instructional theory (Vol. II) (pp.
  215239). Mahwah, NJ Lawrence Erlbaum
  Associates.
• Jonassen, D. H. Hernandez-Serrano, J.(2002).
  Case-based reasoning and instructional design
  using stories to support problem solving.
  Educational Technology Research and Development,
  50(2), 65-77.
• Jonassen, D. H. (2006). Modeling with technology
  Mindtools for conceptual change (3rd ed.).
  Columbus, OH Pearson/Prentice Hall.
• Kintsch, E. (1993). Principles of instruction
  from research on human cognition. In J. M.
  Spector, M. C. Polson, D. J. Muraida,
  Automating instructional design Concepts and
  issues (pp. 22-42). Englewood Cliffs, NJ
  Educational Technology Publications.
• Lowyck, J., Elen, J. (2004). Linking ICT,
  knowledge domains, and learning support for the
  design of learning environments. In N. M. Seel
  S. Dijkstra (Eds.), Curriculum, plans and process
  in instructional design International
  perspectives (pp. 239-256). Mahwah, NJ Erlbaum.
• Lowyck, J., Pöysä, J., van Merriënboer, J.
  (2003). Conditions of ICT-based design for
  learning communities. Technology, Instruction,
  Cognition, and Learning, 1(2), 153-182.
• McCarthy, B. (1996). About learning. Barrington,
  IL Excell Inc.
• Merrill, M. D. (2002). First principles of
  instruction. Educational Technology Research and
  Development, 50(3), 43-59.
• Miller, G. A. (1956). The magical number seven,
  plus or minus two Some limits on our capacity
  for processing information. Psychological Review,
  63, 81-97.
• Milrad, M., Spector, J. M., Davidsen, P. I.
  (2002). Model facilitated learning. In S. Naidu
  (Ed.), Learning and teaching with technology
  Principles and practices (pp. 13-27). London
  Kogan Page.

48
References (3/4)

• Moreno, R. Mayer, R., Spires, H., Lester, J.
  (2001) The Case for social agency in
  computer-based teaching Do students learn more
  deeply when they interact with animated
  pedagogical agents? Cognition and Instruction,
  19(2), 177-213.
• Nelson, L. M. (1999). Collaborative problem
  solving. In C. M. Reigeluth (Ed.), Instructional
  design theories and models A new paradigm of
  instructional theory (Vol. II) (pp.241 267).
  Mahwah, NJ Erlbaum.
• Novak, J. D. (1998). Learning, creating and using
  knowledge Concept maps as facilitative tools in
  schools and corporations. Mahwah, NJ Erlbaum.
• Paas, F., Renkl, A., Sweller, J. (Eds.).
  (2003). Cognitive load theory. Educational
  Psychologist, 63(1), Whole Issue.
• Paivio, A. (1986). Mental representations A dual
  coding approach. Oxford, England Oxford
  University Press.
• Polson, M. C. (1993). Cognitive theory as a basis
  for instructional design. In J. M. Spector, M. C.
  Polson, D. J. Muraida, Automating instructional
  design Concepts and issues (pp. 5-22). Englewood
  Cliffs, NJ Educational Technology Publications.
• Salas, E., Fiore, S. (Eds.) (2004). Team
  cognition Understanding the factors that drive
  process and performance. Washington, DC American
  Psychology Association.
• Schank, R. C., Berman, T. R. Macperson, K. A.
  (1999). Learning by doing. In C. M. Reigeluth
  (Ed.), Instructional design theories and models
  A new paradigm of instructional theory (Vol. II)
  (pp. 161181). Mahwah, NJ Lawrence Erlbaum
  Associates.
• Schvaneveldt, R. W. (Ed.) (1990). Pathfinder
  associative networks Studies in knowledge
  organization. Norwood, NJ Ablex.
• Schwartz, D., Lin, X., Brophy, S., Bransford,
  J. D. (1999). Toward the development of flexibly
  adaptive instructional designs. In C. M.
  Reigeluth (Ed.), Instructional design theories
  and models A new paradigm of instructional
  theory (Vol. II) (pp. 183213). Mahwah, NJ
  Lawrence Erlbaum Associates.

49
References (4/4)

• Seel, N. M. (2001). Epistemology, situated
  cognition, and mental models Like a bridge over
  troubled water. Instructional Science, 29(4-5),
  403-427.
• Seel, N. M. (2003). Model-centered learning and
  instruction. Technology, Instruction, Cognition,
  and Learning, 1(1), 59-86.
• Seel, N., Al-Diban, S., Blumschein, P. (2000).
  Mental models and instructional planning. In J.
  M. Spector T. M. Anderson (Eds.), Integrated
  Holistic Perspectives on Learning, Instruction
  Technolog (pp. 129-158). Dordrecht, The
  Netherlands Kluwer.
• Spector, J. M. (2000, Fall). Trends and issues in
  educational technology How far we have not come.
  Update Semiannual Bulletin 21(2).
• Spector, J. M. (2001). A philosophy of
  instructional design for the 21st century?
  Journal of Structural learning and Intelligent
  Systems 14(4), 307-318.
• Spector, J. M., Anderson, T. M. (2000).
  Integrated and holistic perspectives on learning,
  instruction and technology Understanding
  complexity. Dordrecht Kluwer.
• Spector, J. M., Koszalka, T. A. (2004). The
  DEEP methodology for assessing learning in
  complex domains (Final report the National
  Science Foundation Evaluative Research and
  Evaluation Capacity Building). Syracuse, NY
  Syracuse University.
• Spector, J. M., Dennen, V. P., Koszalka, T. A.
  (2005, July). Causal maps, mental models and
  assessing acquisition of expertise. Proceedings
  of the 11th Annual Human Computer Interaction
  International Conference, Las Vegas, Nevada,
  July, 2005.
• Spiro, R. J., Feltovich, P. J., Jacobson, M. J.,
  Coulson, R. L. (1991, May). Cognitive
  flexibility, constructivism, and hypertext
  Random access instruction for advanced knowledge
  acquisition in ill-structured domains.
  Educational Technology, 31 (5), 24-33.
• Stoyanova, N., Kommers, P. (2002). Concept
  mapping as a medium of shared cognition in
  computer-supported collaborative problem solving.
  Journal of Interactive Learning Research,
  13(1/2), 111-133.
• Suppes, P. (1978). Impact of research on
  education Some case studies. Washington, DC
  National Academy of Education.
• Sweller, J. (2003).  Evolution of human cognitive
  architecture.  The psychology of learning and
  motivation, 43, 215266.
• Taricani, E. M., Clariana, R. B. (2006). A
  technique for automatically scoring open-ended
  concept maps. Educational Technology Research
  Development, 54(1), 65-82.
• van Merriënboer, J. J. G. (1997). Training
  complex cognitive skills A four-component
  instructional design model for technical
  training. Englewood Cliffs, NJ Educational
  Technology Publications.

50
Level 1 Analysis Scenario 1
51
Level 1 Analysis Scenario 2
52
Level 1 Analysis - Biology
53
Level 1 Analysis - Engineering
54
Biology Summary Data
5 Experts 14 Novices
Scenario 1
5 Experts 14 Novices
Scenario 2
55
Engineering Summary Data
5 Experts 18 Novices
Scenario 1
5 Experts 18 Novices
Scenario 2
56
Mental Model Representation
Number
Differences
Time Required
Problem Conceptualizations
Shared Representations
Tasks Accomplished