Accelerating Future Possibilities for Assessment and Learning Technology-Enabled Measurement: Looking Back to Move Ahead

1
Accelerating Future Possibilities for Assessment
and LearningTechnology-Enabled
MeasurementLooking Back to Move Ahead

• Greg Chung

UCLA Graduate School of Education Information
StudiesNational Center for Research on
Evaluation,Standards, and Student Testing
(CRESST) Annual CRESST Conference, Los
AngelesSeptember 9, 2005
2
Organization of Talk

• Technology-Enabled Measurement
• Resolution
• Performance-Sensing Examples
• Circuit analyses
• Rifle marksmanship
• Automated Reasoning
• Data fusion and inferencing

3
Looking Back to Move Ahead

• Look back 3010 years for potential technologies
  and methods
• Internet (ARPA Net/ALOHA Net, 1972), relational
  DBs (IBM, 1970), sensing and telemetry (NASA,
  1960s-70s), online monitoring (Dominick, 1973)
• Constraint networks (Montanari, 1974),
  probabilistic causal reasoning (Suppes, 1970),
  knowledge representation (Minsky, 1975)
• Adaptive/personalized instruction/ teaching
  machines (Glaser, Lumsdaine, Keller, 1960-1970s)

4
Technology-Enabled Measurement

• The use of technology to measure and assess human
  learning and performance
• Principal advantage is that technology-enabled
  measurement provides information about processes
  that cannot be obtained feasibly in any other way
• Order of magnitude increase in the scope,
  frequency, and resolution of what can be measured

5
Resolution

• The degree of detail that can be distinguished in
  students ongoing performance
• Expose black-box processes
• Process-oriented
• Fine-grained

6
Low Resolution
7
High Resolution
We are here
8
Resolution

• How much do you really know about
• How much someone knows(or doesnt know) about
  something?
• What they can do (or cant do)?

9
Examples

• Circuit Analyses
• Performance sensing exposing students problem
  solving processes
• USMC Rifle Marksmanship
• Performance sensing exposing shooters fine
  motor control processes

10
Higher Education Example

• Large-Class Instruction
• Instructor marches along with
• Little direct feedback from students
• Little or no information about what students know

11
How much do you really knowwhether students are
getting it?
12
(No Transcript)
13
Instructors View
14
Instructor Observations

• All students in session participated, drastically
  improved interaction
• Clear and immediate feedback
• Rate of receiving questions and observing
  responses to problems is much higher than
  conventional sessions
• Method exceeds interactivity of one-on-one office
  hour visit

15
Example 2USMC Rifle Marksmanship
16
USMC Rifle Marksmanship
17
How much do you really knowabout these shooters?
18
Measuring Rifle Marksmanship Knowledge and Skills

• Rifle marksmanship is a combination of cognitive,
  affective, motor (gross and fine) skills, and
  uncontrollable equipment and environment
  variables
• Actual process of shooting is a black boxvery
  difficult to observe fine-motor control processes
• Practically impossible to observe all processes
  in parallel at the time the shot is taken

19
Why it Matters
Marine, HaitiNY Times, 3/8/04
Marine, IraqNY Times, 10/18/04
Soldier, Afghanistan 02NY Times, 8/2/04
20
Performance Sensing
21
Trigger Control
22
Sensing Example Remarks

• Able to sense black-box processes that were
  previously unobservableincrease in resolution
• Circuit analysesinference engine in instructors
  head, working toward automating approach
• Marksmanshipinferencing will be computer-based
• Highlights need for automated method for fusing
  data

23
Automated Reasoning
24
Automated Reasoning

• Data Fusion Problem
• Given disparate data types, large quantity of
  data, and inherent uncertainty in the data
• Need a way to reason about the data
• Bayesian networks useful as a way to model the
  phenomena
• Model phenomena causally
• Render probabilistic judgments about whether
  learner is in particular states

25
Circuit Analysis Bayesian Network
26
Towards Individualized Diagnosis and Prescription
Content
probabilities
Recommender
knowledge
individualized feedback and remediation
performance outcome
fine motor processes
27
A Really Bold Assertion

• Technology Will Revolutionize the Field
• Broaden the way we think about how we measure
  human learning and performance
• Technology-enabled measurement affords increased
  resolving power
• Through measurement to knowledge (Onnes, 1882)