Accurately Measuring Skills Assessment

1
Accurately MeasuringSkills Assessment

• Dave Lapointe
• Praxis Technical Group, Inc.

2
Personal Introduction

• Thank you to
• Jeff McSorley
• Satpal Sidhu
• And all the NWPTA organizers and participants
• About me
• Dave Lapointe
• At Praxis for total around 10 years
• Working on learning systems
• Software research and development

3
Topic Introduction

• Some questions
• Some examples
• Some definitions
• Some suggestions
• Questions

4
A Couple Definitions

• Skills Assessment - Identify and measure a
  collection of desirable behaviours within a
  well-defined context.
• Measurement Accuracy - Gauging the success of
  the contexts measurement terms.

5
Some Questions

• What do we want to measure/assess?
• How can we know we are measuring it?
• Is learning happening?
• Are there more effective ways to teach?
• How can we know what is more effective?

6
Example Pattern Recognition
7
Example Process Upset

• Assume clearly defined procedures exist.
• Does passing a quiz ensure qualification?
• Does running a simulator ensure qualification?
• What about
• Stress?
• Problem diagnosis methods?
• Communication with other people?
• Variations on the upset condition?

8
What do we want to measure?

• Clear statements of the objectives - dimension
• Dimension is an unambiguous axis for
  measurement.
• Identify dimensions of the topic. Eg.
• Quizzes for basic information/content
• Process simulations for practice and actionable
  comprehension
• Games to simulate real-life interactions

9
Dimensions

• Each different type of measurement can be defined
  as a dimension
• Dimensions can be independent or interdependent
• Quiz scores are an independent dimension
• Judgment score could be composed of
• Quick response
• Appropriate response
• Prioritization of response steps

10
How many Dimensions?

• Information can be lost through projecting
  dimensions.
• Too many dimensions dilutes meaningful
  measurement.

11
Assessment Strategies

• Adding relevant, measurable dimensions to the
  assessment process increases the likelihood of
  accuracy.
• Each type of assessment is a dimension
• Results clustering is more meaningful with more
  dimensions (up to a point).
• Some dimensions may be a bit subjective, but can
  be accommodated.

12
Measurement

• Reliably measure each dimension
• Draw upon other related learning experiences
• Lots of statistical samples with broad range of
  skill sets.
• Should see results appearing in clusters
  throughout assessment dimensions
• Wide scatter indicates low assessment reliability
• But were only adjusting probabilities.
• Latency.

13
Types of Measurement
14
Quizzes(Measuring knowledge retention)

• Quite straightforward
• Percentages
• Given throughout course

15
Simulations(Measuring Process Knowledge)

• Possible dimensions are
• Appropriate response ranges
• Appropriate response times
• Deviation from ideal state(s)
• Eg
• Several small, quick range responses may
  minimize deviation from ideal states
• Too slow but appropriate range response

16
Games(Measuring Judgment)

• Some additional dimensions
• Procedure comprehension (right order,
  communications)
• Stress response (observe other dimensions while
  increasing system difficulty)
• Deduction/Inference (system interactions -
  checking items, communications, etc).
• Alternative responses (deviation from standard
  response)

17
Game Measurement (contd)

• Metaphor of system health
• Student response demand is transient.
• Overall assessment of all relevant dimensions.
• Relatively easy to calculate.
• Feedback for adjusting play difficulty.
• Effectively, its the stress dimension.
• Dependent dimension
• Eg.
• Deviation from ideal states
• Response timeliness

18
System Health
19
Adaptive Systems

• System Health can serve as the basis
• Increase difficulty
• false information eg. bad data, calibration
  errors, stochastic behaviour
• Partial system failures
• Increased distractions red herrings
• Decrease difficulty
• Intervention by controlling parts of system
• Active guidance to restore System Health

20
Multitask Systems

• Several discreet but related contexts or
  dimensions
• Concurrency is important
• Inference through analysis of clues
• Response to system demands
• Diagnosis using several different modes
• Understanding side-effects
• Interaction with avatars
• Judging what is or isnt reliable data

21
One Theory of Learning

• Activation of prior experience
• Demonstration of skill
• Application of skill
• Integration of skills into real-world activities

Merrill, MD (2003) First Principles of
Instruction
22
Is Learning Happening?

• Interest/Motivation
• Demonstrated knowledge
• We can learn a lot from commercial games
• Very intrinsically motivating (immediate
  gratification)
• Balance between interesting but not frustrating
• Many dimensions of interaction
• Adapt to players actions
• MMORPGs are very popular

23
Is Learning Happening? (contd)

• Can games map directly into learning systems?
• Not really, but they are still effective tools
  for teaching and assessment.
• Stress dimension
• Adaptiveness of system
• System/Player health metaphor as overall
  scorecard
• Motivation

24
More Effective Ways to Teach?

• Research has shown that traditional ways of
  teaching are not the most effective
• People learn in a variety of ways
• Prior experience or context.
• Good/poor results can mean different things
• Learning from incorrect behaviour
• Assessment in context
• Adjust teaching to the student
• Appropriate intervention (mentoring)

25
Measuring Effective Teaching

• Open feedback (critical and positive)
• Tracking student progress after courses
• Experiences of other teachers (throughout
  industry)
• Consensus on terms and assessment for basic
  qualification
• Sharing results

26
Some Strategies forEffective Learning

• Teach through many different ways
• Group learning and assessment
• Computer Based Training
• Classrooms
• Simulations
• Automate only what makes sense to automate

27
Student Directed Content

• Assess how students learn, and adjust course on a
  per student basis.
• Eg. Read or Try
• Some students learn by reading
• Some students learn by trying things out
• Students as teachers
• Forums
• Wikis
• In class
• Learning from others mistakes

28
Conclusions

• We must be aware of the limitations of our
  teaching and testing methods
• Presentation
• Assessment
• Motivation
• We are still only working with probabilities

29
Conclusions (contd)

• Learning is reflected through change in behaviour
• Goal of teaching is directed behaviour change
• Must be very clear about defining
• What we are measuring
• How we are measuring it
• Creating measurable, meaningful dimensions for
  assessment increases the odds of being accurate.
• Validating the measurements through history and
  sample sets.
• We can learn a lot from direct and indirect
  student feedback.

30
Addendum - Capturing Knowledge

• Retaining the knowledge of experience
• Centrally managed knowledge stores
• Often difficult and time consuming to manage
• Can become out-of-date and/or inflexible
• Limited participation
• Look what happened to the Incas
• Other extreme is no better (chaos)
• The Miscellaneous category
• Knowledge can effectively become white noise

31
Addendum - Wiki

• Wikis manage the region in between these two
  extremes.
• Community-driven repository of knowledge.
• Built-in document version control.
• Self-regulating through peer-review and
  recognition.
• Complements other teaching/learning models.
• Can validate effectiveness of teaching systems.
• Community is a large motivator.

32
Open Source model

• Defn Open Source describes practices in
  production and development that promote access to
  the end product's sources. The term open source
  gained popularity with the rise of the Internet
  and its enabling of diverse production models,
  communication paths, and interactive communities

http//en.wikipedia.org