ASSESSMENT OF ACCELERATED LEARNING

1
ASSESSMENT OF ACCELERATED LEARNING

• Harry O'Neil
• Professor of Educational Psychology and
  Technology
• University of Southern California National
  Center for Research on Evaluation, Standards, and
  Student Testing (CRESST)

Defense Science and Technology Advisory
Group July 22-23, 2008
2
Accelerated Learning Definitions

• Any instruction that attempts to control for time
  spent versus content learned (Dee Andrews, email
  7/10/2008)
• Accelerated pace of gaining competency in
  military jobs, assignments, missions (Fletcher,
  email April 1, 2008)
• Moving novices to experts quicker (DARPA)

3
GENERAL RESEARCH QUESTIONS

• Can technology in near term provide technical
  quality improvements and cost savings over other
  testing approaches for accelerated learning?
• real time scoring of open-ended tasks
• baysian statistical approaches for
  diagnosis/prescription
• manage skill decay
• What are the reliability and validity properties
  of these computer-based assessments?

4
PURPOSES OF TESTING AND ASSESSMENT
Individual-TeamOriented
SystemOriented
System monitoring Policy formulation Program
evaluation Program planning Institutional
accountability Personnel accountability Instruct
ional improvement
Individual/team certification Admissions and
selection Placement Individual progress and
student learning Diagnosis
5
Model-Based Assessment

• Starts with model of learning not content
• Draws on elements from learning and assessment
  scientific knowledge
• Empirically developed
• Domain-independent and domain-dependent (domain
  to be measured in the particular instance) are
  combined
• Reusable components result in time and cost
  savings
• Evidence of technical quality obtained
• Computer designed, for either paper based or
  computer administered

6
INSTRUCTION DESIGN ISSUES IN ACCELERATED LEARNING

• Models of Learning
• Analytic Approaches to Assessment
• Empirical approaches to Assessment
• Task based approaches vs. competency approaches

7
CRESST MODEL OF LEARNING
Content Understanding
Collaboration
Learning
Problem Solving
Communication
Self-Regulation
8
Affective/Motivation Model of Learning
Play
Affective Learning
Test Anxiety
Academic Goals
Effort
Self-Efficacy
9
Social Capital
Trust
Teamwork Skills
LearningOrganization
CollectiveEfficacy
Networks
CollectiveEffort
Transparency
10
(No Transcript)
11
Agility in DoD

• Strategic Objective
• Globalization and technological advances have
  contributed to an uncertain security environment.
• Protecting the United States in the face of this
  uncertainty requires greater agility.
• Agility allows the total force to adapt rapidly
  to changing requirements through the flexible use
  of diverse individual and organizational
  capabilities linked to needed military outcomes.
• Competencies
• Enduring, portable, describe individual
  attributes needed to perform successfully on the
  job.
• Human Capital Strategy
• David Chu (6/6/06)

12
COLLABORATIVE PROBLEM-SOLVING
Problem-Solving Strategies
Content Understanding
Group Teamwork Process
Self-Regulation
1. Adaptability 2. Coordination 3. Decision
Making 4. Interpersonal 5. Leadership 6.
Communication
Environmental Science Domain Knowledge
1. Browsing 2. Searching 3. Boolean
Operator Used 4. Feedback Accessing
1. Planning 2. Self-Checking 3. Effort 4.
Self-Efficacy
13
CRESST MEASUREMENT OF PROBLEM SOLVING

• Context is program evaluation of technology-based
  innovations.
• Computer-based administration, scoring and
  reporting
• Content understanding is measured by a knowledge
  map.
• Domain-specific problem-solving strategies are
  measured on a search task by looking at search
  behavior and the use of the information found.
  Search for information on concepts they are
  uncertain of so as to improve their content
  understanding.
• Self-regulation measured by a paper-and-pencil.
• Reporting
• Problem solving would be a profile of three
  scores (content understanding, problem-solving
  strategies, and self-regulation)

14
Analytic Approach Check Validity of
Instructional Strategy in Accelerated Learning

• Often inductive discovery approach
• Usually doesnt result in learning (Kirschner, P.
  A., Sweller, J., Clark, R. E. (2006). Why
  minimal guidance during instruction does not
  work An analysis of the failure of
  constructivist, discovery, problem-based,
  experiential, and inquiry-based learning.
  Educational Psychologist, 41, 75-86.)
• What Works in Distance Learning
• Good instructional practices that can be applied
  to accelerated learning

15
Conceptual Issues

• Retention vs. transfer tradeoff
• Different instructional strategies and different
  assessments
• Transfer has multiple meanings
• Testing knowledge that has not been explicitly
  taught (Mayer) vs. testing knowledge that
  transfers to job setting (Kirkpatrick).

16
Conceptual Issues (Cont.)

• To accelerate learning with same retention with
  standard techniques.
• ISD (teach less) results in approximately 30
  reduction in time.
• ISD plus technology results in an additive
  percentage reduction (x).
• Tailor by prior knowledge.
• -people who have knowledge are not taught it.
• Leverage motivation to accelerate learning

17
Conception Issues (Cont.)

• To accelerate with same transfer with standard
  techniques
• Trade off retention
• Manipulate instruction (e.g. varied example) with
  more time.
• Skill decay
• Declarative knowledge is less often forgotten
• Procedural knowledge is more often forgotten
• Initial acquisition is best predictor of skill
  decay
• Feedback Matters

18
FOUR CHARACTERISTICS OF FEEDBACK

• Complexity of feedback
• What information is contained in the feedback
  messages
• Degree of elaboration
• Timing of feedback
• When is the feedback given to students
• In complex learning delayed better than immediate
• Representation of feedback
• The form of the feedback presented (text vs.
  graphics)
• Frequency of Feedback
• Lowering percentage of feedback
• -Slows down the Acquisition of Knowledge
• -But facilitates Transfer of Knowledge

19
Measurement Challenges

• Performance assessment
• Lack of psychometrics for Simulation/games
• Questions
• How is difficulty estimated?
• How many tasks are needed?
• Live with the reality that performance assessment
  and time constraints lead to few tasks. Conduct
  generalizability studies.
• What is role of gender, ethnicity, and prior
  experiences?
• Best strategy to develop for useful, valid, fair
  and feasible measures?
• Should constructs be reported as single vs.
  multiple scores as profile and/or levels?

20
Acceleration With Non Standard Techniques

• Just in time and right sized instruction
• parcels
• Just in time assessment

21
Instructional Parcels

• Short slices of instruction (lt 5min) targeted to
  a single concept
• Delivered online using video with narration
• Represent the domain using an ontology
• Estimate what the learner knows about a domain
  given performance data on assessments
• Use Bayesian networks to fuse assessment data and
  infer understanding of domain and topics within
  domain
• Conducted two studies to test individualized
  diagnoses and prescription with instructional
  parcels

22
Diagnosis / Remediation System
background
Domain Ontology
knowledge
Model of Knowledge and Performance Dependencies
content
identified knowledge and skill gaps and skill
decay
Recommender
motivation
affective states
physical states
cognitive states
Just in time right sized instruction and
assessment
sensor-based measurements
23
Another Alternative for Accelerated Learning

• Teach people to be smarter
• Goal is to improve cognitive and affective skills
  of personnel
• Rationale
• Effective training is a product of quality of
  instruction and trainees with effective learning
  strategies
• Many trainees lack learning strategies

24
Learning Strategies Course (cont.)

• Initial Evaluation Criteria
• Does curriculum meet its objectives
• Cognitive and affective posttests, user
  acceptance, time savings
• Final Evaluation Criteria
• Trainees with exposure to learning strategies
  learn more or quicker in subsequent courses

25
Research Recommendations

• Invest more money in accelerated learning
  research.
• Common theme of ASB, DSB and NRC studies
• Create mechanism in each Service for transition
  of successful research
• Create a cost-effectiveness rationale that
  rewards agencies that have same transfer while
  reducing time or increases transfer while keeping
  time the same
• Cherry-pick DARPA accelerated learning program
  for transitions POL Amy Kruse, DARPA/DSO

26
Research Recommendations (Cont.)

• Research of motivation and accelerated learning
• Create test beds for cost-effectiveness studies
• Augmented cognitive load theory with improved
  measurement and motivation
• John Sweller-UNSW
• Research for teams accelerated learning
• Create software pipeline from Basic Training to
  A-Schools to C-Schools to fleet
• Diagnostic/prescriptive system
• Just in time instruction and just in time and
  right sized assessment in context of training
  pipelines.

27
CRESST Web Site
http//www.cresst.org or any search engine type
CRESST
honeil_at_usc.edu
28
BACK-UP SLIDES
Page 28
29
Advanced Statistical Methods

• CRESST value-added model features
• 4-Level Latent Variable Regression Hierarchical
  Model
• Applied to multiple-cohorts longitudinal data
• Cohort-to-cohort value-added estimate
• Gap parameter capturing how equitably student
  achievement is distributed within a school
• Possible to adjust differences in student
  characteristics (Free/Reduced Lunch status,
  minority, LEP), cohort characteristics
  (time-varying of FRL in each cohort), and
  school characteristics (time-invariant factors)

30
Modeling Issues (2b)

• A domain ontology rifle marksmanship (Protégé)

Concepts and relations between concepts, in a
database
31
Cognitive Load Theory

• Intrinsic cognitive load depends on the extent
  of element interactivity of the materials or
  tasks that must be learned (van Merriënboer
  Sweller, 2005 Paas, Renkl, Sweller, 2003)
• Germane cognitive load associated with processes
  a learner engages in that are beneficial for
  learning (Sweller, van Merriënboer, Paas, 1998)
• Extraneous cognitive load caused by mental
  activities during learning that are unnecessary
  and so interfere with schema acquisition and
  automation (Paas, Renkl, Sweller, 2003)
• Learning material should be designed to minimize
  extraneous cognitive load and optimize germane
  cognitive load (Cooper, 1998 Sweller, van
  Merriënboer, Paas, 1998)

32
TEAMWORK PROCESSES
33
What Works in Distance Learning

• Develop a core set of research-based DL
  Guidelines
• Instructional Design (Clark)
• Multimedia (Mayer)
• Learning Strategies (Dembo)
• Assessment (Baker)
• Management Strategies (Kazlauskas)
• Self-Regulation and Motivation (ONeil)
• Lessons based on guidelines

ONeil, H. F. (Ed.). (2005). What works in
distance learning Guidelines. Greenwich, CT
Information Age Publishing Inc. ONeil, H. F.
(Ed.). (2003).What works in distance learning
(Report to the Office of Naval Research). Los
Angeles University of Southern California,
Rossier School of Education. Available at
http//www.adlnet.org/index.cfm?fuseactionDLGuid
34
CRESST Cognitive Readiness Competencies
35
Research Recommendations (Cont.)

• Fund NRC Recommendation. National Research
  Council. (2008). Human Behavior in Military
  Contexts. Committee on Opportunities in Basic
  Research in the Behavioral and Social Sciences
  for the U.S. Military. James J. Blascovich and
  Christine R. Hartel, Editors. Board on
  Behavioral, Cognitive, and Sensory Sciences,
  Division of Behavioral and Social Sciences and
  Education. Washington, DC The National Academies
  Press.