WHAT IS COGNITIVE READINESS

1
WHAT IS COGNITIVE READINESS?

• Harry ONeil and Joan Lang
• CRESST/University of Southern California

Teaching and Measuring Cognitive Readiness
Workshop Los Angeles, California January 13, 2009
2
The Cognitive Readiness Definition

• Cognitive readiness is the mental preparation
  (including skills, knowledge, abilities,
  motivations, and personal dispositions) an
  individual needs to establish and sustain
  competent performance in the complex and
  unpredictable environment of modern military
  operations.

Fletcher, J. D. (2004). Cognitive readiness
Preparing for the unexpected. Alexandria, VA
Institute for Defense Analyses.
3
The Cognitive Readiness Definition (contd)

• It includes cognitive skills such as problem
  solving, decision-making, and basic foundational
  skills (e.g. math reasoning and reading
  comprehension).
• In an information-rich and dynamic battle space
  where our adversaries are unpredictable and lack
  systematic doctrine, training cognitive readiness
  will enhance success and survivability.

4
A Cognitive Readiness Model
Attributes/ Competencies
Knowledge
Skills
Relatively hard to teach/assess
Relatively easy to teach/assess
Prerequisites
Domain Knowledge
Adaptability
Decision Making
Problem Solving
Metacognition
Situation Awareness
Communica-tion
Teamwork
Critical Thinking
Creativity
Adaptive Expertise
5
Why is it important?
6
Changing World

• Longer lives
• Multiple careers
• 3 or 4 over lifetime
• Globalization
• Urgent unforeseen societal problems will always
  spring up and need to be solved
• Problems and solutions will diffuse more rapidly
  because of technology and the media
• Problems will be very complex in unpredictable
  environments

7
Agility in DoD

• Strategic Objective
• Globalization and technological advances have
  contributed to an uncertain security environment.
• Protecting the United States in the face of this
  uncertainty requires greater agility.
• Agility allows the total force to adapt rapidly
  to changing requirements through the flexible use
  of diverse individual and organizational
  capabilities linked to needed military outcomes.
• Competencies
• Enduring, portable, describe individual
  attributes needed to perform successfully on the
  job.
• Human Capital Strategy
• David Chu (6/6/06)

8
Key Assessment Methodology Research Test Beds

• Needed for assessment fidelity in simulation/game
  environment
• Provides experimental test bed for learning and
  assessment issues
• Learning variables include time, complexity, cue
  salience, cognitive framing, multi-channel
  stimuli, cognitive load, distraction, degree of
  fidelity, types and frequency of help
• Supports the development of transfer tasks
• Allows comparison of various assessment
  techniques, e.g. knowledge maps vs.
  multiple-choice formats
• Permits experimentation with alternative
  statistical analyses of student process and
  outcomes, e.g., Bayes nets, neural nets, expert
  model-based cut scores

9
Research Test Beds

• Off-the-shelf games (SafeCracker)
• Wraparound instruction and assessment strategies
• Five empirical studies conducted
• Develop own game (research test bed)
• CRESST embedded instructional and assessment
  strategies in game
• Existing Navy simulations (MMTT)
• Laptop version of instructor rating assessment
  for Air Defense
• Measure retention/transfer a la Mayer for Surface
  Warfare
• Initial cognitive readiness study conducted
  (Ayala, 2008)
• Existing Navy simulation (Battlestations 21)
• View game as a test or instruction
• Documented in ONeil, H., Perez, R. (Eds.)
  (2008). Computer Games and Team and Individual
  Learning. United Kingdom Elsevier publications.

10
Vary Cognitive Differences of Game Levels to Test
Cognitive Readiness

• A construct validity approach
• Basic Levelrelatively low need for cognitive
  readiness
• Relatively low cognitive complexity
• Primarily procedural
• Limited decision making
• One best solution
• Multiple adequate solutions
• Advanced Levelrelatively high need for cognitive
  readiness
• High cognitive complexity
• Multiple competing priorities
• Unpredictable environments
• No best solution
• Many potentially disastrous solutions
• Trained procedures/principles may not work.
  Creative solutions may be needed.

11
Measurement Issues
12
Measurement Issues

• Trait vs. state
• High stakes vs. low stakes
• Coaching of motivational component
• Cheating
• Transfer
• Stress situations may decrease experts cognitive
  readiness
• Role of the unaware/implicit/unconscious
  knowledge and motivation
• Lack of psychometrics for simulation/games
• Need affective attributes of cognitive readiness

13
Measurement Issues (contd)

• Performance Assessment (K-12 lessons learned)
• Simulations/games
• Questions
• How is difficulty estimated?
• How many tasks are needed?
• Live with the reality that performance assessment
  and time constraints lead to few tasks. Conduct
  generalizability studies.
• What is role of gender and prior experiences?
• Best strategy to develop for useful, valid, fair
  and feasible measures?
• Should cognitive readiness be reported as single
  vs. multiple scores as profile and/or levels?

14
What Are The Possible Underlying Affective
Attributes of Cognitive Readiness?

• Self-motivation
• Independence
• Controlling impulses
• Completing tasks and follow through
• Self-efficacy
• Risk assessment

• Accept fair blame
• Active choice
• Delay gratification
• Persistent
• Dont procrastinate
• Put forth effort
• Manage anxiety
• Are resilient

Based on Sternbergs Practical Intelligence
15
Measurement of Motivation

• Use of PISA questionnaire
• To measure self-efficacy, effort, test anxiety,
  elaboration, and metacognition

16
New Challenges

• Overlap in conceptual definitions
• Creativity part of many constructs
• Lack of reliable and valid questionnaires for
  selection purposes
• Few state measures for diagnostic/remedial
  purposes
• State worry, state self-efficacy questionnaires
  exist
• In-game process measures
• Lack of applying known psychometric techniques
  into game/simulation scoring (eg. cut scores)

17
Whats Next?

• After workshop refine concept of cognitive
  readiness
• Select/develop cognitive readiness measures
• for selection purposes (trait measures)
• for diagnosis/prescriptive purposes (state
  measures)
• for program evaluation purposes (trait measures)

18
Whats Next? (contd)

• Test/teach cognitive readiness with recent high
  school graduates (RTC) and college graduates
  (OTS, SWOS)
• When contracted, then edited book

19
Cognitive Readiness Workshop
Attributes/ Competencies
Knowledge
Skills
Relatively hard to teach/assess
Relatively easy to teach/assess
Prerequisites
Domain Knowledge
Chung
Adaptability
Decision Making
Problem Solving
Metacognition
Situation Awareness
Communica-tion
Teamwork
Zaccaro
Baker
Bewley
ONeil
Bolstad
Cannon Bowers Bowers
Mayer
Contexts Military (Fletcher Chatelier) Educati
on (Herman) Transfer (Holyoak)
Critical Thinking
Creativity
Adaptive Expertise
Ericsson
Sternberg
20
CRESST Web Site
http//www.cresst.org or any search engine type
CRESST
honeil_at_usc.edu
21
BACK-UP SLIDES
22
CRESSTs Cognitive Readiness Competencies

• Adaptive Expertise Entails a deep comprehension
  of conceptual structure of the problem domain.
  Knowledge must be organized and structures must
  be flexible. Adaptive experts understand when and
  why particular procedures are appropriate or not
  (Zaccaro Banks, 2004).
• Creativity Includes fluency, novel ideas,
  flexibility, synthesizing, analyzing, complexity,
  originality, elaboration (ONeil, Abedi,
  Spielberger, 1994).

23
Cognitive Readiness Competencies

• Situation Awareness Ability to perceive and
  comprehend oneself in relationship relevant to
  present environment (Endsley, 1988).
• Problem-Solving Problem solving is content
  understanding, problem solving strategies, and
  self-regulation (ONeil, 1999).
• Adaptability An effective change in response to
  an altered situation (Mueller- Hanson, 2005).
• Decision-Making Use of information about their
  current situation to help evaluate the utility of
  potential courses of action (ONeil, Chung,
  Herl, 1999).
• Teamwork A trait of the individual that
  predisposes the individual to act as a team
  member. There a six processes (a) adaptability,
  (b) coordination, (c) decision-making, (d)
  interpersonal, (e) leadership, and (f)
  communication (ONeil, Wang, Mulkey, Baker,
  2003).

24
Resiliency Skills

• Hold up well under pressure
• Orient quickly to new demands
• Adapt to changing circumstances
• Use problem-solving strategies

Siebert (2006)