Electronic Advancement Exams

1
Electronic Advancement Exams
IMTA 2003

• Transitioning from Paper-based to Electronic
  Format

Kirk Schultz Robert Sapp Larry Willers
2
Background
INTRODUCTION

• Navy Advancement Center
• Creates Administers Navy Enlisted Advancement
  Exams
• 50 Year Tradition
• Paper Pencil Exam Presentation Process
• Phase I - Baisden Rymsza (2002)

3
Overview
INTRODUCTION

• Electronic Advancement Exams
• Phase II The First Six Months
• Organizational Culture Issues
• Internal/External Processes
• Hardware Solutions
• Integrated Software Solutions

4
Organizational Culture Issues
5
Change Management
ORGANIZATIONAL CULTURE ISSUES

• Organizational Culture Shock
• Building a Motivated Team
• Small Successes / Reduced Program Risk
• Briefing Higher Level Echelons

6
Internal/External Processes
7
Internal Processes
INTERNAL/EXTERNAL PROCESSES

• Exam Production
• Multimedia Item Development
• Exam Administration
• Exam Scoring
• Test Bank Item Analysis

8
Exam Production
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Follow Same Basic Production Process
• Must meet specified schedule
• Follow strict exam writing guidelines
• Perform same reviews
• Provide electronic exam to all in that rating
• Finalize Exam

9
Multimedia Item Development
INTERNAL/EXTERNAL PROCESSES Internal Processes

• If the question is valid without multimedia, then
  do not use it.
• Reflect the Sailors workplace performance
  situation as closely as possible.

10
Multimedia Item Example
INTERNAL/EXTERNAL PROCESSES Internal Processes
11
Multimedia Item Example
INTERNAL/EXTERNAL PROCESSES Internal Processes
12
Multimedia Item Development
INTERNAL/EXTERNAL PROCESSES Internal Processes

• 35 to 45 of exam questions contained
  multimedia.
• 75 of multimedia created specifically for the
  exam.
• Only U.S. government created resources were used.
• Estimates indicate multimedia items will take at
  least 20 times as long to develop as text only
  items.

13
Exam Administration
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Follow Same Basic Administration Process
• Mail exam to Educational Services Officer (ESO)
• ESO Proctors Exam
• Mail Answers back to Navy Advancement Center
• Maintain Exam Integrity and Security

14
Shipboard Exam Administration
INTERNAL/EXTERNAL PROCESSES Internal Processes
Administer Multimedia exams on separate exam day
Mail devices to test sites.

• Return
• Answers
• Options
• Overnight
• SD Cards
• E-mail
• Results

Develop Multimedia Exams
Pre-Test With Fleet ESO's
Test Day
Provide detailed "how to" verbatim
Deliver on SD Card
15
Exam Scoring
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Follow Same Basic Scoring Process
• Answers received as electronic file
• Exam graded electronically
• By-item statistics gathered
• Maintain Scoring Integrity and Security

16
Test Bank Item Analysis
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Follow Same Item Analysis Process
• Multimedia items examined for quality of question
• Adjustments made as necessary
• Items may reused in subsequent exams
• Maintain Exam Item Integrity and Security

17
External Processes
INTERNAL/EXTERNAL PROCESSES

• Sailor Orientation and Training
• Facilitator Orientation and Training
• Command Orientation and Readiness
• Requests for Assistance

18
Sailor Orientation and Training
INTERNAL/EXTERNAL PROCESSES External Processes

• ESO Verbatim
• Training Prior To Electronic Exam
• Job Aid During Exam Administration
• On-line Advancement Strategy Guides

19
Facilitator Orientation/Training
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Frequent Correspondence with ESOs
• Training Prior To Giving Electronic Exam
• Exam Verbatim
• Training on Job Aid
• On-line Advancement Strategy Guides

20
Command Orientation/ Readiness
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Correspondence
• Briefings to Higher Echelons
• Frequent Interaction with Fleet/Shore-based Units
• No Surprises

21
Requests for Assistance
INTERNAL/EXTERNAL PROCESSES Internal Processes

• Prior to Exam
• Exam Day Problems
• Post-Exam Questions

22
Hardware Solutions
23
Hardware Requirements
HARDWARE SOLUTIONS

• Electronic exams require supporting delivery
  technology
• Assumptions
• Insufficient number of computers
• Possibly technologically outdated
• Inappropriate location
• System acquisition and distribution required to
  support electronic exams

24
Hardware Requirements
HARDWARE SOLUTIONS

• Shore based
• Automated Electronic Classrooms
• Navy/Marine Corps Internet
• Allows for Internet delivery
• Shipboard
• Limited number of suitable systems
• Limited Internet access
• Delivery of exam systems required

25
Hardware Requirements
HARDWARE SOLUTIONS

• Phase I evaluated
• PDAs
• Tablet PCs
• Notebook computers
• Phase II validation
• Reexamination of Phase I recommendations

26
Hardware Requirements
HARDWARE SOLUTIONS

• To support exam delivery, systems must be
• Affordable
• Reliable
• Capable
• Deployable
• Acceptable

27
Hardware Requirements
HARDWARE SOLUTIONS

• Affordable unit cost lt 1000
• Reliable off-the-shelf design
• Capable sufficiently robust
• Deployable portable form factor
• Acceptable Sailor familiarity

28
PDA
HARDWARE SOLUTIONS

• Affordable? R
• Reliable? R
• Capable? S
• Deployable? R
• Acceptable? R

29
Tablet PC
HARDWARE SOLUTIONS

• Affordable? S
• Reliable? R
• Capable? R
• Deployable? R
• Acceptable? S

30
Notebook PC
HARDWARE SOLUTIONS

• Affordable? R
• Reliable? R
• Capable? R
• Deployable? R
• Acceptable? R

31
Engineering a Process
HARDWARE SOLUTIONS

• Hardware selection driven by process requirements
• Only scaleable practices adopted
• Three key issues
• Getting the exam to the Sailor
• Getting the results back
• Handling classified content effectively

32
Engineering a Process
HARDWARE SOLUTIONS

• Phase I process characteristics
• Single exam loaded on notebook computer hard
  drive
• On-site evaluator able to provide
  over-the-shoulder support with problems
• Evaluators returned results
• Process not scaleable

33
Exam Delivery
HARDWARE SOLUTIONS

• Exams on hard drives
• Require powering up system to determine exam
• Would have to track computers to users or use an
  authentication process
• Precludes system re-issue
• Will not support delivery to existing assets

34
Exam Results
HARDWARE SOLUTIONS

• Current Process
• Machine scoreable answer sheets express mailed to
  the Advancement Center
• Results returned within three days
• New Process
• Writing results to hard drive requires system
  return for scoring
• E-mailed scores not currently an option
• No system floppy drive

35
Classified Content
HARDWARE SOLUTIONS

• Introduces problems with
• System storage
• Routine procedures such as battery charging
• May require removing system hard drives

SECRET
36
Process Solution
HARDWARE SOLUTIONS

• Exam delivered on separate media
• Secure Digital cards selected
• Economical
• Widely available
• Broad Industry Support
• Notebook computers include SD card readers

37
Advantages
HARDWARE SOLUTIONS

• Writing results to an SD card provides a
  returnable component
• Allows for shipment of unclassified systems
• Supports greater flexibility for make-up exams
• Support distribution to deployed systems

38
Integrated Software Solutions
39
Delivery Software Selection
INTEGRATED SOFTWARE SOLUTIONS

• OutStart Evolution
• NETC Designated LCMS
• Primarily a content development tool
• Questionmark Perception
• Used in Phase I
• Developed for high-stakes testing

40
Delivery Software Selection
INTEGRATED SOFTWARE SOLUTIONS
41
Template Considerations
INTEGRATED SOFTWARE SOLUTIONS

• Display security banner
• Limit the amount of scrolling required
• Reduce the font size
• Reduce the button size

42
Template Considerations
INTEGRATED SOFTWARE SOLUTIONS
Phase I Template
Phase II Template
43
Development System Changes
INTEGRATED SOFTWARE SOLUTIONS

• Exam Development System (EDS)
• Developed in-house
• Currently used for paper-based exam development
• Modifications and Solutions
• Multimedia File Display and Processing
• XML (eXtensible Markup Language) Export
• XML Preview

44
Development System Changes
INTEGRATED SOFTWARE SOLUTIONS
EDS Graphics tab
E-EDS Graphics tab
45
Development System Changes
INTEGRATED SOFTWARE SOLUTIONS

• XML Export
• Conforms to IMS QTI Specifications
• Imports directly into Perception
• XML Preview
• XSL Formatted
• Internet Explorer 6.0
• Does not require Perception

46
Supplemental Data Collection
INTEGRATED SOFTWARE SOLUTIONS

• Sailor personal information
• Sailor feedback
• Collection vehicles
• Perception
• Visual Basic
• Active Server Page

47
Rating/Rate Selection
ASSESSMENT DEVELOPMENT CONSIDERATIONS

• Selection Factors
• Size of rate/rating - 200 Sailors maximum
• Demographic distribution
• Classification of exam
• Appropriateness of multimedia use
• Status of existing multimedia materials
• Number of exam bank control items
• Willingness of Exam Writer
• Marketing impact of a successful exam
• Exam candidate's familiarization with computers

48
Rating/Rate Selection
ASSESSMENT DEVELOPMENT CONSIDERATIONS

• AG3 - Aerographers Mate, Third Class
• STG3 - Sonar Technician - Surface, Third Class
• Professional Military Knowledge (PMK)
• General knowledge questions
• Common to all rate exams

49
Summary Thoughts
CONCLUSION

• Electronic Exams Not if, but when
• Monitor technology advances
• Continue addressing organizational and cultural
  issues
• Conduct ongoing process evaluation