2004 CLEAR Conference

1

• A Ten-Year Review of CBT

By Casey Marks, Ph.D., NCSBN Geraldine Piskorski,
MBA, ASCP Board of Registry Anthony R. Zara,
Ph.D., PV Presented at the 2004 CLEAR Annual
Conference 21st Century Regulation Show Me
What Works September 30, 2004
2
Session Agenda

• Speaker backgrounds
• Before transition to CBT
• Transition
• Operations
• Refinement
• CBT lessons learned
• Conclusions

3
Speaker Background

• Casey Marks, NCSBN
• Geraldine Piskorski, ASCP Board of Registry
• Anthony Zara, Pearson VUE

4
What Were Trying to Accomplish

• Share experiences
• Compare and contrast large vs. small programs
  issues
• Share solutions
• Outline successful practices

5
Some Industry Trends

• Exploding numbers of programs
• More complicated item types
• Security concerns
• Globalization
• Maturing technologies, e.g., Internet
• Increased choices for program sponsors

6
Issues Before Transition to CBT

• Small program
• ASCP/BOR background
• Size and depth of item bank
• Variable vs. fixed length test
• Opportunity to review
• Size of Examinee population
• Availability of computer testing facilities
• Cost of delivery

• Large program
• NCSBN background
• NCLEX facts
• Pre-CAT research program
• Barriers?

7
Foundational Pre-CBT Research

• Dimensionality
• IRT calibration
• Scale stability
• Comparability of results
• Early on, much psychometric research was
  necessary now a bit less, but still need to
  validate findings with your program.
• Mode of administration
• What flavor of CBT? (CAT not right for everyone)

8
Operational Research

• Examinee interface
• Stopping rules
• Test length
• Item pre-testing plan
• Test timing
• Candidate impact
• Beta testing
• Here is where your programs specifics get
  analyzed
• and determined

9
Transition Issues

• Small program
• Transition from testing two days each year to
  year-round testing
• Staffing
• Stakeholder communications

• Large program
• CBT benefits
• Create roadmap to success
• Organizational processes
• Stakeholder communications

10
Benefits of CBT

• More frequent administrations
• (daily testing)
• Measurement accuracy and efficiency
• Advanced item types can be used
• Scheduling on demand
• Increased security
• Instant candidate results
• Professional testing environment
• Reduced time between retakes

11
Roadmap for CBT

• Needs analysis
• Stakeholder analysis
• Building consensus
• Working decision processes
• Communications
• Leverage the experience of others when analyzing
  your program.

12
Organizational Processes

• Organizational Decision Process
• Understanding needs
• Options for solutions
• Contracting Process
• RFP or directed proposals
• Provider options
• Focus on outcomes

13
Operations Issues

• Small program
• Psychometrics
• Administration
• Stakeholder support

• Large program
• Batch to continuous model
• Test development
• Psychometrics
• Administration
• Stakeholder support
• Staffing

14
Program Operations

• Batch to Continuous Different Mindset
• Test Development
• Constant pool rotation -- ability to introduce
  content more quickly
• More items needed
• Continuous item development/reviewing
• Item writer recruitment

15
Program Operations

• Batch to Continuous
• Psychometrics
• Constant item maintenance
• Different item exposure concerns
• Candidate volume spikes/dips due to content
  changes and passing standard changes
• Ability to remove underperforming
  problematic/underperforming items during
  administration

16
Program Operations

• Batch to Continuous
• Test Administration
• Individual issues/incidents
• Ongoing test center monitoring
• Ratio of investigations to candidates increases
• Time from exam registration to result
• Time-dependent registration policies
• Application process
• Fee changes
• Rescheduling issues

17
Program Operations

• Interface with stakeholders
• Test accommodations
• Communicating program changes
• Procedure/documentation changes
• Stakeholder Support
• Direct support
• Information/communication
• Accessible
• Timely
• Constant
• Rumors/urban legends

18
Program Operations

• Batch to Continuous
• Program Staffing
• Dedicated
• Specialized
• Outsourcing
• Customer Service
• The on-demand mentality
• Expectations

19
CBT Program Refinement

• Small program
• Immediate pass/fail results at test center
• Providers - more options
• Quality of graphics

• Large program
• Ongoing research
• Item types
• Providers
• Service

20
Program Refinement - Research

• Ongoing Research
• Methodology and content
• Staff and/or test service
• External
• Implementation dates seemingly arbitrary

21
Program Refinement Item Types

• Advanced item types
• Media
• Stimuli
• Response formats

22
Program Refinement - Providers

• Service Provider Change
• Between 1994 and 2002 same CBT provider
• New test development and administration provider
• (2 for 1) in 2002
• New center network
• New test driver/candidate interface
• Candidate support
• Data reconciliation between test services
• New application/administration/reporting/results
• processing systems
• Seamless

23
Program Refinement - Service

• Innovative item types (beyond MCQ)
• Utilizing same exam administration parameters
• More frequent administrations
• 2 to 4 to 8 per year
• Additional testing time
• Opportunity to demonstrate competence under
  optimal
• exam termination rules
• International administration
• Outside U.S. and territories starting in 2005

24
CBT Lessons Learned

• Small program
• Importance of good communication with all
  stakeholders
• Size of examinee population
• Improved benefits to examinees

• Large program
• Benefits obtained worth it?
• Program complexities
• Stakeholder satisfaction?
• Service?
• Etc.

25
Challenges

• Large program, yet perfection is needed/expected
  daily
• More moving pieces, more expertise needed
• Change is harder to integrate
• The more complex the program, the more difficult
  to educate stakeholders
• Timing of program changes never convenient for
  all stakeholders

26
General Conclusions

• Any large vs. small program things?
• Any gotchas?
• Need to consider all program stakeholders when
  considering procedure changes

27
Conclusions

• CAT is a good thing for the NCLEX
• Boards of Nursing enjoy
• Convenience and speed of system
• The ability to focus on exam content not
  psychometric and/or exam administration
• Candidates like
• Quick registration, scheduling and results
  receipt
• Scheduling options
• Personalized service
• NCSBN appreciates
• Security benefits
• The ability to test content in new and novel ways

28

• Discussion

29
Contact Information

• Casey Marks, Ph.D.
• Director of Testing Services
• NCSBN
• email cmarks_at_ncsbn.org
• Gerry Piskorski, MBA
• Director, Applicant, Certificant and IT
  Activities
• ASCP
• email gpiskorski_at_ascp.org
• Anthony R. Zara, Ph.D.
• Vice President Testing Services
• Pearson VUE
• email tony.zara_at_pearson.com