In Pursuit of Testing Excellence

1
In Pursuit of Testing Excellence
2007 Annual Conference

• The Experts Respond To Your Practical Testing
  Questions

2
Moderator and Coordinator

• Chuck Friedman
• Consultant

3
Panelist

• Reid Klion
• Chief Science Officer
• pan-a TALX Company

4
Panelist
Rose McCallin Director of Examination
Services Colorado Department of Regulatory
Agencies
5
Panelist

• Paul Naylor
• Consultant

6
Speakers

• YOUthe audience!!!!

7
Credentialing and Exam Issues at CLEAR

• Challenges to Quality Electronic and Candidate
  Integrity and Its Implications
• Subject Matter Experts Friends or Foes?
• Getting the Most Out of Your Face-to-Face and
  Virtual Group Meetings
• Get Ready for Continuous Computer- Based Testing
  Examination Quality and Item Security

8
Credentialing and Exam Issues at CLEAR

• Best Practice in Job Analysis and Validation of
  Test Specifications
• Promoting Excellence in Managing ADA
  Accommodations
• Quality Assurance and Regulatory Excellence
• Vital Implementation Issues in Translation
• Innovations and Trends in Testing

9
Responding to YOUR Testing Questions
10
Validity
Reliability
Item Writing
Item Analysis
Item Review
Passing Score
Scoring
Standard Administration
Test Blueprint
Job Analysis
11
(No Transcript)
12
Scoring
Item Review
Reliability
Job Analysis
Item Writing
Validity
Standard Administration
Test Blueprint
Item Analysis
Passing Score
13
What are some alternative ways to conduct a job
analysis?
Job analysis
14
What are some alternative methods to conduct a
job analysis?

• Full Survey
• Literature Search
• In-Depth Interviews
• Focus Groups

15
What are examples of some advanced, alternative
item formats? What are the benefits and
drawbacks of using these formats?
16
Innovative Item Types

• Options are Limitless..
• Performance Testing
• Analogue Work Tasks
• - Visual Refraction
• - Medical Diagnostic Processes
• - Plant Operators
• Streaming Video and Audio
• On-Screen Manipulables

17
..but Reality Constrains

• Pragmatics
• Usability
• Logistics
• Security Salience of Rich Media
• Development Costs Often High
• What are we getting for our money?
• Psychometric Considerations
• Measures More Difficult to Develop than Methods

18
If the testing process is disrupted, (lost answer
sheets/data, power failure, computer crash), what
should my Board tell the candidate and what
action should we take?
19
Stuff Happens

• Force majeure (a greater force)

Tornado Strikes BrooklynAugust 2007
20
What Could Happen?

• Natural Disaster
• Power/Internet Failure
• Local Issue/Emergency
• Many Events Unavoidable
• Can Only Plan to Mitigate Impact
• Policy to Deal with Issues

21
Mitigation of Computer-Based Issues

• Test Provider Issues
• - Redundant Systems
• Backups
• End-User (or Test Taker) Problems
• - Backups of Test Responses
• - Transmission to Servers After Each Screen
  Movement
• -Ability to Resume Test Session

22
What feedback should be provided to candidates on
score reports and results?

23
What feedback should be provided to candidates on
score reports and results?

• Scoring Methods
• Content Domain Scores
• Interpretations
• Reliability Issues
• Cautions
• Aggregate Scores
• How to Reapply

24
Should my Board or program sponsor allow
candidates to review test results including test
questions?
25

Allow Candidate Review?
26

Allow Candidate Review?
27
Allow Candidate Review?

• Candidate Right To Fair, Meaningful Appeals
  Process
• Some Programs Allow Limited Access to Certain
  Test Items and At High Costs
• Only Review M/C Items
• Cost per M/C Item 100
• Process for Review Not Realistic
• If Credit Granted, Not Accredited

28
Allow Candidate Review?

• How Minimize Validity Threats of Exposure
  Provide Candidates Meaningful Appeals Process?
• Independent Review by Qualified 3rd Party
• Non-Disclosure Contract
• Supervised Environment
• Authority to Challenge

29
What should I tell a candidate who failed the
examination by one point?
30
Failed by 1 Point?

• Regulatory Exams Aims of Measurement Validity
  of Interpretations

Maximize Accuracy of P/F Classification Decisions
31
Failed by 1 Point?

• Validity of Interpretations of Performance on
  Regulatory Exams
• Maximize Consist, Accurate Classification
  Decisions
• Consequences of Errors of Measurement
• Candidates Whose Performance Is Far Above or Far
  Below P/F Decision Point
• Candidates Whose Performance Is Close to P/F
  Decision Point (False Positives False
  Negatives)

32
Failed by 1 Point?

• Provide Results Reports that Encourage
  Interpretations Warranted by Precision of
  Classification Consistency

33
Failed by 1 Point?
34
What are the practicalities and economics of
maintaining a small licensing program?
Quality
35
Small Programs Practicalities/Economics
36
Small Programs Practicalities Economics

• Measurement Models CTT or Rasch
• Problems of Dimensionality
• Carefully Consider Logic of Test Specifications,
  Content Dimensions
• Best Practices
• Good Practice Analysis, Test Specs.
  Appropriate, Well-Formulated Test Items Try-Outs

37
Validity As A Unified Concept
38
What are the benefits and drawbacks of computer
based testing as opposed to paper and pencil
delivery?
39
Computerized Assessment - Why?

• Convenience
• - Test Wherever There is a Computer
• - Logistics Vastly Simplified
• - Management of Errata
• - Possibility of Continuous Testing
• - Immediate and Accurate Scoring
• Hanging Chads
• OpScan Issues

40
Facilitation of Data Management

• Data in Electronic Format
• Web-Based Data Immediate Availability
• Centralized Management of Assessment Programs
• Access to Array of Psychometric Data
• (e.g., response latencies, test taker
  activity)
• - Facilitate Integration with Other IT Systems

41
Candidate Perception

• Candidate/Member Preference
• - Familiarity/Perceived Ease of Use
• - Demographic Issues
• Organizational Appearance

42
Why Continue to Use Paper-Pencil?

• Programs That Must Test Massive Numbers
  Concurrently in One Locality
• Where Economics Dictate
• Very Small Programs
• Program Costs Hidden
• Immovable Impediments

43
We are starting a new licensing program which
does not allow grandfathering. What issues need
to be considered?
44
We are starting a new licensing program which
does not allow grandfathering. What issues need
to be considered?

• Security
• SMEs
• Limited Disclosure
• Portfolio Review

45
What are the issues and potential pitfalls in
adapting/translating our examination?
See you soon!
46
What are the issues and potential pitfalls in
adapting/translating our examination?

• Adaptation
• Process
• Beyond Translation
• Back Translation
• Intuitively Valid

47
Adaptation/Translation

• Translators
• Language Skills
• Cultural Skills
• Subject Matter Knowledge
• Trained in Item Writing/Review
• Field Testing
• Ron Hambleton/Stephan Sireci

48
What is the relationship between the examination
process and reciprocity between states? Between
countries?
49
Relationship Between Exam Process Mobility

• Reciprocity
• Jurisdictions Agree to Recognize Each Others
  Practitioners without Require to Demonstrate
  Competence
• License May or May Not Be Required from Other
  Jurisdiction

50
Relationship Between Exam Process Mobility

• Endorsement
• Jurisdiction Recognizes License from Another
  Jurisdiction
• Licensee Relieved of Full Burden of Obtaining
  License in New Jurisdiction

51
Relationship Between Exam Process Mobility

• Major Barriers
• Standards for Entry to Practice Are Not
  Universally Focused On Public Protection
• Professional Interests, Scope-of-Practice
  Protection, etc. Barriers to Entry
• Change Agents
• Mobility
• Labor Supply/Demand
• Communications Innovations

52
What is the impact of test industry consolidation
(mergers, acquisitions) and privatization for
state licensing programs?
53
Consolidation Privatization

• Media Landscape Who Owns What?
• Testing Industry Legacies Then Now
• Licensure Exam Industry Impact of Privatization
  by Regulatory Authorities
• Independent Auditing Mechanisms

54
Consolidation Privatization

• Media Landscape Who Owns What? (Columbia
  Journalism Review http//www.cjr.org/resources )
• Three of Top 5 Print Corporations Affiliated
  with Testing Companies
• Reed Elsevier Harcourt (The Psychological Corp.)
• NCS Pearson Pearson Education (Pearson
  AssessmentsPearson VUE, Promissor, Riverside)
• McGraw-Hill CTB

55
Testing Industry Legacies
56
Testing Industry Legacies Then Now
57
Testing Industry Legacies Then Now
58
Testing Industry Legacies Then Now
59
Testing Industry Legacies Then Now
60
Testing Industry Legacies Then Now
61
Testing Industry Today
More!
62
Privatization Independent Auditing Mechanisms
63
How do we combat threats to security for both
paper-and-pencil test delivery and CBT in the
face of new technology?
64
Security

• Any Exposure of Item is a Security Risk
• Issues Common Across Delivery Modalities
• Content Management Handling
• Item Presentation and Usage
• Candidate Item Review Policies
• Need for Good Proctoring
• Need to Analyze Risk Factors in Program Population

65
Item Security in CBT

• SSL Encryption
• Randomized Forms
• Locked-Down Browsers
• Candidate Education (May be Particularly Useful
  for Certain Certification Programs)
• Signed Attestations

66
Besides test delivery, how can technology impact
licensing programs and state Boards?
67
Impact of Technology

• Data Are Electronic
• Web 2.0 is Real (Internet as Platform)
• - Immediate Availability of Assessment Data
• - Centralized Management of Assessment Programs
• - Facilitate Integration of Test Data with Other
  IT Systems and Processes

68
What is on the horizon in the next five year that
will affect the testing process for licensure and
certification?
69
What is on the horizon in the next five years
that will affect the testing process for
licensure and certification?

• Increased Dependence on Accreditation
• Increased Consolidation in Test Industry
• PP Hangin In There

70
(No Transcript)
71
Speaker Contact Information

• Rose C. McCallin, Ph.D.
• Director, Office of Examination Services
• Colorado Department of Regulatory Agencies
• Division of Registrations
• 1560 Broadway, Suite 880
• Denver, CO 80202
• 303.894.7710
• rose.mccallin_at_dora.state.co.us

72
Speaker Contact Information

• Reid E. Klion, PhD
• Chief Science Officer
• panA TALX Corporation
• rklion_at_panpowered.com

73
Speaker Contact Information

• Paul D Naylor, Ph.D.
• Psychometric Consultant
• naylorpaul_at_msn.com
• [phone]