Title Slide

1
The Future of Testing How You Can Take
Advantage of It, Today!
David MeissnerVice President, Solution
ServicesPrometric
April 23, 2008
2
Introduction

• The Past, Present and Future
• Why Do We Test?
• Why Technology-Based Testing?
• Choosing A Delivery Option
• Computer-Based Testing
• Internet-Based Testing
• Conversion Strategies
• Choosing a Vendor
• Recent Innovations

3
Testing and AssessmentWhere Have We BeenAnd
Where Are We Going?
4

• Prediction is very difficult,
• especially about the future.
• Niels Bohr (also attributed to Mark Twain)

Difficult to see, always in motion is the
future Yoda, (also attributed to George Lucas)
5
A Very Brief History of Testing

• 2200 B.C. Pre-Employment Testing

206 B.C. Psychological Testing
1855 British Civil Service Exams
1917-1945 U.S. Army Testing
1950s U.S. Armed Forces Testing
1960s Widespread K-12 Testing
1968 ASVAB
1970s Mainframe-based CBT
1980s Computer-AdaptiveTests

• 1991 PC-based CBT
• 1993 Performance-based CBT
• 1995 Computer Simulations
• Video-based CBT
• 2005 Computer Mastery Testing
• 2007 Emulation Testing
• 2008 Dynamic Forms Generation

6
Delivery Technologies - 1993

• No Data Transfer Standards
• Cumbersome Localization Process
• Heavy Reliance on Call Center Operations
• Client Server and Mainframe Delivery
• Low Resolution Graphics
• Relatively Simple Item Types
• Limited Data Collection

7
Computer-Based Testing - 1993
8
Internet-Based Testing - 1993
9
Delivery Technologies - 2008

• Vendor-Defined Data Transfer Standards
• Global Availability and Consistency
• Internet-Enabled Services
• Platform Independent Delivery
• High Resolution Graphics and Multimedia
• Robust Item Types
• Rich Data Collection

10
Computer-Based Testing - 2008
11
Internet-Based Testing - 2008
12
Delivery Technologies - 2013

• Independent Data Transfer Standards
• Universal, Multi-Language Exams
• Just-In-Time Delivery
• Internet-Enabled, Personalized Testing
• Immersive and Multimedia Rich
• Performance-Based, Real Time Assessments
• Seamless Integration With the Real-World

13
Why Do We Test?

• To Meet Strategic Objectives
• To Gather Information
• To Maintain Professional Standards
• To Measure Knowledge, Skills and Abilities
• To Certify, License or Credential Individuals
• To

14
Continuous Learning
Content Development
Psychometric Oversight
Item / Test Performance
Training Materials
Learning Performance
Test Material
Analytics Validity, Reliability, Fairness,
Effectiveness
Testing
Training
Shared Resources
Shared Resources
15
Comprehensive Testing System
16
Test Development Services

• Job Analysis
• Identifies what to measure
• Item Development
• Creation and development of test items
• Test Design Construction
• Development of content and structure
• Psychometric Analysis
• Evaluates validity, reliability and fairness

Confidential
16
17
Test Delivery Services

• Test Publishing
• Format, review, and package for distribution
• System Management
• Project, account and program management
• Registration Scheduling
• Authorization, authentication event scheduling
• Test Administration
• Test delivery methodologies and models
• Results Processing
• Test scoring and reporting

Confidential
17
18
Data Management Services

• Data Storage
• Application service provider model
• Data Administration
• Data scrubbing maintenance
• Data Reporting
• Reporting to support programs and organizational
  needs analysis
• Data Integration
• 3rd party software support

Confidential
18
19
Why Technology Based Testing?

• To Increase
• Immediacy
• Convenience
• Efficiency
• Efficacy
• Security
• Face Validity
• Test Taker Satisfaction

20
Immediacy

• Faster Time to Market
• Shorter Test Development Cycle
• Availability of Immediate Scoring Test-Taker
  Feedback
• Nightly Data Feeds To Speed Analysis

21
Convenience

• Tests Available When and Where Test-Takers Choose
• Availability of On-Demand Scheduling and
  Registration
• Test Results Collected and Reported
  Electronically
• Test-Taker Access to Testing History and Other
  Information

22
Efficiency

• Minimized Item Exposure Risk
• Optimized Item Pool Utilization
• Potential for Year-Round Test Delivery
• Normalized Staffing Demand
• Leveraged Operating Costs

23
Efficacy

• Highly Effective Test Delivery Model
• Proven Comparability With Paper and Pencil
• Engaging Item Types
• Real-World, Simulations and Performance-Based
  Content
• Data-Rich Test Results

24
Security

• Paper-Less Delivery
• Streamlined Logistics
• Video/Audio Session Monitoring and Recording
• Effective Item Navigation
• Data Analysis and Forensics
• Fragmentation and Encryption of Data
• Emerging Biometric Capabilities
• Full Session Keystroke Capture

25
Face Validity

• Reference Materials and Exhibits
• Performance-Based Items
• Simulations
• Live Application Testing

26
Test-Taker Satisfaction

• More Frequent Testing
• More Testing Locations
• Potentially Shorter Testing Times
• Immediate Scoring and Performance Feedback
• Engaging Item Types and Real-World Testing

27
Test Delivery OptionsChoosing The Solution
Thats Right For You
28
When To Choose IBT

• Highly Dispersed Testing Population
• Relatively Small Candidate Base (lt1,000)
• Readily Available Proctors
• Very Frequent Content Changes
• Less Rigid Delivery Conditions
• Lower Test Complexity
• Low to Medium Stakes Security
• Need for Lower Test Delivery Price

29
Selected IBT Attributes

• Immediacy
• Ease of Use
• Lower Cost
• 24X7 Availability
• Scalable Systems
• ASP Hosting
• E-commerce Support
• Immediate Scoring and Reporting Availability

• Hardware Independent
• Powerful Authoring Capability
• Prompt Editing Deployment
• Eligibility Processing
• Detailed Reporting
• Full Recovery Capability
• Multi-Language Support
• Proctor Management

30
Thin versus Thick

• Thin Client Delivery
• No Software Installation or Content Download
• Platform Independent
• True anytime, anywhere Delivery
• Thick Client Delivery
• Locally Installed Software and/or Test Content
• Operating System Dependency
• Requires Advance Planning and Preparation

31
When To Choose CBT

• Evenly Distributed Testing Population
• Modest to Large Candidate Base (gt1,000)
• Need for Impartial Proctoring
• Relatively Stable Content
• Desire for Consistency and Precision
• Flexibility to Address any Content
• Mid to High Security Requirements
• Interest in Outsourced Administration

32
Selected CBT Attributes

• Psychometric Rigor
• Advanced Item Types
• Performance-Based Testing
• Integrated Multimedia
• Custom User Interface
• Extensive Physical and Data Security
• Extensibility
• Legal Defensibility
• Immediate Scoring and Reporting Availability

• Highly Scalable
• Precise Item Section Timing
• Exhaustive Data Collection
• Reliable/Consistent Delivery
• Professional Test Center Staff
• Technical Support
• Full Recovery Capability
• Modular (customizable) Systems
• Multi-Language Support

33
Test Delivery OptionsBricks and MortarClient
Server Based Solution
34
Delivery Models

• Linear Testing
• Multiple fixed-length exam forms
• Random or sequential delivery (items, groups,
  sections)
• Static and Dynamic Branching
• Linear on the Fly Testing (LOFT
• Dynamic exam form generation
• Computer Mastery Testing (CMT)
• Variable length testing with pre-defined testlets

35
Item Types and Associated Materials
36

• Supports single response and multiple response
  multiple choice items
• Item responses may have both positive and
  negative score values
• Either Classical Test Theory or Item Response
  Theory Statistics can be generated in real-time

37

• Static images (photographs and illustrations)
  may be associated with stems and individual
  answer options

38

• A variety of supplemental material and
  reference documents can be made available to the
  candidate through the use of one or more exhibit
  buttons
• As with all the design elements of the user
  interface the labels can be customized to meet
  specific needs

39

• The size and placement of the exhibit materials
  is predefined during the publishing process to
  optimize screen real estate and ensure a
  consistent candidate experience

40

• Scenario-based exams, in which a reading
  passage is integral to the presentation of the
  items, benefit from a split screen in which the
  passage and the item(s) are shown on the screen
  simultaneously

41

• A vertical orientation often reduces the amount
  of scrolling required for scenario-based items,
  particularly when multiple items are associated
  with a single passage

42

• Extensive reference materials and multi-part
  bodies of knowledge can be invoked through a
  single exhibit button

43

• This document viewer allows access to multiple
  reference items
• Various tools allow candidates to easily
  navigate dozens, and in some cases, hundreds of
  pages of text.

44

• One or more video clips can be embedded into an
  item
• The controls available to the candidate and the
  appearance of the media player can be fully
  customized

45

• A standard or scientific calculator can be
  embedded into an item

46

• To maximize the space available for other
  elements of the item the calculator can be
  invoked through a dedicated calculator button

47

• The size and placement of the calculator is
  pre-defined to avoid overlapping important
  information

48

• Audio clips may be made available to candidates
  within the APC testing center network, where all
  testing workstations are equipped with headphones

49

• High resolution animations can be used to
  simulate real-world activities within a
  performance-based exam

50

• Essays and other constructed response content
  can be gathered with the aide of integrated word
  processing capabilities

51

• A very effective alternative to multiple choice
  can be the drop down item type

52

• A very large number of potential responses can
  be included in the drop down list, reducing the
  potential of guessing and memorization

53

• Hot-Area or Point and Click items utilize
  images (e.g. photographs, drawings, documents) as
  the means of response capture

54

• Candidates point to the correct response(s)
  with their mouse and click on their selection(s)
• The software then records the coordinates of
  the selection(s) and scores the item

55

• Drag-and-Drop is another alternative item type
  that requires candidates to discern the correct
  answer, or sequence of answers, to the task that
  has been assigned

56

• Candidates point to an object they wish to move
  and use their mouse to drag the object to the
  appropriate location

57

• This particular item requires the wars on the
  left to be placed into the proper sequence on the
  right

58

• Once the sequence is complete it is compared to
  the proper chronological order, with each object
  receiving a positive or negative score based on
  its relative placement

59

• This item type is often used to measure
  matching and identification skills and can be a
  very efficient assessment tool

60

• When taking paper and pencil exams many
  candidates mark their test booklets to indicate
  responses they believe are incorrect

61

• A comparable function, known as strikethrough,
  allows candidates to insert visual reminders to
  themselves of answer options they believe are
  incorrect
• Strikethrough is invoked by simply clicking on
  the text of an answer option

62

• To facilitate recall when using extensive
  reference materials many candidates are
  accustomed to highlighting relevant pieces of text

63

• The ability to highlight text within a scenario
  or passage can be enabled within the driver to
  ensure that the testing process closely
  represents real-world activities
• Highlighting can be invoked by simply dragging
  the cursor over the desired text

64

• Unicode compliance means that exams can be
  delivered in any language (single or double byte)
• Exams are currently being delivered in more
  than 15 languages

65

• Test sponsors may choose to make a review
  screen available throughout the exam to
  facilitate candidate navigation and provide a
  quick status check on their progress

66

• Immediate score reporting can be provided to
  the candidate, both on screen and in printed form
• Reports can be customized to present text and
  graphical representations of performance, and
  where possible, may include a digital photograph
  of the candidate

67
Test Delivery OptionsAnytime,
AnywhereInternet Based Solution
68
Introduction

• Comprehensive Internet-Based testing platform
• Item Authoring
• Test construction
• Proctor/Invigilator Management
• Candidate Registration
• eCommerce Payment
• Test Delivery
• Reporting and Data Management

69
System Overview

• Secure Thin-Client Architecture
• Centrally hosted solution from load-balanced
  servers
• No software or exam content downloads required
• Operating system and browser independent
• Ability to deliver within firewall-protected
  environments
• Full session recovery and backup capability
• Global content management and version control

70
Item Types and Associated Materials
71

• Shown here is the standard item layout for a
  single-response, multiple-choice item

72

• A full-function, online calculator can be made
  available when authoring an item that might
  benefit from this tool

73

• Essays and other writing assignments may be
  developed using the integrated word processor

74

• Test authors may choose to display an onscreen
  score report or to withhold scores until a later
  date
• Onscreen score reports provide a test a graphic
  representation of performance
• Item-level feedback may also be provided

75

• Authors are provided with an extensive tools to
  manage the development and ongoing maintenance of
  exams

76

• All of the exams for which an author is
  authorized to develop/maintain are shown on the
  test inventory
• From this screen existing exams may be modified
  or new exams developed

77

• Item authoring can be accomplished by either
  simple data entry or the importation of content
  from another application
• A complete user manual is continuously
  accessible from the ? symbol in the upper right
  hand corner of the interface

78

• The system offers several delivery styles
• Complete randomization
• Randomization within sections
• Specified sequence within sections
• Specified sequence through the test

79

• The frequency with which a candidate may take
  an exam is customizable to more effectively
  manage item exposure

80

• A wide variety of scoring models can be applied
  to an exam to meet specific measurement criteria

81

• Test section performance can be scored and
  rules established for their impact on overall
  test scores

82

• Test authors can integrate a wide variety of
  multimedia elements into an exam, including
  static graphics, audio, video, animations and
  simulations

83

• An extensive array of item types are supported
  by the system, providing significant flexibility
  to item authors and test developers

84

• Item authoring involves the same data entry
  process used in the creation of test design and
  specifications

85

• Context sensitive help is available for all of
  the fields marked with a ? symbol, providing
  additional technical detail regarding the
  particular feature

86

• To maximize the available space for different
  content components the system offers four
  templates that can present items in either a
  vertical or horizontal format

87

• Item authors can easily access graphics and
  other multimedia components that were previously
  uploaded into the media manager and then embed
  them within item stems, answer options and
  reference materials

88

• To ensure proper placement of the various design
  elements authors may preview items prior to
  publication

89

• Once the exam is complete authors may
  immediately place the test into production or
  into one of several off-line states. This allows
  subsequent authors and/or editors to access the
  exam prior to publication

90

• An extensive array of more than 24 psychometric,
  statistical, and management reports are
  available to the author

91

• Administrators are provided with a series of
  tools to facilitate the effective management of
  exams and candidates

92

• To ensure data-driven decision making
  administrators have access to a variety of
  reports at a technical and business level

93
Converting Paper and Pencil To Computerized
Delivery
94
Conversion Strategies

• Keep Things Simple
• Ease candidates into CBT
• Utilize existing test forms and item types
• Maintain current testing periods
• The Big Bang
• Concentrate changes into one big event
• Leverage advanced CBT functionality
• Reengineer testing schedule

95
Test Development Steps

• Decide on a testing model
• Determine development strategy
• Evaluate existing item bank
• Pre-test new items
• Analyze performance
• Select items for initial final form(s)
• Calibrate items
• Assemble initial final form(s)

96
Test Development Steps (cont.)

• Equate final form(s) to a previous exam form or
  set new cut score
• Attach new pre-test item packets to initial final
  forms (20 of total items)
• Deliver final forms pre-test item packets
• Periodically repeat the above analysis process on
  pre-test items and gradually create new final
  forms
• Repeat until initial item pool is pretested

97
Hypothetical Example Development Requirements

• 100-item form administered currently
• 2,400 exams to delivered per month, on demand
• Pool of 1,000 items exists, but no pre-test data
  have yet been gathered
• No new practice analysis or item development
  needed
• Cut score to be carried over from a previous exam
  form
• One-parameter (Rasch) measurement model to be used

98
Hypothetical Example Initial Fixed Form CBT

• Gather 300 item responses on a live PP form
• Calibrate PP items and establish cut score
• Create an initial oversized beta form for CBT
  delivery (150 items) with 20 of items
  overlapping with PP form.
• Deliver beta form to 300 candidates
• Use beta data to evaluate, select, and calibrate
  items.

99
Hypothetical Example Initial Fixed Form CBT
(cont.)

• Assemble items into a final 100-item final form
• Establish cut score by equating with PP form.
• Create six 25-item packets of pretest items
  (Packets 1-6)
• Deliver CBT Form ABeta Packets (A1-A6) to live
  candidates. Score candidates on the final form
  items but not the beta items.
• When 300 responses are collected on each beta
  packet, repeat the cycle of item analysis,
  calibration, forms assembly, and beta packet
  creation

100
Migration Issues To Consider

• Is the CBT to be equivalent to an existing PP
  Exam?
• Are there additional competencies that should be
  included?
• What new item types are being considered?
• Drop-Down Selection
• Point and Click
• Drag and Drop
• Constructed Response

101
Migration Issues To Consider (cont.)

• What delivery model is being considered?
• Linear, Linear on the Fly, Computerized Mastery,
  etc.
• Will new development be required before the
  computerization phase?
• Will tests be offered continuously or in windows?
• How much pressure will be placed on the test item
  pool?
• How often must the item pool be refreshed for
  technical reasons?
• How many forms will be required?

102
Choosing A Vendor
103
Assess Your Needs And Expectations

• Define the program objectives
• Validate the anticipated stakeholder benefits
• Determine your budget and timeline
• Identify your internal team
• Perform a gap analysis
• Inventory outsource requirements

104
Types of Service Providers

• Implementation Consultants
• Test Development
• Psychometrics
• Test Delivery
• Data Management
• Security
• Marketing

105
Evaluation Criteria

• Relevant Experience
• Organizational Expertise
• Size and Stability
• Technical Capability
• References
• Consultative Approach
• Innovation
• Price

106
Potential Sources of Information

• Network with colleagues
• Association of Test Publishers
• www.testpublishers.org
• National Organization for Competency Assurance
• www.noca.org
• Council on Licensure, Enforcement Regulation
• www.clearhq.org

107
Questions And Ideas
David Meissner david.meissner_at_prometric.com