Maryland Assessment Group Conference Computer Adaptive Testing

1
Maryland Assessment Group ConferenceComputer
Adaptive Testing
2
Session Agenda

• CAT System Overview
• Behind the Scenes of the Maryland CAT
• Next Steps

3
System Functions

• Administrator interaction with the system
• Student Data
• Test Session Management
• Test Delivery download
• Reporting
• Security
• Student interaction
• TestNavTM

4
http//etest.ncs.com/customers/Maryland/index.htm
Starting Point Maryland home page- Entry point
for all Administrator Functions
5
Administrator Functions

6
Log In - Required for All Secure Tasks
Enter User Name and Password Click Log In
7
Student Data
For most schools, student data will be preloaded
8
Test Session Management
Groups of students who test at same time Monitor
the test process Used by proctors and teachers
9
Test Delivery

• TestNav? provides
• Secure Test Delivery
• Access Security
• Desktop Security
• Content Security
• Cross-platform access
• Consistent presentation
• Single item format
• Content staging isolates student from
  interruptions in the network

10
TestNav?
Hyperlink back to question. Hyperlink to items
marked for review. Click End Test To submit
answers for scoring.
11
Reporting
Access determined by authorization
12
Security
Where access is assigned and controlled
13
Requirements for the new MFTP-CAT

• Build upon successes of previous MD CAT system
• Internet delivery
• Work with multiple platforms
• Improve on test security (exposure control)
• Psychometrically sound
• Provide a quality, and fair, assessment to
  Maryland students.

14

• Psychometric Evaluation of MFMT/MFRT CAT
• Throughout the development process there was
  ongoing evaluation of the psychometric
  functioning of the new MFTP-CAT
• Psychometric verification was made of scoring,
  content balancing, item selection, exposure
  control.
• Evaluation involved complex CAT simulations as
  well as comprehensive testing of TestNav?
  delivery and scoring system using the MFTP item
  banks.
• MSDE, along with an independent National expert
  on CAT, have been actively involved in review of
  this evaluation.

15
MFTP-CAT Characteristics

• Internet Delivery (via secure method)
• Cross-Platform Support (now use your Mac!)
• Test Length -- (Min. Std. Error or Maximum
  Items) MFMT 35 items MFRT 30 items
  Minimum Std. Error Criteria 0.30
• Item Selection -- Maximum-likelihood IRT
• Scoring --Bayesian-Modal switching over to
  Maximum-Likelihoodtransformed to MFTP Scale
  Score
• Exposure Control -- Improved Test Bank
  Security
• Content Balancing -- Balanced through course of
  test
• Item Review -- Limited review available

Improved or New Feature (Compared with old
CAT system)
16
Comparability (Validity) Issues

• When tests co-exist with PP counterparts (as
  with the MFTP), or if cut scores based on
  pre-existing PP tests are used, then the
  equivalence of scores across test modes must be
  determined in order to ensure valid score
  interpretations.
• APAs Guideline for Computer-Based Tests and
  Interpretations (1986) when interpreting scores
  from the computerized versions of conventional
  tests, the equivalence of scores from
  computerized versions should be established and
  documented before using norms or cut scores
  obtained from conventional tests. (p. 18)
• AERA/NCME/APA Joint Standards for Educational and
  Psychological Testing -- Standard 4.10 A clear
  rationale and supporting evidence should be
  provided for any claim that scores earned on
  different forms of a test may be used
  interchangeably (AERA, 1999)
• NCS Pearson has been very pro-active in this
  area of research

17
MFTP-CAT Comparability Study

• Fall 2002 Study
• MFMT - 6 LEAs participated MFRT - 8 LEAs
  participated
• Students Randomly pre-assigned to Test Mode
  CAT or Paper-and-Pencil administration
• Analysis (not completed) will focus on comparison
  of student performance across test
  administration conditions.
• Lessons learned

18
Next Steps

• Comparability Results
• Prepare for Next Administration