Test Administration Modes and Test Security

1
Test Administration Modes and Test Security

• Computerized Versus Paper-and-Pencil Format
• Chuah Siang Chee
• AICPA
• Ben-Roy Do
• UIUC

2
Score Validity and Security

• Non-disclosure of test content is preferred for
  most high-stakes examinations.
• Licensure and certification programs do not give
  test questions to examinees after the test.
• Cost of content creation is high, re-use is
  economical.
• When test content is publicized, score
  interpretations and classification decisions may
  be in doubt.
• Proficiency or memorization?
• Security of content is a critical issue when
  tests are used for selection or credentialing.
• Is there some benefit from a particular mode of
  administration?

3
Reducing the Security Risk

• Deliberate planning to minimize item
  over-exposure
• Item exposure algorithms
• Large item pools, multiple test forms
• Insufficient, complicated and expenive!
• Minimize the memorability of test material
• What can we do to reduce item memory?
• Can changing the medium of administration reduce
  item memorization?

4
Study Design

• Which model minimizes item memorization?
• Study to examine mode of administration and item
  memorability
• Administered a test based on a popular
  computerized administration mode and a
  paper-based mode
• Item-level adaptive (CAT) format
• Asked examinees to recall as many items from the
  test as they possibly could

5
Study Design

• Sample
• 289 undergraduate psychology students from a
  large mid-western university
• Assignment to two conditions
• computerized or paper-administered
• blind assignment
• 26 items were taken from a disclosed GMAT item
  pool and administered in each mode
• Multiple-choice question format
• Verbal Critical Reasoning

6
Example Item
FIM PIM CBT 16(12) 49(36) PP

• An overly centralized economy, not the changes in
  the climate, is responsible for the poor
  agricultural production in Country X since its
  new government came to power. Neighboring
  Country Y has experienced the same climatic
  conditions, but while agricultural production has
  been falling in Country X, it has been rising in
  Country Y.
• Which of the following, if true, would most
  weaken the argument above?
• Industrial production also is declining in
  Country X.
• Whereas Country Y is landlocked, Country X has a
  major seaport.
• Both Country X and Country Y have been
  experiencing drought conditions.
• The crops that have always been grown in Country
  X are different from those that have always been
  grown in Country Y.
• Country X's new government instituted a
  centralized economy with the intention of
  ensuring an equitable distribution of goods.

7
Computer Based Test

• One item was presented at a time on the screen
• No review of previously completed test questions
  was permitted
• Randomized presentation of items to simulate CAT
• About half of the sample were given the
  computerized administration
• N 132 (45.7)

8
Paper-Based Test

• A single test form was created
• Review of previous questions was possible
• Answers could be given in any order
• About half the sample were administered the
  paper-based test
• N 157 (54.3)

9
Responses/Coding

• Examinees were asked to recall test items
  immediately after the test was completed
• Responses were coded by 3 independent raters
• Majority decisions on the accuracy of
  memorization were accepted
• Discussion was used to resolve decisions of
  accuracy when there was no initial agreement
  between raters

10
Coding Rules

• Two classifications of accuracy were used
• Full item memory
• Representative of strategies used for
  memorization with aptitude tests
• The item can be identified from the information
  provided
• Enough information was recalled to be of help of
  future examinees
• Partial item memory
• Representative of strategies used for
  memorization for subject tests
• The item can be identified from the information
  provided

11
T-Test
12
Results
13
Results
14
Examinee Performance

• Average Score
• Computer 12.9
• Paper-and-Pencil 11.9
• T-test p 0.06

15
Discussion

• Students remember more computer-based items,
  compared with paper-based
• Possibly due to greater focus/attention on
  computer-based items
• Items were presented one at a time, compared to a
  test form with many items per page on paper.
• However, computer-based tests usually have more
  items in active item pools
• Item selection algorithms are available to reduce
  item exposure
• Greater security over content used in electronic
  administration compared with paper document
  controls
• Randomization of distracters and other strategies
  are possible.

16
Implications

• If you have only one test form, and you
  computerize it, presenting more than one item at
  a time may reduce memorization.
• Memorization rates are relatively low.1 item for
  complete memory2 items for partial/incorrect
  memory
• However, the effect is compounded if there are a
  large number of examinees, or if there is
  collusion between examinees.
• It only takes ONE person to disclose item.