A Comparison of Traditional and Simulation Assessment Memorability and Security

1
A Comparison of Traditional and Simulation
Assessment Memorability and Security

• Alan D. Mead
• PAQ Services, Inc.

not really
2
Exam security is important

• We have all heard about cheating
• When examinees are organized, exams can be stolen
  quickly
• Internet braindump sites can organize examinees
• Smith (2004) found that IT certification exam
  items quickly accumulated on braindump sites

3
Past Literature

• Administration Security
• Revuelta Ponsoda, 1998
• Effects of retesting or compromise
• Lievens, Buyse Sackett, 2005
• Smith, 2004
• Item Theft
• Schnipke Scrams, 1999
• Chang Zhang, 2003

4
Memorability

• Chuah investigated item memorability in a
  cognitive ability test
• Easier items were more memorable
• Relatively easy/hard items were more memorable
• Guessable items were more memorable
• Longer items were more memorable
• Smarter people remembered more
• Item type was also related

5
Enter Simulations

• Simulations
• Much richer and more immersive than traditional
  multiple-choice or Likert assessments
• Computer technology is making sims increasingly
  possible
• Many organizations and stakeholders prefer
  simulations as having greater face validity

6
Security unknown

• Are simulations more or less secure?
• Immersion may be facilitate later recall
• Greater amount of materials might make it harder
  to remember and disseminate
• Greater cost probably makes any theft much more
  expensive to test developers
• Champlain, et al, 1999

7
Present Research

• Compare memorability of three kinds of
  assessments
• Cognitive ability
• Personality
• Bank Teller Work Simulation
• Assess the damage of memorized information
• Are important aspects or trivialities recalled?
• Can Ss effectively communicate memorized
  information about sims?

8
Procedure

• Administer assessments in a counter-balanced
  order
• Administer a distracter task
• Recollection procedure
• Ask Ss to write an email to help another
  test-taker do well on the tests
• Code the emails
• Number of items from each exam
• Damage of the recalled information

9
Hypotheses/Questions

• Hypotheses
• Recency effects
• Information from the cognitive ability test is
  easier to transmit
• Is a simulation more or less memorable than a
  traditional assessment?
• Is recollection and communication of simulation
  content more or less damaging than communication
  of traditional assessment material?

10
Thank you for attending!

• I will be happy to send copies of the paper when
  it is available
• amead_at_alanmead.org