Security in a High Stakes ComputerBased Testing Environment

1
Security in a High Stakes Computer-Based Testing
Environment

• Bruce Biskin
• Chuah Siang Chee
• American Institute of Certified Public Accountants

2
Uniform CPA Examination

• Has been produced by the American Institute of
  Certified Public Accountants (AICPA) since 1917.
• Used for licensing certified public accountants
  in the US by all 54 boards of accountancy since
  1968.
• High Stakes Test
• Since April 2004, the CPA Exam has been
  administered by computer at over 400 secure
  testing centers.

3
Test Security

• Security is a concern with any high stakes test.
• Paper-based program had its own set of risks
• theft of booklets along the distribution path
  from printing to scoring
• copying at the testing centers
• CBT has its own set of risks
• candidates who test early could memorize test
  material and make this information available to
  candidates testing later

4
CBT Vs Paper-and-Pencil

• CBT
• Continuous Testing Window
• Secure computer systems allow for almost
  instantaneous deployment
• Encrypted Test Materials
• Exposure of items can be controlled

• Paper-Pencil
• Single administration of test form
• Transportation of items
• Forms can be lost/stolen
• Test items predetermined

5
Two Programs to Detect Security Breaches

• Crawling the Web for Bad Behavior
• Data forensics for Security Leaks

6
Crawling the Web for Bad Behavior

• Internet has provided an unprecedented capability
  for making information available almost instantly
  to millions of people at an extremely low cost.
• The main goals were to determine whether actual
  Exam content was being posted, offered, sold, or
  shared
• on Web sites
• in public and semi-public discussion groups
• in cyber-markets such as e-bay.

7
Cost

• Web crawling can be very expensive.
• Cost must be weighed against the potential cost
  of failure to identify possible security breaches
  in a timely manner.
• Costs are monetary and political, and can impact
  the credibility of the exam program.
• The Web-crawling strategy them should be based on
  risk and cost-benefit analyses.
• Prioritize activities and constrain scope.
• Conduct work in-house or by contract.

8
Potential Benefits

• True Positives can result in
• Reduced financial and political impact of exposed
  exam material.
• Improved credibility of exam program.
• Maximum validity of decisions based on scores.
• True Negatives can result in
• Maximum confidence in fairness of exam.

9
Potential Costs

• False positives can result in
• High follow-up costs
• Potential legal exposure for false accusation or
  damages if credentialing delayed
• Ill-will of stakeholders
• Decreased validity of scores for decision-making
• False negatives can result in
• Perceptions of unfairness by stakeholders
• Decreased validity of scores for decision-making

10
Base Rates

• Higher incidence of Web-based events will result
  in higher tangible return on investment
• Lower need to replace exam materials through
  early detection
• Demonstrated ability to detect Web-based
  eventsand publicizing actions takenmay
  discourage additional events
• Lower incidence of Web-based events will result
  in higher intangible return on investment
• Greater confidence in the integrity of examinees
  and the associated practice of credential
  holders, but
• Cost per detected incidentif any--may be very
  high

11
Recommendations

• Estimate costs of Web-crawling activities
• Identify resources (internal and external)
• Identify risks and benefits
• When using external resources
• Identify vendors
• Check the scope of services
• Check references
• Ask about quality of service
• Ask about costs vs. benefits
• Match your needs with those of references

12
Data Forensics for Security Leaks

• Another strategy is to identify and evaluate
  changes in statistical characteristics of test
  material, which can signal possible security
  breaches.
• Procedures can be as simple as looking for
  unusual changes item difficulty within and across
  testing windows, or identifying regional shifts
  in item characteristics. They may also be
  complex, involving fitting individual (or groups
  of individual) test-taker response patterns to
  models of aberrant responding.

13
IRT

• Given ability/Theta we can estimate the
  probability of an examinee getting an item
  correct.
• 3-Parameter Logistic Model
• P(?) is the probability of a correct response
  given ability/theta
• D is a scaling constant equal to 1.702
• a, b, and c are the parameters characterizing an
  item.

14
Estimating the Expected P-value

• Compute the probability of a correct response for
  each examinee, given ability/theta.
• The average probability for a given item across
  examinees should approximate the proportion of
  examinees correctly answering the item.

15
Adjusting for Ability

• Natural fluctuations in the average ability of
  examinees.
• Minimize false positives.
• Compute the difference between expected and
  real/observed p-value.
• Difference score indicates if examinees are
  performing above or below expectations.
• Concerned if it is significantly above
  expectations as it may indicate that cheating is
  occurring.

16
Difficulty in Interpreting Results

• Even with corrections, there are some minor
  fluctuations.
• Use historical trend of item behavior.
• Keep in mind sample size when interpreting
  results.

17
Sample Results
18
Conclusions

• Ignoring problem isnt going to help.
• Relatively simple system.
• Relatively inexpensive system.

19
Recommendations

• What do you do after you have flagged a suspect
  item? Have clear policies and procedures
• Automating the system so that it does not become
  an organizational burden.