VVSG 1.1 Test Suite Status

1
VVSG 1.1 Test Suite Status

• Mary Brady
• Manager, NIST Information Systems Group, Software
  and Systems Division, ITL
• http//vote.nist.gov

2
Background

• Status quo Labs have been testing to VVSG 1.0
  (2005) using proprietary, custom tooling and
  review processes
• In 2007-08, NIST developed a set of public test
  suites for VVSG 2.0, to be used as part of the
  EAC Testing and Certification Program
• In 2009, to support VVSG 1.1, test methods for
  new and changed material was back-ported from 2.0
  test suites status quo prevails for everything
  else

3
Test Validation

• NIST test suites have been rigorously traced to
  VVSG requirements
• Reviewed by independent parties, including the
  EAC, VSTL labs, experts, and the public
• Procedural validation has been on-going
• Operating temperature and humidity -- complete
• Usability and accessibility ongoing
• Core functionality, security under consideration
• Pursued obtaining in-house voting systems to aid
  in the evaluation of NIST test suites

4
Recent Activities

• NIST met with each accredited VSTL to discuss
  possible integration of core functionality tests
• Recent activities regarding changes to VVSG 1.1
  will require changes to the NIST developed test
  suites delayed status of VVSG 2.0 warrants
  additional focus on improving testing for VVSG
  1.1, including existing methods for VVSG 1.0
• NIST and EAC met with both accredited VSTLs to
  discuss ways to improve testing for the evolving
  VVSG 1.1 and to integrate NIST-developed tests
  into the certification process

5
Preliminary Findings

• To ensure consistency across testing
  laboratories, agreed upon test methods must be
  developed for all of VVSG 1.1
• Face-to-face meetings revealed differences in
  interpretation of the term readiness test.
  Labs read as acceptance test whereas NIST
  definition was a simple health check
• Some NIST tests require new election definitions,
  which are problematic for the labs
• Labs define test data separately from the test
  case NIST defines them together

6
Additional Findings

• Some NIST tests should be reworked to take into
  account practical test methods, with temperature
  and humidity cited as an example
• Source code analysis represents a large
  percentage of the cost of testing tools remain
  difficult to use
• Reference ballots, including guidance on how
  ballots should be marked (pre-marked,
  hand-marked), election definitions/voting
  patterns to be used, and complexity of
  precincts/ballot styles, would be useful
• Should identify priority areas outside of VVSG
  1.1

7
Action Plan

• Monthly meetings among EAC, NIST, VSTLs, with
  bi-annual face-to-face meetings
• Focus on developing common test methods for new
  VVSG 1.1, starting with information from EAC
  matrix, NIST test suites, and existing VSTL
  definitions
• Validate NIST test suites and integrate into
  certification process
• Expand effort on source code analysis tools and
  work with the VSTLs to develop rules for VVSG
  1.1
• Explore creating a set of reference ballots to be
  used by all labs

8
Discussion/Questions
Page 8