Making every vote count. United States Election Assistance Commission EAC Voting System Certification TGDC Meeting December 9-10, 2009

U.S. Election Assistance Commission Program Roles/Structure Decision Authority (Exec. Director) Program Director Project Manager VSTLManufacturer Technical Reviewers Computer Engineers Appeal Authority (EAC Commissioners )

U.S. Election Assistance Commission Manufacturer Registration Manufacturer submits Manufacturer Registration Application Form (EAC Form 001C) EAC sends letter to manufacturer acknowledging receipt of application form and process initiation Incomplete form: EAC notifies manufacturer and requests information Successful application review EAC notifies manufacturer that it has been registered and issues identification code Manufacturer completes and submit missing information Manufacturer maintains registration up to date EAC includes Manufacturer on listing of registered voting system manufacturers II. Suspension of Registration I. Initial Registration Manufacturer violation of T&C Manual’s procedural requirements EAC provides written notice and deadline for Manufacturer’s response Manufacturer responds or provides cure EAC approves action; prevents suspension Manufacturer does not respond nor provides cure EAC issues notice to Manufacturer of intent to suspend registration Manufacturer submits objections and supporting information EAC makes decision and notifies Manufacturer For information on Effects of Suspension, please, refer to Section of the Testing and Certification Manual.

U.S. Election Assistance Commission Certification Testing and Technical Review Initial Decision on Certification Manufacturer submits certification application to EAC EAC reviews package for completeness and accuracy VSTL develops Test Plan (TP) Incomplete form: EAC notifies Manufacturer and requests information Manufacturer completes and submit missing information Successful application review EAC notifies Manufacturer and provides unique application number EAC approves TP VSTL conducts testing and submits test reports to EAC EAC Program Director reviews Test Report (TR) for completeness Review of TR by EAC Technical Reviewers Manufacturer submits Technical Data Package (TDP) to VSTL EAC Program Director reviews Technical Reviewers response to TR Recommends certification – forwards to Decision Authority for action Refers matter back to Technical Reviewers for additional specified action and resubmission Decision Authority issues an Initial Decision on Certification For more information on Initial Decision on Certification, please, refer to Chapter 4 of the Testing and Certification Manual.

Making every vote count. United States Election Assistance Commission The Testing Process: Test Plans, Test Cases, and Test Reports

Making every vote count. United States Election Assistance Commission The Testing Process from Beginning to End (in an ideal world): 1.Application for Testing 2.Creation, submission, and review of Test Plan 3.Creation, submission and review of test cases 4.Testing of the Voting System 5.Creation, submission and review of the test report (This is how things have actually gone)This is how things have actually gone

Making every vote count. United States Election Assistance Commission The Test Plan Package Test Plan Contains high level test methods for the testing engagement.Test Plan Contains standard lab test methods, system specific test methods, and pass/fail criteria. Contains EAC Requirements Matrix.Requirements Matrix EAC Technical Reviewers review test plan for completeness and to ensure system is tested to the standards. EAC sends comments to test plan to VSTL for review and response. VSTL submits new version of test plan for further review.

Making every vote count. United States Election Assistance Commission Test Case Submission After approval of Test Plan VSTL creates system specific test cases. Test cases are the step by step instructions given to test personnel to test the system.Test cases VSTL is to notify EAC in writing when test cases are completed. EAC may then request the test cases to be submitted to them for review. Test cases are not approved or rejected by EAC. EAC does spot check of various test cases in order to ensure test plan is being followed and system is being tested to the standards.

Making every vote count. United States Election Assistance Commission Testing of the Voting System Test cases used to test the voting system to the VSS or VVSG. VSTLs are responsible for the quality and level of testing. This includes review and approval of 3 rd party lab testing. Maintaining the independence of the testing process. Documentation and resolution of all discrepancies and engineering change orders.

Making every vote count. United States Election Assistance Commission Test Report Submission and Review Test Report Package represents the documentation of the testing performed and the results of that testing. VSTL submits as part of the package: Requirements Matrix Same matrix as before but with the test report information complete Test Report Documents the testing process and results Includes all discrepancies, resolutions, and engineering change orders. VSTL attestation VSTL makes recommendation and asserts all testing was done per the approved test plan.

U.S. Election Assistance Commission Certification Testing and Technical Review Grant of Certification Decision Authority issues Manufacturer an initial decision to grant certification Manufacturer subjects software to a trusted build Manufacturer deposits software in EAC- approved repository Manufacturer provides system identification tools Manufacturer submits documentation to EAC Program Director EAC Program Director reviews documentation and makes recommendations – notifies Decision Authority Decision Authority issues Agency Decision granting certification Certification Number Cert. of Conformance EAC publishes certification documents on website Manufacturer posts mark of certification on all EAC-certified voting systems produced For more information on Grant of Certification, please, refer to Chapter 5 of the Testing and Certification Manual. Manufacturer informs purchasers that changes or mods not tested and certified by EAC shall void the EAC certification

U.S. Election Assistance Commission Decertification For more information on Decertification, please, refer to Chapter 7 of the Testing and Certification Manual. Decertification Policy System is shown not to meet applicable voting system standard System has been modified or changed without following requirements of Manual Failure to follow Manual procedures so the quality, configuration, or compliance of the system is in question I.Decertification Policy – Reasons for decertification (Systems will be decertified only after completion of the process outlined in Chapter 7 of the Testing and Certification Manual)

U.S. Election Assistance Commission Decertification For more information on Decertification, please, refer to Chapter 7 of the Testing and Certification Manual. Report of Investigation* Matter closed – report forwarded to Manufacturer Unsubstantiated Allegation (in whole) Substantiated Allegation Decision Authority issues Notice of Non- Compliance III.Formal Investigations Formal Investigation conducted (7.4.5) Decision Authority authorizes initiation of an EAC Formal Investigation Notice given to Manufacturer of the scope of investigation * When Complete. The report is complete and final when certified and signed by the Decision Authority. * Publication of Report. The report shall not be made public nor released to the public until final.

U.S. Election Assistance Commission Quality Monitoring Program Quality Monitoring Methodology Manufacturing Site Review Field Anomaly Reporting Fielded System Review and Testing For more information on the Quality Monitoring Program, please, refer to Chapter 8 of the Testing and Certification Manual.

Where did the EAC’s program begin? HAVA mandates the EAC to Test, Certify, and Decertify Voting Equipment. December 13, 2005 the EAC approves the 2005 VVSG. January 2007 the EAC’s Voting System Testing and Certification Program begins. Represents the 1 st time the federal government has tested and certified voting systems January 18, 2007 NIST/NVLAP recommends first laboratories for accreditation by EAC. February 21, 2007 EAC accredits first two VSTLs February 22, 2007 EAC receives first application for testing. January 22, 2008 EAC approves first test plan February 6, 2009 EAC certifies first voting system, Microvote EMS 4.0. Making every vote count. United States Election Assistance Commission

Where is the EAC’s Program Now? 13 Registered Manufacturers 3 Voting Systems Certified Premier Assure 1.2 ES&S Unity 3.2 MicroVote EMS Accredited Voting System Test Laboratories (VSTL) CIBER Inc. Wyle Laboratories iBeta Quality Assurance Systest Labs 6 voting systems currently under test in EAC Program6 voting systems 21 Requests for Interpretation issued 13 Notices of Clarification Issued Making every vote count. United States Election Assistance Commission

Making every vote count. United States Election Assistance Commission What have we learned? An adjustment period was necessary All parties adjusting to the new program Labs testing to new standard with new oversight. Manufacturers subject to more rigorous testing and oversight. EAC adjusting to running a new program and the needs of all stakeholders EAC looking to create efficiencies Creating internal SOP’s for all program activities. Looking to create more efficient testing Update the standards Implement test suites Improve communication Synch test schedule and EAC review