1. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Considerations In Designing a Conformity Assessment System for Voting Equipment Presented at The Voting Systems Testing Summit November 28-29, 2005 Sacramento, CA

2. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Purpose of Presentation This presentation discusses: Elements common to conformity assessment systems The EAC system for certification and decertification of voting equipment Considerations for state and local programs Potential for unintended consequences

3. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Certification of a product is a means of providing assurance that it complies with specified standards and other normative documents. Several types of certification systems exist and are under review for the EAC system: Some comprise type testing only Other systems include initial testing and field surveillance Still others include initial testing of a product and assessment of its suppliers’ quality systems, followed by routine audits that take into account the factory quality system and the testing of samples from the factory and the open market.

4. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Components of a Certification System Initial Type Testing Assessment of the Supplier’s Quality System Field Information & Feedback User Involvement

5. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Key Participants EAC State certification authorities Testing Laboratories Vendors State & local election officials

6. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Key Issues for Certification System What is a minimal acceptable system? Are the testing lab/testers/lab assessors qualified? Will the vendor deliver units within manufacturing tolerances to those tested? How will the election officials know if non-compliant units are delivered and what corrective actions can it take? Will election officials and poll workers use the systems as intended?

7. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Primary Consideration National ProgramThe primary concern of the EAC program is that the system design, as represented in the system submitted for testing, meets minimum requirements. State ProgramState certification seeks to assure that: 1.Systems meet specific state requirements. 2.May be used as that state administers elections. 3.The vendor can give reasonable assurance that the units delivered meet and continue to meet the requirements over their useful life.

8. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Primary Consideration Purchase EvaluationWhich system offers the best value among systems offered. Receiving InspectionAre the delivered systems functional and within manufacturing tolerance of those certified by the EAC and state officials? Pre & Post ElectionAre the systems functioning correctly and unaltered from their certified condition?

9. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Coordinated Processes Processes are necessary to implement the components of a conformity assessment system for voting systems and to provide reasonable assurance that the key concerns of such a system are carefully addressed. The best confidence will be gained if the various authorities involved in certifying and selecting voting equipment coordinate their efforts so as to maximize the value contributed from each function.

10. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB EAC / NIST TGDC Technical Guidelines EAC / NIST TGDC Technical Guidelines Candidate System Accredited Test Laboratory Accredited Test Laboratory EAC Technical Review Team Technical Review Team Certified System System Description System Description NIST Software Ref. Lib. NIST Software Ref. Lib. Vendor Quality & Configuration Control Processes Vendor Quality & Configuration Control Processes State Certification State Certification

11. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Certified System System Description System Description NIST Software Ref. Lib. NIST Software Ref. Lib. Vendor Quality & Configuration Control Processes Vendor Quality & Configuration Control Processes State Certification State Certification Field Deployment and Use Purchase Selection Pre & Post Election Testing Pre & Post Election Testing

12. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB EAC / NIST TGDC Technical Guidelines EAC / NIST TGDC Technical Guidelines Accredited Test Laboratory Accredited Test Laboratory EAC Technical Review Team Technical Review Team NIST NVLAP & Software Ref. Lib. NIST NVLAP & Software Ref. Lib. Vendors State & Local Election Officials Citizens Local Receiving Inspectors Local Receiving Inspectors

13. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB System Description The ability to determine that a system is identical to that considered at other steps in the process is critical to the validity of the conformity assessment system.

14. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Typical FCC Submission The FCC routinely requires external AND internal photos accompany all submissions. For Software Controlled Radios source code is required for an equipment grant.

15. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Typical FCC Submission External Photos

16. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Typical FCC Submission Internal Photos

17. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Challenge for Vendors Is there a way by which high confidence many be delivered that a system is exactly the same without also transmitting proprietary information?

18. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Considerations How does each step of the process assist the next? Could the ITA’s include a section in each test report highlight information and observations that would be helpful to state certification personnel? Could state certifiers transmit observations to those making purchasing decisions?

19. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Considerations What is the repeatability of these evaluations? How often will the same system receive the same evaluation if submitted at different time or to different ITA’s?

20. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Considerations How does field performance and field experience information come back into the system to improve it? How will field experience come into the system to improve EAC and state certification processes?

21. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Unintended Consequences Cost of certification will limit the number of systems offered and may eliminate vendors. With more rigorous testing and evaluation vendors will pay for fewer systems to be certified. Vendors will bring new systems to market less often. Some vendors may abandon this business.

22. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Unintended Consequences Will the time and expense of certifying changes and upgrades delay good and needed changes? What is the right balance between careful evaluation of all changes and not making these evaluations so costly that needed improvements are delayed or not introduced at all?

23. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Unintended Consequences How will the necessary coordination of administrative processes and equipment requirements be maintained and evaluated? Election management and equipment requirements must work together. How will this be assured? Where several alternative solutions are possible how will the linkage be maintained between equipment requirements and election management practices?

24. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Proposals Documents are needed that define the interaction and expectations of each participant in the process. Direct communications from each step in the process to succeeding steps may be very helpful. Clarification is needed as to what is not done in preceding steps in the process.

25. © Copyright 2005 TEM Consulting, LP - All Rights Reserved Presentation To EAC Aug. 23, 2005 Hearing, Denver, CORev 1 – 08/16/05 - HSB Conclusion A well constructed certification system provides satisfactory answers to central issues: What is a minimal acceptable system? Are the testing lab/testers/lab assessors qualified? Will the vendor deliver units within manufacturing tolerances to those tested? Will the election officials know if non-compliant units are delivered and what actions can they take? Will election officials and poll workers use the systems as intended?