VVSG and Requirements Management Ed Smith January 13, 2011 Questions/Comments: Ed Smith ed.smith@dominionvoting.com

Opening Thoughts VVSG some of the most important requirements for voting systems and their surrounding development system Poorly executed RM is a cause of project and product failure Requirements elicitation is important Organizing Development Systems Product: software, hardware, voter/pollworker interfaces

Requirements Definition If you don’t have a good requirements document, you needn’t bother with the rest of design controls. Throughout the development process, the requirements document acts as the surrogate for the actual user needs. It’s the ground truth that you come back to over and over to help decide how to implement the design. At each stage of development, the requirements establish the basis for verification of the design Corporate managers I talk to have a lot of misconceptions about this business of requirements, so let’s look a little closer.

Before it all starts… The Introductions Both Volume I and Volume II History, Intent, Summaries of each Volume Much the same for both Introductions Overview in Volume I Glossaries TDP (Technical Data Package) production Volume II, Section 2 Description of the Technical Data Package

Before it all starts… VVSG requirements for Quality Assurance VVSG requirements for Configuration Management The testing also evaluates the completeness of the vendor’s developmental test program, including the sufficiency of vendor tests conducted to demonstrate compliance with stated system design and performance specifications, and the vendor’s documented quality assurance and configuration management practices. The tests address individual system components or elements, as well as the integrated system as a whole. Select a development method Educate Developers with respect to VVSG

Then Development Starts Each section results in inputs to documentation Requirements Specification Functional Specification Engineers’ Documents/Technical Solution Test Specification/Test Plan Test Cases Manufacturing Specification Project Plans Traceability

VVSG CUST PREFERENCE CM PLAN QA PLAN CERT PROG MANUAL REQUIREMENTS SPECIFICATION LAB PROG MANUAL FUNC SPEC Bi-directional traceability EXT REFERENCES PROTOTYPE STATE STATUTE TEST DOCS

Then Development Starts Each section results in inputs to documentation Volumes I and II result in Quality Assurance Planning Audits Product Testing Reliability Accuracy Durability Real World Validation vs. Verification

Then Development Starts Each section results in inputs Overview Voluntary Voting System Guidelines Section 1 Introduction Section 2 Functional Requirements Section 3 Usability and Accessibility Requirements Section 4 Hardware Requirements Section 5 Software Requirements Section 6 Telecommunications Requirements Section 7 Security Requirements

Requirement: voting machine allows access to a range to needs Sub-requirement: …including low vision and lack of motor control of hands Sub-requirement: machine shall have a tactile box with buttons for… and a 3.5mm input Sub-requirement: Braille shall be embossed on each tactile button CM PLAN QA PLAN REQUIREMENTS SPECIFICATION FUNC SPEC Function Spec: Tactile box shall allow for left/right, Volume and speed… Tactile box 3.5mm input shall accept 5 volts, stereo… Braille dots shall be 1mm hemispherical Bi-directional traceability PROTOTYPE TEST DOCS

Tactile box shall allow for left/right, Volume and speed… Function Spec: Tactile box shall allow for left/right, Volume and speed… Tactile box 3.5mm input shall accept 5 volts, stereo… Braille dots shall be 1mm hemispherical CM PLAN QA PLAN REQUIREMENTS SPECIFICATION Test Spec: With an election containing… Ensure function of all tactile buttons Ensure function of 3.5mm input Ensure readability of Braille Test Script: Code an election… Attach tactile box… Scroll through the ballot using… Attach a power supply and input 5V… FUNC SPEC Bi-directional traceability PROTOTYPE TEST DOCS

Requirements---Guiding Principles Must specify what is needed, not the solution (example of machined aluminum housing) Complete to an engineering level of detail Requirements are developed by engineers, not by marketing department or users Capturing the requirements may consume as much as 30 percent of the entire project time budget First bullet: housing example machined aluminum vs. casting or plastic. Reduced time to market; superior EMI shielding, stronger than cast, but much more expensive in volume production. It’s OK to specify the solution if there is a good reason and that reason is explicitly stated. Second bullet: electrical example AC power. 120 (North America), 240 (Europe), or 100 (Japan). 108-132, or 95to 132. What is appropriate for the intended use? Third bullet: 1987 case study example. When I asked for requirements document, they gave me a five-page document prepared by their marketing dept. It was the level of detail you might see on a glossy marketing brochure, grossly inadequate for purposes of implementing the design. You need a lawyer to write a will, and you need an engineer to write a requirements document. Fourth bullet: This is a tough sell for corporate managers from the old school. Right now, FDA is undergoing “reengineering” of our regulatory processes. Management want to see pilot programs, they want results they can brag about at Congressional hearings and industry forums. As I mentioned earlier, part of the solution is for managers to understand the design and development process better and be able to track the paper deliverables. Bean counters need something to count. They are almost as happy counting documents as hardware items.

Requirements---Guiding Principles Unambiguous (objectively verifiable) Quantitative limits expressed with a realistic measurement tolerance Self-consistent Environment completely characterized Completeness and relevance of external references First bullet: Ex. “finish shall be durable”. When challenged, they said that it should withstand repeated cleaning as described in the service manual. OK, but it’s ok to scrape off big chips of paint every time another instrument bumps into it in the O.R. The best approach is to specify some objective durability test, often a consensus standard test method, and let a qualified reviewer determine whether the test is representative of the intended use. Second bullet: Quantitative limits can go to either extreme. If the requirement is for a 10 centimeter hole, the designer must infer what that means, and you are likely to end up with grossly inaccurate solution, or an overly complex and expensive solution. If you say you need a hole big enough to accommodate a mechanic’s gloved hand, that’s better, but the designer has to do some legwork to figure out what the actual dimension needs to be before designing the component and its production process. If you say the diameter must be a minimum of 14 cm to accommodate a mechanic’s gloved hand, the designer has enough information to implement the design, and a qualified reviewer has a basis for determining whether the requirement is is appropriate for the intended use. Last bullet. Completeness and relevance of external references (MIL-ST-810 example)

Lack of RM Allows Scope Creep Developing the wrong product

Requirements imprecision Problems arise when requirements are not precisely stated Ambiguous requirements may be interpreted in different ways by developers and users Consider the term ‘appropriate viewers’ User intention - special purpose viewer for each different document type Developer interpretation - Provide a text viewer that shows the contents of the document