How to Know That What You Want Has Been Done- 1 Claire Lohr Member, SESC Management Board Chair IEEE 829 Working Group How to Know That What You Want Has Been Done STC May 1, 2002

How to Know That What You Want Has Been Done- 2 Topics Sources of the information presented Inclusion of best practices from industry Definition of the steps for recognition of successful completion of software projects Applicable references NOTE: Best practices marked with

How to Know That What You Want Has Been Done- 3 Sources of this Information Highly competitive commercial industries –Semiconductor manufacturing –PC manufacturing –Insurance –Finance –Telecommunications –Etc. Some succeed Some don’t

How to Know That What You Want Has Been Done- 4 The Challenge Current formal definitions of quality are … Too broad for software Real need is to attain the customer’s own specific, detailed quality goals

How to Know That What You Want Has Been Done- 5 Steps for Recognizing Quality Step #1: define what you don’t want Step #2: define what you do want with sufficient detail Step #3: hire and train good people Step #4: prepare to measure the results (test outputs) in parallel with defining the target (requirements) Step #5: refine with business analysts throughout lifecycle

How to Know That What You Want Has Been Done- 6 Step #1 Define what you don’t want (the business case) “ IT buyers are mad and aren’t going to take it anymore … Up to 50% of software cost results from ongoing maintenance and integration activities … We spend so much time and money dealing with software problems, it's ridiculous … Some patches break more things than they fix.” Karyl Scott Information Week January 14, 2002

How to Know That What You Want Has Been Done- 7 Step #2: Define What You Do Want Define what you do want (general) –From the Scott article “The Software Bill of Rights, released this week [Jan ], stipulates that buyers have the right to expect a quality product, accurate delivery schedules, explicit pricing, development accountability, and effective technical support.” –Universally accepted definitions of quality Meets specifications Meets customer expectations Fitness for use

How to Know That What You Want Has Been Done- 8 Define what you do want (specific) –Document requirements and SQA IEEE Std Recommended Practice for Software Requirements Specifications IEEE Std Standard for Software Quality Assurance Plans –Document requirements with Use Cases for more clarity Visual Shows flows Includes everybody –Plan testing in parallel from the beginning IEEE Std Standard for Software Test Documentation Step #2: Define What You Do Want

How to Know That What You Want Has Been Done- 9 Use Case Example Step #2: Define What You Do Want Attend IEEE Track Use case Actor SEPG Member Association Attend STC Order SESC standards Visit computer.org

How to Know That What You Want Has Been Done- 10 Step #3: Good People Hire and train good people (a good process enhances talent does not replace it) –Thorough employment screening –“Boot camp” training for entry level new hires –Employee friendly work environment ($ to $$$$) –Low turnover –Easy access to training Incorporate standards for process and documentation Use real examples online and in class

How to Know That What You Want Has Been Done- 11 Step #4: Prepare Tests and Requirements Prepare to measure the results (test outputs) in parallel with defining the target (requirements) –Helps developers clarify their thinking when answering tester’s questions –Assures testability of the final software –Defines “pass criteria” for the software early in the life cycle –Allows time for acquisition or development of test tools (less use of tools leads to less testing)

How to Know That What You Want Has Been Done- 12 Test Documentation (from IEEE Std ) Test Exe- cution Test Plan Test Design Specs Test Proc Specs Test Case Specs Test Log Test Incident Reports Test Summary Report Always Tailored Real examples available on- line Step #4: Prepare Tests and Requirements

How to Know That What You Want Has Been Done- 13 Test Plan Outline 1. Test-plan identifier 2. Introduction 3. Test items 4. Features to be tested 5. Features not to be tested 6. Approach 7. Item pass/fail criteria 8. Suspension criteria and resumption requirements 9. Test deliverables 10. Environmental needs 11. Responsibilities 12. Staffing and training needs 13. Schedule 14. Risks and contingencies 15. Approvals Step #4: Prepare Tests and Requirements

How to Know That What You Want Has Been Done- 14 Add to the front-end of the lifecycle –Requirements for testing Choose how many levels of testing for this project Choose documentation for this project Choose metrics –Master Test Plan Over 10 million LOC product line (MANY levels of test to sort out) Identify test levels Hand-off criteria between levels for a template example Test Documentation Tailoring Options

How to Know That What You Want Has Been Done- 15 Skip documents –Test Plan Management issues like schedule, and staffing are covered in other documents These issues are not under control of testers Combine documents –Specifications –Procedures –Log Test Documentation Tailoring Options

How to Know That What You Want Has Been Done- 16 Example of combination of Std 829 Specification, Procedures, and Log in MS Word on the next slide for an example of this template with an explanation of all of the fields Tailoring Options

How to Know That What You Want Has Been Done- 17 Tailoring Options

How to Know That What You Want Has Been Done- 18 Design Require- ments Code Unit and System Tests Accept- ance Test Test Plan V1.0 Revise Test Plan and develop the rest of the test documentation Test Doc’s V2.0 Test Doc’s Vx.y Step #4: Prepare Tests and Requirements

How to Know That What You Want Has Been Done- 19 Plan coverage measures for tests; examples: –Source Lines of Code (SLOC) –John Musa’s Operational Profile Use detailed checklists for test case development (examples for web site testing follow) Put the checklists in an understandable framework Step #4: Prepare Tests and Requirements

How to Know That What You Want Has Been Done- 20 Step #4: Prepare Tests and Requirements Web testing checklist framework

How to Know That What You Want Has Been Done- 21 Step #4: Prepare Tests and Requirements

How to Know That What You Want Has Been Done- 22 Step #5: Refine with Business Analysts Refine definition of “correct” for software throughout lifecycle –Testers have constant access to “business” analysts (similar to JAD) They use a combination of formal and informal process There is a continuity of personnel (same business analyst from project to project) Business analysts are physically located nearby Authoritative answers for questions (typical testing challenge) Business analysts help deploy to in-house user community

How to Know That What You Want Has Been Done- 23 Summary Define what you think you want Keep effective communication open to refine requirements Develop specific tests to prove the goals have been met Retain good people and maximize their potential

How to Know That What You Want Has Been Done- 24 References IEEE Std Standard for Software Quality Assurance Plans IEEE Std Standard for Software Test Documentation IEEE Std Recommended Practice for Software Requirements Specifications