CIS-74 Computer Software Quality Assurance Systematic Software Testing Chapter 3: Master Test Planning

Test Plan Hierarchy Master Test Plan (MTP) Unit Test Plan Acceptance Test Plan Integration Test Plan System Test Plan (These last four are covered in Chapter 4.)

MTP The process is more important than the actual document. As the complexity of a testing activity increases, the criticality of a good MTP increases exponentially (see diagram on p. 57).

IEEE Test Plan Template 1. Test Plan Identifier Unique Generated by company Identifies: Version of test plan Level of test plan Version of software covered

IEEE Test Plan Template 2. Table of Contents Added by authors of text Two or more levels preferable

IEEE Test Plan Template 3. References Included in Introduction of IEEE template Given own section by text’s authors Recommended: Project authorization Project plan QA plan Configuration management plan Relevant policies Relevant standards

IEEE Test Plan Template Glossary Added by authors of text

IEEE Test Plan Template Introduction Scope of project - key features, history, etc. Scope of test plan (levels covered and not covered).

IEEE Test Plan Template 6. Test Items Programmatic description of what is to be tested High-level test plan: application or version Low-level test plan: program, unit, module, or build Items to be excluded from testing also

IEEE Test Plan Template Software Risk Issues List of risks from formal risk analysis (covered in Chapter 2).

IEEE Test Plan Template 8. Features to be Tested From customer point of view Basically, those items above the “cut line” from formal risk analysis

IEEE Test Plan Template 9. Features Not to Be Tested From customer point of view Basically, those items below the “cut line” from formal risk analysis Amusing point on pp. 68-69: Managers who raise their eyebrows about this section of a test plan are often the ones who approve schedules that don’t allow enough time to test everything.

IEEE Test Plan Template 10. Approach (Strategy) Heart of test plan Should explain approach for each level Should specify entrance and exit criteria for each level to another

IEEE Test Plan Template 10. Approach (continued) Test environment is most artificial for unit testing, followed by system testing, and then integration testing. Testing environment is closest to reality for acceptance testing.

IEEE Test Plan Template 10. Approach (continued) Methodology decisions What testing levels will be used? When will testers become involved? Will there be a beta? Will there be an alpha? What testing techniques will be used? Etc.

IEEE Test Plan Template 10. Approach (continued) Resource decisions Is dedicated test group sufficient? If not, where else can resources be obtained? Developers? Users? Interns? U. S. contractors? Overseas contractors? Customer support?

IEEE Test Plan Template 10. Approach (continued) Test Coverage Decisions Code coverage measures the percentage of program statements that are executed by a set of test cases. Requirements coverage measures % of business requirements covered. Design coverage measures the % of design covered. Interface coverage measures % of interfaces covered.

IEEE Test Plan Template 10. Approach (continued) Walkthroughs and Inspections Reviews are another form of software evaluation besides testing. Walkthrough - talking/walking through each line of code as a group Inspection - more formal, may be done by a single individual

IEEE Test Plan Template 10. Approach (continued) Configuration Management Can be done in a separate document In MTP, should cover change management and bug-review process

IEEE Test Plan Template 10. Approach (continued) Collection and Validation of Metrics Metrics can be a significant overhead, so it’s necessary to plan exactly which are needed. More info coming in Chapter 10 - The Test Manager.

IEEE Test Plan Template 10. Approach (continued) Tools and Automation Caution! It can take more time to automate than the time it takes to execute tests manually! Be sure automated tests will be re-run often to justify the cost of automation.

IEEE Test Plan Template 10. Approach (continued) Changes to the Test Plan What type of changes require redoing the approval process? How should the test plan be published? How should the review be conducted? Should the test plan go through regular CM?

IEEE Test Plan Template 10. Approach (continued) Meetings and Communications Standard meetings Status reporting Metrics

IEEE Test Plan Template Item Pass/Fail Criteria Pertains to items from Section 6.0 - Test Items - NOT to test cases Examples: Test cases passed and failed Number, type, severity, and location of bugs Usability Reliability Often called “release criteria.”

IEEE Test Plan Template Suspension Criteria & Resumption Requirements Large volumes of bugs Critical bugs Incomplete tasks Code not checked into source-code control system

IEEE Test Plan Template Test Deliverables All testware to be created and maintained for the testing effort.

IEEE Test Plan Template Testing Tasks Authors recommend deleting this section, and listing all testing tasks in Section 16 - Responsibilities.

IEEE Test Plan Template Environmental Needs Hardware Software Documents Security access

IEEE Test Plan Template Responsibilities Authors suggest a matrix… Rows are labeled with testers’ names Columns are labeled with major testing responsibilities

IEEE Test Plan Template Staffing and Training Needs Vary widely, depending on the project Examples: How to use a particular testing tool How to use the organization’s defect-tracking system How to use the org’s configuration management system

IEEE Test Plan Template Schedule Should be based on schedule in Project Plan PLUS testing milestones

IEEE Test Plan Template Planning Risks and Contingencies List of planning risks and contingencies from formal risk analysis (covered in Chapter 2).

IEEE Test Plan Template Approvals Approver(s) should be person(s) who will eventually have to approve the software being ready to move to next level. MTP may require several approvers. Buy-in and commitment is what’s needed, not just signature.

Review Questions

What are the four levels of test plans specified in the IEEE Std What are the four levels of test plans specified in the IEEE Std. 829-1998 Standard for Software Test Documentation?

What are the four levels of test plans specified in the IEEE Std What are the four levels of test plans specified in the IEEE Std. 829-1998 Standard for Software Test Documentation? Unit Integration System Acceptance

When should work start on the MTP?

When should work start on the MTP? At the same time as the requirements specifications and the project plan are being developed. (p. 58)

What does TBD stand for?

What does TBD stand for? To Be Determined

What is the first section of the IEEE Template for Test Planning?

What is the first section of the IEEE Template for Test Planning? Test Plan Identifier

What three pieces of info should be contained in the unique company-generated test plan identifier?

What three pieces of info should be contained in the unique company-generated test plan identifier? Version of test plan Level of test plan (acceptance, integration, unit, or system) Version of software covered by the test plan

What does ISO stand for?

What does ISO stand for? International Standards Organization

What does CMM stand for?

What does CMM stand for? Capability Maturity Model

What three sections do the text’s authors recommend be added near the beginning of the IEEE Template for Test Planning?

What three sections do the text’s authors recommend be added near the beginning of the IEEE Template for Test Planning? Table of Contents References Glossary

What are examples of References recommended by the IEEE standard?

What are examples of References recommended by the IEEE standard? Project Authorization Project Plan QA Plan Configuration Management Plan Relevant Policies Relevant Standards

What are the two main parts of the IEEE test plan template’s Introduction section?

What are the two main parts of the IEEE test plan template’s Introduction section? Scope of the project (features) Scope of the test plan (levels of testing)

What is covered in the 6.0 Test Items section of the IEEE template for test plans?

What is covered in the 6.0 Test Items section of the IEEE template for test plans? Test items, I.e., versions of products, programs, modules, units, etc. that will be tested, AND those that will NOT be tested.

What is the difference between the contents of 6. 0 - Test Items and 8 What is the difference between the contents of 6.0 - Test Items and 8.0 - Features to be Tested?

What is the difference between the contents of 6. 0 - Test Items and 8 What is the difference between the contents of 6.0 - Test Items and 8.0 - Features to be Tested? Test Items covers what to test from the point of view of the developer or build manager, whereas Features to be Tested covers what to test from the point of view of the user/customer.

Which section of a MTP should specify entrance and exit criteria from one level to another?

Which section of a MTP should specify entrance and exit criteria from one level to another? 10 - Approach (Strategy)

Which of the four levels of testing should have a test environment which mirrors the production environment as closely as possible?

Which of the four levels of testing should have a test environment which mirrors the production environment as closely as possible? Acceptance testing

Which of the four levels of testing should be done first, and who normally does it?

Which of the four levels of testing should be done first, and who normally does it? Unit testing, and individual developers.

Which of the four levels of testing should be done second, and who normally does it?

Which of the four levels of testing should be done second, and who normally does it? Integration testing, and developers.

Which of the four levels of testing should be done third, and who normally does it?

Which of the four levels of testing should be done third, and who normally does it? System testing, and the test team.

What approach to creating a testing methodology was cited by the authors in a case study?

What approach to creating a testing methodology was cited by the authors in a case study? Choose a pilot project, create a MTP for it, and then declare that project’s process as Version 1.0 of the organization’s testing methodology.

What two types of events can “sabotage” test plans?

What two types of events can “sabotage” test plans? Development is running late, so won’t provide the testing team the software on schedule. The ship date has been moved forward.

What is the name for the measurement of the percentage of program statements that are executed by a group of test cases?

What is the name for the measurement of the percentage of program statements that are executed by a group of test cases? Code coverage

What is a secondary benefit of doing code coverage?

What is a secondary benefit of doing code coverage? Eliminating code that can no longer be executed because there is no logical path to reach it.

What is the name for such program code that cannot be executed?

What is the name for such program code that cannot be executed? Dead code.

What are three possible explanations for why code coverage is not done more often?

What are three possible explanations for why code coverage is not done more often? --Purchase of, and training on, a new tool. --Some functional level testers aren’t aware of the concept. --Code coverage is a moot point for organizations so resource-constrained that entire programs are not addressed by tests.

What are three other types of coverage measurements?

What are three other types of coverage measurements? --Requirements coverage --Design coverage --Interface coverage

What is another type of software evaluation besides testing?

What is another type of software evaluation besides testing? Reviews (of requirements, design, code)

What are the two most common types of reviews?

What are the two most common types of reviews? Walkthroughs and inspections

Which is more rigorous--walkthroughs or inspections?

Which is more rigorous--walkthroughs or inspections?

What is the name for retesting previously tested features to ensure that a change or bug fix has not introduced new problems?

What is the name for retesting previously tested features to ensure that a change or bug fix has not introduced new problems? Regression testing

What is the name for rerunning tests that revealed a bug to ensure that the bug was fully and actually fixed?

What is the name for rerunning tests that revealed a bug to ensure that the bug was fully and actually fixed? Confirmation testing

What Bugzilla states might a bug get moved to as a result of confirmation testing?

What Bugzilla states might a bug get moved to as a result of confirmation testing? VERIFIED or REOPENED

What does CCB stand for?

What does CCB stand for? Change Control Board “Gatekeepers” is another name for this type of team.

What does the CCB do? Evaluate the severity of bugs, the cost to fix and test them, and the priority for fixing them.

What is a trade-off to consider with respect to automation tools?

What is a trade-off to consider with respect to automation tools? Automated tests take more time to implement than would be required by manual execution. However, if the same tests are executed several times (for regression testing for example), then the automated tests can save time in the long run.

What items are covered in section 11 - Pass/Fail Criteria?

What items are covered in section 11 - Pass/Fail Criteria? Those test items described in section 6 - Test Items.

What are examples of typical pass/fail criteria? “Gatekeepers” is another term used in industry for a CCB.

What are examples of typical pass/fail criteria? Number of test cases passed and failed Number, type, severity and location of bugs Usability Reliability and/or stability

What type of chart can be used to show dependencies between testing activities?

What type of chart can be used to show dependencies between testing activities? Gantt charts.

End of Chapter 3