1. CIS-74 Computer Software Quality Assurance
Systematic Software Testing
Chapter 3:
Master Test Planning

2. Test Plan Hierarchy Master Test Plan (MTP) Unit Test Plan
Acceptance Test Plan
Integration Test Plan
System Test Plan
(These last four are covered in Chapter 4.)

3. MTP The process is more important than the actual document.
As the complexity of a testing activity increases, the criticality of a good MTP increases exponentially (see diagram on p. 57).

4. IEEE Test Plan Template
1. Test Plan Identifier
Unique
Generated by company
Identifies:
Version of test plan
Level of test plan
Version of software covered

5. IEEE Test Plan Template
2. Table of Contents
Added by authors of text
Two or more levels preferable

6. IEEE Test Plan Template
3. References
Included in Introduction of IEEE template
Given own section by text’s authors
Recommended:
Project authorization
Project plan
QA plan
Configuration management plan
Relevant policies
Relevant standards

7. IEEE Test Plan Template
Glossary
Added by authors of text

8. IEEE Test Plan Template
Introduction
Scope of project - key features, history, etc.
Scope of test plan (levels covered and not covered).

9. IEEE Test Plan Template
6. Test Items
Programmatic description of what is to be tested
High-level test plan: application or version
Low-level test plan: program, unit, module, or build
Items to be excluded from testing also

10. IEEE Test Plan Template
Software Risk Issues
List of risks from formal risk analysis (covered in Chapter 2).

11. IEEE Test Plan Template
8. Features to be Tested
From customer point of view
Basically, those items above the “cut line” from formal risk analysis

12. IEEE Test Plan Template
9. Features Not to Be Tested
From customer point of view
Basically, those items below the “cut line” from formal risk analysis
Amusing point on pp : Managers who raise their eyebrows about this section of a test plan are often the ones who approve schedules that don’t allow enough time to test everything.

13. IEEE Test Plan Template
10. Approach (Strategy)
Heart of test plan
Should explain approach for each level
Should specify entrance and exit criteria for each level to another

14. IEEE Test Plan Template
10. Approach (continued)
Test environment is most artificial for unit testing, followed by system testing, and then integration testing.
Testing environment is closest to reality for acceptance testing.

15. IEEE Test Plan Template
10. Approach (continued)
Methodology decisions
What testing levels will be used?
When will testers become involved?
Will there be a beta?
Will there be an alpha?
What testing techniques will be used?
Etc.

16. IEEE Test Plan Template
10. Approach (continued)
Resource decisions
Is dedicated test group sufficient?
If not, where else can resources be obtained? Developers? Users? Interns? U. S. contractors? Overseas contractors? Customer support?

17. IEEE Test Plan Template
10. Approach (continued)
Test Coverage Decisions
Code coverage measures the percentage of program statements that are executed by a set of test cases.
Requirements coverage measures % of business requirements covered.
Design coverage measures the % of design covered.
Interface coverage measures % of interfaces covered.

18. IEEE Test Plan Template
10. Approach (continued)
Walkthroughs and Inspections
Reviews are another form of software evaluation besides testing.
Walkthrough - talking/walking through each line of code as a group
Inspection - more formal, may be done by a single individual

19. IEEE Test Plan Template
10. Approach (continued)
Configuration Management
Can be done in a separate document
In MTP, should cover change management and bug-review process

20. IEEE Test Plan Template
10. Approach (continued)
Collection and Validation of Metrics
Metrics can be a significant overhead, so it’s necessary to plan exactly which are needed.
More info coming in Chapter 10 - The Test Manager.

21. IEEE Test Plan Template
10. Approach (continued)
Tools and Automation
Caution! It can take more time to automate than the time it takes to execute tests manually! Be sure automated tests will be re-run often to justify the cost of automation.

22. IEEE Test Plan Template
10. Approach (continued)
Changes to the Test Plan
What type of changes require redoing the approval process?
How should the test plan be published?
How should the review be conducted?
Should the test plan go through regular CM?

23. IEEE Test Plan Template
10. Approach (continued)
Meetings and Communications
Standard meetings
Status reporting
Metrics

24. IEEE Test Plan Template
Item Pass/Fail Criteria
Pertains to items from Section Test Items - NOT to test cases
Examples:
Test cases passed and failed
Number, type, severity, and location of bugs
Usability
Reliability
Often called “release criteria.”

25. IEEE Test Plan Template
Suspension Criteria & Resumption Requirements
Large volumes of bugs
Critical bugs
Incomplete tasks
Code not checked into source-code control system

26. IEEE Test Plan Template
Test Deliverables
All testware to be created and maintained for the testing effort.

27. IEEE Test Plan Template
Testing Tasks
Authors recommend deleting this section, and listing all testing tasks in Section 16 - Responsibilities.

28. IEEE Test Plan Template
Environmental Needs
Hardware
Software
Documents
Security access

29. IEEE Test Plan Template
Responsibilities
Authors suggest a matrix…
Rows are labeled with testers’ names
Columns are labeled with major testing responsibilities

30. IEEE Test Plan Template
Staffing and Training Needs
Vary widely, depending on the project
Examples:
How to use a particular testing tool
How to use the organization’s defect-tracking system
How to use the org’s configuration management system

31. IEEE Test Plan Template
Schedule
Should be based on schedule in Project Plan PLUS testing milestones

32. IEEE Test Plan Template
Planning Risks and Contingencies
List of planning risks and contingencies from formal risk analysis (covered in Chapter 2).

33. IEEE Test Plan Template
Approvals
Approver(s) should be person(s) who will eventually have to approve the software being ready to move to next level.
MTP may require several approvers.
Buy-in and commitment is what’s needed, not just signature.

34. Review Questions

35. What are the four levels of test plans specified in the IEEE Std
What are the four levels of test plans specified in the IEEE Std Standard for Software Test Documentation?

36. What are the four levels of test plans specified in the IEEE Std
What are the four levels of test plans specified in the IEEE Std Standard for Software Test Documentation?
Unit
Integration
System
Acceptance

37. When should work start on the MTP?

38. When should work start on the MTP?
At the same time as the requirements specifications and the project plan are being developed. (p. 58)

39. What does TBD stand for?

40. What does TBD stand for?
To Be Determined

41. What is the first section of the IEEE Template for Test Planning?

42. What is the first section of the IEEE Template for Test Planning?
Test Plan Identifier

43. What three pieces of info should be contained in the unique company-generated test plan identifier?

44. What three pieces of info should be contained in the unique company-generated test plan identifier?
Version of test plan
Level of test plan (acceptance, integration, unit, or system)
Version of software covered by the test plan

45. What does ISO stand for?

46. What does ISO stand for?
International Standards Organization

47. What does CMM stand for?

48. What does CMM stand for?
Capability Maturity Model

49. What three sections do the text’s authors recommend be added near the beginning of the IEEE Template for Test Planning?

50. What three sections do the text’s authors recommend be added near the beginning of the IEEE Template for Test Planning?
Table of Contents
References
Glossary

51. What are examples of References recommended by the IEEE standard?

52. What are examples of References recommended by the IEEE standard?
Project Authorization
Project Plan
QA Plan
Configuration Management Plan
Relevant Policies
Relevant Standards

53. What are the two main parts of the IEEE test plan template’s Introduction section?

54. What are the two main parts of the IEEE test plan template’s Introduction section?
Scope of the project (features)
Scope of the test plan (levels of testing)

55. What is covered in the 6.0 Test Items section of the IEEE template for test plans?

56. What is covered in the 6.0 Test Items section of the IEEE template for test plans?
Test items, I.e., versions of products, programs, modules, units, etc. that will be tested, AND those that will NOT be tested.

57. What is the difference between the contents of 6. 0 - Test Items and 8
What is the difference between the contents of Test Items and Features to be Tested?

58. What is the difference between the contents of 6. 0 - Test Items and 8
What is the difference between the contents of Test Items and Features to be Tested?
Test Items covers what to test from the point of view of the developer or build manager, whereas Features to be Tested covers what to test from the point of view of the user/customer.

59. Which section of a MTP should specify entrance and exit criteria from one level to another?

60. Which section of a MTP should specify entrance and exit criteria from one level to another?
10 - Approach (Strategy)

61. Which of the four levels of testing should have a test environment which mirrors the production environment as closely as possible?

62. Which of the four levels of testing should have a test environment which mirrors the production environment as closely as possible?
Acceptance testing

63. Which of the four levels of testing should be done first, and who normally does it?

64. Which of the four levels of testing should be done first, and who normally does it?
Unit testing, and individual developers.

65. Which of the four levels of testing should be done second, and who normally does it?

66. Which of the four levels of testing should be done second, and who normally does it?
Integration testing, and developers.

67. Which of the four levels of testing should be done third, and who normally does it?

68. Which of the four levels of testing should be done third, and who normally does it?
System testing, and the test team.

69. What approach to creating a testing methodology was cited by the authors in a case study?

70. What approach to creating a testing methodology was cited by the authors in a case study?
Choose a pilot project, create a MTP for it, and then declare that project’s process as Version 1.0 of the organization’s testing methodology.

71. What two types of events can “sabotage” test plans?

72. What two types of events can “sabotage” test plans?
Development is running late, so won’t provide the testing team the software on schedule.
The ship date has been moved forward.

73. What is the name for the measurement of the percentage of program statements that are executed by a group of test cases?

74. What is the name for the measurement of the percentage of program statements that are executed by a group of test cases?
Code coverage

75. What is a secondary benefit of doing code coverage?

76. What is a secondary benefit of doing code coverage?
Eliminating code that can no longer be executed because there is no logical path to reach it.

77. What is the name for such program code that cannot be executed?

78. What is the name for such program code that cannot be executed?
Dead code.

79. What are three possible explanations for why code coverage is not done more often?

80. What are three possible explanations for why code coverage is not done more often?
--Purchase of, and training on, a new tool.
--Some functional level testers aren’t aware of the concept.
--Code coverage is a moot point for organizations so resource-constrained that entire programs are not addressed by tests.

81. What are three other types of coverage measurements?

82. What are three other types of coverage measurements?
--Requirements coverage
--Design coverage
--Interface coverage

83. What is another type of software evaluation besides testing?

84. What is another type of software evaluation besides testing?
Reviews (of requirements, design, code)

85. What are the two most common types of reviews?

86. What are the two most common types of reviews?
Walkthroughs and inspections

87. Which is more rigorous--walkthroughs or inspections?

88. Which is more rigorous--walkthroughs or inspections?

89. What is the name for retesting previously tested features to ensure that a change or bug fix has not introduced new problems?

90. What is the name for retesting previously tested features to ensure that a change or bug fix has not introduced new problems?
Regression testing

91. What is the name for rerunning tests that revealed a bug to ensure that the bug was fully and actually fixed?

92. What is the name for rerunning tests that revealed a bug to ensure that the bug was fully and actually fixed?
Confirmation testing

93. What Bugzilla states might a bug get moved to as a result of confirmation testing?

94. What Bugzilla states might a bug get moved to as a result of confirmation testing?
VERIFIED or REOPENED

95. What does CCB stand for?

96. What does CCB stand for? Change Control Board
“Gatekeepers” is another name for this type of team.

97. What does the CCB do?
Evaluate the severity of bugs, the cost to fix and test them, and the priority for fixing them.

98. What is a trade-off to consider with respect to automation tools?

99. What is a trade-off to consider with respect to automation tools?
Automated tests take more time to implement than would be required by manual execution. However, if the same tests are executed several times (for regression testing for example), then the automated tests can save time in the long run.

100. What items are covered in section 11 - Pass/Fail Criteria?

101. What items are covered in section 11 - Pass/Fail Criteria?
Those test items described in section 6 - Test Items.

102. What are examples of typical pass/fail criteria?
“Gatekeepers” is another term used in industry for a CCB.

103. What are examples of typical pass/fail criteria?
Number of test cases passed and failed
Number, type, severity and location of bugs
Usability
Reliability and/or stability

104. What type of chart can be used to show dependencies between testing activities?

105. What type of chart can be used to show dependencies between testing activities?
Gantt charts.

106. End of Chapter 3