Test Planning The purpose of test planning  The areas to consider in planning

Planning "Planning is the art and science of envisioning a desired future and laying out effective ways of bringing it about." - Planning, MCDP5 U.S. Marine Corps

Goal of Test Planning The IEEE Standard for Software Test Documentation The purpose of a software test plan is as follows: To prescribe the scope, approach, resources, and schedule of the testing activities. To identify the items being tested, the features to be tested, the testing tasks to be performed, the personnel responsible for each task, and the risks associated with the plan.

Planning Stages Master Test Plan On larger complex projects, create: –Acceptance Test Plan –System Test Plan –Integration Test Plan Test planning CAN'T be separated from project planning. –All important test planning issues are also important project planning issues

Goal of Test Planning It's the planning process that matters, not the resulting document. The ultimate goal of the test planning process is communicating (not recording) the software test team's intent, its expectations.

Test Planning Test planning should be started as soon as possible –When requirements specifications and the project plan begin –Some information will not be available Use TBD (To Be Determined) + When + who

Test Plan IEEE Standard for Software Test Documentation suggests a common form.IEEE Standard for Software Test Documentation suggests a common form Industry or company might have a standard. Use a test plan template. ………Anyway, we should understand its content

Test Plan Content 1.High-Level Expectations 2.People, Places, and Things 3.Definitions 4.Inter-Group Responsibilities 5.What Will and Won't Be Tested 6.Test Phases 7.Test Strategy 8.Resource Requirements 9.Test Schedule 10.Test Cases 11.Bug Reporting 12.Metrics and Statistics 13.Risks and Issues

High-Level Expectations Agree with team members on obvious things –What's the purpose of the test planning process and the software test plan? –What product is being tested? System name, Version number, new release or maintenance, in house or by a third party? –What are the quality and reliability goals of the product? a clear, concise, agreed-on definition of the product's quality and reliability goals.

People, Places, and Things “Pointers to everything that a new tester would ask about”. The test plan should include names, titles, addresses, phone numbers, addresses, and areas of responsibility for all key people on the project Where documents are stored? Where the software can be downloaded from, where the test tools are located? If there are external test labs for configuration testing, where are they located and how are they scheduled?

Definitions A glossary is used to define any terms and acronyms used in the document Depends on the audience of the document Include any product-specific terms as well as technical and testing terms.

Inter-Group Responsibilities High-level tasks performed by functional group (management, test, programmers...so on) Identify tasks and deliverables that potentially affect the test effort. Example An (×) denotes the owner of a task & a dash (-) indicates a contributor.

What Will and Won't Be Tested Won't Be Tested ! –Components of the software that were previously released and have already been tested. –Ready component may be taken as is from another software company –Outsourcing company may supply pre-tested portions of the product

What Will and Won't Be Tested Written in two sections: –Test Items what will be tested from the developer point of view –Features to (NOT) Be Tested what will be tested from the user point of view

Test Strategy Testing approach overall and in each phase –What testing technique to use? –Walkthroughs and Inspections –Manually or with tools? –Process for reviewing, fixing, & promoting bugs –Strategy for updating the test plan (Configuration Managemen) Should include entry and exit criteria from one level to another

Examples of Test Strategies Analytical –Risk-based strategy Perform risk analysis then planning, estimating, designing, and prioritizing the tests based on risk. –Requirements-based strategy Analysis of the requirements specification Methodical –Checklist compiled over the years that suggests the major areas of testing to run or follow an industry-standard

‘entry criteria’ and ‘exit criteria’ Acquisition and supply: the availability of staff, tools, systems and other materials required. Test items: the state that the items to be tested must be in to start and to finish testing. Defects: the number known to be present, the arrival rate, the number predicted to remain, and the number resolved. Tests: the number run, passed, failed, blocked, skipped, and so forth. Coverage: the portions of the test basis, the software code or both that have been tested and which have not.

Test Phases Depends on the development model. It should have entrance and exit criteria for each test phase. Example of Exit Criteria from System Test: All test cases must be documented and run. 90% of all test cases must pass. All test cases dealing with the Billing function must pass. All Medium and High defects must be fixed. Code coverage must be at least 90% (including Integration and Unit testing).

Resource Requirements People. How many, what experience, what expertise? Should they be full-time, part-time, contract, students? Equipment. Computers, test hardware, printers, tools. Office and lab space. Where will they be located? How big will they be? How will they be arranged? Software. Word processors, databases, custom tools. What will be purchased, what needs to be written? Outsource companies. Will they be used? What criteria will be used for choosing them? How much will they cost? Miscellaneous supplies. Disks, phones, reference books, training material. What else might be necessary over the course of the project?

Tester Assignments Assignments identifies the testers and their responsible for each area of the software and for each testable feature. Table High-Level Tester Assignments for WordPad Test AssignmentsTester Character formatting: fonts, size, color, styleAl Layout: bullets, paragraphs, tabs, wrappingSarah Configuration and compatibilityLuis UI: usability, appearance, accessibilityJolie Documentation: online help, rollover helpValerie Stress and loadRon

Test Schedule Maps all of the above into the overall project schedule The amount of test resources on a project typically increases over the course of the development schedule.

Test Schedule Schedule crunch: Table A Test Schedule Based on Fixed Dates DateTesting Task 3/5/2001Test Plan Complete 6/1/2001Test Cases Complete 6/15/2001 – 8/1/2001Test Pass #1 8/15/2001 – 10/1/2001Test Pass #2 10/15/2001 – 11/15/2001Test Pass #3 If some part of the project is delivered to the test group two weeks late and only three weeks were scheduled for testing, what happens? Does the three weeks of testing now have to occur in only one week or does the project get delayed two weeks? Table A Test Schedule Based on Relative Dates DurationStart DateTesting Task 4 weeks7 days after spec completeTest Plan Complete 12 weeksTest plan completeTest Cases Complete 6 weeksCode complete buildTest Pass #1 6 weeksBeta buildTest Pass #2 4 weeksRelease buildTest Pass #3

Test Cases What approach will be used to write them, where the test cases will be stored

Bug Reporting What process will be used to manage the bugs needs to be planned so that each and every bug is tracked from when it's found to when it's fixed

Metrics and Statistics Total bugs found daily over the course of the project List of bugs that still need to be fixed Current bugs ranked by how severe they are Total bugs found per tester Number of bugs found per software feature or area

Risks and Issues Identifying risks of the project—ones that could have an impact on the test effort "What worries you?" For Example:  Testers’ Experience  Testability of some requirements  Modules with many or complicated changes  Security, performance, and reliability issues  Interfaces to other systems

Quiz Name a few typical testing resources that should be considered when test planning True or False: A schedule should be made to meet absolute dates so that there's no question when a testing task or phase is to start and when it is to end

Test Planning Template Test Plan Identifier (current version) Table of Contents References (other plans, Policies, Standards) Glossary (depends on audience) Introduction (Scope) –scope of the plan –scope of the project

Test Planning Template Test Deliverables –A list of all of the documents, tools, and other components that are to be developed and maintained in support of the testing effort –For example: test plans, test design specs, test cases, custom tools, defect reports, test summary reports.

Detailed Planning A level of test is defined by a particular environment, which is a collection of people, hardware, software, interfaces, data, and even the viewpoints of the testerslevel

Detailed Planning Level-specific plans should usually be written in reverse order of execution The "V" model shows: –Acceptance test planned based on requirements –System test planned based on high-level design (and requirements) –Integration testing planned based on detailed design –Unit testing planned based on code