1. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 1 Black Box Software Testing (Professional Seminar) Cem Kaner, J.D., Ph.D. Professor of Computer Sciences Florida Institute of Technology Section:33 Project Planning Summer, 2002 Contact Information: kaner@kaner.com www.kaner.com (testing website) www.badsoftware.com (legal website) I grant permission to make digital or hard copies of this work for personal or classroom use, with or without fee, provided that (a) copies are not made or distributed for profit or commercial advantage, (b) copies bear this notice and full citation on the first page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion, (c) each page bear the notice "Copyright (c) Cem Kaner" or if you changed the page, "Adapted from Notes Provided by Cem Kaner". Abstracting with credit is permitted. The proper citation for this work is Cem Kaner, A Course in Black Box Software Testing (Professional Version), Summer-2002, www.testing-education.org. To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from kaner@kaner.com.www.testing-education.org

2. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 2 Planning and Negotiating the Testing Project The notes here focus on your process of negotiating resources and quality level. Please see Testing Computer Software, Chapter 13 for a detailed discussion of planning and testing tasks across the time line of the project.

3. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 3 Planning the Testing Project Things you can do very early in the project: Analyze any requirements documents for testability, ambiguity. Facilitate inspection and walkthrough meetings. Prepare a preliminary list of hardware configurations and start arranging for loaners. Ask for testing support code, such as debug monitors, memory meters, support for testpoints. Ask for a clear error handling message structure. Discuss the possibility of code coverage measurement (which will require programmer support).

4. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 4 Planning the Testing Project Things you can do very early in the project: Prepare for test automation. This involves reaching agreements in principle on breadth of automation and automation support staffing level. Order automation support equipment. Order external test suites. Learn about the product’s market and competition. Evaluate coding tools that facilitate automation (e.g. test 3rd party custom controls against MS-Test)

5. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 5 Planning the Testing Project: First Principles You can’t nail everything down. You will face difficult prioritization choices, and many constraints will be out of your direct control. You can influence several constraints by opening up your judgments to other stakeholders in the company. This is more than getting a sign-off. Reality is far more important than your ability to cast blame. Your task is to manage project-level risks. This includes risks to the project’s cost, schedule, and interpersonal dynamics, as well as to the product’s feature set and reliability.

6. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 6 Planning the Testing Project: Deciding Your Depth of Testing You have three key objectives: 1Achieve a uniform and well-understood minimum level of testing. 2Be explicit about the level of testing for each area: » mainstream » guerrilla » formally planned » structured regression 3Reach corporate agreement on the level of testing for each area. Just like bug deferrals, depth-of-testing restrictions must rest on sound business decisions.This should not be your decision to make.

7. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 7 Planning the Testing Project: Identify the Tasks Develop a project task list with your staff. Remember, you are looking for realism in estimates of complexity, difficulty, and time. Make the listing and estimation process public, and allow public comment. Identify all potential sources of information. List all main functional areas, and all other cross-functional approaches that you will take in testing the program. List every task that appears to need more than 1/2 day. (Probably you will do this by a group brainstorming session on flipcharts, listing sources on one chart. Gather the sources. List areas and approaches on a few charts. Make one chart for each area of work, and list tasks for each chart. Tentatively assign times to each task, possibly by a museum tour.)

8. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 8 Planning the Testing Project: Estimate the Time 1Assign time estimates to each task. Invite programmers, marketers, and project managers to walk through the charts and provide their own estimates. Explore the bases of differences. You might have underestimated a task’s complexity. Or you might be seeing a priority disagreement. Or wishful thinking. 2Make your best-estimate task list. Provide the time needed to achieve formal, planned testing for each area. Include estimates for structured regression for key areas. This number is probably absurdly huge. 3Add to the list your suggestions for time-cuts to guerrila-level and mainstream-level testing for selected areas. 4Circulate your lists to all stakeholders, call one or more planning meetings, and reach agreements on the level of testing for each area. Keep cutting tasks and/or adding staff until you reach an achievable result.

9. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 9 Planning the Testing Project: Estimate the Time Don’t forget: Budget for meetings and administrative overhead (10-30%). Budget for bug regression (5-15%). Budget for down time. Budget for holidays and vacations. Never develop a budget that relies on overtime. Leave the overtime free for discretionary additional work by the individual tester. Don’t let yourself be bullied or embarrassed, and don’t bully or embarrass your staff, into making underestimates. To achieve a result, insist on cutting tasks, adding staff, or finding realistic ways to improve productivity.

10. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 10 Planning the Testing Project: Identify Dependencies & Milestones You can’t start reviewing the manual until you receive it, right? List these dependencies and point out, in every project meeting, what is due to you that is not yet delivered and holding you up. Try to reach verifiable, realistic definitions for each milestone. For example, if “gamma” is 6 weeks before release, the product can’t have more than 6 weeks of schedule risk at gamma. Negotiate a definition.

11. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 11 Planning the Testing Project: Monitor Progress Put the tasks and agreed-to durations on a timeline and review progress of all staff every week. When prerequisite materials aren’t provided, find other tasks to do instead. When you can’t meet the schedule, due to absent materials, publish new estimates. When design changes invalidate your estimates, publish new estimates. If you’re falling consistently behind (e.g. due to underestimated overhead), publish the problem and ask for fewer tasks, more staff, or some other realistic solution. If one of your staff has trouble with an area, recognize it and deal with it.

12. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 12 Planning the Testing Project: Control Late-Stage Issues Watch out for late changes. Encourage people to be cautious about late bug fixes, and super-cautious about late design changes. Provide interim and final deferred-bug reviews. Take a final inventory of your testing coverage. Carry out final integrity tests. You may have to assess and reporting the reliability of the tested product. Plan for post-release processes. Develop a release kit for product support. Don’t forget the party.

13. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 13 Notes ___________________________________________________________ ___________________________________________________________ ___________________________________________________________ ___________________________________________________________

14. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 14 Project Planning Testing Impossible Software Under Impossible Deadlines Presented at Software Testing Analysis East, 1999

15. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 15 Introduction I didn’t come up with the idea for this talk--the STAR folk suggested it as something that I might be able to provide a few insights into. After all, this is a chronic complaint among testers.... Well, here goes. How should you deal with a situation in which the software comes to you undocumented on a tight deadline? I think that the answer depends on the causes of the situation that you’re in. Your best solution might be technical or political or resume-ical. It depends on how you and the company got into this mess. If it is a mess.

16. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 16 What Caused This Situation? So, let’s start with some questions about causation: Why is the software undocumented? Why do you have so little time? What are the quality issues for this customer?

17. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 17 Why Do You Have So Little Time? A Few Possibilities Time-to-release drives competition Cash flow drives release date Key fiscal milestone drives release date Executive belief that you’ll never be satisfied, so your schedule input is irrelevant Executive belief that testing group is out of control and needs to be controlled by tight budgets Executive belief that you have the wrong priorities (e.g. paperwork rather than bugs) Executive belief that testing is irrelevant Structural lack of concern about the end customer, or Maybe you have an appropriate amount of time.

18. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 18 Quality Issues For This Customer? In-house, “In-house” outsourced External, custom External, large package, packaged External, mass-market, packaged What is quality in this market? What are their costs of failure? » User groups, Software stores, Sales calls, Support calls Over the long term, a good understanding of quality in the market, and as it affects your company’s costs, will buy you credibility and authority.

19. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 19 Political Approaches: Buying Time Many of the ideas in this section will help you if you’re dealing with a single project that is out of control. If your problem is structural (reflects policy or standard practice of the company), then some of these ideas will be counter-productive.

20. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 20 Buying Time: Delaying a Premature Release -2 Take the high road “I want to see knives in the chests, not in the backs.” » Trip Hawkins, founding President, Electronic Arts Communicate honestly. Avoid springing surprises on people. Never sabotage the project. Don’t become The Adversary: If you are nasty and personal, you will become a tool, to be used by someone else. And you will be disposable.

21. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 21 Buying Time: Delaying a Premature Release -3 Search for show-stoppers. If you can, dedicate a specialist within your group to this task. Circulate deferred bug lists broadly. Consider writing mid-project and late-in-project assessments of: » extent of testing (by area) » extent of defects (by area, with predictions about level of bugs left) » deferred bugs » likely out of box experience or reviewers’ experience

22. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 22 Buying Time: Delaying a Premature Release -4 Do regular, effective status reporting. List: your group’s issues (deliverables due to you, things slowing or blocking your progress, etc., areas in which you are behind or ahead of plan) project issues bug statistics Find allies (think in terms of quality-related costs)

23. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 23 Signing Off on the Release? Don’t sign off. Don’t accept the authority to sign off. Don’t pretend to be “QA” when you (obviously) are not. Position yourself as a technical information provider. » Publish pre-release reports that provide data on the stability of the product and the extent of completed testing » Publish a report that lays out the likely issues that will be raised by magazine reviewers (or other third parities) when they see the product » Let the people who want to ship prematurely own their responsibility. Your responsibility is to provide them with the information they need to make that decision.

24. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 24 Technical Approaches Find reference docs (you can get lots of data, even if you can’t get specs.) See the notes on this in the section on Spec-Driven testing Re-use materials across projects. Plan for exploratory testing. Do cost-effective automation. Use tools to find bugs faster » web page syntax checkers, spiders, etc. Facilitate inspections » (Get those bugs out before they reach you.) Cover the obvious legal issues.

25. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 25 Reusable Test Matrix

26. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 26 Group Management Approaches Can you add head count? Will they let you? Can you absorb them? » Do a work flow analysis to figure out what parts of each task could be split off and delegated to a newcomer. Stay organized Use functional outlines Or use a detailed project plan Create and publish explicit coverage reports Get critical processes under control Example: release management

27. Copyright (c) 1994-2001 Cem Kaner. All Rights Reserved. 27 Notes ___________________________________________________________ ___________________________________________________________ ___________________________________________________________ ___________________________________________________________