that focus first on areas of risk. KEY IDEA Run crisp test cycles that focus first on areas of risk.

Test Cycle: What You Do With a Build 1. Receive the product. Formal builds Informal builds Save old builds. 2. Clean your system. Completely uninstall earlier builds. 3. Verify testability. Smoke testing Suspend test cycle if the product is untestable.

Test Cycle: What You Do With a Build 4. Determine what is new or changed. Change log 5. Determine what has been fixed. Bug tracking system 6. Test fixes. Many fixes fail! Also test nearby functionality. 7. Test new or changed areas. Exploratory testing.

Test Cycle: What You Do With a Build 8. Perform regression testing. Not performed for an incremental cycle. Automated vs. manual Important tests first! 9. Report results. Coverage Observations Bug status (new, existing, reopened, closed) Assessment of quality Assessment of testability

Typical Quality Growth The Test Cycle Ship it! Typical Quality Growth Quality Time

Risk Focus: Common and Critical Cases Core functions: the critical and the popular. Capabilities: can the functions work at all? Common situations: popular data and pathways. Common threats: likely stress and error situations. User impact: failures that would do a lot of damage. Most wanted: problems of special interest to someone else on the team.

Test Cycle Convergence Ideal Test Cycle Convergence Good Enough Quality Tests failed: known bad Cycle End Cycle Start Tests not run: Unknown } Quality Build Tests passed: known good Untestable Quality Time

Rapid Bug Investigation Identification Notice a problem. Recall what you were doing just prior to the problem. Examine symptoms of the problem w/o disturbing system state. Consider possibility of tester error. Investigation How can the problem be reproduced? What are the symptoms of the problem? How severe could the problem be? What might be causing the problem? Reality Check Do we know enough about the problem to report it? Is it important to investigate this problem right now? Is this problem, or any variant of it, already known? How do we know this is really a problem? Is there someone else who can help us? Identify Investigate Check

Exercise “I’ve modified the Triangle program so that it now renders an image of the triangle. I’m also worried about the limitations of the input field. Please do another test cycle on it.” -- Your Friendly Programmer

Provide a clear report on how completely you tested Triangle Exercise Provide a clear report on how completely you tested Triangle

An Innocent Question from Management… “On a scale of 0 to 100, how completely did you test the Triangle program?” completely untested completely tested 50 100 Answer with a number. You can also offer a comment, if you have one. Any questions?

What Could We Report? Tasks: Things we are doing. Example: “We are trying to reinstall the server so we can do the smoke tests. We have 28 fix verifications to get through.” Product Coverage: What we have examined. Example: “We have tested printing, performance, and compatibility.” Product Risk: Problems or potential for problems in the product. Example: “We have found 53 bugs.” Agreement: What we specifically contracted to do. Example: “We have implemented 90% of the tests on our test plan. As we agreed, we have started testing for printer compatibility.”

What Could We Report? Project: Schedule, documents, resources, people, or anything else that makes it possible for us to test. Example: “Testing is on schedule. But Jaeline goes on vacation next week.” Mission: The ultimate goal, which may be to ship a product that meets quality criteria, satisfy customers, or some other goal that can be a guiding principle for the testing. Example: “We don’t yet know enough about feature X of the product.” Client Satisfaction: What our clients think of our work. Example: “The project manager is happy with our work.”

Reporting Considerations Reporter safety: What will they think if I made no progress? Client: Who am I reporting to and how do I relate to them? Rules: What rules and traditions are there for reporting, here? Significance of report: How will my report influence events? Subject of report: On what am I reporting? Other agents reporting: How do other reports affect mine? Medium: How will my report be seen, heard, and touched? Precision and confidence levels: What distinctions make a difference? Take responsibility for the communication.