Reporting Bugs Why all bugs aren't always fixed What you can do to make it more likely that the bugs you find are fixed What techniques you can use to isolate and reproduce a bug What a bug's life is like from birth to death How to track the bugs you find manually or with a database

Why not all bugs are fixed There's not enough time It's really not a bug It's too risky to fix It's just not worth it. Ineffective bug reporting

It's really not a bug I learned that every bug doesn't have to be a software bug the hard way. In the late 1980's, I was the head of an independent test team that was testing a critical command and control system for the U.S. military. During one particularly difficult test cycle, we discovered an alarming number of bugs. One of my sergeants told me that I should go inform the development manager (who was a Brigadier General, while I was only a Captain) that his software was the worst on the planet. I remember asking the sergeant, "Are you sure all of the tests are okay?" and he replied, "Yes, sir. It's the software that's bad." In officer training school we were taught to listen to the wisdom of our subordinate leaders, so I marched up to the General's office and said, "Sir, your software is not of the quality that we've come to expect." Well, you can almost guess the ending to this story. The General called together his experts, who informed me that most of the failures were caused by a glitch in our test environment. At that point, the General said to me, "Captain, good initiative, but poor judgment," before I was dismissed and sent back to my little windowless office. The moral of the story is this: If you want to maintain credibility with the developers, make sure your tests are all valid before raising issues with the software. Rick Craig

Principles of Reporting a Bug Report bugs as soon as possible Effectively describe them Be nonjudgmental in reporting bugs (don’t be personnel) Follow up on your bug reports

Report Bugs Early

Effective Bug Description Minimal: –Give an exact sequence of steps that shows the problem. If more than one set of inputs or actions causes the bug, cite a couple of examples, especially if they show a pattern Singular –There should be only one bug per report Isolating a bug –Automation tool just happened to stumble upon those keystrokes while running. Reproducible –Following a predefined set of steps will cause the software to achieve the same state and the bug to occur again “Whenever I type a bunch of random characters in the login box, the software starts to do weird stuff.”

Isolating and Reproducing Bugs After discovering a bug, narrow down the specific steps and conditions that cause the problem. 1.Keep notes of everything you do –Use a keystroke and mouse recording program –Use a video camera 2.Look for time-dependent and race condition problems –Bugs that occur only at certain time –How quickly you enter the data –Slower or faster hardware –Network busy 3.State bugs show up only in certain states of the software (order in which things happen)

What Bugs to Fix? Severity indicates how bad the bug is; the likelihood and the degree of impact when the user encounters the bug Priority indicates how much emphasis should be placed on fixing the bug and the urgency of making the fix Organizations are different in scales used for severity & priority

What Bugs to Fix? Severity 1.System crash, data loss, data corruption, security breach 2.Operational error, wrong result, loss of functionality 3.Minor problem, misspelling, UI layout, rare occurrence 4.Suggestion Priority 1.Immediate fix, blocks further testing, very visible 2.Must fix before the product is released 3.Should fix when time permits 4.Would like to fix but the product can be released as is

What Bugs to Fix?- Examples Software that crashes as soon as you start it up Severity 1, Priority 1 A button should be moved a little further down on the page. Severity 4, Priority 4 Data corruption occur only in very rare instances Severity 1, Priority 3 Misspelling causes every user to have problems installing the software Severity 3, Priority 2

What Bugs to Fix? A bug's priority can change over the course of a project A bug that you originally labeled as Priority 2 could be changed to Level 4 as time starts to run out As a tester monitor the bug's status to make sure that you agree with any changes made, persuade them…..

A Bug's Life Cycle

Open state: tester reports bugs Resolved state: programmer fixes the code, & assigns back to the tester Closed state: tester performs a verification test to confirm that the bug is fixed Review state: project manager or Change Control Board decide whether the bug should be fixed Deferred state: considered for fixing at some time in the future

A Bug's Life Cycle Line from the resolved state back to the open state covers the situation where the tester finds that the bug hasn't been fixed Dotted lines: reopen bugs that was thought to be fixed, tested, and closed or deffered.

Bug-Tracking Systems IEEE 829 Standard for Software Test Documentation Test Incident Report “to document any event that occurs during the testing process which requires investigation.”

Test Incident Report Identifier Summary: short statement of facts & reference to test case Incident Description:detailed description of the bug –Date and time –Tester's name –Hardware and software configuration used –Inputs –Procedure steps –Expected results –Actual results –Attempts to reproduce and description of what was tried –Other observations Impact: severity and priority

Bug-Tracking Systems Manual Bug Reporting ExampleExample Automated Bug Tracking –All users of the system must be able to easily access the system at all times –Easy to use and allow data analysis –Incident reports should be linked to the configuration management system Bug tracking BugAware 7

Test Status Report Formal communication channel between the test manager and the rest of the organization of the progress made by the testing team

Test Summary Report To summarize the results of the testing activities and to provide an evaluation based on the results. There should be a test summary report that corresponds to every test plan. Test Summary Report is a test status report on the last day of the project.

Test Summary Report IEEE Std for Software Test Documentation Template for Test Summary Report Contents 1. Test Summary Report Identifier 2. Summary: References to the test plan 3. Variances: changes other than the plan 4. Comprehensive Assessment 5. Summary of Results 5.1Resolved Incidents 5.2Unresolved Incidents 6. Evaluation 7. Summary of Activities 8. Approvals

Quiz What are the three basic states of a software bug's life cycle and the two common additional states? Suppose that you're running tests on the Windows Calculator and find that 1+1=2, 2+2=5, 3+3=6, 4+4=9, 5+5=10, and 6+6=13. Write a bug title and bug description that effectively describes this problem.