1. 1 CEN 4072 Software Testing PPT2: Tracking the problem

2. 2 What is a problem? A problem is a questionable property of a program run. Life cycle of a problem: - First, a developer is informed about the problem - The developer reproduces the problem to view the circumstance from his own eyes. The way a developer sees a problem may be different than the way the user has seen it - Developer then isolates the circumstance. In other words, they find the piece of code that is generating the problem. - Then the developer fixes the problem. - Lastly, the fix is delivered to the user.

3. 3 Problem reports Problem reports (a.k.a. bug reports, defect reports, fault reports, problem reports, trouble reports, and change requests) contain the information required to fix a problem. Problem reports contain multiple elements. Product release Operating Environment Problem History Expected Behavior Observed Behavior

4. 4 Bugs and product release The version of the program that the problem occurs in. Could also be a different unique identifier that is needed to reproduce the exact version Must assess if the problem occurs only in that release or if it has a broader scope; could help isolate problem

5. 5 Bugs and operating environments The operating environment in which the problem occurred (Ex. Windows 8.1 with the following packages installed) The operating system version information Problem could be present in multiple environments

6. 6 Bugs and system resources Any other resources needed to duplicate the environment that the problem occurs in Ex. Programs that need to be running, external hardware, etc.

7. 7 Bugs and problem history The steps that are necessary to reproduce the problem Must decide which steps are actually contributing to the problem and which are irrelevant

8. 8 Bugs and expected and observed behavior Expected: What should have happened if the problem did not exist What the problem is preventing Observed: What is actually happening What is the problem as observed by the user

9. 9 Managing problems To manage problems can either use… A problem file: Does not scale Only one person can work on the problem at a time Previous fixes, history of the problem is lost Less efficient for a company to use (what happens if a previous revision is more effective than the most current one?) A problem data base: Allows for multiple people to work on problem at one time Saves different revisions of problem so history is not erased

10. 10 Bugzilla problem data base Bugzilla is a web-based general-purpose bug tracker and testing tool originally developed and used by the Mozilla project (Wikipedia definition) Features (as listed on Bugzilla website) : Optimized database structure for increased performance and scalability Excellent security to protect confidentiality Advanced query tool that can remember your searches Integrated email capabilities Editable user profiles and comprehensive email preferences Comprehensive permissions system Proven under fire as Mozilla's bug tracking system

11. 11 Problems: severity Bugs can be classified by many things- the first being severity. There are four main severity classifications: Critical: The defect affects critical functionality or critical data. It does not have a workaround. Example: Unsuccessful installation, complete failure of a feature. Major: The defect affects major functionality or major data. It has a workaround but is not obvious and is difficult. Example: A feature is not functional from one module but the task is doable if 10 complicated indirect steps are followed in another module/s. Minor: The defect affects minor functionality or non-critical data. It has an easy workaround. Example: A minor feature that is not functional in one module but the same task is easily doable from another module. Trivial: The defect does not affect functionality or data. It does not even need a workaround. It does not impact productivity or efficiency. It is merely an inconvenience. Example: Petty layout discrepancies, spelling/grammatical errors.* *http://softwaretestingfundamentals.com/defect-severity/

12. 12 Problems: priority Problem priority is also a classification that is used. It is based on the urgency that the problem is resolved. Immediate – The bug should be resolved immediately. High - This bug should be resolved as soon as possible in the normal course of development activity, before the software is released. Medium – This bug should be repaired after serious bugs have been fixed. Low – It can be resolved in a future major system revision or not be resolved at all.* *http://www.elementool.com/contact/articles/Bug_Tracking_severity_priority.html

13. 13 Problems: identity and notification Every problem is assigned an identifier (also called a PR number or bug number) which is its own unique number that can be used when referring to the bug. A tester can also attach an email address to the problem so whenever the problem is worked on, an email is sent to the tester updating the status of the problem. This notifies the tester (or user) whenever the problem report is updated.

14. 14 The problem life cycle

15. 15 The problem life cycle Unconfirmed - A problem report first enters the data base as unconfirmed

16. 16 The problem life cycle New - If the report is invalid, or a duplicate, it goes straight to the resolved phase. If not, it is a new problem.

17. 17 The problem resolution types Assigned - The problem is assigned to a developer. The developer is in charge of producing the resolution If the resolution is invalid, it is not a problem Duplicate- the problem is already in existence Fixed the problem has been solved by developer Won’t fix - The problem will/can not be resolved by developer (ex. It is a feature) Worksforme - The problem cannot be reproduced developer

18. 18 The problem life cycle Resolved - The problem has been processed ( all roads lead to this)

19. 19 The problem life cycle Verified - The problem has been satisfied with a successful fix. If not, the problem is reopened.

20. 20 The problem life cycle Closed - A new version with the fix is released. Images and information obtained from: http://www.whyprogramsfail.com /

21. 21 The charge of the Software Change Control Board (SCCB) The SCCB is in charge of assessing the impact of the problem delegating tasks to the fix of the problem assessing the “solved” problems deciding if the problem can be closed.

22. 22 Problem driven development The entire software development process can be approached with problem driven development. Initial problem: The product we need does not exist Divide each this broad into sub-problems needed to develop the software Once all of these sub-problems are solved, the product is ready

23. 23 Managing the clutter of problem reports With a large problem report database, it is easy for the database to become cluttered. This can be avoided in a number of ways. Developers are encouraged to do a search for their problem before they submit it as new and to simplify their bug reports This eliminates duplicates in the system Its is also important to “clean house” from time to time, eliminating old problems that are obsolete and no longer occur

24. 24 Problem fixes and releases It is important to decipher problem fixes and releases by following naming conventions. Ex. http://www.whyprogramsfail.com /

25. 25 Problems and tests It is not necessary to insert a failed test into the problem data base. “Once we can repeat a problem, there is no need for a database entry” - http://www.whyprogramsfail.com /

26. 26 Effective problem reports An effective problem report… Is unbiased and sticks with facts only Is clear and simple; keeps cases general Can be easily reproduced Is well structured and readable for all parties

27. 27 Other tools for bug tracking PHPBUGTRACKER- “ a web-based bug tracker with functionality similar to other issue tracking systems, such as Bugzilla. Design focuses on separating the presentation, application, and database layers.” (http://phpbt.sourceforge.net/) ISSUETRACKER- “a support issue tracking system... designed to be user friendly, …includes many features include(ing) things like file uploads, email parsing, email and sms notifications, unlimited users and groups, and much more.” (http://issue-tracker.sourceforge.net/http://issue- tracker.sourceforge.net/) TRAC- “ Trac is an enhanced wiki and issue tracking system for software development projects. Trac uses a minimalistic approach to web-based software project management. Our mission is to help developers write great software while staying out of the way. Trac should impose as little as possible on a team's established development process and policies.” (http://trac.edgewall.org/) SOURCEFORGE- “is a web-based service that offers a source code repository, downloads mirrors, bug tracking and other features. It acts as a central location that software developers can use to control and manage free and open-source software development.” (https://en.wikipedia.org/wiki/SourceForge) GFORGE- “ a free software fork of the web-based project-management and collaboration software originally created for SourceForge, called Savane.” (https://en.wikipedia.org/wiki/GForge)

28. Recap  Reports about problems encountered in the field are stored in a problem database.  A problem report must contain everything relevant to reproduce the problem.  It is helpful to set up a standard set of items that users must provide (product release, operating environment…)  A typical problem life cycle starts with an unconfirmed status and ends with a closed status with a specified resolution  Typically, a software change control board organizes priorities and assignments  Use version control to separate fixes and features during development.  Establish conventions to relate changes to problem reports and vice versa.  Make a problem report obsolete as soon as a test case exists. *Recap and all other information obtained from Why Programs Fail: A Guide to Systematic Debugging, A. Zeller, Elsevier and coinciding website unless otherwise specified 28