1. IS4500 – SOFTWARE QUALITY ASSURANCE SERVICE & FUNCTIONAL TESTING STRATEGIES Copyright © 2012 by Martin Schedlbauer, Ph.D. All Rights Reserved. Do Not Duplicate or Distribute Without Written Consent of the Author. www.neu.edu · m.schedlbauer@neu.edu

2. Objectives Upon completion of this chapter you will be able to: ◦ Test the services or functions provided by a solution ◦ Reduce the number of test cases through equivalence partitioning ◦ Identify common problems through boundary analysis ◦ Map application states and state transitions ◦ Leverage use cases to build better test cases ◦ Use decision tables to understand expected responses ◦ Perform ad-hoc exploratory testing when specifications are unavailable v1.01 2 Solution Validation & Testing

3. Motivation Testing is not an ad hoc activity and does not simply mean that one “bangs against” a solution in order to break it. Rather, testing is a formal activity that is documented in detail. In fact, one of the most important skills of a successful tester is to be detail oriented and methodical. Of course, the degree of formality depends on the type of solution that is being tested and the consequence of latent defects. v1.01 3 Solution Validation & Testing

4. Black Box Testing This module shows how to test software with “blinders on”, i.e., the solution is treated as a “black box” or in some cases as a “grey box”. While programming skills are not necessary to be an effective tester, some understanding of programming is helpful, particularly when it comes to understanding boundary value analysis. v1.01 4 Solution Validation & Testing

5. Exploratory vs Predictive Testing Exploratory testing is an ad hoc approach where testers proceed without a test plan. This approach is used when there is an absence of a formal specification or defined requirements. It is also used during reverse engineering to learn about the solution’s functionality and capabilities. The opposite of exploratory testing is predictive testing, where testing is done based on specific test cases derived from the requirements specification. v1.01 5 Solution Validation & Testing

6. What is a Test Case? A test case defines the expected outcome for a specific set of stimuli. All test cases combined make up a test suite. In practice, the terms test case and test scenario are used interchangeably. Writing test cases presumes that there is a requirements specification, otherwise perform exploratory testing. v1.01 6 Solution Validation & Testing

7. Test-to-Pass vs. Test-to-Fail Test-to-Pass: the tester checks that the solution meets the stated requirements and that it does not break when used in accordance with the specification. Test-to-Fail: the solution is pushed to the edge and is made to break – testers are forcing errors and are using the solution in ways that were not laid out in the specification. v1.01 7 Solution Validation & Testing

8. Practical Approach Of course, testers should start with the test-to-pass approach first to make sure that the solution meets the specification and that there are no defects when used properly. Then testers should “exercise” the solution and see how it behaves when used outside the design boundaries. v1.01 8 Solution Validation & Testing

9. Discussion Question What would test-to-pass and test-to-fail mean for a computer that you are trying to purchase? v1.01 9 Solution Validation & Testing

10. The Test Case Test cases should be properly documented and tracked, ideally in a database such as those provided by testing tools. Documented test cases are critical for logging purposes and to know what exactly has been tested. They are essential for regression testing. Every time some aspect of the solution has changed, all test cases testing potentially affected areas must be re-run. The test case should define the precise inputs and expected outputs; it is a series of stimulus/response pairs. v1.01 10 Solution Validation & Testing

11. Components of a Test Case v1.01 11 Solution Validation & Testing

12. Use Case Analysis in Testing Use cases describe more specific usage scenarios, whereas test cases simply apply a set of inputs to a test object to observe its reaction. As such, use cases are more realistic since they test an actual use of the system. v1.01 12 Solution Validation & Testing

13. Actors & Use Cases A use case is an interaction that a prospective user has with a solution, such as a software application or web site. The prospective user is referred to as the actor. Most actors are human users, but they can also be external systems. The key is that an actor is external, i.e., it represents an outside stimulus. v1.01 13 Solution Validation & Testing

14. Examples of Use Cases Each use case represents a goal that the actor intends to achieve by using the solution. Two examples of use cases for BoatVentures: ◦ The broker intends to capture a new charter lead ◦ The charter guest intends to search for available yachts v1.01 14 Solution Validation & Testing

15. Use Case Diagram During initial analysis and when talking with stakeholders, many analysts like to sketch the actors and the use cases. v1.01 15 Solution Validation & Testing

16. Use Case Narrative The details of each use case are documented in a narrative that: ◦ lists the steps in each successful and unsuccessful scenario that make up the interaction ◦ defines context  pre-conditions: set of conditions that must be true in order for the use case to be successful  post-conditions: set of conditions that will be true once the use case has completed either successfully or unsuccessfully ◦ links to detailed requirements and business rules ◦ links to process models, data models, user interface models, and other artifacts that support the use case v1.01 16 Solution Validation & Testing

17. Use Case Template Most organizations use a standard template for the use case narrative. Although tools exist for organizing use cases, most organizations prefer to use Microsoft Word. When using Word there are two general approaches for writing the documents: ◦ one document per use case or ◦ one document that contains all use cases v1.01 17 Solution Validation & Testing

18. Scenarios and Use Cases Normal scenario Alternate scenario Exception scenario v1.01 18 Solution Validation & Testing

19. Identifying Scenarios The analyst starts by eliciting the normal scenario. Then each step in the basic path is evaluated in collaboration with the subject matter experts and users. At each step, ask: ◦ Are there other ways in which this step can be completed? ◦ What could happen here? ◦ How might this be done differently? ◦ Does it always happen like that? ◦ What should we do if it happens differently? ◦ Is the outcome always the same? When is it different? ◦ Can we assume that the inputs are always as expected? v1.01 19 Solution Validation & Testing

20. Workshop Activity v1.01 20 Solution Validation & Testing Goal: Work with use cases. Time: 10-15 minutesFormat: Individually or in groups Materials: Case study background information in workbook Instructions: Go back to the case study background information and re-read the use case narrative for “Book a Charter”. What are the scenarios that need to be tested? Are there missing scenarios that should be considered?

21. Creating Test Cases When testing solutions, there are often thousand or more possible scenarios to test. Reduce the number of test cases and focus on the most important ones by using: ◦ Use Case Scenario Analysis ◦ Equivalence Class Partitioning ◦ Boundary Value Analysis ◦ Decision Tables, and ◦ State Charting v1.01 21 Solution Validation & Testing

22. Test Cases from Use Cases Each scenario of each use case becomes the basis for a test case. Since use cases are devoid of implementation details, they have to be expanded before they can become fully specified test cases. v1.01 22 Solution Validation & Testing

23. Equivalence Class Partitioning Equivalence partitioning is a methodical approach to reducing the number (often infinite) of possible test cases into a smaller, more manageable, but still effective set of test cases. Equivalence partition should be defined for correct as well as incorrect input values. v1.01 23 Solution Validation & Testing Definition An equivalence class or partition is a set of test cases that exercise the same part of the solution and therefore reveal the same defect if there is one present.

24. Example of Equivalence Partition Let’s take an example from the BoatVentures case. ◦ When booking a charter, guests have to provide the number of days for which they would like to charter a yacht. In this situation any value greater than zero is valid, while any value of zero or less is invalid. Testing all possible scenarios is impossible as there are an infinite number. So, what is the practical set of test cases? v1.01 24 Solution Validation & Testing

25. Boundary Value Analysis Not all values are equally likely to fail, as programmers tend to make more mistakes at the “boundaries” of equivalence partitions. This has to do with the fact that programmers frequently program control logic statements, such as IF or SWITCH, incorrectly. v1.01 25 Solution Validation & Testing

26. Example of Boundary Conditions For example, if BoatVentures states that yacht charters over 7 days receive a 10% discount, then we would expect that there is a control flow statement in the code that tests for that condition. v1.01 26 Solution Validation & Testing if (Booking.Days > 7) Booking.Price *= 0.1;

27. Identifying Boundary Edges When defining equivalence classes, rather than taking random values, it is most revealing to choose values at the boundary edges where programmers are most likely to make mistakes. When a boundary condition has been identified, create an equivalence class for a value just inside the boundary, the last valid data point, and an invalid data point just outside the boundary. v1.01 27 Solution Validation & Testing

28. Other Equivalence Classes Default and empty values Bad data v1.01 28 Solution Validation & Testing

29. Workshop Activity v1.01 29 Solution Validation & Testing Goal: Apply boundary value analysis to create good equivalence partitions. Time: 20-30 minutesFormat: Individually or in groups Materials: Whiteboard, flipchart, or paper Edge and bad data rules from workbook Instructions: Using boundary value analysis and the edge identification rules listed in the workbook, create equivalence classes for valid and invalid inputs for the following requirements and/or business rules: The number of charter guests on a time charter is limited to a maximum of six “Daily Specials” charters must be booked within 48 hours of departure and must be for a minimum of 2 hours but no more than 8 hours Day charters over 4 hours are discounted by 10% if paid with cash What bad data or invalid inputs would you test in addition to the ones discovered above through boundary value analysis?

30. Decision Tables Decision tables provide a methodical way to describe complex business rules where an outcome is based on multiple conditions and simple boundary analysis won’t suffice. A decision table is a tabular representation containing an exhaustive list of all conditions that affect some action, outcome, or calculation. v1.01 30 Solution Validation & Testing

31. Example of a Decision Table Rules for a BoatVentures charter: ◦ If the charter is longer than 4 hours, then a 10% discount is applied ◦ Parties of fewer than 3 guests do not require a mate and are discounted 5% ◦ All off-season charters receive a 10% discount ◦ All discounts are cumulative and multiple discounts can apply Let’s turn this into a decision table… v1.01 31 Solution Validation & Testing

32. Workshop Activity v1.01 32 Solution Validation & Testing Goal: Identify test cases through decision table analysis. Time: 20-25 minutesFormat: Individually or in groups Materials: Whiteboard, flipchart, or paper Instructions: Build a decision table that represents the following rules regarding licensing of charter operators: Operators of time charters must have an OUPV 50GT license for all vessels less than 48’; for those 48’ or over they must have a 100GT endorsement Operators of demise charters must possess at least a 50GT Masters license if they are not employed or contracted through BoatVentures Operators of demise charters employed or contracted through BoatVentures do not require any licenses

33. Solution 12345678 Charter TypeDDDDTTTT Vessel < 48’?YYNNYYNN Employed by BV?YNYNYNYN 50GT License?XXX 100GT License?XXX Master License?XX v1.01 33 Solution Validation & Testing

34. State Charting The testing to this point has concentrated on test cases based on various inputs – both valid and invalid. However, the logic and control flow in the solution has not yet been validated. State charting helps in testing the different solution states and whether the solution progresses through the states in the expected sequence. v1.01 34 Solution Validation & Testing

35. UML State Chart Diagram v1.01 35 Solution Validation & Testing

36. Application of State Charts Navigation Charts ◦ Each screen or page can be viewed as a valid state and getting from one screen or page to another would be a transition Application Mode Charts ◦ Mode maps are applied when an application enters some processing phase where different actions are valid depending on the mode of the interaction v1.01 36 Solution Validation & Testing

37. Example: Navigation Map v1.01 37 Solution Validation & Testing

38. Example: Mode Map v1.01 38 Solution Validation & Testing

39. Workshop Activity v1.01 39 Solution Validation & Testing Goal: Create a navigation map using state charts. Time: 20-25 minutesFormat: Individually or in groups Materials: Whiteboard, flipchart, or paper BoatVentures CRM or other site Optionally a UML modeling tool Instructions: Build a navigation map for the BoatVentures CRM or another web site or application of your choice. You should use LucidChart.com or a similar UML modeling tool if possible.

40. Recording Defects Defects discovered during testing must be recorded and details of the test and the unexpected result must be communicated to the solution development team. The report must include sufficient information for the solution team to understand the defect and be able to replicate it. v1.01 40 Solution Validation & Testing

41. Defect Attributes Let’s inspect the list of attributes that should be recorded for a defect… v1.01 41 Solution Validation & Testing

42. Summary In this module we learned that: ◦ Testing is a methodical activity that requires careful analysis ◦ Test cases are either test-to-pass or test-to-fail ◦ Use cases help in defining test cases; there is one test case for each use case scenario ◦ Equivalence partitioning reduces the number of test cases ◦ Boundary value analysis helps define the right equivalence classes ◦ Decision tables exhaustively list all combination of factors that affect an outcome when testing complex business rules ◦ State chart help analyze and track the mode of an application ◦ Defects are carefully recorded following a template that contains the information needed to reproduce and resolve the error 42 Solution Validation & Testing v1.01