1.
Software Testing

2. SWEBOK Knowledge Areas
Software Requirements
Software Design
Software Construction
Software Testing
Software Maintenance
Software Configuration Management
Software Engineering Management
Software Engineering Process
Software Engineering Models and Methods
Software Quality
Software Engineering Professional Practice
Software Engineering Economics
Computing Foundations
Mathematical Foundations
Engineering Foundations
Today’s topic

3. Can you create “real” software that is bug free?
Alas, no
Our goal: Reduce number of bugs – especially those users are likely to counter
What is dominant means to discover bugs?
Software testing

4. Testing can be used for both
“V & V” Activities
Verification: Check that system complies with requirements (based on your understanding)
Answers: Did we build the system right?
Validation: Check that system actually meets customer’s needs
Answers: Did we build the right system?
Testing can be used for both

5. Bug-Related Terminology
Defect/Bug/Fault: Thing that developer did wrong
Cause of program’s incorrect behavior
Error: Incorrect behavior of program
May not be observable to user
Failure: User-observable error; loss of functionality
Some authors prefer to flip the above defs of fault and error

6. Testing-Related Terminology
Testing: Execute code to reveal defects
Does it behave as expected?
Test/Test Case: One execution of code that may expose bug
Test Suite: Set of test cases
Often used to group related tests

7. How do you completely test a program?
Try all possible inputs—called exhaustive testing
Can we do exhaustive testing in practice?
No. Consider this example…

8. Example: File operation
Takes file name as argument
Each character an 8-bit byte
How many tests cases to exhaustively test?
If you limit file name to 10 characters, there are different test cases
Even if you can run 1000 tests per second, it would still take over 1013 years!
Clearly, exhaustive testing not feasible…
=((2^8)^10)/(1000*60*60*24*365)

9. The Testing Problem: How to choose small set of test cases that reveal all errors?
Fundamental research problem
Essentially unsolvable in general case

10. Decisions when designing tests
What subset of system to test?
Target of test
How to choose test cases?

11. Decisions when designing tests
What subset of system to test?
Target of test
How to choose test cases?

12. What subset of system to test?
smaller subset
Unit testing: Test modules in isolation; smallest “units” of code
Integration testing: Test groups of collaborating units
System testing: Test the complete system
larger subset

13. Unit versus Integration Testing
Problem: Definition of “unit” varies (class? method?...)
How to distinguish:
Unit test
Integration test
Repeatable (context independent)
May involve system time, machine name, etc.
Deterministic
May involve threads, random numbers, etc.
In memory
May involve file system, DB
Fast (less than 0.5 seconds)
May be slow
Tests one property
May test multiple properties

14. Decisions when designing tests
What subset of system to test?
Target of test
How to choose test cases?

15. How to choose test cases?
Blackbox testing: Choose based on module’s possible inputs and outputs
Do not use code
Often test boundary cases
Whitebox testing: Uses internal logic to choose tests
Different levels of code coverage
Aka glass box testing, clear box testing
Regression testing: Keep tests that reveal old bugs
Rationale: “Fixed” bugs come back!

16. Many higher-level testing strategies as well
Acceptance testing: Users test to see if system meets actual use requirements
Serves both validation and verification
Usability testing: Observe users using system with eye on usability of system
Test-driven development: … more to come …
And many more…

17. Tools are helpful and important What affordances might testing tools provide?

18. Tools are helpful and important What affordances might testing tools provide?
Simplify writing test cases (e.g., with library)
Automatically execute tests
Summarize results of tests
Help “jump to” location of failure
Tell how “good” your test cases are (e.g., metrics)
Generate test cases for you

19. Particularly common in testing frameworks (e.g., JUnit, RSpec)
Tools are helpful and important What affordances might testing tools provide?
Particularly common in testing frameworks (e.g., JUnit, RSpec)
Simplify writing test cases (e.g., with library)
Automatically execute tests
Summarize results of tests
Help “jump to” location of failure
Tell how “good” your test cases are (e.g., metrics)
Generate test cases for you

20. Why is automated testing important?

21. From a testing perspective, why might this scenario end badly?
Initial Plan: Implement software, then test at end
Implementation takes 6 months longer than expected
System is huge!
Finally, it’s time to test
System is huge, so testing job is HUGE
Pressure to ship causes skimping on testing
Defects discovered this late are costly to fix

22. Defect Cost Increase (DCI) Principle
The later you discover defect, the more expensive to fix
Time
Bad way:
Better way:
How to achieve?
From Extreme Programming Explained by Beck & Andres

23. Test-Driven Development (TDD)
Idea: Test First!
Write unit tests before functional code
Typically blackbox tests

24. Running Example: Preorder Coffee with Gift Card

25. US and Tasks
Start with Task 1

26. Further Break Down the Task
Represent order info
Represent gift card info
Represent receipt info

27. Next Step: Write Test Code

28. Rule #1: Your test should fail before you implement any code
Establishes a measure of success
Promotes programming incrementally

29. Next: Write code to make the test pass

30. Rule #2: Implement the simplest code possible to make the test pass
Also helps to promote incremental programming
By focusing on small bits of code
Helps resist urge to add unwanted extras
“Gold Plating”
Speculative generality: Adding lots of features to make reusable something that will never be reused

31. Test-Driven Development Cycle

32. Next (Red): Test order information

33. In writing the test, you design the class interface — just enough interface!
1. Create an object
2. Set its attributes
3. Get and check the attribute values

34. Next (Green): Implement the interface you designed for the test
Instance variables
“Setter” methods
“Getter” methods

35. If in the process of building up classes, you realize your design could be improved, then REFACTOR! … and continue going around and around …

36. As you go, you expand upon the systems capabilities
You might do a test for each of these cases:
A gift card with more than enough to cover the cost of the order
A gift card without enough to cover the cost of the order
An invalid gift card number
A gift card with exactly the right amount
A gift card that hasn’t been activated
A gift card that’s expired

37. Despite cons, TDD is a widely advocated practice
Pros/Cons of TDD
Pros:
Yields lots of test cases
More tests leads to increased confidence
Cons:
False sense of confidence?
Non-TDD folks may not understand why writing so many tests and not functionality
Despite cons, TDD is a widely advocated practice