1. Working Software (Testing)
Today’s Topic
Why testing?
Some basic definitions
Kinds of testing
Test-driven development
Code reviews (not testing)

2. Why Testing?
Ideally: we’d prove code correct, using formal mathematical techniques (with a computer, not chalk)
Extremely difficult: for some trivial programs (100 lines) and many small (5K lines) programs
Simply not practical to prove correctness in most cases – often not even for safety or mission critical code

3. Why Testing?
Nearly ideally: use symbolic or abstract model checking to prove that a model is correct
Automatically extract a mathematical abstraction from code
Prove properties with model over all possible executions
In practice, can work well for very simple properties (“this program never crashes in this particular way”), of some programs, but can’t handle complex properties (“this is a working file system”)
Doesn’t work well for programs with complex data structures (like a file system)

4. As a last resort…
… we can actually run the program, to see if it works
This is software testing
Always necessary, even when you can prove correctness – because the proof is seldom directly tied to the actual code that runs
“Beware of bugs in the above code; I have only proved it correct, not tried it” – Knuth

5. NOT a last resort…
Testing is a critical part of every software development effort
Can too easily be left as an afterthought, after it is expensive to correct faults and when deadlines are pressing
The more code that has been written when a fault is detected, the more code that may need to be changed to fix the fault
Consider a key design flaw: better to detect with a small prototype, or after implementation is “finished”?
May “have to ship” the code even though it has fatal flaws

6. Testing and Reviews in Processes
Waterfall
Requirements analysis
Prototyping
Design
Implementation
Testing
Operation

7. Testing and Reviews in Processes
Spiral
Draft a menu of program designs
Analyze risk & prototype
Draft a menu of architecture designs
Analyze risk & prototype
Draft a menu of requirements
Analyze risk & prototype
Establish requirements
Plan
Establish architecture
Plan
Establish program design
Operation
Testing
Implementation

8. Testing and Reviews in Processes
Agile
Do “spike” to evaluate & control risk
Customer provides “stories” (short requirement snippets)
Prioritize stories and plan
Write/run/modify unit tests
Operation
Implement
System and acceptance tests

9. Testing saves lives and money
NIST report, “The Economic Impacts of Inadequate Infrastructure for Software Testing” (2002)
Inadequate software testing costs the US alone between $22 and $59 billion annually
Better approaches could cut this amount in half
Major failures: Ariane 5 explosion, Mars Polar Lander, Intel’s Pentium FDIV bug
Insufficient testing of safety-critical software can cost lives: THERAC-25 radiation machine: 3 dead
We want our programs to be reliable
Testing is how, in most cases, we find out if they are
Ariane 5: exception-handling bug : forced self destruct on maiden flight (64-bit to 16-bit conversion: about 370 million $ lost)
Mars Polar Lander crash site?
THERAC-25 design

10. Today’s Topic Why testing? Some basic definitions Kinds of testing
Test-driven development
Code reviews (not testing)

11. Basic Definitions: Testing
What is software testing?
Running a program
Generally, to reveal faults (bugs)
Could be in the code
Or in the spec
Or in the documentation
Or in the test…

12. Faults vs Failures Fault (defect): a static flaw in a program
What we usually think of as “a bug”
Failure: an observable incorrect behavior of a program as a result of an error
Not every fault ever leads to a failure

13. Bugs
Hopper’s “bug” (moth stuck in a relay on an early machine)
“an analyzing process must equally have been performed in order to furnish the Analytical Engine with the necessary operative data; and that herein may also lie a possible source of error. Granted that the actual mechanism is unerring in its processes, the cards may give it wrong orders. ” – Ada, Countess Lovelace (notes on Babbage’s Analytical Engine)
“It has been just so in all of my inventions. The first step is an intuition, and comes with a burst, then difficulties arise—this thing gives out and [it is] then that 'Bugs'—as such little faults and difficulties are called—show themselves and months of intense watching, study and labor are requisite. . .” – Thomas Edison
Term “bug” erroneously attributed to Grace Hopper, who did have a literal bug

14. Terms: Test (Case) vs. Test Suite
Test (case): one execution of the program, that may expose a bug
Test suite: a set of executions of a program, grouped together
A test suite is made of test cases
Tester: a program that generates tests

15. Exhaustive vs. Representative Testing
Can we test everything?
File system is a library, called by other components of some flight software
Operation Result
mkdir (“/eng”, …) SUCCESS
mkdir (“/data”, …) SUCCESS
creat (“/data/image01”, …) SUCCESS
creat (“/eng/fsw/code”, …) ENOENT
mkdir (“/data/telemetry”, …) SUCCESS
unlink (“/data/image01”) SUCCESS
File system /
/eng
/data
image01
/telemetry

16. Example: File System Testing
How hard would it be to just try “all” the possibilities?
Consider only core 7 operations (mkdir, rmdir, creat, open, close, read, write)
Most of these take either a file name or a numeric argument, or both
Even for a “reasonable” (but not provably safe) limitation of the parameters, there are executions of length 10 to try
Not a realistic possibility (unless we have years to test)
Assuming that you could run 56,235 tests per second!

17. “The Testing Problem”
Cannot execute all possible tests (exhaustive testing): must choose a smaller set
How do we select a small set of executions out of a very large set of executions?
Fundamental problem of software testing research and practice
An open (and essentially unsolvable, in the general case) problem

18. Terms: Coverage Coverage measures or metrics
Abstraction of “what a test suite tests” in a structural sense
Common measures:
Statement coverage
A.k.a line coverage or basic block coverage
Which statements execute in a test suite
Decision coverage
Which boolean expressions in control structures evaluated to both true and false during suite execution
Path coverage
Which paths through a program’s control flow graph are taken in the test suite
Mutation coverage
Ability to detect random variations to the code

19. Practice Problem What inputs would you need to achieve
int foo(int x, int y) {
int z = 0;
if ((x>0) && (y>0)) {
z = x; }
return z;
What inputs would you need to achieve
Statement coverage?
Decision coverage?
Path coverage?
Mutation coverage with a mutation operator that replaces > with <? (Hint: Kill all mutants!)
return z;
z = x;
int z = 0;
if ((x>0) && (y>0)) {
true
false
Control Flow Graph
Answers
Statement: foo(1,1)
Decision: foo(1,1), foo(0,1)
Path: foo(1,1), foo(0,1)
Mutation:
x < 0: foo(-1,1)
y < 0: foo(1,-1)
x < 0 && y < 0: foo(-1,-1)
OR just foo(1,1) would work

20. Terms: Coverage Measures
In general, used to measure the quality of a test suite
Even in cases where the suite was designed for some other purpose (such as testing lots of different use scenarios)
Not always a very good measure of suite quality, but “better than nothing”
We “open the box” in white box testing partly to look at (and design tests to achieve) coverage

21. Today’s Topic Why testing? Some basic definitions Kinds of testing
Test-driven development
Code reviews (not testing)

22. Kinds of testing Whitebox Blackbox Unit Integration System Acceptance
Regression

23. Terms: Black Box Testing
Treats a program or system as a
That is, testing that does not look at source code or internal structure of the system
Send a program a stream of inputs, observe the outputs, decide if the system passed or failed the test
Abstracts away the internals – a useful perspective for integration and system testing
Sometimes you don’t have access to source code, and can make little use of object code
True black box? Access only over a network

24. Terms: White Box Testing
Opens up the box!
(also known as glass box, clear box, or structural testing)
Use source code (or other structure beyond the input/output spec.) to design test cases

25. Stages of Testing
Unit testing is the first phase, done by developers of modules
Integration testing combines unit-tested modules and tests how they interact
System testing tests a whole program to make sure it meets requirements
Acceptance testing by users to see if system meets actual use requirements

26. Stages of Testing: Unit Testing
Unit testing is the first phase, mostly done by developers of modules
Typically the earliest type of testing done
Unit could be as small as a single function or method
Often relies on stubs to represent other modules and incomplete code
Tools to support unit tests available for most popular languages, e.g. JUnit (
Stub: A piece of code that simulates the activity of missing components.
Driver: A piece of code that passes test cases to another piece of code.

27. Stages of Testing: Integration Testing
Integration testing combines unit-tested modules and tests how they interact
Relies on having completed units
After unit testing, before system testing
Test cases focus on interfaces between components, and assemblies of multiple components
Often more formal (test plan presentations) than unit testing

28. Stages of Testing: System Testing
System testing tests a whole program to make sure it meets requirements
After integration testing
Focuses on “breaking the system”
Defects in the completed product, not just in how components interact
Checks quality of requirements as well as the system
Often includes stress testing, goes beyond bounds of well-defined behavior

29. An aspect of System Testing: Functional Testing
Functional testing is when a developer tests a program from a “user’s” perspective – does it do what it should?
It’s a different mindset than unit testing, which often proceeds from the perspective of other parts of the program
Module spec/interface, not user interaction
Sort of a fuzzy line – consider a file system – how different is the use by a program and use of UNIX commands at a prompt by a user?
Building inspector does “unit testing”; you (or user), walking through the house to see if its livable, perform “functional testing”
Kick the tires vs. take it for a spin?

30. Stages of Testing: Acceptance Testing
Acceptance testing by users to see if system meets actual use requirements
Black box testing
By end-users to determine if the system produced really meets their needs
May revise requirements/goals as much as find bugs in the code/system

31. Appropriate at all times: Regression Testing
Changes can break code, reintroduce old bugs
Things that used to work may stop working (e.g., because of another “fix”) – software regresses
Usually a set of cases that have failed (& then succeeded) in the past
Finding small regressions is an ongoing research area – analyze dependencies
“. . . as a consequence of the introduction of new bugs, program maintenance requires far more system testing Theoretically, after each fix one must run the entire batch of test cases previously run against the system, to ensure that it has not been damaged in an obscure way. In practice, such regression testing must indeed approximate this theoretical idea, and it is very costly." - Brooks, The Mythical Man-Month

32. Today’s Topic Why testing? Some basic definitions Kinds of testing
Test-driven development
Code reviews (not testing)

33. Test-Driven Development
One way to make sure code is tested as early as possible is to write test cases before the code
Idea arising from Extreme Programming and often used in agile development
Write (automated) test cases first
Then write the code to satisfy tests

34. Test-Driven Development
How to add a feature to a program, in test-driven development
Add a test case that fails, but would succeed with the new feature implemented
Run all tests, make sure only the new test fails
Write code to implement the new feature
Rerun all tests, making sure the new test succeeds (and no others break)

35. Test-Driven Development Cycle

36. Test-Driven Development Benefits
Results in lots of useful test cases
A very large regression set
Forces attention to actual behavior of software: observable & controllable behavior
Only write code as needed to pass tests
And may get good coverage of paths through the program, since they are written in order to pass the tests
Reduces temptation to tailor tests to idiosyncratic behaviors of implementation
Testing is a first-class activity in this kind of development

37. Test-Driven Development Problems
Need institutional support
Difficult to integrate with a waterfall development
Management may wonder why so much time is spent writing tests, not code
Lots of test cases may create false confidence
If developers have written all tests, may be blind spots due to false assumptions made in coding and in testing, which are tightly coupled

38. Today’s Topic Why testing? Some basic definitions Kinds of testing
Test-driven development
Code reviews (not testing)

39. Not Testing: Code Reviews
Not testing, exactly, but an important method for finding bugs and determining the quality of code
Code walkthrough: developer leads a review team through code
Informal, focus on code
Code inspection: review team checks against a list of concerns
Team members prepare offline in many cases
Team moderator usually leads

40. Not Testing: Code Reviews
Code inspections have been found to be one of the most effective practices for finding faults
Some experiments show removal of 67-85% of defects via inspections
Some consider XP’s pair programming as a kind of “code review” process, but it’s not quite the same
Why?
Can review/walkthrough requirements and design documents, not just code!

41. Testing and Reviews in Processes
Key differences?
More integrated in agile
Part of the “inner loop”
More formal, external, “barrier” in waterfall
In practice, how much testing is done by developers will vary beyond just process
Agile methods tend to encourage heavy unit testing

42. What’s next for you Create your… User stories Tasks and estimates
UML sequence diagrams or spikes
A plan describing
What will be done in a week
What will be done in two weeks
Summary of who did what (customer & you)

43. Copyright (c) Christopher Scaffidi 2009 All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of Oregon State University nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Modified by Scott D. Fleming 2011.