http://flic.kr/p/7iffKj Software Testing

SWEBOK Knowledge Areas Software Requirements Software Design Software Construction Software Testing Software Maintenance Software Configuration Management Software Engineering Management Software Engineering Process Software Engineering Models and Methods Software Quality Software Engineering Professional Practice Software Engineering Economics Computing Foundations Mathematical Foundations Engineering Foundations Today’s topic

Can you create “real” software that is bug free? Alas, no Our goal: Reduce number of bugs – especially those users are likely to counter What is dominant means to discover bugs? Software testing

Testing can be used for both “V & V” Activities Verification: Check that system complies with requirements (based on your understanding) Answers: Did we build the system right? Validation: Check that system actually meets customer’s needs Answers: Did we build the right system? Testing can be used for both

Bug-Related Terminology Defect/Bug/Fault: Thing that developer did wrong Cause of program’s incorrect behavior Error: Incorrect behavior of program May not be observable to user Failure: User-observable error; loss of functionality Some authors prefer to flip the above defs of fault and error

Testing-Related Terminology Testing: Execute code to reveal defects Does it behave as expected? Test/Test Case: One execution of code that may expose bug Test Suite: Set of test cases Often used to group related tests

How do you completely test a program? Try all possible inputs—called exhaustive testing Can we do exhaustive testing in practice? No. Consider this example…

Example: File operation Takes file name as argument Each character an 8-bit byte How many tests cases to exhaustively test? If you limit file name to 10 characters, there are 25610 different test cases Even if you can run 1000 tests per second, it would still take over 1013 years! Clearly, exhaustive testing not feasible… =((2^8)^10)/(1000*60*60*24*365)

The Testing Problem: How to choose small set of test cases that reveal all errors? Fundamental research problem Essentially unsolvable in general case

Decisions when designing tests What subset of system to test? Target of test How to choose test cases?

Decisions when designing tests What subset of system to test? Target of test How to choose test cases?

What subset of system to test? smaller subset Unit testing: Test modules in isolation; smallest “units” of code Integration testing: Test groups of collaborating units System testing: Test the complete system larger subset

Unit versus Integration Testing Problem: Definition of “unit” varies (class? method?...) How to distinguish: Unit test Integration test Repeatable (context independent) May involve system time, machine name, etc. Deterministic May involve threads, random numbers, etc. In memory May involve file system, DB Fast (less than 0.5 seconds) May be slow Tests one property May test multiple properties

Decisions when designing tests What subset of system to test? Target of test How to choose test cases?

How to choose test cases? Blackbox testing: Choose based on module’s possible inputs and outputs Do not use code Often test boundary cases Whitebox testing: Uses internal logic to choose tests Different levels of code coverage Aka glass box testing, clear box testing Regression testing: Keep tests that reveal old bugs Rationale: “Fixed” bugs come back!

Many higher-level testing strategies as well Acceptance testing: Users test to see if system meets actual use requirements Serves both validation and verification Usability testing: Observe users using system with eye on usability of system Test-driven development: … more to come … And many more…

Tools are helpful and important What affordances might testing tools provide?

Tools are helpful and important What affordances might testing tools provide? Simplify writing test cases (e.g., with library) Automatically execute tests Summarize results of tests Help “jump to” location of failure Tell how “good” your test cases are (e.g., metrics) Generate test cases for you

Particularly common in testing frameworks (e.g., JUnit, RSpec) Tools are helpful and important What affordances might testing tools provide? Particularly common in testing frameworks (e.g., JUnit, RSpec) Simplify writing test cases (e.g., with library) Automatically execute tests Summarize results of tests Help “jump to” location of failure Tell how “good” your test cases are (e.g., metrics) Generate test cases for you

Why is automated testing important?

From a testing perspective, why might this scenario end badly? Initial Plan: Implement software, then test at end Implementation takes 6 months longer than expected System is huge! Finally, it’s time to test System is huge, so testing job is HUGE Pressure to ship causes skimping on testing Defects discovered this late are costly to fix

Defect Cost Increase (DCI) Principle The later you discover defect, the more expensive to fix Time Bad way: Better way: How to achieve? From Extreme Programming Explained by Beck & Andres

Test-Driven Development (TDD) Idea: Test First! Write unit tests before functional code Typically blackbox tests

Running Example: Preorder Coffee with Gift Card

US and Tasks Start with Task 1

Further Break Down the Task Represent order info Represent gift card info Represent receipt info

Next Step: Write Test Code

Rule #1: Your test should fail before you implement any code Establishes a measure of success Promotes programming incrementally

Next: Write code to make the test pass

Rule #2: Implement the simplest code possible to make the test pass Also helps to promote incremental programming By focusing on small bits of code Helps resist urge to add unwanted extras “Gold Plating” Speculative generality: Adding lots of features to make reusable something that will never be reused

Test-Driven Development Cycle

Next (Red): Test order information

In writing the test, you design the class interface — just enough interface! 1. Create an object 2. Set its attributes 3. Get and check the attribute values

Next (Green): Implement the interface you designed for the test Instance variables “Setter” methods “Getter” methods

If in the process of building up classes, you realize your design could be improved, then REFACTOR! … and continue going around and around …

As you go, you expand upon the systems capabilities You might do a test for each of these cases: A gift card with more than enough to cover the cost of the order A gift card without enough to cover the cost of the order An invalid gift card number A gift card with exactly the right amount A gift card that hasn’t been activated A gift card that’s expired

Despite cons, TDD is a widely advocated practice Pros/Cons of TDD Pros: Yields lots of test cases More tests leads to increased confidence Cons: False sense of confidence? Non-TDD folks may not understand why writing so many tests and not functionality Despite cons, TDD is a widely advocated practice