1. Software Testing
Introduction

2. IT Map Reqs and Specs Analysis Design Coding Testing Deployment
Prj. Mgmt
Methodologies
Infrastructure
Maintainance
OSs
Databases
AI Systems
Expert Systems
Neural Net Systems
Real-time systems
- Embedded systems
Web Systems
Games
Graphical apps
Office apps
Theory x X
Other

3. References
C. Kaner, J. Falk, H.Q. Nguyen: Testing computer software, Second edition, Wiley & Sons, 1999
G. Everett, R. McLeod: Software Testing. Testing Across the Entire Software Development Life Cycle, Wiley&Sons ,2007
G. Myers: The Art of Software Testing, Second edition, Wiley & Sons, 2004
R. Patton: Software Testing, Sams Publishing, 2005
W. Lewis: Software Testing and Continuous Quality Improvement, Third edition, CRC Press, 2009

4. Course topics Placing testing in the context of software activities
Considering testing as a QA activity
Software errors
Testing types and testing methods
Building testing plans
Building test cases from requirements
Testing management
Test driven programming
Testing environments
Tests automation
Testing reports

5. Examination Marks totaling 11 points
2 marked activities during the semester plus final exam.
Sending activities results in time provides for max points.
Re-examination will only evaluate the final exam. Projects cannot be re-submitted.
1 point for >75% participation of ALL course activities. 0 points for < 40% participation.

6. First computer bug: Mark II (1947)*
List of incidents caused by software bugs:

7. Sample testing scenario
Inspired form Kaner & co
Program to test: adding two numbers of max two digits. The program will read the numbers echoing them and will print the sum. The user has to press ENTER after each number.

8. STEP 1: Simple test Purpose: familiarizing with the program
Check minimal program stability: program often crashes right away
Do not spend too much time
Start the program and add 2 with 3

9. Results of Step 1 Result: Problems? Nothing shows what program this is
No onscreen instructions
How to stop the program?
Numbers alignment
Actions:
Create problem reports
One problem per report
? 2
? 3 5
? _

10. Problem report Report # Program (name/release/version)
Report type (coding, design, suggestion, documentation, hardware, query)
Severity (fatal/serious/minor)
Problem summary
Is reproducible?
Problem description
Suggested fix (optional)
Reported by
Date

11. STEP 2: What else need testing?
Valid Inputs using all digits:
99+99
99+-14
-38+99
56+99
9+9
0+0
0+23
-78+0
Etc.

12. Boundary conditions
Classes of tests: if the same result is expected from two tests, test only one of them
Tests the variant most likely to fail: look at the boundaries of a class
Finding boundary conditions: no magic way, use experience
Programming boundaries(from program listing) vs. testing boundaries(user perspective)
Test both sides of a boundary

13. Step 3: Check the valid cases
Results:
Positive numbers are fine
None of the tests with negatives worked
The computer locked when entering the second digit of a negative number. It also locked on after pressing <Enter>

14. Step 4: Exploring invalid cases
Switching from formal to informal tests
The program significantly crashed therefore switch to informal tests
Keep testing with invalid cases
No formality needed as the program may have to be redesigned
But always write down the results

15. Exploratory tests
: boundary condition. The second 0 acted as <Enter> (100 10)
<Enter>+<Enter> : no input. The program used the last entered number
: how many digits the program accepts. The program accepted the first two digits and ignored the rest
1.2+5: decimal numbers. The decimal point was treated as <Enter>
A+b: invalid chars. Program locked after the first <Enter> and needed computer restart.
<Ctrl-A> + <Ctrl-B>: control characters and function keys. The program locked. On <Ctrl-C> it exited.

16. Step 5: Summarize the program’s behavior
For tester’s use
Helps thinking about the program in order to elaborate a testing strategy later
Identify new things like new boundary conditions
Ex:
The communication style of the program is terse
The program does not deal with negative numbers
The program accepts any char as a valid input until <Enter>
The program does not check if some number is entered before <Enter>

17. Investigate the summary
Assumes there were some reasons for the problems
Fast ?
No characters check
No displayed messages
Retaining the previous variable values
Low memory?
No error handling
No messages
Cannot tell which dimension is more important

18. Storage investigations
Storage of the sum: 1 byte or 2 bytes?
New possible boundary condition: 127/128 as a sum (1 byte storage for positive and negative numbers)
Storage of the input: look at the characters
ASCII codes: ‘0’(48), ‘9’(57),
Consider character validation routine:
IF code < 48 THEN reject
ELSE IF code > 57 THEN reject
ELSE accept

19. Ways for the routine to fail
< mistyped as <=
48 mistyped as 47
48 mistyped as 38
> mistyped as >=
57 mistyped as 58
57 typed as 75
New boundary characters: ‘/’ and ‘:’ beside ‘0’ and ‘9’

20. Speed investigation Adding algorithm changes Potential issue: Carry
Which base?
10: interesting sums are 10, 100
2: interesting sums are 2, 4, …, 128

21. Step 6: Review programmers answers to problem reports
Speed vs. memory: The speed was considered important
Missing title: won’t fix
No instructions: won’t fix, will slow the program
How to stop: <Ctrl-C> ; added message on screen
Numbers alignment: fixed
Crashing on negative numbers: fixed
Assumes the 3’rd char is <Enter>: will fix
Crashing when entering non numbers: won’t fix
Crashing with control chars: won’ t fix

22. Step 7: Review problems which won’t be fixed
Error handling is the most important issue
Tactic: find test cases to make the problem appear under innocent circumstances
Ex: find common keys the user expect the arithmetic program to handle and make a list
Digits, minus and plus signs, spaces, arithmetic expressions (4*3+5), $, %, parentheses, backspace, delete, insert, cursor movements keys

23. Step 8: Restart testing
Redo the valid tests and the fixed issues tests: they worked
Noticed the “Press Ctrl-C to quit” message after every sum. This make the program slower. Make a problem report.
Perform the new tests you imagined. Observed that keys like <Backspace>, <Space>, or <+> do not work and crashes the program. Create a problem report.

24. Future steps
As the development progresses, the list with the formal tests will increase and should be followed each time the program returns to testing.
Minimize the formal testing only after several versions regularly pass them
Be involved in making the stakeholders understanding the severity of the issues

25. What is testing?
Kaner: “A technical investigation of the product under tests conducted to provide stakeholders with quality related information.”
Technical = tools, mathematics, logic, experimentation
Investigation=organized search, questioning
Product = data, documentation, hardware
Stakeholder = someone with interest in the testing effort
Quality = presence or absence of bugs

26. Testing dimensions Someone Something Somewhere TESTING With a purpose
Somehow
Sometime

27. Some fundamental questions in software testing (Kaner)
Why are you testing? [What is the mission of your testing?]
How should you organize your work to achieve your mission? [The strategy problem]
How will you know whether the program passed or failed the test? [The oracle problem]
What would it take to do a complete testing job? [The impossibility of complete testing]
How much testing is enough? [The measurement problem]

28. Testing objectives: the mission
Find important bugs, to get them fixed
Assess the quality of the product
Help managers make release decisions
Block premature product releases
Help predict and control costs of product support
Check interoperability with other products
Find safe scenarios for use of the product
Assess conformance to specifications
Certify the product meets a particular standard
Minimize the risk of safety-related lawsuits
Help clients improve product quality & testability
Help clients improve their processes
Evaluate the product for a third party

29. Some useful oracle heuristics: deciding if a problem is a bug
Black Box Software Testing
Some useful oracle heuristics: deciding if a problem is a bug
Consistent with History: Present function behavior is consistent with past behavior.
Consistent with the Image: Function behavior is consistent with an image that the organization wants to project.
Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products.
Consistent with Claims: Function behavior is consistent with documented or advertised behavior.
Winter 2004

30. Some useful oracle heuristics: deciding if a problem is a bug
Consistent with Specifications or Regulations: Function behavior is consistent with claims that must be met.
Consistent with User’s Expectations: Function behavior is consistent with what we think users want.
Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product.
Consistent with Purpose: Function behavior is consistent with apparent purpose.

31. Few Testing principles(Everett-McLeod)
Business risk can be reduced by finding defects.
Positive and negative testing contribute to risk reduction
Static and execution testing contribute to risk reduction
Automated test tools can contribute to risk reduction
Make the highest risks the first testing priority.
Make the most frequent business activities (the 80/20 rule) the second testing priority.
Statistical analyses of defect arrival patterns and other defect characteristics are a very effective way to forecast testing completion.
Test the system the way customers will use it.
Assume the defects are the result of process and not personality.
Testing for defects is an investment as well as a cost.
Everett-McLeod p.17

32. Testing axioms It's impossible to test a program completely
Software testing Is a Risk-Based exercise
Testing can't show that bugs don't exist
The more bugs you find, the more bugs there are
The more you test software, the more immune it becomes to your tests
Not all the bugs you find will be fixed
When a bug's a bug is difficult to say
Product specifications are never final
Software Testing By Ron Patton

33. How much to test?