Software Testing Introduction

IT Map Reqs and Specs Analysis Design Coding Testing Deployment Prj. Mgmt Methodologies Infrastructure Maintainance OSs Databases AI Systems Expert Systems Neural Net Systems Real-time systems - Embedded systems Web Systems Games Graphical apps Office apps Theory x X Other

References C. Kaner, J. Falk, H.Q. Nguyen: Testing computer software, Second edition, Wiley & Sons, 1999 G. Everett, R. McLeod: Software Testing. Testing Across the Entire Software Development Life Cycle, Wiley&Sons ,2007 G. Myers: The Art of Software Testing, Second edition, Wiley & Sons, 2004 R. Patton: Software Testing, Sams Publishing, 2005 W. Lewis: Software Testing and Continuous Quality Improvement, Third edition, CRC Press, 2009

Course topics Placing testing in the context of software activities Considering testing as a QA activity Software errors Testing types and testing methods Building testing plans Building test cases from requirements Testing management Test driven programming Testing environments Tests automation Testing reports

Examination Marks totaling 11 points 2 marked activities during the semester plus final exam. Sending activities results in time provides for max points. Re-examination will only evaluate the final exam. Projects cannot be re-submitted. 1 point for >75% participation of ALL course activities. 0 points for < 40% participation.

First computer bug: Mark II (1947) *http://www.history.navy.mil/photos/images/h96000/h96566kc.htm List of incidents caused by software bugs: http://www5.in.tum.de/~huckle/bugse.html

Sample testing scenario Inspired form Kaner & co Program to test: adding two numbers of max two digits. The program will read the numbers echoing them and will print the sum. The user has to press ENTER after each number.

STEP 1: Simple test Purpose: familiarizing with the program Check minimal program stability: program often crashes right away Do not spend too much time Start the program and add 2 with 3

Results of Step 1 Result: Problems? Nothing shows what program this is No onscreen instructions How to stop the program? Numbers alignment Actions: Create problem reports One problem per report ? 2 ? 3 5 ? _

Problem report Report # Program (name/release/version) Report type (coding, design, suggestion, documentation, hardware, query) Severity (fatal/serious/minor) Problem summary Is reproducible? Problem description Suggested fix (optional) Reported by Date

STEP 2: What else need testing? Valid Inputs using all digits: 99+99 -99+ -99 99+-14 -38+99 56+99 9+9 0+0 0+23 -78+0 Etc.

Boundary conditions Classes of tests: if the same result is expected from two tests, test only one of them Tests the variant most likely to fail: look at the boundaries of a class Finding boundary conditions: no magic way, use experience Programming boundaries(from program listing) vs. testing boundaries(user perspective) Test both sides of a boundary

Step 3: Check the valid cases Results: Positive numbers are fine None of the tests with negatives worked The computer locked when entering the second digit of a negative number. It also locked on -9+-9 after pressing <Enter>

Step 4: Exploring invalid cases Switching from formal to informal tests The program significantly crashed therefore switch to informal tests Keep testing with invalid cases No formality needed as the program may have to be redesigned But always write down the results

Exploratory tests 100+100: boundary condition. The second 0 acted as <Enter> (100 10) <Enter>+<Enter> : no input. The program used the last entered number 123456+0: how many digits the program accepts. The program accepted the first two digits and ignored the rest 1.2+5: decimal numbers. The decimal point was treated as <Enter> A+b: invalid chars. Program locked after the first <Enter> and needed computer restart. <Ctrl-A> + <Ctrl-B>: control characters and function keys. The program locked. On <Ctrl-C> it exited.

Step 5: Summarize the program’s behavior For tester’s use Helps thinking about the program in order to elaborate a testing strategy later Identify new things like new boundary conditions Ex: The communication style of the program is terse The program does not deal with negative numbers The program accepts any char as a valid input until <Enter> The program does not check if some number is entered before <Enter>

Investigate the summary Assumes there were some reasons for the problems Fast ? No characters check No displayed messages Retaining the previous variable values Low memory? No error handling No messages Cannot tell which dimension is more important

Storage investigations Storage of the sum: 1 byte or 2 bytes? New possible boundary condition: 127/128 as a sum (1 byte storage for positive and negative numbers) Storage of the input: look at the characters ASCII codes: ‘0’(48), ‘9’(57), Consider character validation routine: IF code < 48 THEN reject ELSE IF code > 57 THEN reject ELSE accept

Ways for the routine to fail < mistyped as <= 48 mistyped as 47 48 mistyped as 38 > mistyped as >= 57 mistyped as 58 57 typed as 75 New boundary characters: ‘/’ and ‘:’ beside ‘0’ and ‘9’

Speed investigation Adding algorithm changes Potential issue: Carry Which base? 10: interesting sums are 10, 100 2: interesting sums are 2, 4, …, 128

Step 6: Review programmers answers to problem reports Speed vs. memory: The speed was considered important Missing title: won’t fix No instructions: won’t fix, will slow the program How to stop: <Ctrl-C> ; added message on screen Numbers alignment: fixed Crashing on negative numbers: fixed Assumes the 3’rd char is <Enter>: will fix Crashing when entering non numbers: won’t fix Crashing with control chars: won’ t fix

Step 7: Review problems which won’t be fixed Error handling is the most important issue Tactic: find test cases to make the problem appear under innocent circumstances Ex: find common keys the user expect the arithmetic program to handle and make a list Digits, minus and plus signs, spaces, arithmetic expressions (4*3+5), $, %, parentheses, backspace, delete, insert, cursor movements keys

Step 8: Restart testing Redo the valid tests and the fixed issues tests: they worked Noticed the “Press Ctrl-C to quit” message after every sum. This make the program slower. Make a problem report. Perform the new tests you imagined. Observed that keys like <Backspace>, <Space>, or <+> do not work and crashes the program. Create a problem report.

Future steps As the development progresses, the list with the formal tests will increase and should be followed each time the program returns to testing. Minimize the formal testing only after several versions regularly pass them Be involved in making the stakeholders understanding the severity of the issues

What is testing? Kaner: “A technical investigation of the product under tests conducted to provide stakeholders with quality related information.” Technical = tools, mathematics, logic, experimentation Investigation=organized search, questioning Product = data, documentation, hardware Stakeholder = someone with interest in the testing effort Quality = presence or absence of bugs

Testing dimensions Someone Something Somewhere TESTING With a purpose Somehow Sometime

Some fundamental questions in software testing (Kaner) Why are you testing? [What is the mission of your testing?] How should you organize your work to achieve your mission? [The strategy problem] How will you know whether the program passed or failed the test? [The oracle problem] What would it take to do a complete testing job? [The impossibility of complete testing] How much testing is enough? [The measurement problem]

Testing objectives: the mission Find important bugs, to get them fixed Assess the quality of the product Help managers make release decisions Block premature product releases Help predict and control costs of product support Check interoperability with other products Find safe scenarios for use of the product Assess conformance to specifications Certify the product meets a particular standard Minimize the risk of safety-related lawsuits Help clients improve product quality & testability Help clients improve their processes Evaluate the product for a third party

Some useful oracle heuristics: deciding if a problem is a bug Black Box Software Testing Some useful oracle heuristics: deciding if a problem is a bug Consistent with History: Present function behavior is consistent with past behavior. Consistent with the Image: Function behavior is consistent with an image that the organization wants to project. Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products. Consistent with Claims: Function behavior is consistent with documented or advertised behavior. Winter 2004

Some useful oracle heuristics: deciding if a problem is a bug Consistent with Specifications or Regulations: Function behavior is consistent with claims that must be met. Consistent with User’s Expectations: Function behavior is consistent with what we think users want. Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product. Consistent with Purpose: Function behavior is consistent with apparent purpose.

Few Testing principles(Everett-McLeod) Business risk can be reduced by finding defects. Positive and negative testing contribute to risk reduction Static and execution testing contribute to risk reduction Automated test tools can contribute to risk reduction Make the highest risks the first testing priority. Make the most frequent business activities (the 80/20 rule) the second testing priority. Statistical analyses of defect arrival patterns and other defect characteristics are a very effective way to forecast testing completion. Test the system the way customers will use it. Assume the defects are the result of process and not personality. Testing for defects is an investment as well as a cost. Everett-McLeod p.17

Testing axioms It's impossible to test a program completely Software testing Is a Risk-Based exercise Testing can't show that bugs don't exist The more bugs you find, the more bugs there are The more you test software, the more immune it becomes to your tests Not all the bugs you find will be fixed When a bug's a bug is difficult to say Product specifications are never final Software Testing By Ron Patton

How much to test?