Software testing techniques Introduction Prof. Rimantas Šeinauskas Kaunas University of Technology

Introduction to Software testing techniques2 Objectives 1.To introduce software testing and to explain its importance 2.To introduce structure of teaching content 3.To explain software testing definition

Introduction to Software testing techniques3 SWEBOK project Software Configuration Management Software Construction Software Design Software Engineering Infrastructure Software Engineering Management Software Engineering Process Software Evolution and Maintenance Software Quality Analysis Software Requirements Analysis Software Testing !!!!!!Software Testing

Curricula Module place on the curricula

Content 1.Introduction 2.Verification & Validation ( Repeated from software engineering according I Sommerville textbook) 3.Software testing ( Repeated from software engineering according I. Sommerville textbook) 4. Written quiz Topics of seminars: 5.Unit testing 6.Integration testing 7.Interface testing Introduction to Software testing techniques5

Content 8.Drivers and stubs 9.Sandwich strategy 10.Testing objectives 11. Regression testing 12. Equivalence partitioning 13.Testing based on specifications 14.Testing based on error guessing 15. Input Output analyses Introduction to Software testing techniques6

Content 16. Testing criteria based on control flow 17. Testing criteria based on data flow 18. Cycle testing 19. Mutation testing 20. Object-oriented software testing 21.Graphical user interface testing 22. Testing maturity model 23.Testing tools Introduction to Software testing techniques7

Content 24. Automated software testing 25. Software testability 26. Statistical testing 27. State testing of automata Introduction to Software testing techniques8

Labs 9

10 The “software crisis” By the 1980’s, “quality” in software became a goal; SEI was born “software engineering” became popular the life cycle was studied software developers and testers began to work together by the 1990’s, testing tools became available

Introduction to Software testing techniques11 First: a riddle about testing A mathematician, a physicist, and an engineer are told: “ All odd numbers are prime. ” –The mathematician says, “That’s silly, nine is a non- prime odd number. –The physicist says, “Let’s see, 3 is prime, 5 is prime, 7 is prime -- looks like it’s true.” –The engineer says, “let’s see, 3 is prime, 5 is prime, 7 is prime, 9 is prime, 11 is prime -- looks like it’s true.”

Introduction to Software testing techniques12 Testing: our experience Software to be tested Test Case Output

Introduction to Software testing techniques13 When to stop? Software to be tested Test Case Output Enough? No Verification Yes Test Coverage Test Case Generation

Introduction to Software testing techniques14 Testing example Sorting Program Sorting Program SPECS: Takes a list of numbers; returns a sorted list. {1,3,2} {1,2,3} {3,2,3} {} {-1, -2} Just a list. A sorted list. Repeated entry. Empty list. Negative numbers. Test Cases {1, 2, 3} Output {1, 2, 3} Output {2, 3, 3} Output {} Output {-2, -1} Output

Introduction to Software testing techniques15 Manual testing Software to be tested Test Case Output Enough ? No Verification Test Coverage Test Case Generation Yes

Introduction to Software testing techniques16 Automated testing Software to be tested Test Case Output Coverage Evaluator Test Case Gener ator Verifie r OR Test Oracle Test Specs

Introduction to Software testing techniques17 Software testing? “The process of executing computer software in order to determine whether the results it produces are correct”, Glass “The process of executing a program with the intent of finding errors”, Myers “Program testing can be used to show the presence of bugs, but never their absence”, Dijkstra

Introduction to Software testing techniques18 Software testing? “The aim is not to discover errors but to provide convincing evidence that there are none, or to show that particular classes of faults are not present”, Hennell “Testing is the measure of software quality”, Hetzel The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.” IEEE/ANSI,

Introduction to Software testing techniques19 The point of view “If our goal is to show the absence of errors, we will find very few of them” “If our goal is to show the presence of errors, we will discover a large number of them”

Introduction to Software testing techniques20 Testing efforts 50%Brooks/Myers, 80%Arthur Andersons’ Director of testing in North America

Introduction to Software testing techniques21 Testers/programmers 1:5-10 Mainframes i.e.,1 tester for every 5 to 10 developers 2:3 Microsoft 2:1 Lotus (for Windows) 1:2 Average of 4 major companies, Microsoft, Borland, WordPerfect, Novell

Introduction to Software testing techniques22 Software testing problems poorly expressed requirements informal design techniques nothing executable until coding stage Huge input set: consider testing software that categorises an exam grade: 101 inputs

Introduction to Software testing techniques23 Exhaustive software testing Exhaustive software testing is intractable Even if all possible inputs could be identified, the problem of identifying expected results is undecidable Weyuker has shown that there is no algorithm that can determine if a given statement, branch or path will be exercised!

Introduction to Software testing techniques24 Software testing definition Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the specified expected behavior.

Introduction to Software testing techniques25 Dynamic This term means testing always implies executing the program on valued inputs. Static analysis techniques, such as peer review and inspection are not considered; nor is program execution on symbolic inputs, or symbolic evaluation;

Introduction to Software testing techniques26 Finite Clearly the tests are not selected literally from an infinite set of possible tests, but a very large finite one (the set of all bit strings shorter than some length). Nonetheless, for even simple programs, so many test cases are theoretically possible that exhaustive testing could require even years to execute. This is why in practice the number of tests can generally be considered infinite. However, the number of executions which can realistically be observed in testing must obviously be finite, and -more than this- it must be manageable. Indeed, testing always implies a trade-off between limited resources and schedules, and inherently unlimited test requirements: this conflict points to well known problems of testing, both technical in nature (criteria for deciding test adequacy) and managerial in nature (estimating the effort to put in testing);

Introduction to Software testing techniques27 Selected The many proposed test techniques essentially differ in how they select the (finite) test set, and testers must be aware that different selection criteria may yield largely different effectiveness. The problem of identifying the most suitable selection criterion under given conditions is still under research.

Introduction to Software testing techniques28 Expected It must be possible to decide whether the observed outcomes of program execution are acceptable or not, otherwise the testing effort would be useless. The observed behavior may be checked against user's expectations (commonly referred to as testing for validation) or against a functional specification (testing for verification). The test pass/fail decision is referred to as the oracle problem, which can be addressed with different approaches.

Introduction to Software testing techniques29 Static techniques Software testing is a means for quality evaluation. In the SQA chapter of the SWEBOK, activities and techniques for quality analysis are categorized into: static techniques (no code execution), and dynamic techniques (code execution). Both categories are useful. We would like to emphasize that static techniques are equally important for the purpose of building quality in a software product. Static techniques are covered into the SQA KA description

Introduction to Software testing techniques30 Testing activity In the years, the view of software testing has evolved towards a more constructive attitude. Testing is no longer seen as an activity that starts only after the coding phase is complete, with the limited purpose of finding mistakes. Software testing is nowadays seen as an activity that encompasses the whole development process, and is an important part itself of the actual construction. Indeed, planning for testing starts since the early stages of requirement analysis, and test plans and procedures are systematically and continuously refined as the development proceeds. These activities of planning and designing tests constitute themselves a useful input to designers for highlighting potential weaknesses.

Introduction to Software testing techniques31 Error prevention As more extensively discussed in the SQA chapter, the right attitude towards quality is one of prevention: it is better to avoid problems, rather than repairing them. Testing must be seen as a means primarily for demonstrating that the prevention has been effective, but also for identifying anomalies in those cases in which, for some reason, it has been not. Finally, it is worth recognizing that even after a good testing, the system could still contain mistakes. The remedy to system failures after delivery is provided by (corrective) maintenance actions. The maintenance issues of software engineering are covered into the Software Evolution and Maintenance chapter of the SWEBOK

Introduction to Software testing techniques32 General Content A.Testing Basic Concepts and Definitions B.Test Levels C.Test Techniques D.Test-related Measures E.Managing the Test Process F.Test Tools.

Introduction to Software testing techniques33 A. Testing Basic Concepts and Definitions When the outcome of a test execution diverges from the expected, a failure is observed, whose cause is a fault. These and other basic definitions within software testing area, as well as an introduction to the terminology, will be provided in next two chapters

Introduction to Software testing techniques34 B. Testing levels Software testing is usually performed at different levels along the development process. That is to say, the object of the test can vary: a whole program, part of it (functionally or structurally related), a single module. The latter subject is expanded in Part B Test Levels. This consists of two (orthogonal) subsections: B.1 lists the phases in which the testing of large software systems is traditionally subdivided. In B.2 testing for specific conditions or properties is instead considered, and is referred to as "Objectives of testing". Clearly not all listed types of testing apply to every system, nor has every possible type been listed, but those most generally applied.

Introduction to Software testing techniques35 Test objective Testing is conducted in view of a specific purpose (test objective), which is stated more or less explicitly, and with varying degrees of precision. Stating the objective in precise, quantitative terms allows for establishing control over the test process.

Introduction to Software testing techniques36 Testing aims One of testing aims is to expose failures (as many as possible), and many popular test techniques have been developed for this objective. These techniques variously attempt to "break" the program, by running identified classes of (deemed equivalent) executions: the leading principle underlying such techniques is being as much systematic as possible in identifying a representative set of program behaviors (generally in the form of subclasses of the input domain). However, a comprehensive view of the KA of testing as a means for quality must include other as important objectives for testing, e.g., reliability measurement, usability evaluation, contractor’s acceptance, for which different approaches would be taken. Note that the test objective varies with the test object, i.e., in general different purposes are addressed at the different levels of testing.

Introduction to Software testing techniques37 Testing criteria The test objective determines how the test set is identified, both with regard to its consistency -how much testing is enough for achieving the stated objective?- and its composition -which test cases should be selected for achieving the stated objective?-, although usually the "for achieving the stated objective" part is left implicit and only the first part of the two italicized questions above is posed. Criteria for addressing the first question are referred to as test adequacy criteria, while for the second as test selection criteria

Introduction to Software testing techniques38 C. Test Techniques Several Test Techniques have been developed in the last two decades according to various criteria, and new ones are still proposed. "Generally accepted" techniques will be described from chapter three.

Introduction to Software testing techniques39 D. Testing measure Sometimes, it can happen that confusion is made between test objectives and techniques. For instance, branch coverage is a popular test technique. Achieving a specified branch coverage measure should not be considered per se as the objective of testing: it is a means to improve the chances of finding failures (by systematically exercising every program branch out of a decision point). To avoid such misunderstandings, a clear distinction should be made between test measures which evaluate the test set, like measures of coverage, and those which instead provide an evaluation of the program under test, based on the observed test outputs, like reliability

Introduction to Software testing techniques40 E. Managing the Test Process Testing concepts, strategies, techniques and metrics need to be integrated into a defined and controlled process, which is run by people. The test process should support testing activities and testing teams, from test planning to test outputs evaluation, in such a way as to provide justified assurance that the test objectives are met cost- effectively. Issues relative to Managing the Test Process.

Introduction to Software testing techniques41 F. Test Tools Software testing is a very expensive and labor- intensive part of development. For this reason, tools are instrumental to support test activities. Moreover, in order to enhance cost- effectiveness ratio, a key issue has always been pushing test automation as much as possible. Existing tools and concepts related to automating the constituent parts of test process are addressed in different chapters

RECOMMENDED REFERENCES FOR SOFTWARE TESTING K. Beck, Test-Driven Development by Example, Addison-Wesley, P. C. Jorgensen, Software Testing: A Craftsman's Approach, second edition, CRC Press, 2004 C. Kaner, J. Bach, and B. Pettichord, Lessons Learned in Software Testing, Wiley Computer Publishing, S. L. Pfleeger, Software Engineering: Theory and Practice, second ed., Prentice Hall, 2001