Software Testing and Quality Assurance Practical Considerations (1) 1.

Slides:



Advertisements
Similar presentations
Analyzing Regression Test Selection Techniques
Advertisements

Introduction to Software Testing Chapter 1 Model-Driven Test Design Paul Ammann & Jeff Offutt
Testing and Quality Assurance
Copyright A. Andrews, 2003 Regression Testing Department of Computer Science.
Testing Concurrent/Distributed Systems Review of Final CEN 5076 Class 14 – 12/05.
Software Quality Assurance Plan
Dynamic Service Composition with QoS Assurance Feb , 2009 Jing Dong UTD Farokh Bastani UTD I-Ling Yen UTD.
CSE 331 SOFTWARE DESIGN & IMPLEMENTATION REGRESSION TESTING Autumn 2011.
Software Evolution Managing the processes of software system change
COMP8130 and 4130Adrian Marshall 8130 and 4130 Test Execution and Reporting Adrian Marshall.
Applied Software Project Management Andrew Stellman & Jennifer Greene Applied Software Project Management Applied Software.
(c) 2007 Mauro Pezzè & Michal Young Ch 24, slide 1 Documenting Analysis and Test.
Software Testing. “Software and Cathedrals are much the same: First we build them, then we pray!!!” -Sam Redwine, Jr.
Paul Ammann & Jeff Offutt
High Level: Generic Test Process (from chapter 6 of your text and earlier lesson) Test Planning & Preparation Test Execution Goals met? Analysis & Follow-up.
Software Reengineering 2003 년 12 월 2 일 최창익, 고광 원.
Software Re-engineering
SEG Software Maintenance1 Software Maintenance “The modification of a software product after delivery to correct faults, to improve performance or.
Dr. Pedro Mejia Alvarez Software Testing Slide 1 Software Testing: Building Test Cases.
Introduction to Software Testing Chapter 5.2 Program-based Grammars Paul Ammann & Jeff Offutt
Fruitful functions. Return values The built-in functions we have used, such as abs, pow, int, max, and range, have produced results. Calling each of these.
Applied Software Project Management Andrew Stellman & Jennifer Greene Applied Software Project Management greene.com 1 Applied Software.
Introduction to Software Testing Chapter 9.3 Challenges in Testing Software Test Criteria and the Future of Testing Paul Ammann & Jeff Offutt
Objectives Understand the basic concepts and definitions relating to testing, like error, fault, failure, test case, test suite, test harness. Explore.
By Touseef Tahir Software Testing Basics. Today's Agenda Software Quality assurance Software Testing Software Test cases Software Test Plans Software.
Introduction to Software Testing Chapter 8.1 Building Testing Tools –Instrumentation Paul Ammann & Jeff Offutt
Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.
Historical Aspects Origin of software engineering –NATO study group coined the term in 1967 Software crisis –Low quality, schedule delay, and cost overrun.
1 Introduction to Software Testing. Reading Assignment P. Ammann and J. Offutt “Introduction to Software Testing” ◦ Chapter 1 2.
Testing as a Driver for Development Change Wall Street Systems Graham Thomas.
Software Testing and Quality Assurance Practical Considerations (4) 1.
Introduction to Software Testing Paul Ammann & Jeff Offutt Updated 24-August 2010.
Software Maintenance Speaker: Jerry Gao Ph.D. San Jose State University URL: Sept., 2001.
Chapter 1: Fundamental of Testing Systems Testing & Evaluation (MNN1063)
Introduction to Software Testing Chapter 9.2 Program-based Grammars Paul Ammann & Jeff Offutt
Introduction to Software Testing (2nd edition) Chapter 4 Putting Testing First Paul Ammann & Jeff Offutt August.
Introduction to Software Testing EDITION 1 Chapter 6 Practical Considerations (partial) Paul Ammann & Jeff Offutt
Mutation Testing Breaking the application to test it.
T EST T OOLS U NIT VI This unit contains the overview of the test tools. Also prerequisites for applying these tools, tools selection and implementation.
Testing JUnit Testing. Testing Testing can mean many different things It certainly includes running a completed program with various inputs It also includes.
Foundations of Software Testing Chapter 5: Test Selection, Minimization, and Prioritization for Regression Testing Last update: September 3, 2007 These.
Introduction to Software Testing Model-Driven Test Design and Coverage testing Paul Ammann & Jeff Offutt Update.
REGRESSION TESTING Software Quality Engineering NC Zunaira Tariq Bese 19B Software Quality Engineering NC Zunaira Tariq Bese 19B.
Mutation Testing Laraib Zahid & Mariam Arshad. What is Mutation Testing?  Fault-based Testing: directed towards “typical” faults that could occur in.
Software Testing and Quality Assurance Syntax-Based Testing (2) 1.
Introduction to Software Testing (2nd edition) Chapter 5 Criteria-Based Test Design Paul Ammann & Jeff Offutt
Applied Software Project Management SOFTWARE TESTING Applied Software Project Management 1.
Introduction to Software Testing Chapter 6 Practical Considerations
Applied Software Testing
Paul Ammann & Jeff Offutt
Regression Testing with its types
Generating Automated Tests from Behavior Models
Introduction to Software Testing Chapter 9.2 Program-based Grammars
Paul Ammann & Jeff Offutt
Verification and Testing
Coverage-Based Test Design CS 4501 / 6501 Software Testing
Aditya P. Mathur Purdue University
Applied Software Implementation & Testing
Introduction to Software Testing Chapter 2 Model-Driven Test Design
It is great that we automate our tests, but why are they so bad?
Improving Test Suites for Efficient Fault Localization
Introduction to Software Testing Chapter 5.2 Program-based Grammars
Test Automation CS 4501 / 6501 Software Testing
Test Case Test case Describes an input Description and an expected output Description. Test case ID Section 1: Before execution Section 2: After execution.
Department of Computer Science Regression Testing.
Managing the Test Process CS 4501 / 6501 Software Testing
Regression Testing.
Test Cases, Test Suites and Test Case management systems
Paul Ammann & Jeff Offutt
Testing.
Presentation transcript:

Software Testing and Quality Assurance Practical Considerations (1) 1

Reading Assignment P. Ammann and J. Offutt “Introduction to Software Testing” ◦ Chapter 6  Section 6.1 2

3 Outline Regression Testing ◦ Automated tool support Managing Tests in a Regression Test Suite ◦ Choosing which regression tests to run

4 Regression Testing Most software today has very little new development ◦ Correcting, perfecting, adapting, or preventing problems with existing software ◦ Composing new programs from existing components ◦ Applying existing software to new situations Definition The process of re-testing software that has been modified

5 Regression Testing Because of the deep interconnections among software components, ◦ changes in one method can cause problems in methods that seem to be unrelated Not surprisingly, most of our testing effort is regression testing Large regression test suites accumulate as programs (and software components) age

6 Automation and Tool Support Too many tests to be run by hand Tests must be run and evaluated quickly ◦ often overnight, or more frequently for web applications Testers do not have time to view the results by inspection Regression tests must be automated

7 Automation and Tool Support Types of tools : ◦ Capture / Replay – Capture values entered into a GUI and replay those values on new versions ◦ Version control – Keeps track of collections of tests, expected results, where the tests came from, the criterion used, and their past effectiveness ◦ Scripting software – Manages the process of obtaining test inputs, executing the software, obtaining the outputs, comparing the results, and generating test reports

8 Managing Tests in a Regression Suite Test suites accumulate new tests over time Test suites are usually run in a fixed, short, period of time ◦ Often overnight, sometimes more frequently, sometimes less At some point, the number of tests can become unmanageable ◦ We cannot finish running the tests in the time allotted

9 Managing Tests in a Regression Suite We can always add more computer hardware But is it worth it? How many of these tests really need to be run ?

10 Policies for Updating Test Suites Which tests to keep can be based on several policies ◦ Add a new test for every problem report ◦ Ensure that a coverage criterion is always satisfied

11 Policies for Updating Test Suites Sometimes harder to choose tests to remove ◦ Remove tests that do not contribute to satisfying coverage ◦ Remove tests that have never found a fault (risky !) ◦ Remove tests that have found the same fault as other tests (also risky !)

12 Policies for Updating Test Suites Reordering strategies ◦ If a suite of N tests satisfies a coverage criterion, the tests can often be reordered so that the first N-x tests satisfies the criterion – so the remaining tests can be removed

13 When a Regression Test Fails Regression tests are evaluated based on whether the result on the new program P is equivalent to the result on the previous version P-1 ◦ If they differ, the test is considered to have failed

14 When a Regression Test Fails Regression test failures represent three possibilities : ◦ The software has a fault – Must fix the fix ◦ The test values are no longer valid on the new version – Must delete or modify the test ◦ The expected output is no longer valid – Must update the test Sometimes hard to decide which !!

15 Evolving Tests Over Time Changes to external interfaces can sometimes cause all tests to fail ◦ Modern capture / replay tools will not be fooled by trivial changes like color, format, and placement ◦ Automated scripts can be changed automatically via global changes in an editor or by another script Adding one test does not cost much – but over time the cost of these small additions start to pile up

16 Choosing Which Regression Tests to Run When a small change is made in the software, what portions of the software can be impacted by that change ? Change Impact Analysis How does a change impact the rest of the software ?

17 Choosing Which Regression Tests to Run More directly, which tests need to be re- run ? ◦ Conservative approach : Run all tests ◦ Cheap approach : Run only tests whose test requirements relate to the statements that were changed ◦ Realistic approach : Consider how the changes propagate through the software

18 Choosing Which Regression Tests to Run Clearly, tests that never reach the modified statements do not need to be run Lots of clever algorithms to perform CIA have been invented ◦ Few if any available in commercial tools

19 Rationales for Selecting Tests to Re-Run Inclusive : A selection technique is inclusive if it includes tests that are “modification revealing” ◦ Unsafe techniques have less than 100% inclusiveness Precise : A selection technique is precise if it omits regression tests that are not modification revealing

20 Rationales for Selecting Tests to Re-Run Efficient : A selection technique is efficient if deciding what tests to omit is cheaper than running the omitted tests ◦ This can depend on how much automation is available General : A selection technique is general if it applies to most practical situations

21 Key Points We spend far more time on regression testing than on testing new software If tests are based on covering criteria, all problems are much simpler ◦ We know why each test was created ◦ We can make rationale decisions about whether to run each test ◦ We know when to delete the test ◦ We know when to modify the test Automating regression testing will save much more than it will cost