18 October 2013. Why do we care?  Therac-25 (1985) 6 massive radiation overdoses  Multiple space fiascos (1990s) Ariane V exploded after 40 seconds.

Slides:



Advertisements
Similar presentations
Test process essentials Riitta Viitamäki,
Advertisements

Testing and Quality Assurance
Software Quality Assurance Plan
LIFE CYCLE MODELS FORMAL TRANSFORMATION
Software Failure: Reasons Incorrect, missing, impossible requirements * Requirement validation. Incorrect specification * Specification verification. Faulty.
(ADAPTED FROM DIANE POZEFSKY) Software and System Testing.
15 November Essay 1  Methodologies Points on the spectrum All can adapt to changes Required vs. permitted  Releases vs. iterations  Spool’s.
Testing: Who 3, What 4, Why 1, When 2, How 5 Lian Yu, Peking U. Michal Young, U. Oregon.
SE 450 Software Processes & Product Metrics Reliability: An Introduction.
Validating and Improving Test-Case Effectiveness Author: Yuri Chernak Presenter: Lam, Man Tat.
Software Testing. “Software and Cathedrals are much the same: First we build them, then we pray!!!” -Sam Redwine, Jr.
CSC 395 – Software Engineering Lecture 9: Testing -or- How I Stopped Worrying and Learned to Love the Bug.
EE694v-Verification-Lect5-1- Lecture 5 - Verification Tools Automation improves the efficiency and reliability of the verification process Some tools,
Systems Analysis and Design in a Changing World, 6th Edition
School of Computer ScienceG53FSP Formal Specification1 Dr. Rong Qu Introduction to Formal Specification
March 16, Calendar Next week: Thursday meeting instead of Tuesday (May 25) Web update later today.
ECI 2007: Specification and Verification of Object- Oriented Programs Lecture 0.
Formal verification Marco A. Peña Universitat Politècnica de Catalunya.
Introduction to Software Testing
Testing Test Plans and Regression Testing. Programs need testing! Writing a program involves more than knowing the syntax and semantics of a language.
Software Development Unit 6.
CSCI 5801: Software Engineering
Formal Methods 1. Software Engineering and Formal Methods  Every software engineering methodology is based on a recommended development process  proceeding.
CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie.
©Ian Sommerville 2000Software Engineering, 6th edition. Chapter 19Slide 1 Verification and Validation l Assuring that a software system meets a user's.
Software Quality Assurance Lecture #8 By: Faraz Ahmed.
Software Testing. Definition To test a program is to try to make it fail.
1 Software Testing (Part-II) Lecture Software Testing Software Testing is the process of finding the bugs in a software. It helps in Verifying and.
University of Palestine software engineering department Testing of Software Systems Fundamentals of testing instructor: Tasneem Darwish.
Pragmatic Projects Prepared by Doug Glidden. Pragmatic Projects Pragmatic Teams Ubiquitous Automation Ruthless Testing It’s All Writing Great Expectations.
CompSci 230 Software Design and Construction
Towers of Hanoi. Introduction This problem is discussed in many maths texts, And in computer science an AI as an illustration of recursion and problem.
Software Testing.
Software testing basic. Main contents  Why is testing necessary?  What is testing?  Test Design techniques  Test level  Test type  How to write.
Software Testing Testing principles. Testing Testing involves operation of a system or application under controlled conditions & evaluating the results.
 CS 5380 Software Engineering Chapter 8 Testing.
Software testing Main issues: There are a great many testing techniques Often, only the final code is tested.
Dr. Tom WayCSC Testing and Test-Driven Development CSC 4700 Software Engineering Based on Sommerville slides.
By Ian Jackman Davit Stepanyan.  User executed untested code.  The order in which statements were meant to be executed are different than the order.
Introduction to Software Testing. Types of Software Testing Unit Testing Strategies – Equivalence Class Testing – Boundary Value Testing – Output Testing.
16 October Reminder Types of Testing: Purpose  Functional testing  Usability testing  Conformance testing  Performance testing  Acceptance.
Well-behaved objects Main concepts to be covered Testing Debugging Test automation Writing for maintainability Objects First with Java - A Practical.
NIST (2002) cost to US $59B annual Why do we care?  Therac-25 (1985) 6 massive radiation overdoses  Multiple space fiascos (1990s) Ariane V exploded.
Software Quality and Testing CPS 109: Program Design and Construction October 28, 2003.
Requirements Specification. Welcome to Software Engineering: “Requirements Specification” “Requirements Specification”  Verb?  Noun?  “Specification”
What is Testing? Testing is the process of finding errors in the system implementation. –The intent of testing is to find problems with the system.
Software Defects.
Formal Methods.
Verification of FT System Using Simulation Petr Grillinger.
Scientific Debugging. Errors in Software Errors are unexpected behaviors or outputs in programs As long as software is developed by humans, it will contain.
Software Engineering. Acknowledgement Charles Moen Sharon White Bun Yue.
Software Engineering1  Verification: The software should conform to its specification  Validation: The software should do what the user really requires.
ISBN Prentice-Hall, 2006 Chapter 8 Testing the Programs Copyright 2006 Pearson/Prentice Hall. All rights reserved.
Software Quality Assurance and Testing Fazal Rehman Shamil.
29 March Software Quality and Testing. Why do we care? Therac-25 (1985) Multiple space fiascos (1990s) Ariane V exploded after 40 seconds (conversion)
1 Object-Oriented Analysis and Design with the Unified Process Figure 13-1 Implementation discipline activities.
Static Techniques for V&V. Hierarchy of V&V techniques Static Analysis V&V Dynamic Techniques Model Checking Simulation Symbolic Execution Testing Informal.
1 The Software Development Process ► Systems analysis ► Systems design ► Implementation ► Testing ► Documentation ► Evaluation ► Maintenance.
CS 160 and CMPE/SE 131 Software Engineering April 12 Class Meeting Department of Computer Science Department of Computer Engineering San José State University.
Building Valid, Credible & Appropriately Detailed Simulation Models
Introduction to Software Testing Maili Markvardt.
Week#3 Software Quality Engineering.
CSCE 548 Secure Software Development Risk-Based Security Testing
Software Engineering (CSI 321)
Software Testing.
Testing and Test-Driven Development CSC 4700 Software Engineering
Project Management: Inspections and Reviews Formal Specifications
Department of Computer Science Abdul Wali Khan University Mardan
© Oxford University Press All rights reserved.
Overview Activities from additional UP disciplines are needed to bring a system into being Implementation Testing Deployment Configuration and change management.
Presentation transcript:

18 October 2013

Why do we care?  Therac-25 (1985) 6 massive radiation overdoses  Multiple space fiascos (1990s) Ariane V exploded after 40 seconds (conversion) Mars Pathfinder computer kept turning itself off (system timing) Patriot missile misquided (floating point accuracy)  Millenium bug (2000)  Microsoft attacks (ongoing)  Healthcare.gov  NIST: cost to US, $59 billion annual (2002)

Healthcare.gov  As large as Windows XP  Last minute change to require log in for any viewing  Failed test with 200 users Still failing with thousand Capacity was supposed to be twice that  Project manager didn’t know there were problems  Bypassed their own deploy rules for security  Ignored the subcontractor that said it was broken  Stray code that was never cleared  Caching techniques not used

Quality and testing  “Errors should be found and fixed as close to their place of origin as possible.” Fagan  “Trying to improve quality by increasing testing is like trying to lose weight by weighing yourself more often.” McConnell

Life Testing  Used regularly in hardware Addresses “normal use”  n specimens put to test  Test until r failures have been observed  Choose n and r to obtain the desired statistical errors  As r and n increase, statistical errors decrease  Expected time in test = mu 0 (r / n) Where mu 0 = mean failure time

Butler and Finelli  “The Infeasibility of Experimental Quantification of Life-Critical Software Reliability”  In order to establish that the probability of failure of software is less than in 10 hours, testing required with one computer (1990s technology) is greater than 1 million years

Testing Classification  Purpose  Scope  Access  Risk-based  Structured vs Free Form

How to identify what to test  New features  New technology  Overworked developers  Regression  Dependencies  Complexity  Bug history  Language specific bugs  Environment changes  Late changes  Slipped in “pet” features  Ambiguity  Changing requirements  Bad publicity  Liability  Learning curve  Criticality  Popularity

Working backwards  Here’s the case I’m worried about  How could I have gotten here? Different order of entry Skipping intialization Reverse state traversal

Best (and Worst) Testing Practices (Boris Beizer)  Unit testing to 100% coverage: necessary but not sufficient for new or changed  Integration testing: at every step; not once  System testing: AFTER unit and integration testing  Testing to requirements: test to end users AND internal users  Test execution automation: not all tests can be automated  Test design automation: implies building a model. Use only if you can manage the many tests  Stress testing: only need to do it at the start of testing. Runs itself out  Regression testing: needs to be automated and frequent  Reliability testing: not always applicable. statistics skills required  Performance testing: need to consider payoff  Independent test groups: not for unit and integration testing  Usability testing: only useful if done early  Beta testing: not instead of in-house testing

Usability Testing  Frequency with which the problem occurs Common or rare?  Impact of the problem if it occurs Easy or difficult to overcome?  Persistence of the problem One-time problem or repeated?

Wizard of Oz Testing  Inputs and outputs are as expected  How you get between the two is “anything that works”  Particularly useful when you have An internal interface UI choices to make Children’s Intuitive Gestures in Vision-Based Action Games CACM Jan 2005, vol. 48, no. 1, p. 47

Number of Usability Test Users Needed Usability problems found = N(1-(1-L) n ) N = total number of usability problems in the design L = proportion of usability problems discovered by a single user n = number of users L=31%

Using as an Estimator  Found 100 problems with 10 users  Assumption: each user finds 10% of problems  How many are left?  found = N(1-(1-L) n ) 100 = N(1-(1-.1) 10 ) N = 100/( )=154  54 left

Historical Data  Lots of variants based on statistical modeling  What data should be kept?  When are releases comparable?  Dangers with a good release Test forever Adversarial relation between developers and tester  Dangers with a bad release Stop too soon

Capture-recapture model  Estimate animal populations: How many deer in the forest? ○ Tag and recount ○ If all tagged, assume you’ve seen them all  Applied to software by Basin in 73  Number of errors = |e 1 | * |e 2 | / |e 1 ∩ e 2 | where e n = errors found by tester n 2 testers: 25, 27, 12 overlap: 56 total errors  What’s wrong with this model (aside from the fact the denominator can be 0)? Assumptions about independence of testers

Performance Test Tools  What do they do? Orchestrate test scripts Simulate heavy loads  What is available? JMeter (Apache project) Grinder (Java framework)

Other Test Tools  Tons of test tools  One starting point

Other Ways of Improving Quality  Reviews and inspections  Formal specification  Program verification and validation  Self-checking (paranoid) code  Deploy with capabilities to repair

Formal Methods and Specifications  Mathematically-based techniques for describing system properties  Used in inference systems Do not require executing the program Proving something about the specification not already stated Formal proofs Mechanizable Examples: theorem provers and proof checkers

Uses of Specifications  Requirements analysis rigor  System design Decomposition, interfaces  Verification Specific sections  Documentation  System analysis and evaluation Reference point, uncovering bugs

Examples  Abstract data types Algebras, theories, and programs ○ VDM (Praxis: UK Civil aviation display system CDIS) ○ Z (Oxford and IBM: CICS) ○ Larch (MIT)  Concurrent and distributed systems State or event sequences, transitions ○ Hoare’s CSP ○ Transition axioms ○ Lamport’s Temporal Logic  Programming languages!