Advanced Software Engineering: Software Testing COMP 3702 (Lecture1) Sada Narayanappa Seif Azgandhi Anneliese Andrews Thomas Thelin Carina Andersson

A Andrews - Software Engineering: Software Testing'06 27 dm 3, 31 kg 4 KB read/write magnetic core memory 64 KB read-only memory

A Andrews - Software Engineering: Software Testing'06

Software on Wings 4 M LOC

A Andrews - Software Engineering: Software Testing'06 Facts about testing System development:  1/3 planning  1/6 coding  1/4 component test  1/4 system test [Brooks75]

A Andrews - Software Engineering: Software Testing'06 Implementation Analysis Design Function test System Test

A Andrews - Software Engineering: Software Testing'06 Good enough quality To claim that any given thing is good enough is to agree with all of the following propositions:  It has sufficient benefits  It has no critical problems  The benefits sufficiently outweigh the problems  In the present situation, and all things considered, further improvement would be more harmful than helpful James Bach, IEEE Computer, 30(8):96-98, 1997.

A Andrews - Software Engineering: Software Testing'06 Quality attributes – ISO 9126

A Andrews - Software Engineering: Software Testing'06 Quality attributes – McCall

A Andrews - Software Engineering: Software Testing'06 Why use testing?  Risk mitigation  Faults are found early  Faults can be prevented  Reduce lead-time  Deliverables can be reused  …

A Andrews - Software Engineering: Software Testing'06 Why do faults occur in software?  Software is written by humans  Who know something, but not everything  Who have skills, but aren’t perfect  Who don’t usually use rigorous methods  Who do make mistakes (errors)  Under increasing pressure to deliver to strict deadlines  No time to check, assumptions may be wrong  Systems may be incomplete  Software is complex, abstract and invisible  Hard to understand  Hard to see if it is complete or working correctly  No one person can fully understand large systems  Numerous external interfaces and dependencies

A Andrews - Software Engineering: Software Testing'06 Origins of defects Defect sources Lack of education Poor communication Oversight Transcription Immature process Impact of software artifacts Errors Faults / Defects Failures Impact from user’s view Poor quality software User dissatisfaction Fault model

A Andrews - Software Engineering: Software Testing'06 Whoops, that’s my calculator

A Andrews - Software Engineering: Software Testing'06 Testing, Verification & Validation Definition 1  Verification  is the product right?  Validation  is it the right product? Definition 2  Verification  satisfies the conditions at the start of the phase  Validation  satisfies the requirements Testing The process of evaluating a program or a system

A Andrews - Software Engineering: Software Testing'06 Definitions  Failure is an event, fault is a state of the software caused by an error  Error – human mistake  Fault / Defect – anomaly in the software  Failure – inability to perform its required functions  Debugging / Fault localization – localizing, repairing, retesting.  A test case consists of:  A set of inputs  Execution conditions  Expected outputs

A Andrews - Software Engineering: Software Testing'06 Scripted and non-scripted testing  In scripted testing test cases are pre-documented in detailed, step-by-step descriptions  Different levels of scripting possible  Scripts can be manual or automated  Non-scripted testing is usually manual testing without detailed test case descriptions  Can be disciplined, planned, and well documented exploratory testing  or ad-hoc testing

A Andrews - Software Engineering: Software Testing'06 Test oracle  An oracle is the principle or mechanism by which you recognize a problem  Test oracle provides the expected result for a test, for example  Specification document  Formula  Computer program  Person  In many cases it is very hard to find an oracle  Even the customer and end user might not be able to tell which is the correct behaviour

A Andrews - Software Engineering: Software Testing'06 Principle 1 – purpose of testing Testing is the process of exercising a software component using a selected set of test cases, with the intent of  Revealing defects  Evaluating quality

A Andrews - Software Engineering: Software Testing'06 Principles 2: A good test case – When the test objective is to detect defects, then a good test case is one that has high probability of revealing a yet undetected defect(s) 3: Test result – The results should be inspected meticulously 4: Expected output – A test case must contain the expected output

A Andrews - Software Engineering: Software Testing'06 Principles 5: Input – Test cases should be developed for both valid and invalid input conditions 6: Fault content estimation – The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component 7: Test organization – Testing should be carried out by a group that is independent of the development group

A Andrews - Software Engineering: Software Testing'06 Principles 8: Repeatable – Tests must be repeatable and reusable 9: Planned – Testing should be planned 10: Life cycle – Testing activities should be integrated into the software life cycle 11: Creative – Testing is a creative and challenging task

A Andrews - Software Engineering: Software Testing'06  Test Maturity Model

A Andrews - Software Engineering: Software Testing'06 Goals of the course Knowledg e Skills Attitudes A test specialist who is trained as an engineer should have knowledge of test-related principles, processes, measurements, standards, plans, tools, and methods, and should learn how to apply them to the testing tasks to be performed.

A Andrews - Software Engineering: Software Testing'06

A Andrews - Software Engineering: Software Testing'06

A Andrews - Software Engineering: Software Testing'06 Lectures  Theory + discussions  Cover the basic parts of software testing  Introduction  Black-box testing, Reliability, Usability  Inspections, white-box testing  Lifecycle, documentation  Organization, tools  Metrics, TMM  Guest lecture, Technical Overview Technical / Manageri al Managerial EconomicEconomic

A Andrews - Software Engineering: Software Testing'06 Lab sessions Preparation, Execution, Report  Black-box testing  Usage-based testing and reliability  White-box testing  Inspection and estimation  Software process simulation

A Andrews - Software Engineering: Software Testing'06 Project: Option 1  Learn a specific area of software testing  Collect and summarize research information  Critical thinking beyond the written information  Present information in a structured way  Peer review

A Andrews - Software Engineering: Software Testing'06 Examination  Written exam based on the book and lab sessions  Lab sessions (approved)  Project (F / G / VG)

A Andrews - Software Engineering: Software Testing'06 Schedule  Read  Course program  Projects in Software Testing  Check homepage  Not decided  Extra Lab dates

A Andrews - Software Engineering: Software Testing'06 This week  Read course program  Lecture 2 tomorrow  Project  Read Projects in Software Testing  Exercise on Thursday  Decide subject (deadline Thursday 17/3)  Lab  Prepare lab 1  Read Burnstein 1-3  Prepare Burnstein 4,12

A Andrews - Software Engineering: Software Testing'06 Project: Option 1  Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc.  Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool.  Practical: Use an existing technique to test a system or design and implement a prototype for a system.

A Andrews - Software Engineering: Software Testing'06 Project: Option 1  Read Projects in Software Testing  Divide in groups (2-3 persons)  Discuss with me 

A Andrews - Software Engineering: Software Testing'06 Project: Option 2  Experiment participation.  Compare Fault finding  Unit test  code inspection  Two sessions 3-4 less total  Full credit