Chapter 1: Fundamental of Testing Systems Testing & Evaluation (MNN1063)

Fundamental of Testing 0Why is testing necessary? 0What is testing? 0General testing principles 0Fundamental test process 0The psychology of testing 0Code of ethics

Why is testing necessary? 0 Human make mistakes 0 Mistakes – unimportant, expensive or dangerous 0 Must check – some things can always go wrong 0 Mistakes may comes from: 0 Bad assumption 0 Blind spot

Software Systems Context 0 Software that did not work as expected? 0 Error on a bill, delay on credit card processing, website that did not load correctly etc. 0 Different software carry different level of risk 0 Problems – trivial, costly and damaging (loss of money, time or business reputation, or results in injury or death RISK: A factor that could result in future negative consequences; usually expressed as impact and likelihood

Causes of Software Defects 0 People who design or build the software can make mistakes 0 Mistakes means flaws in the software => defects/bugs/faults 0 Defects may caused system to fail to do what it should do (or do something it shouldn’t) => causing failure Error (Mistakes) Error (Mistakes) Defects (Bugs/Faults) Defects (Bugs/Faults) Failure

Do our mistakes matter? 0 Think about the consequences of mistakes… 0 Human being, programmer, tester => can make error 0 Error => defects => defect code executed => systems failure 0 Systems and project are complicated

Defects & Failures 0 Errors in specifications, design and implementation 0 Errors in use of the systems 0 Environmental conditions 0 Intentional damage 0 Potential consequences of earlier errors, intentional damage, defects and failures

When do defects arise? Requirement 1 Correct requirement Designed to meet requirement Build to meet design Product works as expected Requirement 2 Correct requirement Designed to meet requirement Mistakes made in build Mistakes made in build Product has bugs in it Product has bugs in it

When do defects arise? Requirement 3 Correct requirement Mistake made in design Built to meet design Product has design flaws Requirement 4 Mistakes made in requirement Design to meet requirement Built to meet design Wrong product delivered

What is cost of defects? 0 The cost of finding and fixing the defects rises considerably across the life cycle Cost Time Cost of finding and fixing defects increases over time RequirementsDesignBuildTestLive use

Role of Testing in Software Development, Maintenance & Operations 0 We have seen human errors can caused defect or fault at any stage of software development life-cycle 0 Rigorous testing is necessary during the maintenance & development to identify defects => to reduce failures in operational environment 0 To increase quality of operational systems 0 Executing tests helps us move towards improved quality of product and services

Testing & Quality 0 Testing helps us to measure quality of software in terms of: 0 The number of defects found 0 The number of tests run 0 The system covered by the test 0 Both functional and non-functional software requirements 0 Testing gives us confidence in quality 0 Poor test may uncover few defects

What is Quality? 0 Project aim to deliver software that meet customer needs & correct specification 0 Validation – “ is this the right specification” 0 Verification – “is the system correct to specification” 0 Need to understand what customer understand by quality and what their expectation are

What is root cause analysis? 0 Detect failures => track them back to their root cause 0 Several ways & involving a group brainstorming ideas and discussion 0 Assignment #1: 0 Find 3 ways to do Root Cause Analysis. 0 Discuss how it works.

How much testing is enough? 0 We have a choice: 0 Test everything? 0 Test nothing? 0 Test some part? 0 Example: One-digit field – how many test is enough? 0 Real example: one screen with 15 input fields 0 Pressures on project: time & budget 0 Need test approach which provides the right amount of testing 0 Risk assessment – helps to decide how much testing to do

What is testing? 0 Recall the common objectives of testing 0 What is testing? – checking if the software is OK? 0 Lets discuss an analogy of: driving test 0 The format of the driving test and conduct of examiner: 0 The test is planned & prepared 0 The test has known goals 0 The test is therefore carried out to show that the driver satisfied the requirement for driving & demonstrate that they are fit to drive

Defining software testing 0 ISTQB definition of software testing 0 Process – Testing is a process rather a single activity – there are series of activities 0 All life cycle activities – software development life cycle (discuss this in Chapter 2) 0 Requirements 0 Design (to prevent defects introduced into code) – test via the test design

Defining software testing 0 Both static and dynamic 0 Dynamic testing – execute codes & demonstrate the results 0 Static – reviewing documents including the source code 0 Planning – to manage the testing: plan, control, report testing progress, close testing 0 Preparation – selecting test conditions & design test cases 0 Evaluation – check results and evaluate software under test & completion criteria 0 Software products and related work product – not only test code, requirements & design specification

Defining software testing 0 Objectives for testing 0 Determine the software product satisfy specified requirements 0 Demonstrate that the software products are fit for purpose 0 Detect defects 0 Objectives may be different in each stage

General testing principles 0 Principle 1: Testing shows the presence of defects 0 Principle 2: Exhaustive testing is impossible 0 Principle 3: Early testing 0 Principle 4: Defect clustering 0 Principle 5: Pesticide paradox 0 Principle 6: Testing is context dependent 0 Principle 7: Absence-of-errors fallacy

Fundamental test process 0 Fundamental test process: 0 Planning and control 0 Analysis and design 0 Implementation and execution 0 Evaluation exit criteria and reporting 0 Test closure activities

Test Planning and control 0 We make sure we understand the goal and objectives of the customer, stakeholders, the project and the risks 0 Based on the understanding, we set the goal & objectives for the testing itself 0 To help us, we may have: 0 Test policies – rules for testing 0 Test strategy – Overall high-level approach

Test planning and control 0 Test planning has the following major task: 0 Determine the scope and risks; Identify the objectives of testing 0 Determine the test approach (techniques, test items, coverage, identifying and interfacing the teams involve in testing, testware) 0 Implement the test policy and/or the test strategy 0 Determine the required test resources (people, test environment, PCs) 0 Schedule test analysis and design tasks, test implementation, execution and evaluation 0 Determine the exit criteria

Test planning and control 0 Test control is an on-going activity 0 We compare the actual progress with planned progress 0 Test control has the following major tasks: 0 Measure & analyze the results of reviews and testing 0 Monitor and document progress 0 Provide information on testing 0 Initiate corrective actions 0 Make decisions

Test Analysis and Design 0 Test analysis and design has the following major tasks: 0 Review the test basis (product risk analysis, requirements, architecture, design specification and interface) 0 Identify test conditions (behavior and structure) 0 Design the test (Chapter 4) 0 Evaluate testability of the requirements and systems 0 Design the test environment set-up and identify any required infrastructure and tools

Test implementation and Execution 0 Implementation: 0 We take the test conditions and make them into test cases and set up the test environment 0 Having the high-level design – we now build them 0 We set up the environment 0 Build our test data 0 Execution: 0 Execute the test cases 0 Log the outcome of test execution 0 Compare actual result with expected result

Evaluating exit criteria and reporting 0 Is an activity where test execution is assessed against the defined objectives 0 Evaluating exit criteria involve following major tasks: 0 Check the test logs against the exit criteria 0 Assess if more test needed 0 Write a test summary report