Software Quality Assurance & Testing Mistake in coding is called error, Error found by tester is called defect, Defect accepted by development team then it is called bug, Product does not meet the requirements then it Is failure.”

Importance of Test  There are hundreds of stories about failures of computer systems that have been attributed to errors in software  There are many reasons why systems fail but the issue that stands out the most is the lack of adequate testing. Example Pepsi - $42 Billion Error Philippines Due to a software error, 800,000 bottle caps were produced with number 349 instead of one. It was equivalent to $42 billion in prize money instead of $40,0000

Testing Levels

Testing Goals Based on Test Process Maturity Bezier Testing Levels Level 0 There is no difference between testing and debugging Level 1 The purpose of testing is to show the correctness (software works). Level 2 The purpose of testing is to show that the software doesn’t work. Level 3 The purpose of testing is not to prove anything specific, but to reduce the risk of using the software. Level 4 Testing is a mental discipline that helps all IT professionals develop higher quality software.

Bezier Testing Levels  Level 0 is the view that testing is the same as debugging.  In Level 1 testing, the purpose is to show correctness  In Level 2 testing, the purpose is to show failures  Level 3 testing shows the presence, not absence or failure. If we use software, we expose some risk. The risk may be small and unimportant, or the risk may be great and the consequences catastrophic, but risk is always there. In level 3 testing, both testers and developers work together to reduce risk.

Bezier Testing Levels  Once the testers and developers are on the same “team,” an organization can progress to real Level 4 testing. Level 4 testing means that the purpose of testing is to improve the ability of the developers to produce high quality software.

Debugging and Testing  Debugging is done in the development phase by the developer  In debugging phase identified bugs are fixed  Testing is done in the tester phase by the tester.  In testing, locating and identifying of the bugs are included.

Software Bug  A software bug is an error, flaw, mistake, failure, or fault in a computer program or system  Bug is terminology of Tester  Softare bug produces an incorrect or unexpected result, or causes it to behave in unintended ways

Software Fault  A static (physical)defect*, imperfection, flaw in the software.  short between wires  break in transistor  infinite program loop  A fault means that there is a problem within the code of a program which causes it to behave incorrectly. * defect: Mismatch between the requirements

Software Error  Error is an incorrect internal state that is the demostration of some fault.  In other words; Error is a deviation from correctness or accuracy. Example  Suppose a line is physically shortened to 0. (there is a fault). As long as the value on line is supposed to be 0, there is no error.  Errors are usually associated with incorrect values in the system state.

Software Failure  A failure means that the program has not performed as expected, and the actual performance has not met the minimum standards for success.  External, incorrect behavior with respect to the requirements or other description of the expected behavior Example  Suppose a circuit controls a lamp (0 = turn off, 1 = turn on) and the output is physically shortened to 0 (there is a fault).  As long as the user wants the lamp off, there is no failure.

eBay Crash  eBay in 1999 was a very large Internet auction house –top 10 Internet business – Market value of $22 billion – 3.8 million users as of March 1999 – Access allowed 24 hours 7 days a week  In June 6, 1999 eBay system was unavailable for 22 hours with problems ongoing for several days  Stock drops by 6.5%, $billion lost revenues Problems blamed on Sun server software

Ariane 5 Rocket Crash (June 4, 1996)  Ariane 5 rocket exploided 37 seconds after lift-off on  Error was due to software bug  Conversion of a 64-bit floating point number to a 16- bit integer resulted in an overflow  Due to the overflow, the computer cleared its memory  Ariane 5 interpreted the memory dump as an instruction to its rocket nozzles  Testing of full system under actual conditions not done due to budget limits Estimated cost: 60 million $

Nine Causes of Software Errors 1. Faulty requirements definition 2. Client developer communication failures 3. Deliberate deviations from software requirements 4. Logical design errors 5. Coding errors 6. Noncompliance with documentation and coding instructions 7. Shortcomings of the testing process 8. User interface and procedure errors 9. Documentation errors

Example: Failure & Fault & Error Consider a medical doctor making a diagnosis for a patient.  The patient enters the doctor’s office with a list of failures (that is, symptoms).  The doctor then must discover the fault, or root cause of the symptom. To aid in the diagnosis, a doctor may order tests that look for anomalous internal conditions.  In our terminology, these anomalous internal conditions correspond to errors.

Cause-and-effect relationship  Faults can result in errors.  Errors can lead to system failures  Errors are the effect of faults.  Failures are the effect of errors  Bug in a program is a fault.  Possible incorrect values caused by this bug is an error.  Possible crush of the operating system is a failure

Cause & Effect Relationship

Origins of Faults  Specification Mistakes – incorrect algorithms, incorrectly specified requirements (timing, power, environmental)  Implementation Mistakes – poor design, software coding mistakes,  Component Defects – manufacturing imperfections, random device defects, components wear-outs  External Factors – operator mistakes, radiation, lightning,

Program Example class numZero { public static int numZero (int[] x) { // if x == null throw NullPointerException // else return the number of occurrences of 0 in x int count = 0; for (int i = 1; i < x.length; i++) if (x[i] == 0) count++; return count; } }

The Fault in the Example  The fault in this program is that it starts looking for zeroes at index 1 instead of index 0, as is necessary for arrays First Input: numZero ([2, 7, 0]) correctly evaluates to 1, Second Input: numZero ([0, 7, 2]) incorrectly evaluates to 0.  In both of these cases the fault is executed.  Both of these cases result in an error,  Only the second case results in failure.

Error States in the Example To understand the error states, we need to identify the state for the program.  The state for numZero consists of values for the variables x, count, i and the program counter (PC).  The state at the if statement on the first iteration of the loop is ( x = [2, 7, 0], count = 0, i = 1, PC = if)  This state is in error, because the value of i should be 0 on the first iteration.

Error States in the Example (for the first input) However  The value of count is correct  The error state does not affect the output,  The software does not fail. Finally:  A state is in error if it is not the expected state, even if all of the values in the state are acceptable.

Error States in the Example (for the second input) For the second input:  The state for numZero consists of values for the variables x, count, i and the program counter (PC).  The state is (x = [0, 7, 2], count = 0, i = 1, PC = if)  The error affects the variable count  The error is present in the return value of the method.  The failure results.

Distinguish Testing from Debugging  The definitions of fault and failure allow us to distinguish testing from debugging.  Big difference is that debugging is conducted by a programmer and the programmer fix the errors during debugging phase.  Tester never fixes the errors, but rather find them and return to programmer.

Testing versus Debugging  Testing activity is carried down by a team of testers, in order to find the defect in the software.  Test engineers run their tests on the piece of software and if they encounter any defect (i.e. actual results don't match expected results), they report it to the development team.  Testers also have to report at what point the defect occurred and what happened due the occurrence of that defect. All this information will be used by development team to DEBUG the defect.

Testing versus Debugging  Debugging is the activity which is carried out by the developer.  After getting the test report from the testing team about defect(s), the developer tries to find the cause of the defect.  He has to go through lines of code and find which part of code in causing that defect.  After finding out the bug, he tries to modify that portion of code and then he rechecks if the defect has been finally removed.  After fixing the bug, developers send the software back to testers.

What is Software Quality ? According to the IEEE Software Quality is:  The degree to which a system, component, or process meets specified requirements.  The degree to which a system, component, or process meets customer or user needs or expectations.

Importance of Software Quality  Software is a major component of computer systems (about 80% of the cost) – used for – communication (e.g. phone system, system) – health monitoring, – transportation (e.g. automobile, aeronautics), – economic exchanges (e.g. ecommerce), – entertainment, – etc.  Software defects are extremely costly in term of – money – reputation – loss of life

Software Quality Factors

 Correctness – accuracy, completeness of required output –datedness, availability of the information  Reliability – maximum failure rate  Efficiency – resources needed to perform software function  Integrity – software system security, access rights  Usability – ability to learn, perform required task

Software Quality Factors  Maintainability – effort to identify and fix software failures (modularity, documentation, etc)  Flexibility – degree of adaptability (to new customers, tasks, etc)  Testability – support for testing (e.g. log files, automatic diagnostics, etc)

Software Quality Factors  Portability – adaptation to other environments (hardware, software)  Reusability – use of software components for other projects  Interoperability – ability to interface with other components/systems

What is Software Quality Assurance?  According to the IEEE Software quality assurance is:  A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements.  A set of activities designed to evaluate the process by which the products are developed or manufactured. Contrast with: quality control.

Software Quality Assurance  Verification – are we building the product right ? – performed at the end of a phase to ensure that requirements established during previous phase have been met  Validation – are we building the right product ? – performed at the end of the development process to ensure compliance with product requirements

Three General Principles of Quality Assurance  Know what you are doing  Know what you should be doing  Know how to measure the difference

Verification & Validation  Verification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase.  Validation: The process of evaluating software at the end of software development to ensure compliance with intended usage.

Difference Between Verification &Validation  Verification is preventing mechanism to detect possible failures before the testing begin.  It involves reviews, meetings, evaluating documents, plans, code, inspections, specifications etc.  Validation occurs after verification and it's the actual testing to find defects against the functionality or the specifications

The Difference between Verification &Validation  Verification is usually a more technical activity that uses knowledge about the individual software artifacts, requirements, and specifications.  Validation usually depends on domain knowledge; that is, knowledge of the application for which the software is written.  For example, validation of software for an airplane requires knowledge from aerospace engineers and pilots.