1. Testing
Xiaojun Qi

2. Testing There are two basic types of testing “V & V”
Execution-based testing
Non-execution-based testing
“V & V”
Verification: Determine if the workflow was completed correctly
Validation: Determine if the product as a whole satisfies its requirements (specification)
Term “Testing” is used instead of “V & V”

3. Software Quality
The quality of software is the extent to which the product satisfies its specifications.
Defect is a generic term for
Fault: It happens when a human makes a mistake
Failure: It is the observed incorrect behavior of the software product as a consequence of a fault
Error: It is the amount by which a result is incorrect.
Every software professional is responsible for ensuring that his or her work is correct
Quality must be built in from the beginning

4. Quality Issues: Software Quality Assurance
The members of the SQA group must ensure the quality of the software process and thereby ensure the quality of the product.
Test at the end of each workflow
Test when the product is complete
Quality assurance must be applied to the process itself
Develop various standards to which the software must conform
Establish the monitoring procedures for assuring compliance with those standards.

5. Quality Issues: Managerial Independence
There must be managerial independence between the development group and the SQA group
Neither group should have power over the other
More senior management must decide whether to
Deliver the faulty product on time, or
Test further and deliver the product late
The decision must take into account the interests of the client and the development organization

6. Non-Execution-Based Testing
Non-execution-based testing means that testing software without running test cases.
Review software (Walkthrough or Inspection) is a common non-execution-based testing method. Its underlying principles are:
We should not review our own work
Group synergy: A document is painstakingly checked by a team of software professionals with a broad range of skills.

7. Non-Execution-Based Testing: Walkthroughs
A walkthrough team consists of four to six members
It includes representatives of
The team responsible the current workflow
The manager responsible for the current workflow
The team responsible for the next workflow
The SQA group
The clients

8. Managing Walkthroughs
The walkthrough is preceded by preparation
A list of items
Items not understood
Items that appear to be incorrect
The walkthrough team is chaired by the SQA representative since
The quality of the product is a direct reflection of the professional competence of the SQA group.

9. Managing Walkthroughs (Cont.)
In a walkthrough we detect faults for later correction, not correct them
A correction produced by a committee is likely to be of low quality
The cost of a committee correction is too high
Not all items flagged as faults are actually incorrect
A walkthrough should not last longer than 2 hours so there is no time to correct faults

10. Managing Walkthroughs (contd)
Since verbalization leads to fault finding, there are two ways of conducting a walkthrough:
Participant-driven
Document-driven
Document-driven is likely to be more thorough since:
The majority of faults are spontaneously detected by the presenter
A walkthrough should never be used for performance appraisal

11. Non-Execution-Based Testing: Inspections
An inspection has five formal steps
An overview of the document to be inspected
The preparation: List the statistics of fault types ranked by frequency
The inspection: Find and document the faults
The rework: The individual responsible for the document resolves all faults and problems noted in the written report.
The follow-up: Every issue has been resolved and all fixes must be checked to ensure that no new faults have been introduced.

12. Inspections (Cont.) An inspection team has four members
Moderator: Both manager and leader
A member of the team performing the current workflow
A member of the team performing the next workflow
A member of the SQA group
IEEE standard also recommends:
Reader: Leads the team through the design
Recorder: Produce a written report of the detected faults.

13. An Essential Component of an Inspection: Fault Statistics
Faults are recorded by severity
Major faults: They cause premature termination or damage a database
Minor faults
Faults are recorded by fault type
Examples of design faults:
Not all specification items have been addressed
Interface faults: Actual and formal arguments do not correspond
Logic faults:

14. Use of Fault Statistics
We compare current fault rates with those of previous products for a given workflow.
We take action if there are a disproportionate number of faults of a particular type in an artifact
Check other code artifacts and take corrective action
Redesigning from scratch is a good alternative
We carry forward fault statistics to the next workflow
We may not detect all faults of a particular type in the current inspection

15. Successful Stories on Inspections
IBM inspections showed up
82 percent of all detected faults (1976); 70 percent of all detected faults (1978); 93 percent of all detected faults (1986)
Switching system
90 percent decrease in the cost of detecting faults (1986)
JPL
Four major faults, 14 minor faults per 2 hours (1990) ; Savings of $25,000 per inspection; The number of faults decreased exponentially by phase (1992)
However, fault statistics should never be used for performance appraisal

16. Comparison of Inspections and Walkthroughs
Walkthrough: Two-step, informal process
Preparation
Analysis
Inspections: Five-step, formal process
Overview
Inspection
Rework
Follow-up

17. Strengths and Weaknesses of Reviews (Walkthrough or Inspection)
A review is an effective way to detect a fault.
Faults are detected early in the process
Weaknesses:
Reviews are less effective if the process is inadequate. The two solutions to this weakness are:
Large-scale software should consist of smaller, largely independent pieces
The documentation of the previous workflows has to be complete and available online

18. Metrics for Inspections
Inspection rate: design pages inspected per hour
Fault density: faults per page inspected or faults per 1000 lines of code (KLOC) inspected
Fault detection rate: the number of faults detected per hour
Fault detection efficiency: number of major and minor faults detected per person-hour
Question: Does a 50 percent increase in the fault detection rate mean that
Quality has decreased? Or
The inspection process is more efficient?

19. Execution-Based Testing
Organizations spend up to 50 percent of their software budget on testing
But delivered software is frequently unreliable
“Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence” (Dijkstra; 1972)
The only information that can be deduced from the particular test is that the product runs correctly on that particular set of test data.

20. What Should Be Tested? Definition of execution-based testing
“The process of inferring certain behavioral properties of the product based, in part, on the results of executing the product in a known environment with selected inputs”
This definition has troubling implications
“Inference”: We have a fault report, the source code, and often nothing else. This makes the inference difficult.
“Known environment”: We never really can know our environment, either the hardware or the software
“Selected inputs”: Sometimes we cannot provide the inputs we want
Solution:
Simulator is needed: A simulator is a working model of the environment in which the product executes.

21. Items Should be Tested: Utility
The extent to which a user’s needs are met when a correct product is used under conditions permitted by its specifications
Ease of use
Useful functions
Cost effectiveness

22. Items Should be Tested: Reliability
A measure of the frequency and criticality of product failure
Mean time between failures: How often the product fails
Mean time to repair: How long it takes, on average, to repair the failure
Time (and cost) to repair the results of a failure

23. Items Should be Tested: Robustness
A function of
The range of operating conditions
The possibility of unacceptable results with valid input
The effect of invalid input

24. Items Should be Tested: Performance
The extent to which the product meets its constraints with regard to response time or space requirements.
Real-time software is characterized by hard real-time constraints
If data are lost because the system is too slow
There is no way to recover those data

25. Items Should be Tested: Correctness
A product is correct if it satisfies its output specifications, independent of its use of computing resources, when operated under permitted conditions.
Technically, correctness is
Not necessary
Example: C++ compiler
Not sufficient
Example: trickSort

26. Importance of the Correctness of Specifications
Incorrect specification for a sort:
Function trickSort which satisfies this specification:
Figure 6.1

27. Importance of the Correctness of Specifications (Cont.)
Corrected specification for the sort:

28. Testing versus Correctness Proofs
A correctness proof is a mathematical technique for showing that a product is correct, in other words, that it satisfies its specification.
It is an alternative to execution-based testing

29. Example of a Correctness Proof
The code segment to be proven correct
Its corresponding
flowchart

30. Add the followings:
Input specification
Output specification
Loop invariant
Assertions

31. Example of a Correctness Proof (Cont.)
Loop Invariant: A mathematical expression that holds irrespective of whether the loop has been executed 0, 1, or many times.
An informal proof (using induction) appears in Section That is: given the input specification, it can be proven that loop invariant holds. Furthermore, it can be proven that after n iterations the loop terminates and the output specification is satisfied.  The code segment therefore is mathematically proven to be correct.

32. Correctness Proof Mini Case Study
Dijkstra (1972):
“The programmer should let the program proof and program grow hand in hand”
“Naur text-processing problem” (1969)
Episode 1: Naur constructed a 25-line procedure and informally proved its correctness
Episode 2: Reviewer in Computing Reviews found 1 fault in 1970.
Episode 3: London found 3 more faults in 1971.
Episode 4: Goodenough and Gerhart found 3 further faults in 1975.
Lesson: Even if a product has been proved correct, it must STILL be tested.

33. Three Myths of Correctness Proving
Software engineers do not have enough mathematics for proofs
Most computer science majors either know or can learn the mathematics needed for proofs
Proving is too expensive to be practical
Economic viability is determined from cost–benefit analysis
Proving is too hard
Many nontrivial products have been successfully proved
Tools like theorem provers can assist us

34. Difficulties with Correctness Proving
Can we trust a theorem prover?
How do we find input and output specifications, and loop invariants or their equivalents in other logics?
What if the specifications are wrong?
We can never be sure that specifications or a verification system are correct

35. Correctness Proofs and Software Engineering
Correctness proofs are a vital software engineering tool. They are appropriate:
When human lives are at stake
When indicated by cost–benefit analysis
When the risk of not proving is too great
Also, informal proofs can markedly improve the quality of the product
Use the assert statement

36. Who Should Perform Execution-Based Testing?
Programmers should not test their own code artifacts since:
Programming is constructive
Testing is destructive
A successful test finds a fault
Solution:
The programmer debugs the module
The programmer does informal testing
The SQA group then does systematic testing

37. Test Cases All test cases must be
Planned beforehand, including the expected output, and
Retained afterwards for regression testing
Testing stops only when the product has been irrevocably discarded