1. A Population Size Estimation Problem Eliezer Kantorowitz Software Engineering Department Ort Braude College of Engineering kantor@cs.technion.ac.il

2. 11/7-2006@2006 Eliezer Kantorowitz2 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

3. 11/7-2006@2006 Eliezer Kantorowitz3 Estimating Population Size Two steps 1.Make an observation 2.Employ an estimator on the number of observed items Example: Industrial quality assurance 1.Count the number of defects in a sample 2.Estimate the defect population size from the defects counted in the sample

4. 11/7-2006@2006 Eliezer Kantorowitz4 Partial Observation Methods A Partial Observation Methods is an observation method that do not produce a count all the relevant items Example: Due to poor lighting, some of the defect items in the sample cannot be seen

5. 11/7-2006@2006 Eliezer Kantorowitz5 This talk is about estimators applicable when using partial observation methods

6. 11/7-2006@2006 Eliezer Kantorowitz6 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

7. 11/7-2006@2006 Eliezer Kantorowitz7 Counting Wild Animals Capture Recapture Estimators Example: Counting the gazelle population in upper Galilee Problem: We can only observe a part of the n members of the gazelle population Solution: We capture n tag gazelles in a trap. The gazelles are tagged and freed We assume that the freed gazelles are evenly mixed with the remaining n-n tag gazelles

8. 11/7-2006@2006 Eliezer Kantorowitz8 Capture Recapture (CR) - 2 We put a new trap and capture m gazelles of which m tag are recaptured gazelles The gazelle population size n may be estimated as assuming

9. 11/7-2006@2006 Eliezer Kantorowitz9 Capture Recapture (CR) -3 A number of different CR estimators corresponding to different sets of assumptions have been developed The essence of CR is that we enter (inject) a KNOWN number of tagged animals into the unknown number of animals. This known number can be employed in later statistical analysis

10. 11/7-2006@2006 Eliezer Kantorowitz10 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

11. Problem discussed in the following : Estimating the Number of Defects in Software Users Requirements Document (URD)

12. 11/7-2006@2006 Eliezer Kantorowitz12 Users Requirements Document (URD) Prepared by software analysts and users Part of software ordering contract In one case 55% of all defects (“bugs”) were URD defects URD validation usually done by inspection

13. 11/7-2006@2006 Eliezer Kantorowitz13 Example: a URD used in ourexperiments PURPOSE Manage a costume shop, which rents and sells costumes. Control the inventory and customer databases. Manage orders and invoices. CUSTOMER DATABASE - SYSTEM ACTIVITIES Enter new customers. Automatic updates of the customer ’ s database. List of customers active over the last three years. List of customers ordered by the age of the children. List of customers ordered by their purchase and rental transactions.

14. 11/7-2006@2006 Eliezer Kantorowitz14 URD validation Usually done by inspection

15. 11/7-2006@2006 Eliezer Kantorowitz15 Inspection Method (Fagan 1986) The inspected document is presented by its originator to a team of human inspectors Each inspector inspects the entire document and records the found defects Meeting of all inspectors, where defects found by different inspectors are checked and combined into one list

16. 11/7-2006@2006 Eliezer Kantorowitz16 Inspection Problem Usually an inspector sees only a part of all defects Different inspectors usually see different sets of defects A team of j+1 inspectors usually detects more defects than a team of j inspectors Inspection costs proportional to j

17. 11/7-2006@2006 Eliezer Kantorowitz17 Fault Detection Ratio (FDR) as Function of Inspector Team Size FDR=(number detected faults)/(total number of faults)

18. 11/7-2006@2006 Eliezer Kantorowitz18 Using Capture Recapture (CR) CR was adapted to the inspection problem Defects detected by more than one inspector play a similar role to that of recaptured gazelles Extensive experiments suggest that CR is not providing sufficient accurate estimates

19. 11/7-2006@2006 Eliezer Kantorowitz19 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

20. 11/7-2006@2006 Eliezer Kantorowitz20 Defect Injection In CR methods we freed a KNOWN number of tagged animals In defect injection methods we enter (inject) a KNOWN number of defects into the document

21. 11/7-2006@2006 Eliezer Kantorowitz21 Defect Injection Method n injected – number of injected defects n injected-detected – number of detected injected defects n real - number of real defects (the unknown) n detected-real – number of detected real defects Estimated number or real defects: n real = n detected-real (n injected / n injected-detected )

22. 11/7-2006@2006 Eliezer Kantorowitz22 Problems of Defect Injection The injected defects must “represent” the real defects “correctly”

23. 11/7-2006@2006 Eliezer Kantorowitz23 Defect Types Distribution

24. 11/7-2006@2006 Eliezer Kantorowitz24 Examples of Injected Defects Inconsistent Information Lists of customers entered by different techniques that contradict each other (lines 3 and 26). Cancellation of an order that was reserved is illegal (lines 28 and 32). The systems do not keep customer data for more than three years (lines 5 and 10). There is not enough information about the customers in the system (lines 6 and 10). An article that was reserved cannot be sold. (lines 27 and 33). Missing functionality: … Missing information: …

25. 11/7-2006@2006 Eliezer Kantorowitz25 Defect Injection Summary Common method for software documents Sufficient accurate estimates Difficult to produce “representative” defects Laborious

26. 11/7-2006@2006 Eliezer Kantorowitz26 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

27. 11/7-2006@2006 Eliezer Kantorowitz27 The Experimentators Eliezer Kantorowitz Arie Guttman Lior Arzi Assaf Harel

28. 11/7-2006@2006 Eliezer Kantorowitz28 Experiments - 1 Computer Science students at Technion –250 freshmen –69 senior Industry engineers –25 engineers Two experiments from literature –57 senior Computer Science students All together 401 persons involved

29. 11/7-2006@2006 Eliezer Kantorowitz29 Experiments - 2 Employed requirements documents –Costume shop information system –Missile launcher –Railroad system (in experiments from litterature) Data of good quality –401 persons –Careful preparation

30. 11/7-2006@2006 Eliezer Kantorowitz30 Typical Results Y axis is the number of inspectors that detected the different defects. The two “ easiest to detect ” defects were detected by 6 inspectors each

31. 11/7-2006@2006 Eliezer Kantorowitz31 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

32. 11/7-2006@2006 Eliezer Kantorowitz32 The Model - 1 i (fault number) P 0,1 1 n max n-1 0 P i,1 (probability that fault i is detected by 1 inspector) The linearity assumption

33. 11/7-2006@2006 Eliezer Kantorowitz33 The Model - 2 The linearity assumption P i,1 - probability that one inspector detects defect i. n max – defects 0 ≤ i <n max can be detected

34. 11/7-2006@2006 Eliezer Kantorowitz34 The Model - 3 P 0,1 - The probability that one inspector detects the “easiest to detect” defect P 0,1 ε[0,1] - A measure of the ease of detection FDR max – The inspectors are able to detect the proportion FDR max of the n defects, i.e. FDR max n defects FDR max ε[0,1] – a measure of the domain knowledge of the inspectors

35. 11/7-2006@2006 Eliezer Kantorowitz35 The Model – 4 The probability that j inspectors will detect defect i may be estimated: j inspectors are expected to detect FDR(j)n defects : For n →∞

36. 11/7-2006@2006 Eliezer Kantorowitz36 Kantorowitz Estimator Example of application: A quality assurance manager can employ this estimator to estimate the number of inspectors j required to detect the proportion FDR(j) of all faults. The coefficients FDR max and P 0,1 must somehow be estimated This estimator is implicitly the cost function required in a Total Quality Management (TQM). The number of inspectors j represent the costs, while FDR(j) represents the quality

37. 11/7-2006@2006 Eliezer Kantorowitz37 Application example: What is the optimal inspector team size? Teams of 2 detects the largest number of defects per inspector

38. 11/7-2006@2006 Eliezer Kantorowitz38 Application example: Comparing Engineers with Students P 0,1 0.990.401 st year Students 0.990.74Industry Engineers Experiment with Missile launcher user requirements document Example: 4 student teams achieve FDR=053 while only two engineer teams do FDR=0.54, i.e. an engineer detected about twice as many defects as a student max FDR

39. 11/7-2006@2006 Eliezer Kantorowitz39 Summary of my estimator Based on a property of the data observed in a large number of experiments The estimator was derived by modeling the observed property of the data Sufficient accurate Measuring the two coefficients of the model P 0,1 and FDR max is laborious, however, their numerical values may be estimated from similar cases

40. 11/7-2006@2006 Eliezer Kantorowitz40 Table of Contents The problem Capture Recapture Estimators Estimating number of software defects Defect injection estimators Our experiments Our estimator Conclusions and Research plans

41. 11/7-2006@2006 Eliezer Kantorowitz41 Surveyed Estimators for the incomplete Counting Problem Capture Recapture Defect Injection My estimator

42. 11/7-2006@2006 Eliezer Kantorowitz42 My Estimator vs. Capture Recapture Estimators Our estimator was sufficient accurate for estimating the number of defects in use requirements documents, while the CR estimators were not sufficient accurate Were the data in the extensive CR experiments of sufficient good quality?

43. 11/7-2006@2006 Eliezer Kantorowitz43 Why did My Estimator Work My estimator exploited a property of the data (the linearity assumption) The property was detected through careful extensive experimentation

44. 11/7-2006@2006 Eliezer Kantorowitz44 Looking for Similar Applications Can the approach of this research be useful in other areas where the employed observation method only count part of the relevant items?