1. 5/2008Copyright 2008 Eliezer Kantorowitz1 Estimating the Required Number of Software Code Inspectors E. Kantorowitz Software Engineering Dept., Ort Braude College of Engineering

2. Experimental Validation Collaborators 1996- Arie Guttman ALior Arzi Assaf Harel Dmitry Tilman Alex Raginsky T. Kuflik 5/2008Copyright 2008 Eliezer Kantorowitz2

3. Software Faults Software Requirements (what it should do) Document faults –Missing functionality –Missing Data –Inconsistencies Code (Software) faults –Does not implement the requirements correctly 5/2008Copyright 2008 Eliezer Kantorowitz3

4. Fault Detection Methods 1.Inspection, e.g. 80% of faults 2.Testing code, remaining 20% 5/2008Copyright 2008 Eliezer Kantorowitz4

5. Inspection Process (Fagan 1986) 5/2008Copyright 2008 Eliezer Kantorowitz5 1. Each inspector inspects the document and lists the detected faults 2. The team meets and validates the faults. Invalid faults are removed. Faults detected during the meeting are added.

6. Inspection results Inspector 4Inspector 3Inspector 2Inspector 1 0111Fault 0 1010Fault 1 0110Fault 2 0010Fault 3 0100Fault 4 5/2008Copyright 2008 Eliezer Kantorowitz6 Inspector 1 found only Fault 0 Fault 1 was found by two inspectors Fault 0 is by definition the “easiest to detect fault”, as it was detected by the largest number (3) of inspectors PROBLEM: Are the 4 detected faults 80% of all faults ?

7. Statistical Quality Assurance Sampling : Population size known –Example: We sample 5 bicycles from a batch of 100 Capture-Recapture: Population size unknown –Example: We capture 5 gazelles in the Galilee mountains. What is the total number of Gazelles in Galilee 5/2008Copyright 2008 Eliezer Kantorowitz7

8. 5/2008Copyright 2008 Eliezer Kantorowitz8 Capture Recapture (CR) [Laplace 1786] How many Gazelles live in Galilee? Catch n m gazelles, mark and free them After some time we put a trap and catch n m2 marked an n 2 unmarked gazelles The total number of gazelles n may be estimated using Lincoln-Petersen estimator:

9. 5/2008Copyright 2008 Eliezer Kantorowitz9 Example Capture recapture We capture and release 10 marked gazelles A later trap captures 5 marked and 10 unmarked gazelles We estimate that we have captured 5/10=0.5 of all gazelles The captured 10+5=15 gazelles are thus 50% of all gazelles, i.e. there are 30 gazelles

10. Assumptions of the Lincoln-Petersen Estimator No immigration, emigration, births or deaths between the release and recapture times. The probabilities of being caught are equal for all individuals (including marked ones). Marks (or tags) are not lost and are always recognizable. 5/2008Copyright 2008 Eliezer Kantorowitz10

11. Further Estimators Different estimators exist for different assumption combinations CR experiments may be repeated to improve estimation accuracy 5/2008Copyright 2008 Eliezer Kantorowitz11

12. 5/2008Copyright 2008 Eliezer Kantorowitz12 Capture Recapture (CR)in Software Re-detection of a fault by n different inspectors corresponds mathematically to n recaptures in CR estimation How to estimate the number of inspectors required to detect, say, 80% of all faults

13. Kantorowitz Estimator 5/2008Copyright 2008 Eliezer Kantorowitz13

14. 5/2008Copyright 2008 Eliezer Kantorowitz14 Developing the Estimator 1.Experimental investigation of ease of detection of different faults 2.Discovering the nearly linear distribution 3.Developing the estimator from this distribution

15. 5/2008Copyright 2008 Eliezer Kantorowitz15 The Experiment Injecting faults in the two software requirements documents –Mask shop –Missile launcher for Columbian mafia 205 students and 25 industry engineers acted as inspectors

16. 5/2008Copyright 2008 Eliezer Kantorowitz16 Typical Result The Linear Model P 0,1 N_FDR max

17. Assumption The assumption that all faults have the same detection probability is not met Assume linear probability distribution 5/2008Copyright 2008 Eliezer Kantorowitz17

18. 5/2008Copyright 2008 Eliezer Kantorowitz18 The Linear Distribution P i,j - Probability that j inspectors detect fault i P 0,1 - Probability that one inspector detects the easiest to detect fault N_FDR max – The largest number of faults the em[ployed inspectors are able to detect The Linear distribution :

19. 5/2008Copyright 2008 Eliezer Kantorowitz19 Estimator Concepts FDR- Fault Detection Ratio is the ratio between the number of detected faults and the total number of faults FDR(j) – FDR expected when employing j inspectors FDR max – the maximal FDR that the employed inspectors may achieve FDRmax<1 when the domain knowledge of the inspectors is incomplete

20. Developing the Estimator 5/2008Copyright 2008 Eliezer Kantorowitz20 where

21. 5/2008Copyright 2008 Eliezer Kantorowitz21 Kantorowitz Estimator A continuous version of the estimator is obtained by Producing the estimator:

22. 5/2008Copyright 2008 Eliezer Kantorowitz22 Experimental Validation Software Requirements ParticipantsTeams SizeTeamsDurationSystemReference senior3106 hoursRailroad[4]1 senior396 hoursRailroad[5]2 senior3720 minutesCostumes3 senior3740 minutesCostumes4 senior3930 minutesCostumes5 senior3830 minutesMissile launcher6 freshmen3735 minutesMissile launcher7 freshmen3840 minutesCostumes8

23. The Experiment We employed two software requirements documents –Information system for a mask shop –Missile launcher for the Columbian mafia We injected faults into the documents and inspected them 5/2008Copyright 2008 Eliezer Kantorowitz23

24. 5/2008Copyright 2008 Eliezer Kantorowitz24 4 rd Year Computer Science Students User Requirements Document Inspection P 0,1 =0.72 FDR max =0.68

25. Is the Linear Probability Distribution Assumption Satisfactory ? 5/2008Copyright 2008 Eliezer Kantorowitz25 We tried both a parabolic model: And an exponential model The linear, parabolic and exponential model produced nearly the same results. We prefer therefore the simple linear model

26. Linear, Parabolic and Exponential Estimations and Observed Standard Deviations 5/2008Copyright 2008 Eliezer Kantorowitz26

27. 5/2008Copyright 2008 Eliezer Kantorowitz27 Code Inspection Experiments TeamsPopulationProgramming Language Experiment 12 Teams of 2 inspectors + 1 team of 3 insp. 3 rd year MIS students Java1 10 Teams of 2 inspectors + 1 team of 1 insp. Industry Software Engineers C#2

28. 5/2008Copyright 2008 Eliezer Kantorowitz28 3 rd year Information System Students – Java Code Inspection P 0,1 =0.75 FDR max =0.85

29. 5/2008Copyright 2008 Eliezer Kantorowitz29 Software Engineers in Information Systems Industry

30. 5/2008Copyright 2008 Eliezer Kantorowitz Coode Inspection Results FDR max P 0,1 PopulationExperiment 0.930.58Students1 1.000.75Engineers2

31. 5/2008Copyright 2008 Eliezer Kantorowitz31 Inspection Metrics The inspection is characterized by; P 0,1 The probability that one inspector detects the easiest to detect fault FDR max the fraction of all faults that the inspectors are able to detect These two numbers in the [0;1] interval characterize the combination: inspectors/document/inspection method

32. Unexpected Observation The P 0,1 and FDR max values measured with 4 th year Technion CS students and senior CS students at University of Minnesota were nearly the same! Both inspected information system requirements documents –2 page mask shop –22 page railway system 5/2008Copyright 2008 Eliezer Kantorowitz32

33. 5/2008Copyright 2008 Eliezer Kantorowitz33 First and Last Year Students

34. Comparing Code Inspection of Students and Engineers 5/2008Copyright 2008 Eliezer Kantorowitz34

35. Inspection Costs Estimations Checking 200 lines in 40 minutes is optimal Inspection costs = j▪ hours ▪ hour costs, where j is the number inspectors Our estimator FDR(j), thus expresses Quality(costs) Useful for TQM (Total Quality Management) 5/2008Copyright 2008 Eliezer Kantorowitz35

36. 5/2008Copyright 2008 Eliezer Kantorowitz36 Conclusions Experiments suggest that the estimator is sufficient accurate for both requirements and code Inspection The inspection process metrics P 0,1 and FDR max introduced by the estimator may be reused in similar inspections

37. Conclusion The number of required inspectors may be estimated ahead of the experiment The estimator facilitates inspection costs estimation Useful for TQM 5/2008Copyright 2008 Eliezer Kantorowitz37

38. The End 5/2008Copyright 2008 Eliezer Kantorowitz38

39. 5/2008Copyright 2008 Eliezer Kantorowitz39 3 rd year Information System Students – Java Code Inspection

40. 5/2008Copyright 2008 Eliezer Kantorowitz40 Detection Profile Method (DPM) [Wohlin and Runeson 1998] The number M k of inspectors that detected defect k is a measure to its ease of detection We number the faults in their ease of detection order as 1,2,…,k,… We fit M(k) = ae -bk to the given M K values The number of faults is estimated as the k value for which M(k)=0

41. 5/2008Copyright 2008 Eliezer Kantorowitz41 Detection Profile Method (DPM) [Wohlin and Runeson 1998]

42. 5/2008Copyright 2008 Eliezer Kantorowitz42 Nielsen Landauer estimator (1993) Determined experimentally the number N of usability problems that n inspectors detects Regression analysis produced the estimator N=(1-(1-L)n) where L=0.31

43. Capture Recapture Assumptions Possible assumptionYesNo Marked gazelles evenly distributed in the population √√ Number gazelles constant (none borne/dead √√ Gazelles learn to beware of traps√√ 5/2008Copyright 2008 Eliezer Kantorowitz43 8 possible assumption combinations, which is right?

44. Example: Estimate for Software Requirements Documents with Observed Standard Deviations 5/2008Copyright 2008 Eliezer Kantorowitz44

45. 5/2008Copyright 2008 Eliezer Kantorowitz45 Examples For experienced inspectors we may assume P 0,1 =1 and FDR max =1 We can then estimate for a team of j=2 inspectors that FDR(2)=2/3 In other words two expert inspectors are expected to detect 2/3 of all Faults. If the two inspectors have, for example, detected 6 faults then it is expected that there are still 3 undetected faults left Of all faults. The estimator may be employed by a manager who wishes To estimate the number of ins

46. 5/2008Copyright 2008 Eliezer Kantorowitz46 Using the Estimator A manager that want to estimate how many inspectors are needed to detect 80% of all faults may compute the table and see that 4 inspectors will do the job FDR(j)j 2/32 3/43 4/54 For different values of P 0,1 and FDR max a different estimate will be achieved

47. 5/2008Copyright 2008 Eliezer Kantorowitz47 3 rd Year Students

48. 5/2008Copyright 2008 Eliezer Kantorowitz48 When P 0,1 and FDR max are estimated j – number of inspectors Compute j that ensures that FDR(j)>quality required

49. 5/2008Copyright 2008 Eliezer Kantorowitz49 Interpolation Assume that abilities of student that just finished are half way between 3 rd year students and Engineers, i.e. P 0,1 =(0.58+0.75)/2=0.66 FDR max =(0.93+1.00)/2=0.96. Using equation We can now estimate that two (j=2) students that just finished their studies are expected to detect FDR(j)=0.49, i.e. about half of all faults