1. The Erik Jonsson School of Engineering and Computer Science Software Testing Process Control A Novel Approach João W. Cangussu University of Texas at Dallas Department of Computer Science cangussu@utdallas.edu www.utdallas.edu/~cangussu Association for Software Engineering Excellence September 13, 2005

2. The Erik Jonsson School of Engineering and Computer Science Motivation ➋ Can we achieve a level o control for the STP that is close to what is observed in other engineering disciplines? ❶ Current techniques for the Software Testing Process (STP) are mostly based on guidelines and semi-formal methods.

3. The Erik Jonsson School of Engineering and Computer Science Goal Increase the controllability and prediction capabilities of the Software Testing Process by developing a control mechanism to correct for deviations in the Testing Process

4. The Erik Jonsson School of Engineering and Computer Science Testing Process Control Status ✔ Several recommended procedures are available. None is based on formal models. There is a lack of control algorithms. ✔ COCOMO: For cost and effort estimation; no feedback loop. ✔ Reliability models: For the estimation of software reliability; once again, no feedback loop. So what do Level 5 companies/groups do? ➊ Often chaotic.

5. The Erik Jonsson School of Engineering and Computer Science The Problem Software Test Process input parameters observed quality (Q o ) expected quality (Q e ) Test Manager error = Q e - Q o Control Mechanism set of solutions

6. The Erik Jonsson School of Engineering and Computer Science System Testing t- time cp 1 cp 2 cp 3 cp 4 cp 5 cp 6 cp 7 cp 8 cp 9 where cp i = check point i r0r0 approximation deadline t0t0 number of remaining defects rfrf schedule set by the test manager observed Estimated

7. The Erik Jonsson School of Engineering and Computer Science Control Approach Actual STP wf+wfwf+wf  +  wf+wfwf+wf STP State Model r observed (t) r expected (t) Initial Settings (w f,  ) r error (t) Controller  ’  w’ f Test Manager + +  wfwf

8. The Erik Jonsson School of Engineering and Computer Science Why a State Model? ➊ State Variable approach has been used to successfully model different types of system ✔ Predator/Prey Model (Voltera) ✔ Economic Model (Samuelson) ✔ Warfare Model (Lanchester) ➋ Enables the use of Control Theory techniques ✔ Feedback Control - Friedland, Luenberger ✔ Optimization - Chong, Zak ✔ System Identification - Ljung

9. The Erik Jonsson School of Engineering and Computer Science Modeling Deming states: All models are wrong, some models are useful.

10. The Erik Jonsson School of Engineering and Computer Science Assumption I The rate at which the velocity of the number of remaining errors changes is directly proportional to the net applied effort (e n ) and inversely proportional to the complexity of the program under test.

11. The Erik Jonsson School of Engineering and Computer Science Assumption II for an appropriate The effective test effort (e f ) is proportional to the product of the applied work force and the number of remaining errors.

12. The Erik Jonsson School of Engineering and Computer Science Assumption III for an appropriate constant . The error reduction resistance (e r ) is proportional to the error reduction velocity ( r ) and is inversely proportional to the overall quality of the test phase (  ). e r opposes r.

13. The Erik Jonsson School of Engineering and Computer Science State Model

14. The Erik Jonsson School of Engineering and Computer Science Analogy Block Dashpot Rigid surface External force Xcurrent Xequilibrium X: Position Number of remaining errors Spring Force Effective Test Effort Software Mass of the block Software complexity Quality of the test process Viscosity Spring To err is Human.

15. The Erik Jonsson School of Engineering and Computer Science Parameter Calibration ➋ Good historic data help avoid initial over/under estimation ➊ Models are approximations and need to be calibrated ➌ Calibration algorithm considerably decreases the effect of initial over/under estimation

16. The Erik Jonsson School of Engineering and Computer Science Model Parameters ➊ Parameters to Estimate ✔ s c (software complexity) ✔ γ (quality of the testing process) ✔ R 0 ✔ ζ ✔ ξ ➋ Automatic Parameter Calibration

17. The Erik Jonsson School of Engineering and Computer Science Estimating s c scsc 11 22 33 nn M 1 : KLOC M 2 : Cyclomatic Complexity M 3 : Halstead Metric M n : Information Flow Metric

18. The Erik Jonsson School of Engineering and Computer Science Estimating γ  11 22 33 nn Q1Q1 Q2Q2 Q3Q3 QnQn Where Q i is a quality factor

19. The Erik Jonsson School of Engineering and Computer Science Estimating the α weights Quality Factor Q i Q1Q1 Q2Q2 …QnQn Q 1 : test plan adequacy 12…1/4 Q 2 : test team experience 1/21…3 ………1… Q n : testing tool adequacy/automation 41/3…1 The eigenvector associated with the largest eigenvalue of this matrix give us the appropriated weights

20. The Erik Jonsson School of Engineering and Computer Science Estimating R 0 λ1 λ2λ1 λ2 λ 1 λ 2 Approximator R 0 Estimator Data from ongoing testing R 0 Correction R0R0 R 0c Turn key solution: does not depend on historical or “unavailable” data from previous projects does not depend on initial values of Internal parameters

21. The Erik Jonsson School of Engineering and Computer Science Estimating ξ and ζ

22. The Erik Jonsson School of Engineering and Computer Science Parametric Control r - number of remaining errors t- time R0R0 RfRf r(T) r(T+  t) tt t0t0

23. The Erik Jonsson School of Engineering and Computer Science Computing Δw f and Δ γ

24. The Erik Jonsson School of Engineering and Computer Science Results – Control and Time Prediction

25. The Erik Jonsson School of Engineering and Computer Science Results – Control and Time Prediction

26. The Erik Jonsson School of Engineering and Computer Science Results – Control and Time Prediction

27. The Erik Jonsson School of Engineering and Computer Science Results – T E X78

28. The Erik Jonsson School of Engineering and Computer Science Results – Transformer Project

29. The Erik Jonsson School of Engineering and Computer Science Results – Feedback Application

30. The Erik Jonsson School of Engineering and Computer Science Results – More Projects

31. The Erik Jonsson School of Engineering and Computer Science Results – More Projects 21.2 weeks 21 weeks 13.6 weeks 13.4 weeks 76463518 21.4 weeks 21 weeks 13.6 weeks 13.4 weeks 75859016 21.6 weeks 21 weeks 13.6 weeks 13.4 weeks 75853514 18 weeks21 weeks10 weeks 13.4 weeks 55745812 16 weeks 21 weeks 10 weeks 13.4 weeks 5572819 EstimatedActualEstimatedActual 90% Defect Reduction 70% Defect Reduction Estimated R 0 Observ- ed Defect # Week

32. The Erik Jonsson School of Engineering and Computer Science Convergence of Completion Time Estimates

33. The Erik Jonsson School of Engineering and Computer Science Convergence of Completion Time Estimates 68.5% required at most 11% of total time to converge 90% required at most 21% of total time to converge

34. The Erik Jonsson School of Engineering and Computer Science Estimates for R 0

35. The Erik Jonsson School of Engineering and Computer Science Estimates for R 0

36. The Erik Jonsson School of Engineering and Computer Science Comparing R 0 Estimates

37. The Erik Jonsson School of Engineering and Computer Science Comparing R 0 Estimates

38. The Erik Jonsson School of Engineering and Computer Science Convergence of R 0 Estimates

39. The Erik Jonsson School of Engineering and Computer Science Convergence of R 0 Estimates

40. The Erik Jonsson School of Engineering and Computer Science Sensitivity Analysis ➊ Changes in the process’s parameters are more effective at early stages of the process ➋ Under certain conditions, late changes can make the process slow down instead of speed up (Brooks’s law). ➌ Improvements in the quality of the process is better than increases the size of the test team.

41. The Erik Jonsson School of Engineering and Computer Science Conclusions ❶ The state model is reasonably accurate in modeling the behavior of the STP ➋ The parameter identification technique provides accurate results for model calibration and defect estimation ➌ Static and dynamic analysis has shown consistency of the model with the real world

42. The Erik Jonsson School of Engineering and Computer Science Future Work ❶ Develop a modular hierarchy of models for the other phases of the SDP and interconnect them ➌ Develop a secure web-based tool to host the techniques ➋ Investigate alternative control techniques ✔ Stochastic control ✔ Fuzzy control