1. Software Process Control1 Aditya P. Mathur Department of Computer Sciences Purdue University, West Lafayette Visiting Profesor, BITS, Pilani Research collaborators: João Cangussu (CS, UT Dallas) Ray. A. DeCarlo (ECE, Purdue University) Monday November 10, 2003 Newton's Law of Motion in Software Development Processes? Presentation at: Indian Institute of Technology, Kanpur, India

2. Software Process Control2 Research Question Can we control the Software Development Process in a manner similar to how physical systems and processes are controlled ? The central problem in control is to find a technically feasible way to act on a given process so that the process adheres, as closely as possible to some desired behavior. The fundamental control problem (Ref: Control System Design by G. C. Goodwin et al., Prentice Hall, 2001) Furthermore, this approximate behavior should be achieved in the face of uncertainty of the process and in the presence of uncontrollable external disturbances acting on the process.

3. Software Process Control3 Research Methodology 1.Understand how physical systems are controlled? 2. Understand how software systems relate to physical systems. Are there similarities? Differences? 3. Understand the theory and practice of the control of physical systems. Can we borrow from this theory? 4.Adapt control theory to the control of SDP and develop models and methods to control the SDP. 5.Study the behavior of the models and methods in real-life settings and, perhaps, improve the model and methods. 6.Repeat steps 6 and 7 until you are thoroughly bored or get rich.

4. Software Process Control4 Feedback Control Specifications Program Effort + f(e) Additional effort What is f ? - Required Quality Observed Quality

5. Software Process Control5 Software Development Process: Definitions A Software Development Process (SDP) is a sequence of well defined activities used in the production of software. An SDP usually consists of several sub-processes that may or may not operate in a sequence. The Design Process, the Software Test Process, and the Configuration Management Process are examples of sub-processes of the SDP.

6. Software Process Control6 Software Development Process: A Life Cycle Requirements Elicitation Requirements Analysis Integrate/Test Design Code/Unit test System test More testDeploy Not all feedback loops are shown.

7. Software Process Control7 Current Focus Software Test Process (STP): System test phase Objective: Control the STP so that the quality of the tested software is as desired. Quantification of quality of software: Number of remaining errors Reliability

8. Software Process Control8 Problem Scenario cp 1 cp 2 cp 3 cp 4 cp 5 cp 6 cp 7 cp 8 cp 9 cp i = check point i rfrf schedule set by the manager Approximation of how r is likely to change r0r0 observed deadline r - number of remaining errors t- time t0t0

9. Software Process Control9 Our Approach Controller r error (t)  ’  w’ f + + wf+wfwf+wf  +  wf+wfwf+wf r observed (t) r expected (t) Actual STP scsc r0r0 STP State Model scsc r0r0 Initial Settings (w f,  )  wfwf Test Manager w f : workforce  : quality of the test process

10. Software Process Control10 Physical and Software Systems: An Analogy Dashpot Rigid surface External force Xequilibrium X: Position Number of remaining errors Spring Force Effective Test Effort Block Software Mass of the block Software complexity Quality of the test process Viscosity Xcurrent Spring To err is Human

11. Software Process Control11 Physical Systems: Laws of Motion [1] First Law: Every object in a state of uniform motion tends to remain in that state of motion unless an external force is applied to it. Does not (seem to) apply to testing because the number of errors does not change when no external effort is applied to the application.

12. Software Process Control12 Physical Systems: Laws of Motion [2] Newton’s Second Law: The relationship between an object's mass m, its acceleration a, and the applied force F is F = ma. CDM First Postulate: The relationship between the complexity S c of an application, its rate of reduction in the number of remaining errors, and the applied effort E is E=S c. r..

13. Software Process Control13 Physical Systems: Laws of Motion [3] Third Law: For every action force, there is an equal and opposite reaction force. When an effort is applied to test software, it leads to (mental) fatigue on the tester. Unable to quantify this relationship.

14. Software Process Control14 CDM First Postulate The magnitude of the rate of decrease of the remaining errors is directly proportional to the net applied effort and inversely proportional to the complexity of the program under test. This is analogous to Newton’s Second Law of motion.

15. Software Process Control15 CDM Second Postulate The magnitude of the effective test effort is proportional to the product of the applied work force and the number of remaining errors. for an appropriate . Analogy with the spring: Note: While keeping the effective test effort constant, a reduction in r requires an increase in workforce.

16. Software Process Control16 CDM Third Postulate The error reduction resistance is proportional to the error reduction velocity and inversely proportional to the overall quality of the test phase. for an appropriate . Analogy with the dashpot: Note: For a given quality of the test phase, a larger error reduction velocity leads to larger resistance.

17. Software Process Control17 State Model : Disturbance x(t) = Ax(t) + B u(t). Force (effort) balance equation:

18. Software Process Control18 Computing the feedback-Question Question: What changes to the process parameters will achieve the desired r(T+  T) ? r(T): the number of remaining errors at time T r(T+  T): the desired number of remaining errors at time T+  T Given:

19. Software Process Control19 Computing the feedback-Answer From basic matrix theory: The largest eigenvalue of a linear system dominates the rate of convergence. Hence we need to adjust the largest eigenvalue of the system so that the response converges to the desired value within the remaining weeks (  T). This can be achieved by maintaining: Obtain the desired eigenvalue.

20. Software Process Control20 Computing the feedback-Calculations ( max ) Compute the desired max Given the constraint: We know that the eigenvalues of our model are the roots of its characteristic polynomial of the A matrix.

21. Software Process Control21 Computing the feedback-Calculations ( max ) We use the above equation to calculate the space of changes to w and  such that the system maintains its desired eigenvalue. f

22. Software Process Control22 Computing the feedback-Input to the Manager The space of changes in the workforce and the quality of the process is made available to the test manager in the form of suggestions for possible process changes. The test manager may decide to select a combination of these values for implementation or simply ignore them. So far, in each of the two commercial studies we carried out, the manager ignored the suggestions given using the model.

23. Software Process Control23 Case Study I: The Razorfish Project Project Goal: translate 4 million lines of Cobol code to SAP/R3 A tool has been developed to achieve the goal of this project. Goal of the test process: (a) Test the generated code, not the tool. (b) Reduce the number of errors by about 85%.

24. Software Process Control24 Razorfish Project Test Process output 1 run output 2 Transformer S SAP R/3 S Cobol Select a Test Profile input continue testing yes modify = no

25. Software Process Control25 Razorfish Project: Results (intermediate) 85% reduction achieved. If the process parameters are not altered then the goal is reached in about 35 weeks. Prediction using feedback Prediction using the model Project data Expected behavior

26. Software Process Control26 Alternatives from Feedback: STP Quality Desired eigenvalue=-0.152 Improving quality alone will not help in achieving the goal.

27. Software Process Control27 Alternatives from Feedback: Workforce Desired eigenvalue=-0.152 Changing the workforce alone can produce the desired results.

28. Software Process Control28 Alternatives from Feedback: STP quality and workforce Set of valid choices for changing the quality and the workforce

29. Software Process Control29 Razorfish Project Results (final) The project was completed in 32 weeks. The model predicted 85% error reduction in 35 weeks.

30. Software Process Control30 Case Study 2: Company X P1: Week 9 Start of the study 1 week = 5 working days Estimated R 0 = 557 70% reduction – 10 weeks 90% reduction - 16 weeks

31. Software Process Control31 Company X P1: Week 12 No recalibration Estimated R 0 = 557 70% reduction – 10 weeks (confirming previous prediction) 90% reduction - 14 weeks Note: for 90% error reduction, the change in 14wks vs. 16wks from the previous slide is due to an increase in the number of testers from 5 to 7

32. Software Process Control32 Company X P1: Week 14 Recalibration Estimated R 0 = 758 (agressive) 70% reduction – 13.6 weeks 90% reduction - 21.6 weeks

33. Software Process Control33 Company X P1: End of Phase 1

34. Software Process Control34 Company X P1: Summary 21.2 weeks 21 13.6 weeks 13.4 weeks 76463518 21.4 weeks 21 13.6 weeks 13.4 weeks 75859016 21.6 weeks -- 13.6 weeks --75853514 14 weeks--10 weeks 55745812 16 weeks -- 10 weeks --5572819 EstimatedActualEstimatedActual 90% Defect Reduction 70% Defect Reduction Estimated R 0 Observ- ed Defect # Week

35. Software Process Control35 Summary Analogy between physical and software systems presented. The notion of feedback control of software processes introduced. Two case studies described. Parameter estimation techniques used for model calibration. Made use of system identification techniques.

36. Software Process Control36 Ongoing Research Expansion of the model to include the entire SDP: ongoing project in collaboration with Guidant Corporation (detailed model). Sensitivity analysis (completed, IEEE TSE May 2003) r is more sensitive to changes in the model parameters during the early stages of the test process than during the later stages. An improvement in the quality of the STP is more effective than an increase in the workforce. Brook’s Law was also observed during the analysis.

37. Software Process Control37 Physical Systems: Control Controllability Is it possible to control X (r) by adjusting Y (workforce and process quality)? Observability Does the system have distinct states that cannot be unambiguously identified by the controller ? Robustness Will control be regained satisfactorily after an unexpected disturbance?