1. Aditya P. Mathur Professor Department of Computer Sciences Purdue University, West Lafayette Friday June 11, 2010 Software Cybernetics: The Next Frontier 15th Anniversary Celebration of Society for Design and Process Science (SDPS), Dallas, TX, USA 1

2. 2 Collaborators Kai-Yuan Cai, Beijing University of Aeronautics and Astronautics Joao Cangussu, Microsoft Ray DeCarlo, Purdue Scott Miller, Purdue

3. 3 Key Question Can we model and control a software development process using theories and techniques available to control engineers?

4. 4 What is Software Cybernetics? A (new) discipline that aims at the development of models and control methodologies for software and the software development processes.

5. 5 Focus of Software Cybernetics Formalization and quantification of feedback mechanisms in software processes and systems Adaptation of principles and techniques of control theory to software processes and systems Application of the principles of software related theories and engineering to control systems and processes Integration of the theories of software engineering and control engineering

6. 6 Software and Control Software Code Development Maintenance Evolution Control A finite or infinite sequence of actions applied to the controlled object to achieve a given apriori goal for the controlled object.

7. 7 Software Cybernetics: Domain Modeling and Simulation: Abdel-Hamid and S.E. Madnick. Software Project Dynamics. Prentice Hall 1991. Role of feedback in software evolution [Basili et al. 1994; Lehman, 1996]

8. 8 Software Cybernetics: Domain Modeling and control for decision support: Software synthesis [Sridharan et al. 2003] Software test process control [Cangussu et al. 2002] Adaptive testing [Cai 2002] Software development process control, incremental lifecycle model [Miller et al. 2008]

9. 9 Software Cybernetics: Domain Modeling and control for reliable operation: Eliminating Concurrency Bugs with Control Engineering [Kelly et al. IEEE Computer 2009]

10. 10 Feedback Control: The basic idea Specifications Program Effort + f(e) Additional effort What is f ? - Required Quality Observed Quality

11. 11 Software Development Process: Definitions A Software Development Process (SDP) is a sequence of well defined activities used in the production of software. An SDP usually consists of several sub-processes that may or may not operate in a sequence. The Design Process, the Software Test Process, and the Configuration Management Process are examples of sub-processes of the SDP.

12. 12 Software Development Process: A Life Cycle Requirements Elicitation Requirements Analysis Integrate/Test Design Code/Unit test System test More testDeploy Not all feedback loops are shown. Techniques proposed here can be adapted to any life cycle model.

13. 13 Study 1: Modeling the system test phase Software Test Process (STP): System test phase; physical system analogy Objective: Control the STP so that the quality of the tested software is as desired. Quantification of quality of software: Number of remaining errors Reliability

14. 14 Study 2: Modeling the development process Software Development Process (SDP): All phases, queuing system anaogy [Miller et al. COMPSAC 2004] Objective: Offer change suggestions capable of achieving a specified deviation from the predicted system trajectory. Quantification of quality of software: Number of remaining errors Reliability

15. 15 The System Test Phase: Parametric Control Cangussu et al. TSE 2002

16. 16 Problem Scenario cp 1 cp 2 cp 3 cp 4 cp 5 cp 6 cp 7 cp 8 cp 9 cp i = check point i rfrf schedule set by the manager Approximation of how r is likely to change r0r0 observed deadline r - number of remaining errors t- time t0t0

17. 17 Our Approach Controller r error (t)  ’  w’ f + + wf+wfwf+wf  +  wf+wfwf+wf r observed (t) r expected (t) Actual STP scsc r0r0 STP State Model scsc r0r0 Initial Settings (w f,  )  wfwf Test Manager w f : workforce  : quality of the test process

18. 18 Physical and Software Systems: An Analogy Dashpot Rigid surface External force X equilibrium X: Position Number of remaining errors Spring Force Effective Test Effort Block Software Mass of the block Software complexity Quality of the test process Viscosity X current Spring To err is Human

19. 19 Physical Systems: Laws of Motion [1] First Law: Every object in a state of uniform motion tends to remain in that state of motion unless an external force is applied to it. Does not (seem to) apply to testing because the number of errors does not change when no external effort is applied to the application.

20. 20 Physical Systems: Laws of Motion [2] Newton’s Second Law: The relationship between an object's mass m, its acceleration a, and the applied force F is F = ma. CDM First Postulate: The relationship between the complexity S c of an application, its rate of reduction in the number of remaining errors, and the applied effort E is E=S c. r..

21. 21 Physical Systems: Laws of Motion [3] Third Law: For every action force, there is an equal and opposite reaction force. When an effort is applied to test software, it leads to (mental) fatigue on the tester. Unable to quantify this relationship.

22. 22 CDM First Postulate The magnitude of the rate of decrease of the remaining errors is directly proportional to the net applied effort and inversely proportional to the complexity of the program under test. This is analogous to Newton’s Second Law of motion.

23. 23 CDM Second Postulate The magnitude of the effective test effort is proportional to the product of the applied work force and the number of remaining errors. for an appropriate . Analogy with the spring: Note: While keeping the effective test effort constant, a reduction in r requires an increase in workforce.

24. 24 CDM Third Postulate The error reduction resistance is proportional to the error reduction velocity and inversely proportional to the overall quality of the test phase. for an appropriate . Analogy with the dashpot: Note: For a given quality of the test phase, a larger error reduction velocity leads to larger resistance.

25. 25 State Model : Disturbance x(t) = Ax(t) + B u(t). Force (effort) balance equation:

26. 26 Computing the feedback-Question Question: What changes to the process parameters will achieve the desired r(T+  T) ? r(T): the number of remaining errors at time T r(T+  T): the desired number of remaining errors at time T+  T Given:

27. 27 Computing the feedback-Answer From basic matrix theory: The largest eigenvalue of a linear system dominates the rate of convergence. Hence we need to adjust the largest eigenvalue of the system so that the response converges to the desired value within the remaining weeks (  T). This can be achieved by maintaining: Obtain the desired eigenvalue.

28. 28 Computing the feedback-Calculations ( max ) Compute the desired max Given the constraint: We know that the eigenvalues of our model are the roots of its characteristic polynomial of the A matrix.

29. 29 Computing the feedback-Calculations ( max ) We use the above equation to calculate the space of changes to w and  such that the system maintains its desired eigenvalue. f

30. 30 Computing the feedback-Input to the Manager The space of changes in the workforce and the quality of the process is made available to the test manager in the form of suggestions for possible process changes. The test manager may decide to select a combination of these values for implementation or simply ignore them. In each of the two commercial studies we carried out, the manager ignored the suggestions given using the model.

31. 31 Case Study: The Razorfish Project Project Goal: translate 4 million lines of Cobol code to SAP/R3 A tool has been developed to achieve the goal of this project. Goal of the test process: (a) Test the generated code, not the tool. (b) Reduce the number of errors by about 85%.

32. 32 Razorfish Project Test Process output 1 run output 2 Transformer S SAP R/3 S Cobol Select a Test Profile input continue testing yes modify = no

33. 33 Razorfish Project: Results (intermediate) 85% reduction achieved. If the process parameters are not altered then the goal is reached in about 35 weeks. Prediction using feedback Prediction using the model Project data Expected behavior

34. 34 Alternatives from Feedback: STP Quality Desired eigenvalue=-0.152 Improving quality alone will not help in achieving the goal.

35. 35 Alternatives from Feedback: Workforce Desired eigenvalue=-0.152 Changing the workforce alone can produce the desired results.

36. 36 Alternatives from Feedback: STP quality and workforce Set of valid choices for changing the quality and the workforce

37. 37 Razorfish Project Results (final) The project was completed in 32 weeks. The model predicted 85% error reduction in 35 weeks.

38. 38 The incremental development process: Queuing model and control Miller et al. COMPSAC 2008

39. 39 Interconnected dynamical systems * C1C2C4 C3 uy C: Component (sub-system) u: System input y: System output x: System state vector * DeCarlo and Saeks, 1980

40. 40 Incremental software development: Components R R: Requirements

41. 41 Component data dependencies

42. 42 System of systems model

43. 43 System of systems model L11: Linear feedback from sub-system system outputs to sub-system inputs L12: Entry of external inputs into subsystem inputs L21, L22: Similar relationships for system outputs

44. 44 Workforce productive capability : productivity measure : workforce size : remaining work items : process quality : calibrated average productive capability/FTE

45. 45 Productivity state model

46. 46 Productivity state model: behavior (Csikszementmihalyi, 1988) Cangussu et al. [2002]

47. State model of a queue a1a1 a2a2 b1b1 b2b2 b3b3 (x 1 ) (x 2 ) (x 3 ) 47

48. Coordination constraints x: features coded c(x): step function; test case execution threshold Cubic splines used for smoothing the threshold function. 48

49. Additional models Defect detection; failure analysis: Lotka-Volterra predator-prey population dynamics model 49 Defect introduction: Based on Barry Boehm’s COQUALMO model Proportional splitting model: defects in code versus defects in test case code

50. Simulation study: System description Incremental development process Two serial increments First increment: 10 features, 70 tests, 70 regression tests Second increment: 20 features, 130 tests, 130 regression tests 29 FTE: 5 assigned to each development activity except feature and test correction that are assigned 5 each. 50

51. Simulation study: Sample results 51

52. Simulation study: Sample results 52

53. 53 Summary Analogy between physical and software systems presented. The notion of feedback control of software processes introduced. Two case studies described. Parameter estimation techniques used for model calibration. Made use of system identification techniques.

54. 54 Challenges Control objects: humans Time constants: large; process changes are slow Parameter estimation: need for data collection; must account for the impact of changes in the development environment Tool: needed to assist the manager evaluate multiple change scenarios

55. 55 Thank you!