1. CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved CSE7315M27 Slide 1 January 7, 2006 SMU CSE 7315 / NTU SE 584-N Planning and Managing a Software Project Module 27 Project Tracking and Oversight

2. CSE7315M27 Slide # 2 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Objectives of This Module To present an overview of tracking and oversight To discuss some basic terminology regarding measurement

3. CSE7315M27 Slide # 3 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Tracking and Oversight Occur Continuously Manage Risks Define the Approach Generate Detailed Plans Understand the Need Execute and Monitor text, chapters 6, 15

4. CSE7315M27 Slide # 4 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved SW Project Monitoring Typical Symptoms of a Problem We’re six months behind schedule, and nobody knew it! Why did it take us so long to find out? January

5. CSE7315M27 Slide # 5 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved SW Project Monitoring Symptoms of Another Problem They’ve been working on that module for eight weeks and everyone else is waiting for it. Did the developers in charge know how many people depend on the module?

6. CSE7315M27 Slide # 6 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved SW Project Monitoring Other Examples The project manager promised a new feature to the customer -- but never told any of the programmers! – “I thought you knew about this!” The software takes up too much disk space. – Nobody ever thought it would get that big.

7. CSE7315M27 Slide # 7 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved SW Project Tracking and Oversight Purpose To provide adequate visibility into actual progress so that management can take effective actions when the software project’s performance deviates significantly from the software plans

8. CSE7315M27 Slide # 8 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved SW Tracking and Oversight Goals from the SEI CMM/CMMI 1) Actual results and performance are tracked against software plans – Plans are revised to reflect actual performance and changes in requirements or commitments 2) Corrective actions are taken and managed to closure when actual performance deviates significantly from software plans

9. CSE7315M27 Slide # 9 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved 3PM Today SW Tracking and Oversight Goals from the SEI CMM/CMMI 3) Changes to software commitments are communicated to all affected groups and individuals

10. CSE7315M27 Slide # 10 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved SW Project Tracking & Oversight Practices Recommended by SEI Use a software development plan for tracking and communicating status Track schedule, size, effort, computer resources, technical activities and risks Hold periodic reviews and take corrective actions Revise plans and schedules to reflect changes -- using a defined procedure Review customer commitments on a regular basis

11. CSE7315M27 Slide # 11 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Tracking Progress

12. CSE7315M27 Slide # 12 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Things You Can Estimate and Track Costs Sizes Quality Reliability Schedules Staffing etc.

13. CSE7315M27 Slide # 13 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Establish a Data Base Know about your organization – Performance on past projects – Lessons Learned Know about your industry and competitors – What is best in class? – Improvement rates Historical Data Base - Data - Lessons - etc. Facts to help you manage

14. CSE7315M27 Slide # 14 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Example of Experience vs. History History: for C++ doing your kind of software, you should be generating – 25 lines of code per day during the coding phase, with – 3 errors per 1000 lines of code during module test Actual experience on your project: – 40 lines of code per day, with – 0.5 errors per 1000 lines of code during module test

15. CSE7315M27 Slide # 15 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Optimist’s Conclusion Do you have a solid reason to explain this difference? Ask questions. Why are you better? – Is the process different? – Are the people a lot better? – Are the tools better? We are doing much better than in the past!

16. CSE7315M27 Slide # 16 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Pessimist’s Conclusion Ask questions. Find out what is really happening. – Are the tests being performed? – Is the coverage adequate? – Are there higher rates of customer complaints after shipment? Our testing is no good (perhaps because it is being rushed due to deadlines)

17. CSE7315M27 Slide # 17 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Knowing the Competition Can Give You Insights But what if the norm in your industry is an improvement of 25%? And what if your competitors have all switched to Java and are 50% more productive as a result? We improved our “C” language productivity by 15%

18. CSE7315M27 Slide # 18 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Measurement

19. CSE7315M27 Slide # 19 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Every Measure Should Have a Purpose -- You Want to Get Information Data Analysis Information

20. CSE7315M27 Slide # 20 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved But for Every Analysis there are Two Possible Results Information - tells you something right – We are (or are not) on schedule – Our risks are (or are not) under control Misinformation - tells you something wrong – We are (or are not) on schedule – Our risks are (or are not) under control And there will always be changes in the organization when you measure it

21. CSE7315M27 Slide # 21 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Key Issues Define how to interpret measurements – To form a basis of consistent analysis Choose consistent display techniques – So people know how to interpret the data Define how to use each measurement – You must also demonstrate that you are using it that way, so people will believe you – Given any measure, people will change to make it look to their advantage you want to make their behavior change in a positive way

22. CSE7315M27 Slide # 22 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Organizational Framework There must be an Organizational Framework for understanding the importance of measurements – i.e., people do not sabotage the data collection effort – And people do not abuse measures – And people do not draw wrong conclusions

23. CSE7315M27 Slide # 23 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Achieving an Effective Organizational Framework Educate everyone in the proper use of measurements Develop the right measures – Involve those who are being measured – Measure only what you can benefit from Use the measurements – To make decisions about the product and the process – But NOT to make decisions about people

24. CSE7315M27 Slide # 24 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved You Want to Achieve Optimal Performance Don’t over-measure or under-measure Don’t over-test or under-test Don’t over-inspect or under-inspect etc. Track the things that represent your greatest risks and concerns Remember that it costs time and money to track - make it worthwhile Track the things that represent your greatest risks and concerns Remember that it costs time and money to track - make it worthwhile

25. CSE7315M27 Slide # 25 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Basic Definition and Terminology Issues Levels of Measurement What to Measure Who Cares about What Measures

26. CSE7315M27 Slide # 26 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Levels of Measurement What do we mean by a measure? Does everyone mean the same thing? How do we resolve discrepancies? Consider the example on the next slide:

27. CSE7315M27 Slide # 27 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Information may be Measured in Many Ways! Cost is best measured in staff days! Why not dollars ? I measure it by drop in stock price. However they measure it, they had better figure out how to reduce cost!

28. CSE7315M27 Slide # 28 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Information, Measures and Data Process in Execution Data Information Need Measure

29. CSE7315M27 Slide # 29 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Example Process in Execution Units Produced Head- count Lines of Code $ SpentMonths Productivity Units Per Month LOC per Staff Month $ per Line of Code

30. CSE7315M27 Slide # 30 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Definitions: Data Data (or Primitive Measures) The fundamental, factual quantities that characterize a process or product – Specific facts that are countable or otherwise obtainable – Minimum amount of analysis

31. CSE7315M27 Slide # 31 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Key Issues: Data Consistent definitions Effective collection Examples: – Hours worked – Number of employees – Lines of code

32. CSE7315M27 Slide # 32 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Consistent Definitions What do you mean by a “day” of work? Does everyone mean the same thing? Such questions should be asked about any data you propose to collect, such as lines of code, people, dollars, defects, variance, tests completed, complaints You must define what you want … -- How many hours per day? -- What is a “line” of code? -- Which lines to include in LOC? You must define what you want … -- How many hours per day? -- What is a “line” of code? -- Which lines to include in LOC?

33. CSE7315M27 Slide # 33 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved How Long Is a Staff Day? Salaried Staff Staff Day OvertimeRegular Time Hourly StaffSalaried StaffHourly Staff UnpaidPaidUnpaidPaid

34. CSE7315M27 Slide # 34 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Effective Collection Can you collect accurate and consistent data? Can you collect it efficiently? How will the organization and the process change when you collect the data?

35. CSE7315M27 Slide # 35 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Definitions: Measures Measure (sometimes called Compound Measure) The result of counting or otherwise quantifying an attribute of a process or product – How we quantify – Something we can analyze and interpret – Generally associated with specific graphs and rules of interpretation

36. CSE7315M27 Slide # 36 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Key Issues: Measures Proper interpretation How measures are displayed Effective use of measures Examples: – lines of code per hour – ratio of current to historical test performance – turnover rate

37. CSE7315M27 Slide # 37 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Display and Interpretation: Consider the Following Graph

38. CSE7315M27 Slide # 38 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Effective Use You must define how you will use each measure – A measure can typically be used many ways – Some of these ways can be ineffective or even harmful – People tend to fear measurement because of past experience with misuse You must also demonstrate that you are using it as planned, so people will believe you Given any measure, people will change to make it look to their advantage – So you want to make their behavior change in a positive way

39. CSE7315M27 Slide # 39 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Example: A Measure & Its Impact Information Need: Productivity Measure 1: Lines of code per day Use: reward those who produce the most lines of code per day Result: people may produce bloated, inflated code in order to look good Measure 2: requirements met and tested, weighted by complexity of requirement Use: track against history and use to identify process bottlenecks Result: people may use the data to make the process more efficient, resulting in lower cost

40. CSE7315M27 Slide # 40 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved A Less Common Measure That May be Very Effective Information Need: Productivity Measure 3: Number of customer complaints Use: reward those who produce the code that is responsible for the fewest customer complaints Result: developers may pay particular care to the customer’s need, resulting in future sales and business success

41. CSE7315M27 Slide # 41 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Definitions: Information Information A measured quantity that provides insight into a software issue or concept or goal. – What and why we measure – An indicator – A standard of measurement that ties to a goal or purpose

42. CSE7315M27 Slide # 42 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Key Issues: Information Selecting a measure that provides the desired insight or information regarding the goal or purpose Examples: – Comparisons (planned vs. actual) – Indices (performance ratio, productivity, return on investment)

43. CSE7315M27 Slide # 43 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Focusing on the Goals and Needs Customer satisfaction can be measured in terms of response to a survey that biases customer response and hides their real views Or it can be measured in terms of something that accurately reflects customer views The same applies to measures of productivity, quality, cycle time, defect density, etc. Each measure should provide information that addresses some higher level objective, such as “reducing cost” or “on time delivery”

44. CSE7315M27 Slide # 44 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Select the Right Measure There are many ways to measure a desired quantity and different organizations measure differently The key is to spend time to select the right measure, so you satisfy the information needs in support of the higher level objective(s)

45. CSE7315M27 Slide # 45 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Good and Not So Good Measures Goal: Produce software more efficiently Information Need: Efficiency Measure 1: tests completed per week Result: easy tests done first; corners cut in testing; hard problems ignored or deferred Measure 2: rework Result: process and methods are improved to reduce rework, resulting in more efficient software development

46. CSE7315M27 Slide # 46 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Rework Is an Excellent Measure of Efficiency or Productivity It tells you where you are wasting resources However … Rework is a lagging indicator. – It does not spot problems in advance

47. CSE7315M27 Slide # 47 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved What Should We Measure? ProductProjectProcess determines success of determines quality of root causes Process Measures – Effectiveness of the process – How well are we following the process? – Risk monitoring Product Measures – Performance and quality – How well is the product meeting its requirements? Project Measures – The state of the project – How are we doing relative to cost, schedule, staffing, etc.?

48. CSE7315M27 Slide # 48 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved ProductProjectProcess Attributes What Resources Quality Time Are We On Schedule? Expenses vs. Budget? How Fast can we Manufacture? What Is our Cycle Time? Post-release Defects? What will it Cost? What is our Productivity? Customer Satisfaction? In-process Defects? Performance Meets Perfor- mance Goals? Meets Mgt. Goals? Does it Work? What Attributes Can We Measure? We want attributes that relate to our goals – time, resources, performance, quality etc. The following type of matrix can help:

49. CSE7315M27 Slide # 49 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Who Cares About What? Managers usually care about project measures - that’s how they are judged -- But if the project is in trouble they need to know more Developers usually care about product measures -- that’s what they are evaluated by Both should care about process measures -- this is usually where you learn the reasons why a project or a product is in trouble

50. CSE7315M27 Slide # 50 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Summary of Module Measure to know what is really happening Avoid misinterpreting the data Define how to measure, interpret, and display the data There are many ways to measure a given quantity – choose one that works best for you

51. CSE7315M27 Slide # 51 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved References Baumert, John H., and Mark S. McWhinney, Software Measures and the Capability Maturity Model, CMU/SEI-92-TR-25, ESC-TR-92-025, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pa., 1992. DeMarco, Tom, Controlling Software Projects: Management, Measurement, and Estimation, New York, Yourdon Press, 1982. Grady, Robert B. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, N.J., Prentice-Hall, Inc., 1992. ISBN 0-13- 720384-5.

52. CSE7315M27 Slide # 52 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved Possible Exam Questions  Explain why tracking and oversight are important for project management  Explain the difference between an information need and a measure.  Explain several risks with defining and collecting data properly

53. CSE7315M27 Slide # 53 January 7, 2006 CSE 7315 - SW Project Management / Module 27 - Project Tracking and Oversight Copyright © 1995-2006, Dennis J. Frailey, All Rights Reserved END OF MODULE 27