1. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 1 1/11/2004 Day 4, Part 3 Software Tracking and Oversight

2. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 2 1/11/2004 Outline Overview Tracking Progress Measurement Selecting Measures - The Goal / Question / Metric Paradigm Recommended Measures for Tracking and Oversight Measurement Issues Corrective Action

3. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 3 1/11/2004 SW Project Tracking and Oversight Typical Symptoms of a Problem We’re six months behind schedule, and nobody knew it! Why did it take us so long to find out? January

4. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 4 1/11/2004 SW Project Tracking and Oversight Symptoms of Another Problem They’ve been working on that module for eight weeks and everyone else is waiting for it. Did the developers in charge know how many people depend on the module?

5. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 5 1/11/2004 SW Project Tracking and Oversight Other Examples The project manager promised a new feature to the customer -- but never told any of the programmers! –“I thought you knew about this!” The software takes up too much disk space. –Nobody ever thought it would get that big.

6. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 6 1/11/2004 SW Project Tracking and Oversight Purpose To provide adequate visibility into actual progress so that management can take effective actions when the software project’s performance deviates significantly from the software plans

7. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 7 1/11/2004 SW Tracking and Oversight Goals from the SEI CMM 1) Actual results and performance are tracked against software plans –Plans are revised to reflect actual performance and changes in requirements or commitments 2) Corrective actions are taken and managed to closure when actual performance deviates significantly from software plans

8. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 8 1/11/2004 3PM Today SW Tracking and Oversight Goals from the SEI CMM 3) Changes to software commitments are communicated to all affected groups and individuals

9. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 9 1/11/2004 SW Project Tracking & Oversight Practices Recommended by SEI CMM Use a software development plan for tracking and communicating status Track schedule, size, effort, computer resources, technical activities and risks Hold periodic reviews and take corrective actions Revise plans and schedules to reflect changes -- using a defined procedure Review customer commitments on a regular basis

10. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 10 1/11/2004 Tracking Progress

11. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 11 1/11/2004 Things you can Estimate and Monitor Costs Sizes Quality Reliability Schedules Staffing etc.

12. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 12 1/11/2004 Establish a Data Base Know about your organization –Performance on past projects –Lessons Learned Know about your industry and competitors –What is best in class? –Improvement rates Historical Data Base - Data - Lessons - etc. Facts to help you manage

13. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 13 1/11/2004 Example of Experience vs. History History: for C++ doing your kind of software, you should be generating –25 lines of code per day during the coding phase, with –3 errors per 1000 lines of code during module test Actual experience on your project: –40 lines of code per day, with –0.5 errors per 1000 lines of code during module test

14. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 14 1/11/2004 Optimist’s Conclusion Do you have a solid reason to explain this difference? Ask questions. Why are you better? –Is the process different? –Are the people a lot better? –Are the tools better? We are doing much better than in the past!

15. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 15 1/11/2004 Pessimist’s Conclusion Ask questions. Find out what is really happening. –Are the tests being performed? –Is the coverage adequate? –Are there higher rates of customer complaints after shipment? Our testing is no good (perhaps because it is being rushed due to deadlines)

16. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 16 1/11/2004 Knowing the Competition Can Give You Insights But what if the norm in your industry is an improvement of 25%? And what if your competitors have all switched to Java and are 50% more productive as a result? We improved our “C” language productivity by 15%

17. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 17 1/11/2004 Measurement

18. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 18 1/11/2004 Data Analysis Information Basic Reasons for Measurement Every measurement should have a purpose –You want to get information

19. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 19 1/11/2004 But for Every Analysis there are Two Possible Results Information - tells you something right –We are (or are not) on schedule –Our risks are (or are not) under control Misinformation - tells you something wrong –We are (or are not) on schedule –Our risks are (or are not) under control And there will always be changes in the organization when you measure it

20. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 20 1/11/2004 Key Issues Define how to interpret measurements –To form a basis of consistent analysis Choose consistent display techniques –So people know how to interpret the data Define how to use each measurement –You must also demonstrate that you are using it that way, so people will believe you –Given any measure, people will change to make it look to their advantage you want to make their behavior change in a positive way

21. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 21 1/11/2004 Typical Graph of a Measure

22. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 22 1/11/2004 Organizational Framework There must be an Organizational Framework for understanding the importance of measurement –i.e., people do not sabotage the data collection effort –And people do not abuse measurements –And people do not draw wrong conclusions

23. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 23 1/11/2004 Achieving an Effective Organizational Framework Educate everyone in the proper use of measurements Develop the right measures –Involve those who are being measured –Measure only what you can benefit from Use the measurements –To make decisions about the product and the process –But NOT to make decisions about people

24. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 24 1/11/2004 You Want to Achieve Optimal Performance Don’t over-measure or under-measure Don’t over-test or under-test Don’t over-inspect or under-inspect etc. Track the things that represent your greatest risks and concerns Remember that it costs time and money to track - make it worthwhile Track the things that represent your greatest risks and concerns Remember that it costs time and money to track - make it worthwhile

25. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 25 1/11/2004 Example: A Measure & Its Impact Information Need: Productivity Measure 1: Lines of code per day Use: reward those who produce the most lines of code per day Result: people may produce bloated, inflated code in order to look good Measure 2: requirements met and tested, weighted by complexity of requirement Use: track against history and use to identify process bottlenecks Result: people may use the data to make the process more efficient, resulting in lower cost

26. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 26 1/11/2004 Good and Not So Good Measures Goal: Produce software more efficiently Information Need: Efficiency Measure 1: tests completed per week Result: easy tests done first; corners cut in testing; hard problems ignored or deferred Measure 2: rework Result: process and methods are improved to reduce rework, resulting in more efficient software development But rework is a lagging indicator - it does not spot problems in advance

27. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 27 1/11/2004 What Should we Measure? ProductProjectProcess determines success of determines quality of root causes Process Measures –Effectiveness of the process –How well are we following the process? –Risk monitoring Product Measures – Performance and quality – How well is the product meeting its requirements? Project Measures – The state of the project – How are we doing relative to cost, schedule, staffing, etc.?

28. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 28 1/11/2004 ProductProjectProcess Attributes What Resources Quality Time Are We On Schedule? Expenses vs. Budget? How Fast can we Manufacture? What Is our Cycle Time? Post-release Defects? What will it Cost? What is our Productivity? Customer Satisfaction? In-process Defects? Performance Meets Perf. Goals? Meets Mgt. Goals? Does it Work? What Attributes can we Measure? We want attributes that relate to our goals –time, resources, performance, quality etc. The following type of matrix can help:

29. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 29 1/11/2004 Managers -- project measures –That’s what they are measured by –But if the project is in trouble they need to know more Developers -- product measures –That’s what they are measured by Both should care about process measures –This is usually where you learn the “root causes” of problems Who Cares about What?

30. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 30 1/11/2004 Selecting Measures: The Goal/Question/Measure Paradigm (see Basili in reference list)

31. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 31 1/11/2004 GOAL –What do we want to accomplish QUESTIONS –The answers tell us whether we are meeting the goals or help us understand how to meet them MEASURES –Measurements that answer the questions The Goal/Question/Measure Paradigm

32. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 32 1/11/2004 Steps of the GQM Paradigm 1) Determine your goals -- and prioritize them –Don’t select too many –Make sure they are your real goals 2) For each goal, determine what questions would help you understand whether you are achieving the goal or how to achieve it

33. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 33 1/11/2004 Steps of the GQM Paradigm 3) For each question, determine if it can be answered by measurement –If so, determine what can be measured –What it would cost to measure in terms of difficulty in measurement, effect on process etc –The side effects of the measurement 4) Select the measures with the best payoff

34. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 34 1/11/2004 Example: GQM Approach Goal: Improve Efficiency by Reducing the Cost of Rework Question 1: What does rework cost us? Question 2: How many defects are in the software when we release it? Question 3: What are the sources of defects and rework?

35. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 35 1/11/2004 Possible Measures for Question 2 This is impossible to know But can be approximated with other measures 1) Number of Known Defects at release incomplete, but perhaps correlated to the total 2) Number of defects detected after release 3) Total defects over the life of the product

36. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 36 1/11/2004 Analyzing Defect Data Release12 Months Retirement Total Defects Known at Release Known after 12 Months Total Defects in Product Life

37. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 37 1/11/2004 Measures are Not Goals !! If the measure becomes the goal, you usually end up at the wrong place (example: lines of code per day) The measure tells you how you are doing, but there may be other measures that tell you other aspects of how you are doing You Get What You Measure!

38. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 38 1/11/2004 Prioritize Goals You cannot always measure everything Determine which goals are most important Also make sure that the goals are consistent Are people getting consistent direction?

39. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 39 1/11/2004 It Helps to Have a Vision Vision Key Features or Goals Prioritized List of Goals Focus Measures on Top Priority Goals 1. Learn to Sail 2. Select the Best Beer 3. Save $ 1. Save $ 2. Learn to Sail 3.....

40. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 40 1/11/2004 Forms of Payoff for Measurement i.e., Which Measures to Use Low cost of collection and storage Minimal impact of collection (morale, disruption, etc.) High information communicated One measure answers multiple questions One measure supports multiple goals

41. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 41 1/11/2004 Look for Questions and Measures that Do Dual Duty Goal 1Goal 2 Q1Q2Q3Q4Q5 M1M2M3M4M5M6M7

42. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 42 1/11/2004 Other Attributes of Good Measures Completeness –Addresses all issues of concern Benefit is high relative to costs of collection, storage and analysis All parts of the organization benefit Alternative measures are available if primary choices are not available

43. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 43 1/11/2004 Tie Measures to the Process -- A Measure for Every Phase Phase Measure Require ments Test Plan ning Design Integr ation Coding Staffing X X XXX X Requirements Stability XXX Etc. X Design Complexity X Code Complexity

44. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 44 1/11/2004 Risk: “Running out of memory space” Possible Measures : – Predicted memory space used (update each week or month, depending on risk) – Size of code produced by compiler – Level of experience of average staff Tie Measures to Risks Any risk may result in measures for monitoring Choice depends on anticipated causes of risk

45. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 45 1/11/2004 Another Example Risk Measure Risk: “Not meeting schedule” Cause: Not enough experienced staff Possible Measures: Staffing level Unplanned turnover Average staff experience, etc.

46. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 46 1/11/2004 Recommended Measures

47. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 47 1/11/2004 1) Problem Reports Complaints help you improve Reports should be categorized by such properties of problems as: –Kind, severity, source (process step), effort to correct, time of discovery This can tell you where you need to focus improvement efforts Problem reports are a direct measure of customer satisfaction

48. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 48 1/11/2004 Problem Report Analysis

49. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 49 1/11/2004 Some Issues with Measuring Problem Reports How to handle duplicates How to handle different symptoms caused by the same root problem How to handle reports that are the result of multiple root problems Urgency vs. cost vs. other factors in deciding on priority

50. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 50 1/11/2004 2) Customer Satisfaction Has to be measured by the customer Surveys Degree of cooperation Do they buy your product? etc. Business survival depends on satisfying the customer better than the competition

51. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 51 1/11/2004 3) Requirements Stability TBDs + New + Changes Total Requirements

52. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 52 1/11/2004 Requirements Stability: Another Way to Display

53. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 53 1/11/2004 4) Rate Charts These tell you the rate at which something is happening - usually compared with some plan Measure something discrete that represents genuine progress toward the objectives of the project If you measure well, rate charts can give you a better handle on where you are than something like “hours spent”

54. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 54 1/11/2004 Rate Charts Example Completion vs. history and plans Today Deadline

55. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 55 1/11/2004 Staffing: actual vs. plan - I

56. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 56 1/11/2004 Staffing: actual vs. plan - II

57. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 57 1/11/2004 Staffing: actual vs. plan - III

58. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 58 1/11/2004 Requirements Implemented

59. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 59 1/11/2004 Another Variation for Iterative Lifecycles

60. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 60 1/11/2004 Rate Charts Can Be Used for Many Things Examples: -- Modules designed/coded/tested/integrated -- Requirements designed/coded/tested/integrated -- Problems corrected -- High priority complaints resolved

61. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 61 1/11/2004 Characteristics of Applications for Which Rate Charts Are a Good Choice Discrete Events Easily Measured Clear Definition of Completion Many Occurrences

62. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 62 1/11/2004 5) Resource Use Staffing on project Staff-hours worked Memory requirements for product CPU capacity required Amount of reused code You can predict beforehand and then measure actuals When actuals deviate from plan, you know that you need to act

63. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 63 1/11/2004 Example of Memory Use Measure

64. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 64 1/11/2004 Example of Reuse Measure

65. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 65 1/11/2004 The Estimation Spreadsheet Provides Initial Plan Data Staffing levels Headcount Dollars spent Size in Lines of Code Size in Memory Reuse etc.

66. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 66 1/11/2004 6) Defects Defects indicate fundamental process problems that are causing you to waste time and money and produce lower quality products But people resist measuring defects because they represent “bad news” It takes a cultural change, sometimes, to get defects viewed as a good thing to measure

67. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 67 1/11/2004 Kinds of Defects to Measure Any product/byproduct can have defects Requirements analysis defects –Collect at inspections and reviews Design defects –Collect at design walkthroughs, inspections, reviews Coding defects –Collect during peer reviews and tests Testing defects (defects in testing procedures)

68. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 68 1/11/2004 Evaluate problem reports Consider defect containment methods (we will cover this under “rework” in the advanced course) How to Measure Defects Keep track of them at inspections, walkthroughs, and reviews

69. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 69 1/11/2004 Measurement Issues

70. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 70 1/11/2004 Frequency How Often to Measure Frequency should depend on degree of risk and cost of measurement –Too often results in high cost and causes disruption of the process –Too seldom can result in failure to see problems soon enough

71. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 71 1/11/2004 Synchronization Getting the True Picture

72. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 72 1/11/2004 Update Estimates and Predictions

73. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 73 1/11/2004 Predictions Get Better as the Project Makes Progress Require- ments Analysis Design Code & Module Test SW Integ. & Test Released Software Predictions based on actual code Predictions based on process and design info Actuals known

74. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 74 1/11/2004 Margin A Frequently Misunderstood Term What does “leave 50% margin” mean? Code Margin 66 2/3 50 100 Code Margin

75. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 75 1/11/2004 Peak vs. Average “Software must be able to handle 100 screens per minute” Peak? (Worst Case) Average over some interval? How often to measure?

76. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 76 1/11/2004 Units Often Subject to Misinterpretation Memory Size: words, bytes, bits??? Program Execution Time: –Hours since shipped? {was it delivered?} –Hours since received by customer? {was it installed?} –Hours since installed? {was it used?} –CPU execution time since installed? CPU Time: cycles, instructions, MIPS, FLOPS, ? Beware of innocent misinterpretations

77. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 77 1/11/2004 Interplay of Resources Few Things Happen in Isolation Sometimes, system resources interact with each other, giving misleading evidence of performance Memory Capacity: 800 cycles per micro- second CPU 500 MIPS I/O Channel 200 MIPS Display 400 MIPS CPU may run at only 200 MIPS because of memory saturation

78. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 78 1/11/2004 Taking Corrective Action and Tracking to Closure

79. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 79 1/11/2004 Is This Enough? Identify problems during reviews Assign action items for each problem Document the problems and action items Assure that there are enough resources (for example, time) to follow through on the action items Send out reminders about action items Review the action items at the next meeting Continue until the actions are closed

80. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 80 1/11/2004 Summary Measure to know what is really happening Ask questions when results are inconsistent with experience or plans Beware of misinformation from faulty analysis Relate measurements to goals –Use the G/Q/M Paradigm But don’t let measures substitute for the real goals

81. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 81 1/11/2004 Summary (continued) Tie measures to the process Find measures that pay off –Dual duty –High benefit to cost ratio Make sure the people being measured support the data collection effort Rate charts are powerful tools for tracking and oversight

82. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 82 1/11/2004 Summary (concluded) Know the side effects and pitfalls of your measurements –How they will change behaviors –Proper units –Misuse –etc. Take corrective action when problems are identified Track action items to closure

83. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 83 1/11/2004 References - I Basili, Victor & David Weiss, “A Methodology for Collecting Valid Software Engineering Data,” IEEE Transactions on Software Engineering, Vol SE-10, no. 6, Nov., 1984, p728. Baumert, John H., and Mark S. McWhinney, Software Measures and the Capability Maturity Model, CMU/SEI-92-TR-25, ESC-TR-92-025, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pa., 1992. DeMarco, Tom, Controlling Software Projects: Management, Measurement, and Estimation, New York, Yourdon Press, 1982.

84. Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 4, Part 3, Page 84 1/11/2004 References - II Grady, Robert B. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, N.J., Prentice-Hall, Inc., 1992. ISBN 0-13- 720384-5. Paulk, Mark, et. al., Capability Maturity Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-2, February, 1993.