1. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 1 SMU CSE 8314 / NTU SE 762-N Software Metrics and Quality Engineering Module 22 Principles of Measurement

2. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 2 Contents Introduction Some Principles of Measurement Theory Issues with Measuring Software

3. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 3 Risk Management Plan Execute Measure Engineer Quality Metrics in the Larger Picture Metrics are used to Monitor & Monitoring is used to Manage Risk

4. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 4 Introduction

5. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 5 Metrics are Powerful Because of the power of metrics they can tell you a lot But it is easy to misuse them Proper use of metrics requires understanding of some basic rules and principles

6. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 6 Data Analysis Information Why Measure? Every measurement should have a purpose – You want to get information

7. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 7 But for Every Analysis there are Two Possible Results Information - tells you something right – We are (or are not) on schedule – Our risks are (or are not) under control Misinformation - tells you something wrong – We are (or are not) on schedule – Our risks are (or are not) under control And there will always be changes in the organization when you measure it

8. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 8 Key Issues Define how to interpret measurements – To form a basis of consistent analysis Choose consistent display or graphing techniques – So people know how to interpret the data We will address these throughout the next several modules, at several levels of detail

9. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 9 Some Principles of Measurement Theory (how to interpret measurements correctly)

10. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 10 Importance of Measurement “ When you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of a science.” Lord Kelvin, 1800’s

11. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 11 Why Study Measurement Theory The theory of measurement offers us principles that we should be careful not to violate We will consider several of them here. First we will consider some observations about software engineering and software metrics by (somewhat) neutral observers

12. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 12 “Software Engineering is Still an Aspiration...... because Computer Science is not yet a science” Ruth Ravenel, U. of Colorado, Dept of Electrical and Computer Engineering, 1995

13. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 13 “Lemmingengineering” “The process of engineering systems by blindly following techniques the masses are following, without regard to the appropriateness of those techniques.” Alan Davis, IEEE Software, 9/93.

14. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 14 Failure to Use Measurement Theory “Despite the fact that the basis for software metrics lies in measurement theory, it has been largely ignored by both practitioners and researchers. The result is that much work in software metrics is theoretically flawed.” Norman Fenton, IEEE Transactions on Software Engineering, 3/94

15. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 15 Measurement is...... the process by which numbers and symbols are assigned to attributes of real world entities so as to describe them according to defined rules – The assignment of numbers must preserve intuitive and empirical observations about the attributes and entities

16. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 16 Preservation of Attributes Example “House A is bigger than House B” is a meaningful statement only if the number assignment of “size” preserves our intuitive notion of houses and their sizes. House AHouse B

17. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 17 But Intuitions Vary Is “size” defined by area? – or by number of rooms? – or by the cost to construct? We must define a model that reflects a specific viewpoint before we measure. – The model must specify an entity to be measured and an attribute of that entity. – I.e., what do you want to measure and what do you want to know about it?

18. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 18 Four Issues 1) The Properties of Numbers 2) Are Means Meaningful? 3) The Problem of Small Sample Size 4) Are the Variables Independent?

19. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 19 1) Properties of Numbers ScaleYesterdayToday Centigrade018 Fahrenheit3264 Is it twice as hot today as it was yesterday? The properties of the number system may not necessarily apply to the attribute being measured Consider temperature:

20. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 20 Twice as... “Twice as” is a meaningful concept for real numbers It is probably not a meaningful concept for temperature The error we make is assuming that properties of the number system apply to the thing being measured

21. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 21 Some Types of Scales Nominal: something is or is not X Ordinal: there is a sequence Interval: there is a known distance between consecutive members of a sequence Ratio: scalar multiplications are meaningful Absolute: all mathematical operations are meaningful

22. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 22 Example - Assigning a Scale to Test Failures {Blue, Green, Yellow, Red}. This is only an ordinal scale – It makes no sense to add, subtract, multiply or divide the values. – The difference between “red” & “yellow” may be un-comparable to the difference between “yellow” and “green”

23. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 23 But Suppose we Replace with a Numeric Scale 4 = Blue3 = Green 2 = Yellow1 = Red We are tempted to make statements like these: “The average test error improved from 2.2 to 3.1” “The average test error improved by 47%”

24. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 24 Other Examples My code is 10% smaller than yours – But what about the languages, the comments, the clarity, the performance, etc.? The average response from our customers is “good” [on a scale of very poor, poor, good, very good] – but the scale is not an interval or ratio scale, so what does “average” mean? – Does “half very good and half poor” mean “good”?

25. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 25 And Old Favorite I don’t know about that This year you will get a 10% pay cut But next year you will get a 20% pay raise So it will be like giving you 5% raises for two years

26. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 26 2) Are Means Meaningful? Consider some well known examples: “The average family has 2.4 children” “The average worker is 63% male and 37% female” “The average car has 3.4 customer complaints in the first 3 months of ownership” The mean or average is a statistical concept that may have no meaning in a real situation

27. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 27 Meaningless Means If my family is average, then I have 2.4 children I will buy 2.4 sets of clothing for the children I have made my 3 complaints but haven’t made the.4 yet That average employee must be an interesting medical specimen

28. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 28 Another Meaningless Mean Half of the people think we should turn left And half of the people think we should turn right So we will average and go straight ahead

29. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 29 Meaningful Means Clearly, averages have some statistical validity and can be useful in some situations, such as: – Determining how big to make schools – Evaluating child health care costs – Comparing cars for reliability – Evaluating diversity in the workplace But clearly they also have no validity in other situations

30. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 30 3) The Problem of Small Sample Size Suppose you have a large population and you want to determine its properties by selecting a “typical” sample population of size “n”. What conclusions can you draw from this sample? How reliable are those conclusions?

31. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 31 Small n vs. Large n Many statistical properties only apply to “large n”, where individual quirks can be smoothed. – A sample size of n < 17 is generally considered “small” For many of cases, n must be much larger than this WHY? Because individual items have undue impact on the results when n is small.

32. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 32 Example “40% of the students have blonde hair” Suppose your population size is 1000 If your sample size is 100, and 40 of them are blonde, this is a reasonable conclusion If your sample size is 5 and 2 have blonde hair, this is a much less reliable conclusion

33. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 33 Misuse of Statistics for Small Sample Sizes “We measured 10 programs and concluded that our typical program has 23.7 defects per 1000 lines of code” Statistically speaking, can you draw a meaningful conclusion from only 10 programs? What percent is this of the total population? 100%? 10%? 1%?

34. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 34 4) Are the Variables Independent? Many standard statistical manipulations assumeindependent variables But many software engineering situations have variables that influence each other and thus are dependent

35. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 35 Example: Comparing A and B FactorRating ARating B Clarity of Code2.52.2 Complexity of Code3.43.0 Size of Code2.02.6 Total7.97.8 A is “better”

36. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 36 Dangers of Ignoring Measurement Theory We attach undue credibility to numbers that may be meaningless or at least much less meaningful than we think they are We delude ourselves into thinking we have a sound basis for decisions We may reach wrong conclusions because we misunderstand what the numbers tell us

37. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 37 Issues with Measuring Software

38. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 38 Issues with Software Software is not bound by the laws of physics or hardware constraints – Be careful not to rely on hardware metrics where the theory assumes limits to physical behavior E.g., if you increase input by.0001%, output changes by 10000000000%. Rarely possible in hardware, but very easy in software.

39. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 39 Issues with Software (continued) Many software products are NOT code – specifications – tests – user guides – etc. If you only measure the code, you will probably not really understand your software or its development process

40. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 40 Summary Learn the principles of measurement theory Understand what attribute you are measuring before you start to measure Don’t assume the properties of the number system apply to the attribute being measured Beware of misuse of means Beware of small “n” Beware of dependent variables

41. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 41 References Department of Defense, Joint Logistics Commanders Joint Group on Systems Engineering, Practical Software Measurement, a Guide to Objective Program Insight (version 2.1), Naval Undersea Warfare Center, c/o John McGarry, mcgarry@ada.npt.navy.mil. Fenton, Norman E. Software Metrics: A Rigorous Approach, Chapman & Hall, London SE1 8HN, 1991. ISBN 0-442-31355-1.

42. CSE 8314 - SW Metrics and Quality Engineering Copyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M22 8/20/2001Slide 42 END OF MODULE 22