1. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 1 SMU CSE 8314 / NTU SE 762-N Software Measurement and Quality Engineering Module 22 Principles of Measurement Part 1

2. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 2 Contents Introduction Some Principles of Measurement Theory Issues with Measuring Software

3. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 3 Risk Management Plan Execute Measure Engineer Quality Measurement in the Larger Picture Monitoring is used to Manage Risk Measurements are used to Monitor

4. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 4 Introduction

5. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 5 Measurements are Powerful Because of the power of measurements they can tell you a lot But it is easy to misuse them Proper use of measurement requires understanding of some basic rules and principles

6. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 6 Data Analysis Information Why Measure? Every measurement should have a purpose – You want to get information

7. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 7 But for Every Analysis there are Two Possible Results Information - tells you something right – We are (or are not) on schedule – Our risks are (or are not) under control Misinformation - tells you something wrong – We are (or are not) on schedule – Our risks are (or are not) under control You need to make sure you know what the data are really telling you

8. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 8 Another Issue with Measurement Whenever you measure an organization, there will be changes in the organization. – It takes resources to measure, which causes things to be slightly less efficient – People change when they are measured so that they will look more favorable If you’re not careful, you may not measure the true situation

9. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 9 Key Issues Define how to interpret measurements – To form a basis of consistent analysis Choose consistent display or graphing techniques – So people know how to interpret the data We will address these throughout the next several modules, at several levels of detail

10. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 10 Some Principles of Measurement Theory (how to interpret measurements correctly)

11. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 11 Importance of Measurement “ When you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of a science.” Lord Kelvin, 1800’s

12. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 12 Why Study Measurement Theory The theory of measurement offers us principles that we should be careful not to violate We will consider several of them here. First we will consider some observations about software engineering and software measurement by (somewhat) neutral observers

13. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 13 “Software Engineering is Still an Aspiration...... because Computer Science is not yet a science” Ruth Ravenel, U. of Colorado, Dept of Electrical and Computer Engineering, 1995

14. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 14 “Lemmingengineering” “The process of engineering systems by blindly following techniques the masses are following, without regard to the appropriateness of those techniques.” Alan Davis, IEEE Software, 9/93.

15. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 15 Failure to Use Measurement Theory “Despite the fact that the basis for software metrics lies in measurement theory, it has been largely ignored by both practitioners and researchers. The result is that much work in software metrics is theoretically flawed.” Norman Fenton, IEEE Transactions on Software Engineering, 3/94

16. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 16 Metrology Metrology is the study of measurement Recently, software measurement experts have turned to metrology to bring order to software metrics One of their first conclusions was that the term “metrics” is not well defined As a result, there is a general movement away from use of the term “metrics” and toward standard metrology terminology Several ISO standards define the terminology

17. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 17 ISO Standards Related to Measurement ISO International Vocabulary of Basic and General Terms in Metrology (1993) – Defines basic terminology for measurement ISO/IEC 15939 Software Measurement Process (2002) – Defines the activities and tasks necessary to have a successful measurement program ISO/IEC 12207 Software Development Process (1995) – Defines terms and concepts associated with software development processes

18. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 18 Some Basic Terms Measure (singular noun): a variable to which a value is assigned as a result of measurement Data (plural noun): a collection of values assigned to measures Example: – Lines of code – a measure (of size) – 2345, 5432, 18432 – specific data indicating the size values of three different programs

19. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 19 More Basic Terms Attribute: a property or characteristic of an entity that can be distinguished quantitatively or qualitatively – Example: size Base Measure: a measure of an attribute – Example: lines of code Derived Measure: a measure defined in terms of other measures – Example: lines of code per month

20. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 20 Measurement is...... the process by which numbers and symbols are assigned to attributes of real world entities so as to describe them according to defined rules – The assignment of numbers must preserve intuitive and empirical observations about the attributes and entities

21. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 21 Preservation of Attributes Example “House A is bigger than House B” is a meaningful statement only if the number assignment of “size” preserves our intuitive notion of houses and their sizes. House AHouse B

22. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 22 But Intuitions Vary Is “size” defined by area? – or by number of rooms? – or by the cost to construct? We must define a model that reflects a specific viewpoint before we measure. – The model must specify an entity to be measured and an attribute of that entity. – I.e., what do you want to measure and what do you want to know about it?

23. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 23 Four Issues 1) The Properties of Numbers 2) Are Means Meaningful? 3) The Problem of Small Sample Size 4) Are the Variables Independent?

24. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 24 1) Properties of Numbers ScaleYesterdayToday Centigrade018 Fahrenheit3264 Is it twice as hot today as it was yesterday? The properties of the number system may not necessarily apply to the attribute being measured Consider temperature:

25. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 25 Twice as... “Twice as” is a meaningful concept for real numbers It is not a meaningful concept for temperature (See discussion of scales – next three slides) The error we make is assuming that properties of the number system apply to the thing being measured

26. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 26 Some Types of Scales (1 of 3) Nominal: a scale that places entities in categories – but without any ordering – Example: defects are design related coding related or requirements related. Ordinal: there is a ranking or ordering – Example: defect severity is minor significant major

27. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 27 Some Types of Scales (2 of 3) Interval: there is a fixed distance between consecutive members of a sequence, but multiplication is not meaningful – Example: the “McCabe Complexity” of a program. Can be any positive integer But if one program has complexity 3 and another has complexity 9, it does not mean the second one is 3 times as complex – Or consider the temperature example 2 slides back If it is 10 degrees one day and 30 the next, it is not “3 times as hot”

28. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 28 Some Types of Scales (3 of 3) Ratio: A fixed distance between consecutive sequence members, but multiplication is meaningful – Example: Size of a software program – A program with 900 lines of code is 3 times a big as one that is 300 lines of code Absolute: all mathematical operations are meaningful

29. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 29 Example - Assigning a Scale to Test Failures {Blue, Green, Yellow, Red} This is only an ordinal scale – It provides a ranking but nothing else – It makes no sense to add, subtract, multiply or divide the values. – The difference between “red” and “yellow” is not comparable to the difference between “yellow” and “green”

30. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 30 But Suppose we Replace with a Numeric Scale 4 = Blue3 = Green 2 = Yellow1 = Red We are tempted to make statements like these: “The average test error improved from 2.2 to 3.1” “The average test error improved by 47%”

31. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 31 Other Examples My code is 10% smaller than yours – But what about the languages, the comments, the clarity, the performance, etc.? The average response from our customers is “good” [on a scale of very poor, poor, good, very good] – but the scale is not an interval or ratio scale, so what does “average” mean? – Does “half very good and half poor” mean “good”?

32. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 32 And Old Favorite I don’t know about that This year you will get a 10% pay cut But next year you will get a 20% pay raise So it will be like giving you 5% raises for two years

33. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 33 2) Are Means Meaningful? Consider some well known examples: “The average family has 2.4 children” “The average worker is 63% male and 37% female” “The average car has 3.4 customer complaints in the first 3 months of ownership” The mean or average is a statistical concept that may have no meaning in a real situation

34. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 34 Meaningless Means If my family is average, then I have 2.4 children I will buy 2.4 sets of clothing for the children I have made my 3 complaints but haven’t made the.4 yet That average employee must be an interesting medical specimen

35. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 35 Another Meaningless Mean Half of the people think we should turn left And half of the people think we should turn right So we will average and go straight ahead

36. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 36 Meaningful Means Clearly, averages have some statistical validity and can be useful in some situations, such as: – Determining how big to make schools – Evaluating child health care costs – Comparing cars for reliability – Evaluating diversity in the workplace But clearly they also have no validity in other situations

37. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 37 3) The Problem of Small Sample Size Suppose you have a large population and you want to determine its properties by selecting a “typical” sample population of size “n”. What conclusions can you draw from this sample? How reliable are those conclusions?

38. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 38 Small n vs. Large n Many statistical properties only apply to “large n”, where individual quirks can be smoothed. – A sample size of n < 17 is generally considered “small” For many of cases, n must be much larger than this WHY? Because individual items have undue impact on the results when n is small.

39. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 39 Example “40% of the students have blonde hair” Suppose your population size is 1000 If your sample size is 100, and 40 of them are blonde, this is a reasonable conclusion If your sample size is 5 and 2 have blonde hair, this is a much less reliable conclusion

40. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 40 Misuse of Statistics for Small Sample Sizes “We measured 10 programs and concluded that our typical program has 23.7 defects per 1000 lines of code” Statistically speaking, can you draw a meaningful conclusion from only 10 programs? What percent is this of the total population? 100%? 10%? 1%?

41. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 41 4) Are the Variables Independent? Many standard statistical manipulations assume independent variables But many software engineering situations have variables that influence each other and thus are dependent

42. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 42 Example: Comparing A and B FactorRating ARating B Clarity of Code2.52.2 Complexity of Code3.43.0 Size of Code2.02.6 Total7.97.8 A is “better”

43. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 43 Dangers of Ignoring Measurement Theory We attach undue credibility to numbers that may be meaningless or at least much less meaningful than we think they are We delude ourselves into thinking we have a sound basis for decisions We may reach wrong conclusions because we misunderstand what the numbers tell us

44. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 44 Issues with Measuring Software

45. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 45 Issues with Software Software is not bound by the laws of physics or hardware constraints – Be careful not to rely on hardware measurement where the theory assumes limits to physical behavior E.g., if you increase input by.0001%, output changes by 10000000000%. Rarely possible in hardware, but very easy in software.

46. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 46 Issues with Software (continued) Many software products are NOT code – specifications – tests – user guides – etc. If you only measure the code, you will probably not really understand your software or its development process

47. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 47 Summary Learn the principles of measurement theory Understand what attribute you are measuring before you start to measure Don’t assume the properties of the number system apply to the attribute being measured Beware of misuse of means Beware of small “n” Beware of dependent variables

48. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 48 References Department of Defense, Joint Logistics Commanders Joint Group on Systems Engineering, Practical Software Measurement, a Guide to Objective Program Insight (version 2.1), Naval Undersea Warfare Center, c/o John McGarry, mcgarry@ada.npt.navy.mil. Fenton, Norman E. Software Metrics: A Rigorous Approach, Chapman & Hall, London SE1 8HN, 1991. ISBN 0-442-31355-1.

49. CSE 8314 - SW Measurement and Quality Engineering Copyright © 1995-2005, Dennis J. Frailey, All Rights Reserved CSE8314M22 version 5.09Slide 49 END OF MODULE 22