1. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 SMU CSE 8314 Software Measurement and Quality Engineering Module 22 Principles of Measurement Part 1

2. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 2 Contents  Introduction  Some Principles of Measurement Theory  Issues with Measuring Software

3. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 3 Risk Management Plan Execute Measure Engineer Quality Measurement in the Larger Picture Monitoring is used to Manage Risk Measurements are used to Monitor

4. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 Introduction

5. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 5 Measurements are Powerful  Because of the power of measurements they can tell you a lot  But it is easy to misuse them  Proper use of measurement requires understanding of some basic rules and principles

6. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 6 Data Analysis Information Why Measure?  Every measurement should have a purpose –You want to get information

7. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 7 But for Every Analysis there are Two Possible Results Information - tells you something right –We are (or are not) on schedule –Our risks are (or are not) under control Misinformation - tells you something wrong –We are (or are not) on schedule –Our risks are (or are not) under control You need to make sure you know what the data are really telling you

8. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 8 Another Issue with Measurement Whenever you measure an organization, there will be changes in the organization. –It takes resources to measure, which causes things to be slightly less efficient –People change when they are measured so that they will look more favorable If you’re not careful, you may not measure the true situation

9. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 9 Key Issues  Define how to interpret measurements –To form a basis of consistent analysis  Choose consistent display or graphing techniques –So people know how to interpret the data We will address these throughout the next several modules, at several levels of detail

10. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 10 Some Principles of Measurement Theory (how to interpret measurements correctly)

11. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 11 Importance of Measurement “ When you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of a science.” Lord Kelvin, 1800’s “ When you cannot measure, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind: it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of a science.” Lord Kelvin, 1800’s

12. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 12 Why Study Measurement Theory  The theory of measurement offers us principles that we should be careful not to violate  We will consider several of them here.  First we will consider some observations about software engineering and software measurement by (somewhat) neutral observers

13. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 13 “Software Engineering is Still an Aspiration...... because Computer Science is not yet a science” Ruth Ravenel, U. of Colorado, Dept of Electrical and Computer Engineering, 1995

14. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 14 “Lemmingengineering” “The process of engineering systems by blindly following techniques the masses are following, without regard to the appropriateness of those techniques.” Alan Davis, IEEE Software, 9/93. “The process of engineering systems by blindly following techniques the masses are following, without regard to the appropriateness of those techniques.” Alan Davis, IEEE Software, 9/93.

15. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 15 Failure to Use Measurement Theory “Despite the fact that the basis for software metrics lies in measurement theory, it has been largely ignored by both practitioners and researchers. The result is that much work in software metrics is theoretically flawed.” Norman Fenton, IEEE Transactions on Software Engineering, 3/94 “Despite the fact that the basis for software metrics lies in measurement theory, it has been largely ignored by both practitioners and researchers. The result is that much work in software metrics is theoretically flawed.” Norman Fenton, IEEE Transactions on Software Engineering, 3/94

16. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 16 Metrology  Metrology is the study of measurement  Recently, software measurement experts have turned to metrology to bring order to software metrics  One of their first conclusions was that the term “metrics” is not well defined  As a result, there is a general movement away from use of the term “metrics” and toward standard metrology terminology  Several ISO standards define the terminology

17. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 17 ISO Standards Related to Measurement  ISO International Vocabulary of Basic and General Terms in Metrology (1993) –Defines basic terminology for measurement  ISO/IEC 15939 Software Measurement Process (2002) –Defines the activities and tasks necessary to have a successful measurement program  ISO/IEC 12207 Software Development Process (1995) –Defines terms and concepts associated with software development processes

18. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 18 Some Basic Terms  Measure (singular noun): a variable to which a value is assigned as a result of measurement  Data (plural noun): a collection of values assigned to measures  Example: –Lines of code – a measure (of size) –2345, 5432, 18432 – specific data indicating the size values of three different programs

19. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 19 More Basic Terms  Attribute: a property or characteristic of an entity that can be distinguished quantitatively or qualitatively –Example: size  Base Measure: a measure of an attribute –Example: lines of code  Derived Measure: a measure defined in terms of other measures –Example: lines of code per month

20. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 20 Measurement is...... the process by which numbers and symbols are assigned to attributes of real world entities so as to describe them according to defined rules –The assignment of numbers must preserve intuitive and empirical observations about the attributes and entities

21. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 21 Preservation of Attributes Example “House A is bigger than House B” is a meaningful statement only if the number assignment of “size” preserves our intuitive notion of houses and their sizes. House AHouse B

22. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 22 But Intuitions Vary  Is “size” defined by area? –or by number of rooms? –or by the cost to construct?  We must define a model that reflects a specific viewpoint before we measure. –The model must specify an entity to be measured and an attribute of that entity. –I.e., what do you want to measure and what do you want to know about it? –Examples (next two slides)

23. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 23 Models for Size of a House A.In this model, the size of a house is based on how many people can comfortably live there  The size of a house is measured by the number of bedrooms and the number of bathrooms B.In this model, the size of a house is based on how much it will cost to construct it  The size of a house is measured by the square feet of living space

24. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 24 Models for Size of HTML Software Used for Web Sites A.In this model, the size of software is based on how much disk space it occupies  The size of software is measured by the number of bytes of memory it requires when stored on a disk B.In this model, the size of software is based on how much it will cost to construct it  The size of software is measured by the number of web pages

25. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 25 Four Issues that Must Be Understood when Measuring 1) The Properties of Numbers 2) Are Means Meaningful? 3) The Problem of Small Sample Size 4) Are the Variables Independent?

26. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 26 1) Properties of Numbers ScaleYesterdayToday Centigrade018 Fahrenheit3264 Is it twice as hot today as it was yesterday? The properties of the number system may not necessarily apply to the attribute being measured Consider temperature:

27. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 27 Twice as...  “Twice as” is a meaningful concept for real numbers  It is not a meaningful concept for temperature  (See discussion of scales – next three slides) The error we make is assuming that properties of the number system apply to the thing being measured

28. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 28 Some Types of Scales (1 of 3)  Nominal: a scale that places entities in categories – but without any ordering –Example: defects are  design related  coding related or  requirements related.  Ordinal: there is a ranking or ordering –Example: defect severity is  minor  significant  major

29. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 29 Some Types of Scales (2 of 3)  Interval: there is a fixed distance between consecutive members of a sequence, but multiplication is not meaningful –Example: the “McCabe Complexity” of a program. Can be any positive integer  But if one program has complexity 3 and another has complexity 9, it does not mean the second one is 3 times as complex –Or consider the temperature example 2 slides back  If it is 10 degrees one day and 30 the next, it is not “3 times as hot”

30. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 30 Some Types of Scales (3 of 3)  Ratio: A fixed distance between consecutive sequence members, AND multiplication is meaningful –Example: Size of a software program –A program with 900 lines of code is 3 times a big as one that is 300 lines of code  Absolute: all mathematical operations are meaningful

31. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 31 Example - Assigning a Scale to Test Failures {Blue, Green, Yellow, Red}  This is only an ordinal scale –It provides a ranking but nothing else –It makes no sense to add, subtract, multiply or divide the values. –The difference between “red” and “yellow” is not comparable to the difference between “yellow” and “green”

32. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 32 But Suppose we Replace with a Numeric Scale 4 = Blue3 = Green 2 = Yellow1 = Red We are tempted to make meaningless statements like these: “The average test error improved from 2.2 to 3.1” “The average test error improved by 47%”

33. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 33 Other Examples My code is 10% smaller than yours –But what about the languages, the comments, the clarity, the performance, etc.? The average response from our customers is “good” [on a scale of very poor, poor, good, very good] –but the scale is not an interval or ratio scale, so what does “average” mean? –Does “half very good and half poor” mean “good”?

34. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 34 An Old Favorite I don’t know about that This year you will get a 10% pay cut But next year you will get a 20% pay raise So it will be like giving you 5% raises for two years

35. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 35 2) Are Means Meaningful? Consider some well known examples: “The average family has 2.4 children” “The average worker is 63% male and 37% female” “The average car has 3.4 customer complaints in the first 3 months of ownership” The mean or average is a statistical concept that may have no meaning in a real situation

36. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 36 Meaningless Means  If my family is average, then I have 2.4 children  I will buy 2.4 sets of clothing for the children  I have made my 3 complaints but haven’t made the.4 yet  That average employee must be an interesting medical specimen

37. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 37 Another Meaningless Mean  Half of the people think we should turn left  And half of the people think we should turn right  So we will average and go straight ahead

38. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 38 Meaningful Means  Clearly, averages have some statistical validity and can be useful in some situations, such as: –Determining how big to make schools –Evaluating child health care costs –Comparing cars for reliability –Evaluating diversity in the workplace  But clearly they also have no validity in other situations

39. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 39 3) The Problem of Small Sample Size Suppose you have a large population and you want to determine its properties by selecting a “typical” sample population of size “n”.  What conclusions can you draw from this sample?  How reliable are those conclusions?

40. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 40 Small n vs. Large n  Many statistical properties only apply to “large n”, where individual quirks can be smoothed. –A sample size of n < 17 is generally considered “small”  For many of cases, n must be much larger than this WHY? Because individual items have undue impact on the results when n is small.

41. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 41 Example “40% of the professors at SMU have gray hair”  Suppose you examine 1000 professors and 400 have gray hair. Then this is probably a sound conclusion.  What if you examine 100 professors and 40 of them are gray. Is this a reasonable conclusion?  Suppose you examine 5 and 2 have gray hair? This is a much less reliable conclusion.

42. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 42 Possible Misuse of Statistics for Small Sample Sizes “We measured 10 programs and concluded that our typical program has 23.7 defects per 1000 lines of code”  Statistically speaking, can you draw a meaningful conclusion from only 10 programs?  What percent is this of the total population? 100%? 10%? 1%?

43. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 43 4) Are the Variables Independent?  Many standard statistical manipulations assume independent variables  But many software engineering situations have variables that influence each other and thus are dependent

44. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 44 Example: Comparing A and B FactorRating ARating B Clarity of Code2.52.2 Complexity of Code3.43.0 Size of Code2.02.6 Total7.97.8 A is “better”

45. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 45 Dangers of Ignoring Measurement Theory  We attach undue credibility to numbers that may be meaningless or at least much less meaningful than we think they are  We delude ourselves into thinking we have a sound basis for decisions  We may reach wrong conclusions because we misunderstand what the numbers tell us

46. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 46 Issues with Measuring Software

47. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 47 Issues with Software  Software is not bound by the laws of physics or hardware constraints –Be careful not to rely on hardware measurement where the theory assumes limits to physical behavior  E.g., if you increase input by.0001%, output changes by 10000000000%. Rarely possible in hardware, but very easy in software.

48. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 48 Issues with Software (continued)  Many software products are NOT code –specifications –tests –user guides –etc.  If you only measure the code, you will probably not really understand your software or its development process

49. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 49 Summary  Learn the principles of measurement theory  Understand what attribute you are measuring before you start to measure  Don’t assume the properties of the number system apply to the attribute being measured  Beware of misuse of means  Beware of small “n”  Beware of dependent variables

50. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 50 References  Department of Defense, Joint Logistics Commanders Joint Group on Systems Engineering, Practical Software Measurement, a Guide to Objective Program Insight (version 2.1), Naval Undersea Warfare Center, c/o John McGarry, mcgarry@ada.npt.navy.mil.  Fenton, Norman E. Software Metrics: A Rigorous Approach, Chapman & Hall, London SE1 8HN, 1991. ISBN 0-442- 31355-1.

51. Copyright 1995-2007, Dennis J. Frailey CSE8314 - Software Measurement and Quality Engineering CSE8314 M22 - Version 7.09 51 END OF MODULE 22