1. 1 SIM5102 Software Evaluation Framework for software measurement

2. 2 Classifying software measurement Three classes of entities: ➢ Processes – collections of software related activities usually associated with some timescale ✔ Different SE proceses: development, maintenance, testing, reuse, configuration and management process, etc ➢ Products – any artefacts, deliverable or documents that result from a process activity ➢ Resources – entities required by a process activity

3. 3 Classifying Entities Within each entity class, there can be two types of attributes: Internal attributes of a products, process or resource are those that can be measured by examining the product, process or resource on its own, separate from its behaviour. External attributes of a product, process or resource relates to its environment. The behaviour of the product, process or resource is important, rather than the entity itself.

4. 4 What Managers Want Bigger, faster, better, more.... Improve productivity Improve quality - fitness for purpose - conformance to specification - degree of excellence - timeliness Improve projection Reduce risk ---> external attribute ---> why do we care about internal attribute?

5. 5 External Attribute External attributes are what managers want External attributes can involve aspects that are difficult to quantify, and often introduce subjectivity Internal metrics can be measured objectively, and usually fairly easily We usually measure internal attributes to predict measures of external attributes

6. 6 Sample Process Measures Number of test executed Number of test passed Progress Number of tests scheduledVolumeTest process Number of tests passed divided by number of test executed. Performance Percentage of tasks complying with standard procedures or directives. Process compliance Percentage of total defects found in same phase where it was introduced. Phase containment Staff hours.Effort Calendar dates.Milestones Number of working days.Elapsed timeDevelopment Process Possible MeasuresAttributesProcess Entities

7. 7 Sample Process Measures Number of change requests awaiting service Estimated effort (person hours) for pending request SizeChange request backlog Dollars per year Staff hours per change request CostMaintenance Number of working daysElapsed timeDetailed design Possible MeasuresAttributesProcess Entities

8. 8 Sample Product measures (1) Ratio of unchanged physical lines to total physical lines, comments and blanks excluded Percent reused Physical source lines of code Logical source statements LengthModule Defects per KLOC Defects per function point Defect density Number of modules Number of bubbles in a data-flow diagram Number of use cases in a use case diagram Number of function points Number of physical source lines of code Number of memory bytes of words required (or allocated) SizeSystems Possible MeasuresAttributesProduct Entities

9. 9 Sample Product measures (2) Staff hoursEffort to fix An ordered set of severity classesSeverity Name of activity where introduceOrigin Type namesTypeDefect Language nameProgramming Language Name of production methodHow produced Type namesStatement typeLines of code Number of pagesLengthDocument McCabe’s complexityNumber of linearly independent flow paths Unit Possible MeasuresAttributesProduct Entities

10. 10 Sample Resource measures Yes/no (a binary classification)Is used? Name of typeTypeCASE tools Year of domain experience Year of programming experience Experience Number of people assignedTeam sizeAssigned staff Possible MeasuresAttributesResource Entities

11. 11 Goal-based Measurement The primary question in goal-based measurement is: “What do we want to know or learn?” instead of “what measures shall we use?” The answers will depend on the organisation’s goals, therefore no fixed set of measures is universally adequate.

12. 12 GQM Approach The Goal-Question-Metric (GQM) approach provides a framework involving three steps: Goal : List the major goals of the development or maintenance project Question : Derive from each goal the questions that must be answered to determine if the goals are being met Metric : Decide what must be measured in order to be able to answer the questions adequately

13. 13 GQM Example Goal : Evaluate effectiveness of coding standard Questions: Who is using What is coder What is code standard? Productivity? quality Measures : Proportion Experience Code size Effort Errors.. of coders of coders (LOC, function - using - with standard Points etc) standard - with language - using - witch environment language - etc

14. 14 Template for Goal Definition Purpose: To (characterize, evaluate, predict, motivate etc) the (process, product, model, measure etc) in order to (understand, assess, manage, engineer, learn, improve etc) it. Example: To evaluate maintenance process in order to improve it.

15. 15 Template for Goal Definition Perspective : Examine the (cost, effectiveness, correctness, defects, changes, product measures etc) from the viewpoint of the (developer, manager, customer etc). Example : Examine the cost from the viewpoint of the manager.

16. 16 Template for Goal Definition Environment : The environment consists of the following: process factors, people factors, problem factors, methods, tools, constraints etc. Example: The maintenance staff are poorly motivated programmers who have limited access to tools.

17. 17 Examples Purpose : Determine a way to improve the productivity of our development team by evaluating their current productivity Perspective : Examine the team personality factors, expertise of the development organisation, team’s analysis and design techniques, knowledge of programming languages used by our development team Environment and constraints : Payroll applications programming in C++ 100 software developers with 5 or more years experience in C++ Customers are businesses Do not maintain a reusable module database Examine new projects completed and sold from 1/1/2002 to 31/12/2004.

18. 18 Examples Purpose : Evaluate working environment in order to identify opportunities to improve the productivity of our development team Perspective : Examine the ratio of work time to break time of our employees, the accommodations, incentives, extra-curricular activities offered, workspace (room and desk size, ventilation) where our employees work from the point of view of the employees themselves Environment and constraints : Payroll applications programming in C++ 100 software developers with 5 or more years experience in C++ Customers are businesses Do not maintain a reusable module database Examine new projects completed and sold from 1/1/2002 to 31/12/2004.

19. 19 Examples Purpose : Evaluate the impact of various CASE tools on the productivity of the development team Perspective : Examine the effectiveness of using various CASE tools to help in the development of our product from the point of view of the developers and testers Environment and constraints : Payroll applications programming in C++ 100 software developers with 5 or more years experience in C++ Customers are businesses Do not maintain a reusable module database Examine new projects completed and sold from 1/1/200 to 31/12/2004

20. 20 GQM, Entities and Attributes We can relate the GQM template to the Entity&Attribute framework. A Goal or question can be associated with at least one pair of Entities and Attributes. Once a measure is obtained, it can be related back to questions and goals.

21. 21 Chapter 3 Fenton and Pfleeger book until page 87. V.R. Basili and H.D. Rombach, The Tame Project: Towards Improvement-Oriented Software Environments, IEEE Trans. Software Engineering, vol. 14, pp.758-773 (1988). Reading

22. 22 SEI CMM (Capability Maturity Model) Five levels of process maturity: Initial Essentially uncontrolled Repeatable Product management procedures defined and used Defined Visibility through defined processes Managed Quality management strategies defined and used Optimizing Process Improvement strategies defined and used

23. 23 CMM Levels Continuously Improving process Predictable process Standard Consistent process Disciplined process Initial (1) Repeatable (2) Defined (3) Managed 4) Optimising (5)

24. 24 CMM Level 1 Level 1 – The Initial Level At the initial, the organisation typically does not provide a stable environment for developing and maintaining software. Success depends entirely on having a very good manager and an experienced and effective software team. Inputs to the process are ill-defined; transition from inputs to outputs is undefined and uncontrolled. Similar projects may vary widely in their productivity and quality characteristics due to lack of adequate structure and control. Baselines are needed to provide a starting point for measuring improvement as maturity increases.

25. 25 CMM Level 2 Level 2 – The Repeatable Level At the repeatable level the organisation establishes policies for managing a software project and procedures to implement those policies. New projects are planned and managed based on experience with similar projects and basic software management controls have been installed. Software process capability of level 2 organisations is disciplined because planning and tracking of the software project is stable and earlier successes can be repeated.

26. 26 A repeatable Process (level 2) Input Requirements Control Budget Schedule Standards Output Code Documentation Resources Staff Tools Construct the system

27. 27 CMM Level 3 Level 3 – The Defined Level At the defined level the standard process for developing and maintaining software across the organisation is documented, including both software engineering and management processes, and these processes are integrated into a coherent whole. Management has good insight into technical progress on all projects because the software process is well defined. The software process capability of level 3 organisations is standard and consistent because both software engineering and management activities are stable and repeatable.

28. 28 The Defined Level (level 3) Design method Test Plans Requirements Tools, Staff Inspection Criteria System Design Tools, Staff Tested Modules Tools, Staff SW

29. 29 CMM Level 4 Level 4 – The Managed Level At the managed level, the organisation sets quantitative quality goals for both software products and processes. There is an organisational measurement program where productivity and quality are measured for important software process activities across all projects. The software process capability of level 4 organisations is quantifiable and predictable because the process is measured and operates within measurable limits.

30. 30 The Managed Level (level 4) Reporting requirements form senior management Directives for new employees Problems With Early versions Design defects Requirements Design Software Redesign directive Manage Define, Design Build, Test

31. 31 CMM Level 5 Level 5 – The Optimising Level At the optimising level, the entire organisation is focused on continuous process improvement. Software project teams in level 5 organisations analyse defects in order to determine their causes. The software process capability of level 5 organisations improves continuously, striving to improve the range of their process capability. The spiral model is an example of such a dynamically changing process, responding from feedback form early activities in order to reduce risks in later ones.

32. 32 Spiral Model (Bohem, 1988) DETERMINE GOALS, ALTERNATIVES, CONTSTRAINTS EVALUATE ALTERNATIVES AND RISKS PLAN DEVELOP AND TEST Alternatives, Constraints,, Risk analysis Constraints,, Alternatives, Budget Development plan Integration and test plan Requirements, Life-cycle plan Implementation plan Acceptances test System test Unit test Code Detail design Proto- type Proto- type Proto- type Validated requirements Validated Verified design Software requirements Software design Constraints,, Concepts of operations Budget Proto- type

33. 33 Process Maturity and Measurement Entities Characteristics Type of Measures to use level 1 Heuristic Baseline 2 Process dependent on Project management individuals 3 Process defined and Product institutionalised 4 Measured process Process plus feedback for control 5 Improvement fed back Process plus feedback for to the process changing the process

34. 34 Combining GQM with Process Maturity Since process maturity suggest that one can only measure what is visible, the incorporation of process maturity with GQM helps identify what measures will be most useful to the organisation at that particular moment in time.

35. 35 GQM and CMM GQM helps to understand why we measure an attribute. Process maturity (e.g. CMM) suggests whether we are capable of measuring that attribute in a meaningful way.

36. 36 Reading literature Fenton and book: chapter 3

37. 37 Factor Criteria Metrics The earliest approach (mid 1970s) developed by Mc Call et al. The principal aim of FCM is not measurement as such, but rather, to obtain a quantitative view of quality. Very much oriented to user’s/ customer’s perspective of quality–fitness for purpose

38. 38 In Class exercise To improve the accuracy of software project managers' cost estimates at the specification stage of a project within the telecomunication systems division at aardvark international. - purpose -perspective -environment -questions -metrics