Lero© 2010 Software Quality & Process Improvement Dr. Ita Richardson Lero – the Irish Software Engineering Research Centre and Department of Computer Science & Information Systems University of Limerick 1 Lero© 2010

LECTURE 3: MEASURING THE SOFTWARE PROCESS

Lero© 2010 Why Measure? Business & Industry use quantitative approach Measure: – Product quality – Productivity – Efficiency – Machine reliability Why should Software Engineering be different?

Lero© 2010 Quantitative Measures Length of project activity? Time remaining in project? Productivity of software engineer? Bottlenecks in system? % of time for test vs development No of errors detected Time spent fixing errors

Lero© 2010 Measurement as feedback Performance indicators Match against best practice Make required improvements Results in continuous process performance

Lero© 2010 Measurement Measurement in chaotic environment is generally not useful Need consistent and standard process Use balanced set of indicators Measure versus goals

Lero© 2010 Types of measures Process-related measures – Number of change requests – Cycle time – Effectiveness of the process Project-related measures – Project development time – Cost of work to date – Deviation of costs – Staff productivity

Lero© 2010 Types of measures Product / Customer –related measures – User-reported bugs – Customer satisfaction – Source code size

Lero© 2010 Principles of SP Measurement Define clear objectives – Key performance indicators Management support – Supporting Business Goals Roles and responsibilities Cover both process and product Reflect business & engineering objectives

Lero© 2010 Issues to be considered How to measure business value of SPI? – Increased productivity – Early error detection and correction – Cost of SPI How to measure qualitative benefits? – Product availability – Customer satisfaction – Employee morale

Lero© 2010 Measurement Tools Understanding Data – Histograms – Bar Charts – Pareto Charts – Scatter diagrams – Run charts Analyse Data – Control Charts

Lero© 2010 Using the Data How do we analyse the data? What can we do within the organisation? Do we need more information? Should cause us to ask questions as well as answer questions

Lero© 2010 Histograms Frequency counts Distribution of observed values Easy to compare distributions

Lero© 2010 Histogram

Lero© 2010 Bar Charts Sets of discrete values Data associated with individual entities

Lero© 2010 Bar Chart

Lero© 2010 Scatter Diagrams Comparing one variable with another ‘Dependent’ / ‘Independent’ Cause and Effect

Lero© 2010 Scatter Diagram

Lero© 2010 Run Charts Values arranged in time sequence Basis for Control Charts Temporal Behaviour of Process

Lero© 2010 –,, © S-Cube – 20/ Run Chart

Lero© 2010 Sequential Run Chart As Run Charts BUT some Grouping of Data exists

Lero© 2010 When did the training occur? Sequential Run Chart

Lero© 2010 Pareto Charts Ranks outcomes Frequency counts in descending order Temporal

Lero© 2010 Variation Total Variation = Common Cause Variation + Assignable Cause Variation

Lero© 2010 Common Cause Variation Natural Results are common across groups Stable consistent pattern Predictable Unexpected results are rare

Lero© 2010 Assignable Cause Variation Causes which could have been prevented Outside influences Shifts in quality – People – Processes – Tools

Lero© 2010 Stable process In statistical control No assignable causes – As they have been removed from the process Variability due to Common Causes Use Statistical Process Control – Control Charts

Lero© 2010 Control Charts How process has behaved R-Chart – Range charts – Smallest: Largest value – Don’t expect to exceed limits of range variation X-Chart – in control – Sub-group averages don’t fall outside upper/lower variation control For examples of Control Charts: see the internet

Lero© 2010 Statistical Process Control Improve process though causal analysis Control Limits – Estimated for process – Cannot assign arbitrarily – +- 3 sigma (Std Deviations) Centreline – Observed process average

Lero© 2010 Statistical Process Control Common Cause Variation – Assignable cause must be eliminated Rational Sub-Grouping & Rational Sampling – To eliminate common cause variation – To ensure narrow control limits

Lero© 2010 Out-of-control Tests Instability where one of the following exist: Single point outside 3-sigma Two / three successive values fall on the same side and outside 2-sigma Four / five successive values fall on the same side and outside 1-sigma Eight successive values fall on the same side of centreline – Tests 2-4 for X-Charts only

Lero© 2010 What do we do with Results? Remove assignable causes then Change the process then Continually improve

Lero© 2010 Remove assignable causes Find reasons for their existence Prevent their recurrence Introduces stability to process May use qualitative data May use other quantitative data

Lero© 2010 Change the process Removing assignable causes requires: Identifying changes Designing changes Implementing changes

Lero© 2010 Continually improve Reduce variability Improves quality Improves cost Improves time-to-market Ultimate requirement is STABLE & CAPABLE process - Need to consider Change Management

Lero© 2010 Framework for Change Management (Willman, 1996) Pressure for Change Leadership & Vision Capable People Effective Rewards= Successful Implementation Pressure for change Leadership & Vision Capable People ? = Evaporation Pressure for change Leadership & Vision Capable People Effective Rewards=Frustration Pressure for change Leadership & Vision ?Effective Rewards=Disengagement Pressure for change ?Capable People Effective Rewards=Disillusionment ?Leadership & Vision Capable People Effective Rewards=Disinterest

Lero© 2010 Measurement as feedback Performance indicators Match against best practice Make required improvements Results in continuous process performance

Lero© 2010 Acknowledgement The information presented in these slides has been collected from a variety of sources including: – Software Quality Assurance: From Theory to Implementation by Daniel Galin, 2003 – Software Process Improvement: Practical Guidelines for Business Success by Sami Zahran, 1998 – Research carried out by post-doctoral researchers and PhD students at Lero – the Irish Software Engineering Research Centre, Ireland under the supervision of Dr. Ita Richardson – Software Process: Improvement and Practice (journal) – The SPIRE Handbook: Better, Faster, Cheaper Software Development in Small Organisations, edited by Marty Sanders (Version 1, 1998) and Jill Pritchet (Version 2, 2000) The research presented in this lecture has been partially supported by Science Foundation Ireland funded through Global Software Development in SMEs Cluster Grant (no 03/IN3/1408C) and Lero – the Irish Software Engineering Research Centre (CSET grant no 03CE2/I303.1).