OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Metrics in the context of the CMM/SPICE SPIN-UK, 29 September 1998

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Some views on measurement… “What is not measurable, make measurable.”- Galileo “When you can measure what you are speaking about and express it in numbers you know something about it; but when you cannot measure it, when you cannot express it in numbers your knowledge is of a meagre and unsatisfactory kind.” - Lord Kelvin “…This remark [above] has been quoted with an approval it does not altogether deserve - it does not, for example, do justice to the work of Darwin or Pasteur, to name only two of Kelvin’s contemporaries.” - G.U.Yule (statistician) “You can’t control what you can’t measure.” - DeMarco “Metrics is crap.” - Dijkstra

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy What is measurement?

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Empirical relational system real world

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Empirical relational system Formal relational system measurement real worldmathematical world

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Empirical relational system Formal relational system Results measurement mathematics and statistics real worldmathematical world

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Empirical relational system Relevant empirical information Formal relational system Results measurement interpretation mathematics and statistics real worldmathematical world

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Empirical relational system Relevant empirical information Formal relational system Results measurement interpretation mathematics and statistics decisions and actions real worldmathematical world

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Characteristics... good measurement should not (in general) conflict with empirical insights sound measurement systems reflect the level of understanding of those things that are measured good measurement encourages improved understanding

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Empirical relational system Relevant empirical information Formal relational system Results refined measurement more sophisticated interpretation mathematics and statistics better decisions and actions real worldmathematical world

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Measurement - state of practice… can be self serving, but increasing awareness of role often not well founded, but effort is invested data collected but often not valued (even when useful)

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Effective measurement require an appreciation of... measurement fundamentals –how do we measure? context –what do we need to measure? –what can we measure? soft (human) factors –why should I measure/be measured?

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Measurement within the Capability Maturity Model:

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Initial Repeatable Defined Managed Optimizing The Capability Maturity Model Organizational abilities are described as five levels of increasing capability

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Defined maturity levelkey process areas activities commitment ability verification M & A common features measurement and analysis in each KPA from levels 2 to 5 ISM TP PR OPD IGC SPE OPF

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Measurement within the CMM... is a common feature of every key process area at each described maturity level measurement and analysis common feature is least well described –SEI recognize need for organizations to derive their own measures –SEI say nothing about how to measure in practice we tend to find measurement and analysis is the last common feature to be satisfied for any given KPA –can use this as an informal assessment ‘short cut’.

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy...Measurement within the CMM... When investigating measurement capability look for... –a complementary view of quality throughout the organization –recognition of why data is collected, and understanding of definitions and analysis –data verification (accuracy known) and measures validated –information used to make decisions and direct actions

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy...Measurement within the CMM Level 4 –measure products and process quantitatively Software Quality Management Quantitative Process Management –aspires to production engineering (cf project engineering) –still controversial

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Measurement within SPICE (ISO 15504)

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy The SPICE Architecture ORG ENG SUPP PROJ CUS Proc 4 Proc 2 Proc 1 Proc 4 Eng. 3 Develop S/W Design Proc Practice 1 Practice 2 Practice 3 Eng 3.4 Establish traceability Process Capability Well defined

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy A comparison of the CMM and SPICE SPICE... –is more complex this is a good thing and a bad thing –it may be unrealistic –but does require thought –is structured by process...not capability - so may make arbitrary target setting more difficult allows all processes to be addressed to some extent (c.f. CMM’s SPE) –has process capability defined by generic processes

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Measurement within SPICE... Measures required throughout (like CMM)... –Generic Practices (define process capability of every process) level 2 - plan process (I.e. estimation) level 2 - track with measurement –note - “use of measurement implies measures have been defined and selected and data has been collected” level 3 - use well defined data level 4 - measurement oriented (like CMM level 4)

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy...Measurement within SPICE......Measures required throughout (like CMM) –Processes engineering 2.2 hints at ISO9126 project 2.5 and size, effort,cost schedule resources risks project define quality metrics organization define process measures organization set performance expectations organization define measures of impact

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy...Measurement within SPICE introduced at low capability level (like CMM) states expectation of what measurement requires (unlike CMM) –(even at low capability level) minimal guidance (examples) on what to measure (less than CMM) implies a distinct capability to define and assess measures

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy Summary Measurement within the Capability Maturity Model within SPICE (ISO 15504)

OXFORD SOFTWARE ENGINEERING Software Engineering Services & Consultancy C.C.Shelley +44 (0) O X F O R D S O F T W A R E E N G I N E E R I N G L I M I T E D