The value of benchmarking IT projects Harold van Heeringen Software Cost Engineer, Sogeti Nederland B.V. ISBSG president NESMA board The Russian Managers Association Location: Moscow Date: October 2013

Семь раз отмерь, один отрежь Сравнить божий дар с яичницей Умом Россию не понять, Аршином общим не измерить Ничто не существует, пока оно не измерено

3 Overview Benchmarking Software Project Industry Functional Size Measurement Software Metrics Historical Data: ISBSG Project Benchmark Example Organization Benchmark Other uses for Benchmarking Conclusions and final remarks Scope: Software Development

4 Benchmarking (wikipedia) Benchmarking is the process of comparing one's business processes and performance metrics to industry bests or best practices from other industries. performance metricsbest practices Benchmarking is used to measure performance using a specific indicator (cost per unit of measure, productivity per unit of measure, cycle time of x per unit of measure or defects per unit of measure) resulting in a metric of performance that is then compared to othersindicator This then allows organizations to develop plans on how to make improvements or adapt specific best practices, usually with the aim of increasing some aspect of performance. Benchmarking may be a one-off event, but is often treated as a continuous process in which organizations continually seek to improve their practices.

5 Where are we? “Even the most detailed navigation map of an area is useless if you don’t know where you are” ? ? ?

6 Benchmarking Senior Management of IT departments/organizations need to make decisions need to make decisions based on ‘where they are’ and ‘where they want to go’. Benchmarking is about determining ‘where you are’ compared to relevant peers, in order to make informed decisions. But how to measure and determine where you are?

7 Software project industry Low ‘performance metrics’ maturity Few Performance Measurement Process implemented Few Benchmarking processes implemented Most organizations don’t know how good or how bad they are in delivering or maintaining software. These organizations are not able to assess their competitive position, nor able to make informed strategic decisions to improve their competitive position.

8 But… Best in Class organizations deliver software up to 30 times more productively than Worst in Class organizations High Productivity, High Quality More functionality for the users against lower costs Shorter Time to Market – competitive advantage! Worst in Class organizations will find themselves in trouble in an increasingly competitive market Outperformed by competition Internal IT departments get outsourced Commercial software houses fail to win new contracts Important to know where you are! Benchmark is essential!

9 Performance means balance Delivery to time and budget Project Productivity Project Speed Project Quality Actual vs. Estimated Cost Actual vs. Estimated Duration Size / Effort Size / Duration Post delivery defects / Size

10 Difficulty – low industry maturity How to measure metrics like productivity, quality, time-to- market in such a way that a meaningful comparison is possible? Comparing apples to apples

11

12 Functional Size Measurement Function Point Analysis (NESMA, IFPUG or COSMIC) Measure the functional user requirements – size in function points; ISO standards – objective, independent, verifiable, repeatable; Strong relation between functional size and project effort needed; What to do with the results? Project effort/duration/cost estimation Benchmarking/performance measurement Use in Request for Proposal management (answer price/FP questions) What about historical data? Company data (preferably for estimation) Industry data (necessary for external benchmarking)

13 Unit of Measure (UoM) Why are Function Points the best UoM to use in Benchmarking? Functionality is of value for the client/business. More functionality means more value. More Lines of code (technical size) is not necessarily of value. Function Points are measured independent from technical requirements 500 FP of functionality implemented in Java SOA architecture = 500 FP of functionality implemented in Cobol mainframe Function Points are measured independent from implementation method 500 FP delivered in an agile development project = 500 FP delivered in a COST package implementation

14 Software metrics – some examples Productivity Productivity Rate: #Function points per staff month PDR: #Effort hours per function Point Quality Defect Density: #Defects delivered per 1000 function points Time to Market Speed: #Function points delivered per calendar month

15 Performance Measurement Measure the size of completed projects Project size in Function Points Product size in Function Points Collect and analyze the data Effort hours, duration, defects Normalize the data when necessary Store the data in the corporate database Benchmark the project, internally and externally Report metrics and trends Different reports for different stakeholders Depending on goals of the stakeholder

16 Historical data: ISBSG repositories International Software Benchmarking Standards Group Independent and not-for-profit Grows and exploits two repositories of software data: New development projects and enhancements (> 6000 projects) Maintenance and support (> 1000 applications) Everybody can submit project data DCQ on the site Anonymous Free benchmark report in return

17 ISBSG Mission: “To improve the management of IT resources by both business and government, through the provision and exploitation of public repositories of software engineering knowledge that are standardized, verified, recent and representative of current technologies”. All ISBSG data is validated and rated in accordance with its quality guidelines current representative of the industry independent and trusted captured from a range of organization sizes and industries Industry leaders around the world contribute to the ISBSG’s development, offering the highest metrics expertise worldwide

18

19 Example – project benchmark Project X was completed and the following data was collected: Primary programming language: Java Effort hours spent: 5730 Duration: 11 months Defects found after delivery: 23 The functional size of the project was measured: 411 FP Software metrics: Project Delivery Rate: 5730/411 = 13,9 h/FP Project Speed: 411/11 = 37,4 FP per calendar month Defect Density: (23/411) *1000 = 56,0 defects/1000 FP

20 Example: Benchmark ISBSG ‘New Developments & Enhancements’ Select the right ‘peer group’ Data Quality A or B Count approach: IFPUG 4.x or NESMA Primary Programming Language = ‘Java’ 300 FP < Project Size < 500 FP

21 Results – project benchmark PDR N488 Minimum0,1 Percentile 102,5 Percentile254,7 Median9,8 Percentile 7518,4 Percentile 9028,9 Maximum621,3 Average15,2 Project Delivery Rate: 5730/411 = 13,9 h/FP Project Speed: 411/11 = 37,4 FP per calendar month Defect Density: (23/411) *1000 = 56,0 defects/1000 FP Speed N428 Minimum9,4 Percentile 1023,1 Percentile 2532,5 Median53,8 Percentile 7595,4 Percentile 90130,2 Maximum476,0 Average70,9 Defect Density N154 Minimum0,0 Percentile 100,0 Percentile 250,0 Median3,7 Percentile 7517,9 Percentile 9040,1 Maximum366,5 Average18,6 This project was carried out less productive and slower than market average, and the quality is worse than average.

22 Organization Benchmark Analysis: - Until 2010, the organization was improving - After 2010/2011, the trends go the wrong way - Recommendation: find the cause and draw up improvement plan

23 Other uses for Benchmarking Vendor selection, based on productivity, speed or quality metrics Definition of SLA agreements (or other KPI’s) based on market average performance Establish a baseline from which to measure future improvement Explain to the client/business that a project was carried out in a ‘better-than-average’ way, while the client may perceive otherwise

24 Conclusions and final remarks Benchmarking is essential in the strategic management of an organization. It helps understanding the competitive position and it help to identify the ‘problem areas’, or ‘improvement areas’. There are proven tools, techniques, models and historical data available to carry out benchmarks in a fairly low-cost way. This presentation focused on software development, because this is usually the main concern of the business/client. Benchmarking of other aspects of IT is similar, but with other performance metrics.

Harold van haroldveendam haroldvanheeringen Software Cost Engineer NESMA board member ISBSG president COSMIC International Advisory Council, NL representative