1. 1 How To Benchmark Applications Development or Maintenance: Theory and Practice David G. Rogers

2. UKSMA Conference Oct 2005David G. Rogers2 How to benchmark Apps, in a nutshell 1. Sponsor the benchmark at a senior level 2. Understand the risks, costs and timescales 3. Be actively involved in the benchmark Passive benchmarking (“Speak when you’re spoken to”) is bad for your healthPassive benchmarking (“Speak when you’re spoken to”) is bad for your health 4. Cater for the measurement problems unique to Applications Watch the UKSMA website – launching initiative to solve major AM measurement problemWatch the UKSMA website – launching initiative to solve major AM measurement problem 5. Plan round the comparison problems 30 mins 10 mins

3. UKSMA Conference Oct 2005David G. Rogers3 Senior sponsorship –In-house – IT director IT director –Outsourced – Customer’s IT director (not just contract manager) Customer’s IT director (not just contract manager) Supplier’s relationship and delivery managers (not just contract manager) Supplier’s relationship and delivery managers (not just contract manager) –Give the benchmark the level of management commitment warranted by the risks

4. UKSMA Conference Oct 2005David G. Rogers4 The Risks The result might ruin formerly win-win relationships, damage careers, cost many jobs The result might ruin formerly win-win relationships, damage careers, cost many jobs –Relationship of the Applications service supplier with its customers is at risk, whether in-house or outsourced –What will senior management do if a benchmark result says: “Your AM costs 3 times the market average”? Or “ … 1/3 the market average”? –First, they will decide whether they believe it. If they don’t, reputations are damaged If they don’t, reputations are damaged If they do … If they do …

5. UKSMA Conference Oct 2005David G. Rogers5 The Risks The result might be wrong The result might be wrong –Mistakes abound Not necessarily (but possibly) by the benchmarker Not necessarily (but possibly) by the benchmarker The quality of the benchmark is your responsibility. Don’t delegate all responsibility for quality to the benchmarker The quality of the benchmark is your responsibility. Don’t delegate all responsibility for quality to the benchmarker Watch the detail Watch the detail –Check all data going into the process –Ensure all services and all costs are reported (this may seem obvious, but …) –Ensure in writing that you have the right to check for possible arithmetic errors by the benchmarker (they are only human) Build in cross-checks where possible Build in cross-checks where possible

6. UKSMA Conference Oct 2005David G. Rogers6 The Costs Major cash costs in benchmarking Applications: Major cash costs in benchmarking Applications: –Benchmarker’s fee –FP counting costs could easily be higher Staffing Staffing –One senior (reporting to the Sponsor) manager responsible –Full-time benchmark manager –System experts when required

7. UKSMA Conference Oct 2005David G. Rogers7 Timescales Only passive benchmarks keep to the benchmarker’s schedule Only passive benchmarks keep to the benchmarker’s schedule –In a “Passive benchmark” you: Do only what the benchmarker tells you Do only what the benchmarker tells you Supply only the information you are asked for Supply only the information you are asked for Sit back and wait for The Answer Sit back and wait for The Answer –Passive benchmarking is bad for your health!

8. UKSMA Conference Oct 2005David G. Rogers8 Some other stuff you MUST get right Objectives Objectives –Crucial but usually easy if outsourced Primary reason for outsourcing: 48% say “Reduce cost” Primary reason for outsourcing: 48% say “Reduce cost” –(that explains a lot … imagine recruiting senior executives on the same principle …) –Crucial but slippery if in-house Like-for-like comparisons Like-for-like comparisons –Very hard to achieve … you have to help the benchmarkers Releases Releases –Very hard to match output to input … don’t leave it all to the benchmarker

9. UKSMA Conference Oct 2005David G. Rogers9 Application Maintenance The key metric: £ / FP maintained The key metric: £ / FP maintained Commercially crucial measurement Commercially crucial measurement –(see Risks above!)

10. UKSMA Conference Oct 2005David G. Rogers10 How do you obtain £ / FP? £ : the price to you of running AM £ : the price to you of running AM FP : the size of the maintained portfolio FP : the size of the maintained portfolio How is FP obtained? Count the FPs: +/- 7.5%, but usually much too expensive Count the FPs: +/- 7.5%, but usually much too expensive “Fast counts” etc: less accurate (+/- 20% or more) “Fast counts” etc: less accurate (+/- 20% or more) –Too inaccurate if results are commercially important Much used: BACKFIRING Count Source Lines Of Code (SLOC), and “backfire” to FPs using average ratios Much used: BACKFIRING Count Source Lines Of Code (SLOC), and “backfire” to FPs using average ratios

11. UKSMA Conference Oct 2005David G. Rogers11 How accurate is backfiring? In one recent benchmark, benchmarker claimed +/- 10% In one recent benchmark, benchmarker claimed +/- 10% Most experts say +/- 100% to 400% Most experts say +/- 100% to 400% The experts differ – but if the latter, benchmarking represents a HUGE commercial risk … The experts differ – but if the latter, benchmarking represents a HUGE commercial risk … … so it is financially important to find out … so it is financially important to find out

12. UKSMA Conference Oct 2005David G. Rogers12 Initiative launched to accumulate proof Nothing even remotely confidential: only size matters Nothing even remotely confidential: only size matters –No dates, times, costs, prices, regions –No names except the verifying UKSMA member The result will bear the imprimatur of UKSMA, and will be in the public domain The result will bear the imprimatur of UKSMA, and will be in the public domain Counted FPs Counting method SLOCLanguage 99,999 e.g. IFPUG 4.1 999,999… ………… …………

13. UKSMA Conference Oct 2005David G. Rogers13 The desired outcome The usefulness of SLOC as a metric for use in benchmarks will be finally and permanently quantified The usefulness of SLOC as a metric for use in benchmarks will be finally and permanently quantified This will benefit EVERYONE WHO EVER BENCHMARKS APPLICATIONS This will benefit EVERYONE WHO EVER BENCHMARKS APPLICATIONS

14. UKSMA Conference Oct 2005David G. Rogers14 Primary contact for this UKSMA initiative: David@RogersTeam.co.uk +44 7812 189 672 Any questions, suggestions or offers of data? David Rogers is an EDS employee, but in this initiative is acting solely on UKSMA’s behalf. David@RogersTeam.co.uk