Applied Analytics in Business Plans Lessons learnt during the SRC15 business planning process Robert Murray – Scottish Water Analytics Team Leader – 27th February 2014

Our key observations 1. Ensure that presentation is clear and objective 2. Start early and respond to the changing needs of your stakeholders 3. Invest significant time in validation 5. Prepare the right level of detail 6. Address the data challenge 4. Think about how multiple sources of evidence will be used

1. Ensure that presentation is clear and objective presenting statistical data - stakeholder landscape

Presenting Statistical Data Present data in context

Presenting Statistical Data Ensure that visuals are fair

Presenting Statistical Data Choice of colour can make a difference

Presenting Statistical Data Avoid chart junk

Presentation should enable consistent and correct interpretation across all interested parties regardless of experience and capability. - Business plans are key public documents. - Greater emphasis on customer engagement and involvement: - Customer research. - Customer Forum. - Customer education. Stakeholder landscape

2. Start early and respond to the changing needs of your stakeholders demand – development model – other considerations

Demand Profile Period of maximum engagement Expect frequent scenario requests Diminishing opportunity to deliver big changes and degree of stability expected Expect questions about assumptions | confidence | overlaps | cause and effect

Demand Profile How can you maximise acceptance and use of the analysis during this time? How can you encourage early engagement and development during this time?

Analysis Spiral

Maturity Business Use Present Feedback Development Engage stakeholders every 6 months. Deliberately start with the simplest approach Focus on big details. Increase frequency of cycles to 2-3 months. Focus on small details, scenarios, testing

- Phase 1: - Limited need to document - Stakeholder requirements easy to track - Informal deployment of results. - Focus on developing the best solution. - Phase 2: - Produce frequent and short information papers. - Need to track emerging requirements – open / in progress / resolved - Formalise transfer of results to establish audit trail. - Track changes and keep records of why changes have been made. - Focus on developing the most appropriate solution. Other considerations

3. Invest significant time in validation examples of approach taken in Scottish Water - benefits

- Encourage extensive internal challenge and collaboration - Undertake statistical tests. - Undertake ‘in-the-field’ studies or obtain physical observations. - Collaborate with external experts: - In the process. - Independent of process. - Review against external sources: - Literature - Available industry data Our approach to validation

- Testing assumptions of technique applied. - Cross-validation (leave-one-out, 10-fold, time-based), e.g. Example Statistical Tests data for testingdata for analysis All Data Output, e.g. a model FITNESS evaluation

Example Statistical Tests

Field Observations

University Partnerships Industry Collaboration

- Internal collaborationsense checks and ensures buy in and use - Statistical testsessential practice, quick and robust - Field studieseasy to understand and harder to discredit - External experts introduces independence and new capabilities - External sourcesexplores biases Benefits

4. Think about how multiple sources of evidence will be used sources of information – methods

“How much do we need to invest in water mains renewal?” Could be investigated in a number of different ways: - Historic run-rates - End of life model - Replace mains with known problems - Comparative analysis - Economic optimisation model - Gut feel … finance constrained … and many others Water Mains Capital Maintenance

Jury vs. Judge Expanded evidence pool that is potentially ambiguous and lacking consensus Narrower evidence pool that is potentially more vulnerable to error or biases

Potential basic Jury Methods - Use the average - Use the median - Use the minimum or maximum - Use the majority vote - Use a weighted combination Or … compare the approaches and use the best estimator

Selecting the best estimator data for testingdata for analysis All Data Estimator 1 FITNESS 1 Estimator 2 FITNESS 2 Estimator 3 FITNESS 3

Selecting the best estimator

5. Prepare the right level of detail effort vs. use

Prepare the right level of detail amount of effort to prepare extent of use Detailed Technical Reports Information / SG Papers Presentations Data Tables & Charts Walkthroughs

6. Address the data challenge Our Approach

The Data Challenge That’s all really great, BUT how good is your data?

Sensitivity Testing Average Deterioration Rate : or ~1% per year. Random Error in GIS AgeChange in Deterioration Rate 0 – 2.5%< 1% 2.5 – 10%1 – 2.7% 10 – 20%2.7 – 4.6% 20 – 30%4.6 – 7.6% 30 – 40%7.6 – 9.5% 40 – 50%9.5 – 11.6%

Expert Knowledge - Generally viewed as a less objective than empirical data - Situation can be greatly improved by using a structured elicitation process - Expect questions about corporate biases. - Two ways to address these concerns: - Obtain physical ‘in-field’ observations and compare. - Compare against available industry data.

Our key observations 1. Ensure that presentation is clear and objective 2. Start early and respond to the changing needs of your stakeholders 3. Invest significant time in validation 5. Prepare the right level of detail 6. Address the data challenge 4. Think about how multiple sources of evidence will be used