10 April May 2008 Measuring what’s important Monitoring Outcomes Doug Gosling

10 April May 2008 Measuring what’s important My messages Monitoring outcomes and the effectiveness of services is undeveloped Monitoring outcomes - individual and whole service - is not technically difficult There are complex barriers to effective monitoring There are important lessons from our wider lives!

10 April May 2008 Measuring what’s important My outcomes for today? Your immediate reaction to what I present, esp. enthusiasm, challenge? What you take back as useable ideas? How you monitoring of outcomes improves in (define period)? Understand (or be reminded about) chiasmus – fun but irrelevant?! It may be comparatively evidence light, but is intended to be idea rich

10 April May 2008 Measuring what’s important Wider Life We are more experienced at buying outcome based services than we might think – though not in social care Examples Key Lessons: Trust, which grows over time Buyer has measure(s) of success

10 April May 2008 Measuring what’s important Monitoring Outcomes CSIP survey (June 2006): No authorities (15) were monitoring outcomes CSIP think tanks (Autumn 2007): No authorities (30) were monitoring service outcomes Commissioning network request: limited response, still some confusion of outputs and outcomes

10 April May 2008 Measuring what’s important Monitoring outcomes: the essentials Agree outcome(s) expected: service and individual Identify what shows they have been achieved Collect information to measure achievement - individual Collate individual information to obtain measure of service effectiveness USE THE RESULTS TO ACHIEVE CHANGE

10 April May 2008 Measuring what’s important Monitoring outcomes: Connie Outcome: Wears her hearing aid without reminder Measure: Whether staff have had to help or remind her Information: Daily log records and reviews, where progress and achievement over period is recorded BUT, given her forgetfulness, will she ever acquire the skill and/or do it unprompted? (In which case, does it remain an output task: insert Connie’s hearing aid?)

10 April May 2008 Measuring what’s important Collating outcome measures Representation of an individual's progress on several fronts, e.g. the outcome star Collating measures from several service users to monitor the effectiveness of the service

10 April May 2008 Measuring what’s important Example of star Ref: And The London Housing Foundation and Triangle Consulting, via

10 April May 2008 Measuring what’s important Suggested Benefits For clients More accurate assessment of their progress Validates clients journey of recovery Client centred tool For staff Identifies appropriate interventions Guides the work Improves confidence For organisations Better assessment leads to better outcomes Improved effectiveness Demonstrability of results

10 April May 2008 Measuring what’s important Service effectiveness All service users have outcomes, but variable number and focus Progress monitored on simple scale: 0= no progress 1= progress, less than 50% 2= progress above 50 but not complete 3= full achievement Effectiveness score and profile

10 April May 2008 Measuring what’s important Service Effectiveness Service user Individual Outcomes Achievement Profile TOTAL SCORE MAX SCORE EFFECT’ NESS (%) Total 0 Total 1 Total 2 Total TOTAL FOR SERVICE

10 April May 2008 Measuring what’s important Service Effectiveness - example Service user Individual Outcomes Achievement Profile TOTAL SCORE MAX SCORE EFFECT’ NESS (%) Total 0 Total 1 Total 2 Total TOTAL FOR SERVICE

10 April May 2008 Measuring what’s important Barriers to effective monitoring? Contracts not based on purchasing outcomes, accepted across the whole system Care management – focus on outcomes User/informal carers – are they involved enough in identifying and agreeing individual outcomes? Contract monitoring resources – numbers and competences? Integration of different levels of outcomes, especially timescale and strategic – individual Knowledge barriers – cause and effect, satisfaction and effectiveness, outputs and outcomes OVERALL SHIFT IN CULTURE AND PRACTICE

10 April May 2008 Measuring what’s important Remember the lessons from wider life Mutual trust Measure(s) of whether you have got what you paid for The supplier expects you to check you have a satisfactory outcome

10 April May 2008 Measuring what’s important Have I been effective? Immediate – have more than 75% of you rated me X or higher on all scales? (Is this actually satisfaction?) Have you got greater knowledge and/or confidence about monitoring outcomes? How many of you? How much greater? How much have the systems you work with moved on in say six to twelve months? Are you monitoring service effectiveness? Sewing ideas: germination and growth are influenced by a multitude of other factors.

10 April May 2008 Measuring what’s important Finally: chiasmus You must measure what is important You must not make important what you can measure