Abt Associates | pg 1 Performance Management Systems and Evaluation: Towards a Mutually Reinforcing Relationship Jacob Alex Klerman (Abt Associates) APPAM/HSE Conference “Improving the Quality of Public Services” Moscow, June 2011

Abt Associates | pg 2 Performance Management Systems and Evaluation Performance ManagementEvaluation

Abt Associates | pg 3 Performance Management Systems and Evaluation Performance Management  The need is clear –“What gets measured gets done” –If you know what you want done; you need to manage against it –To manage against it, you need to measure it Evaluation

Abt Associates | pg 4 Performance Management Systems and Evaluation Performance Management  The need is clear –“What gets measured gets done” –If you know what you want done; you need to manage against it –To manage against it, you need to measure it Evaluation  But what do you want done? –Are you sure?  That’s the role of rigorous impact evaluation –Dirty little secret: much of what we do—much of what seems “plausible”—has minimal impact (or even hurts)

Abt Associates | pg 5  Current Practice  A Better Way  Closing Thoughts Outline

Abt Associates | pg 6  Everyone wants better program outcomes –We might even be willing to spend more if we could prove better outcomes  Proving “better outcomes” requires rigorous impact evaluation –Many apparently plausible programs (and program innovations) don’t work –Naive evaluation methods give the wrong answer  Rigorous impact evaluation is challenging –Requiring large samples –And, the smaller the projected incremental impact, the larger the required samples Rigorous Impact Evaluation Is Crucial

Abt Associates | pg 7 Current Evaluation Practice Isn’t Very Useful  Asks the wrong questions: Does the program “work”? –i.e., Should we shut the program down? –Big programs address major social problems –The programs aren’t going away

Abt Associates | pg 8 Current Evaluation Practice Isn’t Very Useful  Asks the wrong questions: Does the program “work”? –i.e., Should we shut the program down? –Big programs address major social problems –The programs aren’t going away  The right question is often: How can we make the program better? –Which program model works better? –Would some minor—and affordable—change in program design help? –For which subgroups does our program work? Target the program at them

Abt Associates | pg 9  Answering up/down evaluation question requires (relatively) small samples –For a training program, perhaps 500-2,000 case  Answering practitioners’ questions requires much large samples –For a training program, perhaps 10,000+ cases  At current evaluation cost—$1,000+ per case—we can’t afford to answer practitioner’s questions –Especially if the change in outcomes will be at best small –And that’s a big problem because CQI/kaizen suggests that major improvement often comes from lots of small improvements The Realities of Sample Size and Cost To answer practitioner’s questions, we’re going to need to get the cost way down

Abt Associates | pg 10  Negotiate access to sites, including convincing them to deny service to some applicants –  Customize randomization for each site –  Detailed process analysis at each site –  Detailed survey follow-up – Steps in a Current Evaluation Is there another way? Sometimes, yes …

Abt Associates | pg 11  Current Practice  A Better Way  Closing Thoughts Outline

Abt Associates | pg 12  We have just argued that collecting information on outcomes drives costs  Performance measurement systems already collect information on outcomes –Presumably on the key outcomes  So, when we can measure outcomes through the performance measurement system –Costs will be much, much lower –Allowing large samples –A key requirement for evaluating incremental changes  Will only work when both treatment and control are “in the system” (e.g., incremental changes) At the Back End—Leverage Ongoing Performance Management Systems

Abt Associates | pg 13  Currently research is “top down” –Someone outside the system decides to evaluate X –Then, evaluator tries to convince sites to adopt X; and to deny all services to a control group At the Front End—A Learning Organization

Abt Associates | pg 14  Currently research is “top down” –Someone outside the system decides to evaluate X –Then, evaluator tries to convince sites to adopt X; and to deny all services to a control group  Alternative is “bottom up” –Ask sites to suggest what to evaluate –Form a committee—site representatives, central program staff, substance experts, evaluation experts –Ask them to select from among the suggestions –Ask sites to volunteer to implement the selected suggestions –Randomize at the site level; control condition is “current practice”, not “no service” At the Front End—A Learning Organization Cutting time and costs

Abt Associates | pg 15 In Summary NowBetter Negotiate access to sites (expensive and time consuming) They volunteer Customize randomization for each site (expensive and time consuming Site level randomization Detailed process analysis (expensive) Skip this Collect detailed survey outcome data (very, very expensive) Use Performance Management System data And when your costs drop sharply, CQI if feasible; i.e., you can test little changes

Abt Associates | pg 16  Current Practice  A Better Way  Closing Thoughts Outline

Abt Associates | pg 17 A True Learning Organization  Performance measurement is an ongoing task  CQI/Continuous Quality Improvement; i.e., –Proposing small changes to SOP/Standard Operating Procedures –Rigorously evaluating those small changes –Adopting those that can be shown to “help”  … Should also be an ongoing task  The key insight of “kaizen” is that improved outcomes arise from the accumulation of lots of such small changes Data collected as part of Performance Management Systems makes such CQI feasible

Abt Associates | pg 18  Site level randomization needs lots (50-200) of, relatively similar, sites  Central organization controls resources –Much easier to get volunteers, when volunteering is the only way to get more resources When Will this Work? We’re looking for test cases. Any volunteers?

Performance Management Systems and Evaluation: Towards a Mutually Reinforcing Relationship Jacob Alex Klerman APPAM/HSE Conference “Improving the Quality of Public Services” Moscow, June 2011