1. RBA Results-Based Accountability The Fiscal Policy Studies Institute www.raguide.org www.resultsaccountability.com TM Book - DVD Orders amazon.com resultsleadership.org

2. SIMPLE COMMON SENSE PLAIN LANGUAGE MINIMUM PAPER USEFUL

3. Results-Based Accountability is made up of two parts: 2. Performance Accountability is about the well-being of CUSTOMER POPULATIONS for Programs – Agencies – Service Systems 1. Population Accountability is about the well-being of WHOLE POPULATIONS for Communities – Cities – Counties – States - Nations 1. Population Accountability 2. Performance Accountability

4. Results-Based Accountability COMMON LANGUAGE COMMON SENSE COMMON GROUND

5. Common Language

6. THE LANGUAGE TRAP Too many terms. Too few definitions. Too little discipline Benchmark Target IndicatorGoal Result Objective Outcome Measure Modifiers Measurable Core Urgent Qualitative Priority Programmatic Targeted Performance Incremental Strategic Systemic Lewis Carroll Center for Language Disorders Measurable urgent systemic indicators Core qualitative strategic objectives Make up your own jargon.

7. DEFINITIONS Children born healthy, Children ready for school Safe communities, A Healthy Green City, Prosperous Economy Rate of low-birthweight babies, Percent ready at K entry, crime rate, air quality index, unemployment rate 6. How much did we do? 7. How well did we do it? 8. Is anyone better off? RESULT or OUTCOME INDICATOR or BENCHMARK PERFORMANCE MEASURE A condition of well-being for children, adults, families or communities. A measure which helps quantify the achievement of a result. A measure of how well a program, agency or service system is working. Three types: Population Performance Children born healthy Rate of low-birthweight babiesPercent ready at K entry Children ready for school crime rate Safe communities air quality index A Healthy Green City unemployment rate Prosperous Economy INDICATOR RESULT PERFORMANCE MEASURE Population Performance = Customer Results 3. 4. 5.

8. From Ends to Means ENDS MEANS From Talk to Action Population Performance RESULT INDICATOR PERFORMANCE MEASURE Customer result = Ends Service delivery = Means From Talk to Action

9. 1. Safe Communities 2. Crime rate 3. Average police response time 4. People live in safe stable affordable housing 5. % spending >30% of income on housing 6. People have living wage jobs and income 7. % of people with living wage jobs and income 8. % of participants in job training who get living wage jobs IS IT A RESULT, INDICATOR OR PERFORMANCE MEASURE? RESULT INDICATOR PERF. MEASURE RESULT INDICATOR RESULT INDICATOR PERF. MEASURE |.........

11. Translation Guide/Rosetta Stone Not the Language Police Ideas 1. A condition of well-being for children, adults, families & communities 2. 3. etc. Group 1Group 2Group 3 etc. RESULTOUTCOMEGOAL TRANSLATION Back to the Idea

12. Results – Indicators – Performance Measures in Amharic, Cambodian, Laotian, Somali, Spanish, Tigrigna, Vietnamese

13. Population Accountability

14. Community Outcomes for Christchurch, NZ 1. A Safe City 2. A City of Inclusive and Diverse Communities 3. A City of People who Value and Protect the Natural Environment 4. A Well-Governed City 5. A Prosperous City 6. A Healthy City 7. A City for Recreation, Fun and Creativity 8. City of Lifelong Learning 9. An Attractive and Well-Designed City

15. CARDIFF, WALES Community Outcomes 1. People in Cardiff are healthy 2. Cardiff has a clean attractive and sustainable environment 3. People in Cardiff are safe and feel safe 4. Cardiff has a thriving and prosperous economy 5. People in Cardiff achieve their full potential. 6. Cardiff is a great place to live, work and play. 7. Cardiff has a fair, just and inclusive society.

16. VERMONT Population Outcomes (Act 186) 1. Vermont has a prosperous economy. 2. Vermonters are healthy. 3. Vermont’s environment is clean and sustainable. 4. Vermont’s communities are safe and supportive. 5. Vermont’s families are safe, nurturing, stable, and supported. 6. Vermont’s children and young people achieve their potential. 7. Vermont’s elders and people with disabilities and people with mental conditions live with dignity and independence in settings they prefer. 8. Vermont has open, effective, and inclusive government at the State and local levels.

17. VERMONT Population Outcomes (Act 186) 1. Vermont has a prosperous economy. 2. Vermonters are healthy. 3. Vermont’s environment is clean and sustainable. 4. Vermont’s communities are safe and supportive. 5. Vermont’s families are safe, nurturing, stable, and supported. 6. Vermont’s children and young people achieve their potential. (A) Pregnant women and young people thrive. (B) Children are ready for school. (C) Children succeed in school. (D) Youth choose healthy behaviors. (E) Youth successfully transition to adulthood. 7. Vermont’s elders and people with disabilities and people with mental conditions live with dignity and independence in settings they prefer. 8. Vermont has open, effective, and inclusive government at the State and local levels.

20. Getting from Talk to Action

21. Leaking Roof (Results thinking in everyday life) Experience Measure Story behind the baseline (causes) Partners What Works Action Plan Cm of Water 9. BASELINE ? Fixed Not OK 10. Turning the Curve Action Plan # 2

22. Results / Outcomes Indicators Baseline & Story Experience

23. Criteria for Choosing Indicators as Primary vs. Secondary Measures Communication Power Proxy Power Data Power Does the indicator communicate to a broad range of audiences? Does the indicator say something of central importance about the result? Does the indicator bring along the data HERD ? Quality data available on a timely basis.

24. Choosing Indicators Worksheet Outcome or Result_______________________ Candidate Indicators Communication Power Proxy Power Data Power H M L H Measure 1 Measure 2 Measure 3 Measure 4 Measure 5 Measure 6 Measure 7 Measure 8 H D ata D evelopment A genda Safe Community H M L H M L H H H L

25. Three Part Indicator List for each Result Part 1: Primary Indicators Part 2: Secondary Indicators Part 3: Data Development Agenda ● 3 to 5 “Headline” Indicators ● What this result “means” to the community ● Meets the Public Square Test ● Everything else that’s any good (Nothing is wasted.) ● Used later in the Story behind the Curve ● New data ● Data in need of repair (quality,timeliness etc.)

26. The Matter of Baselines Baselines have two parts: history and forecast H M L History Forecast Turning the Curve Point to Point OK?

27. 45 people per day 75 people per day 2010 28 people per day MADD

28. Rebound

29. Newcastle, UK Nov 08 – Jan 09 8.5

34. - Rosell “If I include you, you will be my partner. If I exclude you, you will be my judge.”

35. Performance Accountability

36. Compliment to Traditional Evaluation Methods: DEMONSTRATING the CONTRIBUTION of complex change efforts… requires 3 elements: A Curve to Turn 2 …and it had a timely relationship to…. 3 …. a turn in the curve. c FPSI 1 We tried a bunch of stuff that had a credible chance of making a difference … Bonus: a background trend stayed the same or got worse. 4

37. Rebound

38. “All performance measures that have ever existed for any program in the history of the universe involve answering two sets of interlocking questions.”

39. How Much did we do? ( # ) How Well did we do it? ( % ) Quantity Quality Performance Measures

40. Effort How hard did we try? Effect Is anyone better off? Performance Measures

41. Effort Effect How Much How Well Performance Measures

42. How much service did we deliver? Performance Measures How well did we deliver it? How much change / effect did we produce? What quality of change / effect did we produce? Quantity Quality Effect Effort Output Input

43. How much did we do? Performance Measures How well did we do it? Is anyone better off? Quantity Quality Effect Effort # %

44. How much did we do? Education How well did we do it? Is anyone better off? Quantity Quality Effect Effort Number of students Student-teacher ratio Number of high school graduates Percent of high school graduates

45. How much did we do? Education How well did we do it? Is anyone better off? Quantity Quality Effect Effort Number of students Student-teacher ratio Percent of 9 th graders who graduate on time and enter college or employment after graduation Number of 9 th graders who graduate on time and enter college or employment after graduation

46. How much did we do? Special Education How well did we do it? Is anyone better off? Quantity Quality Effect Effort # students # assessments # IEP’s Teacher retention rate Rate of disproportional representation % IEP’s on time % proficient (reading, math) % graduation % working or in school after graduation # proficient (reading, math) # graduation # working or in school after graduation 56

47. How much did we do? Pediatric Practice How well did we do it? Is anyone better off? Number of children treated Percent of patients waiting less than 30 min in the waiting room Quantity Quality Effect Effort # children fully immunized (in the practice) % children fully immunized (in the practice)

48. How much did we do? Drug/Alcohol Treatment Program How well did we do it? Is anyone better off? Number of persons treated Percent of staff with training/ certification Number of clients off of alcohol & drugs - at exit - 12 months after exit Percent of clients off of alcohol & drugs - at exit - 12 months after exit Quantity Quality Effect Effort

49. How much did we do? Fire Department How well did we do it? Is anyone better off? Number of responses Response Time Quantity Quality Effect Effort # of fires kept to room of origin % of fires kept to room of origin

50. How much did we do? General Motors How well did we do it? Is anyone better off? # of production hrs # tons of steel Employees per vehicle produced # of cars sold $ Amount of Profit $ Car value after 2 years Quantity Quality Effect Effort Source: USA Today 9/28/98 % Market share Profit per share % Car value after 2 years

51. How much did we do? Not All Performance Measures Are Created Equal How well did we do it? Is anyone better off? Least Important Quantity Quality Effect Effort Most Important Least Most Also Very Important Quality

52. Quantity Quality Efficiency, Admin overhead, Unit cost Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety Customer Satisfaction (quality service delivery & customer benefit) Cost / Benefit ratio Return on investment Client results or client outcomes Effectiveness Value added Productivity Benefit value Process Input Effect Effort Cost 1. Did we treat you well? 2. Did we help you with your problems? * Product Output Impact RBA Categories Account for All Performance Measures (in the history of the universe) * World’s simplest complete customer satisfaction survey Total Quality Mgmt (TQM)

53. How much did we do? The Matter of Control How well did we do it? Is anyone better off? Quantity Quality Effect Effort Least Control PARTNERSHIPS Most Control

54. The Matter of Use 1.The first purpose of performance measurement is to IMPROVE PERFORMANCE. 2. Avoid the performance measurement equals punishment trap. ● Create a healthy organizational environment. ● Start small. ● Build bottom-up and top-down simultaneously.

55. CHARTS on the WALL

56. Advanced Baseline Display Your Baseline Comparison Baseline Goal (line) Target or Standard Instead: Count anything better than baseline as progress. Avoid publicly declaring targets by year if possible. ● Create targets only when they are: FAIR & USEFUL x Aspiration NOT Punishment

57. How much did we do? Performance Accountability Types of Measures found in each Quadrant How well did we do it? Is anyone better off? # Clients/customers served # Activities (by type of activity) % Common measures e.g. client staff ratio, workload ratio, staff turnover rate, staff morale, % staff fully trained, % clients seen in their own language, worker safety, unit cost % Skills / Knowledge (e.g. parenting skills) # % Attitude / Opinion (e.g. toward drugs) # % Behavior (e.g.school attendance) # % Circumstance (e.g. working, in stable housing) # % Activity-specific measures e.g. % timely, % clients completing activity, % correct and complete, % meeting standard Point in Time vs. 2 Point Comparison # %

58. How much did we do? Performance Accountability Types of Measures found in each Quadrant How well did we do it? Is anyone better off? # Clients/customers served # Activities (by type of activity) % Common measures e.g. client staff ratio, workload ratio, staff turnover rate, staff morale, % staff fully trained, % clients seen in their own language, worker safety, unit cost % Skills / Knowledge (e.g. parenting skills) # % Attitude / Opinion (e.g. toward drugs) # % Behavior (e.g.school attendance) # % Circumstance (e.g. working, in stable housing) # % Activity-specific measures e.g. % timely, % clients completing activity, % correct and complete, % meeting standard Point in Time vs. 2 Point Comparison

59. “Data is the unblinking eye of reform.” Barbary Curley, Area Director, Hyde Park Office, Boston

60. How much did we do? Service: __________________________________ How well did we do it? Is anyone better off? Primary customers # studentspatients persons trained Primary activity # hours of instruction diagnostic testsjob coursesalarms responded to If your service works really well, how are your customer's better off? # students who graduate patients who fully recover fires kept to room of origin % Unit cost Workload ratio % of ___x___ that happen on time persons who get jobs School HospitalJob Training Fire Department

61. How much did we do? Choosing Headline Measures and the Data Development Agenda How well did we do it? Is anyone better off? Quantity Quality Effect Effort # Measure 1 ---------------------------- # Measure 2 ---------------------------- # Measure 3 ---------------------------- # Measure 4 ---------------------------- # Measure 5 ---------------------------- # Measure 6 ---------------------------- # Measure 7 ---------------------------- #1 Headline #2 Headline #3 Headline #1 DDA #2 DDA #3 DDA % Measure 8 ---------------------------- % Measure 9 ----------------------------- % Measure 10 --------------------------- % Measure 11 --------------------------- % Measure 12 --------------------------- % Measure 13 --------------------------- % Measure 14 --------------------------- # Measure 15 ---------------------------- # Measure 16 ---------------------------- # Measure 17 ---------------------------- # Measure 18 ---------------------------- # Measure 19 ---------------------------- # Measure 20 ---------------------------- # Measure 21 ---------------------------- % Measure 15 ---------------------------- % Measure 16 ---------------------------- % Measure 17 ---------------------------- % Measure 18 ---------------------------- % Measure 19 ---------------------------- % Measure 20 ---------------------------- % Measure 21 ----------------------------

62. Select 3 to 5 Performance Measures ACROSS THE BOTTOM OF THE ORG CHART 3 - 5 20 – 60 – 20 Rule

63. LR UR Primary v. Secondary Direct v. Indirect Internal v. External Baseline & Story

68. School Improvement Planning

69. Required Plans Consolidated Plan School Improvement Plan Technology Plan Professional Development Plan 301 Plan NCA Accreditation District Assessment Plan School Reform Plan Customer Result, Goal or Purpose All students proficient in reading math and science All teachers and para-professionals are highly qualified All ELL students proficient in reading math and science All students are safe and drug free All students graduate from H.S. All Parents are Involved Technology use is integrated to improve student achievement Prepared for the Arizona Department of Education by the Fiscal Policy Studies Institute www.raguide.org Draft March 2004www.raguide.org Headline Performance Measures ( and Baseline Picture - Charts on the Wall) Data source Targets Story - Common crosscutting analysis - Analysis specific to given measures Partners - Basic partners in all plans - Special partners for specific plans What works - Action Plan Crosscutting strategies - Strategies specific to given measures Action Plan and Budget Appendices Demographics Secondary Measures Data Dev Agenda Information and Research Agendas About causes About what works XXXXXXXXXX XXXXXXXXXXXX XXX XXXXXXX % proficient Reading, Math and Science % highly qualified teachers % retention highly qualified teachers Ave class size Rate of drug use Rate of violence Rate of expulsions % positive attitudes % ELL proficient Reading, Math and Science H.S. graduation rate % Parents involved % Parents involved in technology % Teachers technology literate % Students technology literate UNIFIED PLANNING STRUCTURE FOR EDUCATION TH 123 - 24

70. Putting Population Performance Accountability TOGETHER and

71. Contribution relationship Alignment of measures Appropriate responsibility THE LINKAGE Between POPULATION and PERFORMANCE POPULATION ACCOUNTABILITY Healthy Births Rate of low birth-weight babies Children Ready for School Percent fully ready per K-entry assessment Self-sufficient Families Percent of parents earning a living wage CUSTOMER RESULTS # persons receiving training Unit cost per person trained # who get living wage jobs % who get living wage jobs PERFORMANCE ACCOUNTABILITY POPULATION RESULTS Job Training Program

72. Performance measure Population Indicator

73. Every time you present your program, Use a two-part approach. Result: to which you contribute to most directly. Indicators: Story: Partners: What would it take?: Your Role: as part of a larger strategy. Population Accountability Program: Performance measures: Story: Partners: Action plan to get better: Performance Accountability Your Role

74. Different Kinds of Progress 1. Data a. Population indicators Actual turned curves: movement for the better away from the baseline. b. Program performance measures: customer progress and better service: How much did we do? How well did we do it? Is anyone better off? 2. Accomplishments: Positive activities, not included above. 3. Stories behind the statistics that show how individuals are better off.

75. Board of Directors Meeting AGENDA 1. New data 2. New story behind the curves 3. New partners 4. New information on what works. 5. New information on financing 6. Changes to action plan and budget 7. Adjourn 1. New data 2. New story behind the curves 3. New partners 4. New information on what works. 5. New information on financing 6. Changes to action plan and budget 7. Adjourn

76. SUMMARY

78. 3 - kinds of performance measures. How much did we do? How well did we do it? Is anyone better off? RBA in a Nutshell 2 – 3 - 7 2 - kinds of accountability Population accountability Performance accountability 7 - questions from ends to means in less than an hour. plus language discipline Results & Indicators Performance measures Baselines & Turning the Curve

79. EXERCISES

80. 2017 2019 TOTAL: 25.6 million

81. 2016 2018 TOTAL: 1.2 million

82. 2015 2017 TOTAL: 271,000

83. 2015 2017 TOTAL: 9,079

84. 2016 2018 9.8 million 5.0 million TOTAL: 14.8 million

85. Creating a Working Baseline from Group Knowledge Now Indicator or Performance Measure 65% Not OK? Backcasting Forecasting

86. Turn the Curve Exercise: Population Well-being 5 min:Starting Points - timekeeper and reporter - geographic area - two hats (yours plus partner’s) 10 min:Baseline - pick a result and a curve to turn - forecast (to 2018 ) – OK or not OK? 15 min:Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes 15 min:What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works 10 min:Report convert notes to one page Two pointers to action

87. ONE PAGE Turn the Curve Report: Population Result: _______________ Indicator (Lay Definition) Indicator Baseline Story behind the baseline --------------------------- --------------------------- (List as many as needed) Partners --------------------------- --------------------------- (List as many as needed) Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost Sharp Edges 4. --------- Off the Wall

88. Turn the Curve Exercise: Program Performance 5 min:Starting Points - timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s) 10 min:Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast (to 2017 ) – OK or not OK? 15 min:Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes 15 min:What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works 10 min:Report convert notes to one page Two pointers to action

89. Program: _______________ Performance Measure (Lay definition) Performance Measure Baseline Story behind the baseline --------------------------- --------------------------- (List as many as needed) Partners --------------------------- --------------------------- (List as many as needed) Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost ONE PAGE Turn the Curve Report: Performance Sharp Edges 4. --------- Off the Wall

90. Turn the Curve Exercise – Lessons Talk to Action in an hour 1. How was this different from other processes? What worked and what didn’t work? 2. Why did we ask for: a. Results before indicators? b. Forecast? c. Story? d. No cost / low cost? e. Two hats? f. Crazy idea? g. Only 3 best ideas? 3. Do you think a lay audience could understand the reports? 4. How many think you could lead this exercise with a small group? (2+ curves at the same time)

91. SUMMARY CLOSING EVALUATION

92. 3 - kinds of performance measures. How much did we do? How well did we do it? Is anyone better off? RBA in a Nutshell 2 – 3 - 7 2 - kinds of accountability Population accountability Performance accountability 7 - questions from ends to means in less than an hour. plus language discipline Results & Indicators Performance measures Baselines & Turning the Curve

93. Next Steps: 1. What’s one thing I can personally do with what I learned today? 2. What’s one thing I would like to ask someone else to do?

94. Resources www.raguide.org www.resultsaccountability.com RBA Facebook Group Book - DVD Orders amazon.com resultsleadership.org

96. THANK YOU !

97. School District Performance School (Performance Measures) Partners Community School = Contribution Measure baseline Community Quality of Life Community School Contribution Relationships (Performance Measures) FPSI ==

98. Next Generation CONTRACTING p. 28

99. Next Generation Contracting Contract Provisions Provision 1. Specify the 3 to 5 most important performance measures (from the How well did we do it? and Is anyone better off? categories). Provision 2. Specify that the contractor will use a continuous improvement process (the RBA 7 Questions). Provision 3. Specify how the funder and contractor will work in partnership to maximize LR customer results (quarterly meetings using the 7 questions as the agenda). Provision 4. Specify that the funder will work with the funding community to simplify and standardize contracting and performance reporting.

100. Provision 5: : Clear articulation of role in population/community well-being using the language of contribution not attribution. Provision 6: 10% for quality management and administration. Provision 7: Multi-year funding using 3 year rolling contracts Provision 8: Use of targets that are fair and useful. Provision 9: Fund flexibility and virtual funding pool: transfer of up to 10% across line items and program lines. Provision 10: Request for Results: Getting past the sometimes negative effects of competitive RFP contracting or tendering. Next Generation Contracting Contract Provisions

103. RESULTS BASED BUDGETING Do we need it? Is it working? POPULATION ACCOUNTABILITY PERFORMANCE ACCOUNTABILITY