“If I include you, you will be my partner. If I exclude you, you will be my judge.” - Rosell

PERFORMANCE ACCOUNTABILITY

Results-Based Accountability is made up of two parts: Population Accountability is about the well-being of WHOLE POPULATIONS for Communities – Cities – Counties – States - Nations Introduction and the difference between population and performance accountability: We are going to talk about two different kinds of accountability: Accountability for whole populations, like all children in Los Angeles, all elders in Chicago, all residents of North Carolina. This first kind of accountability is not the responsibility of any one agency or program. If we talk for example about “all children in your community being healthy,” who are some of the partners that have a role to play? Notice that the traditional answer is “It’s the health department.” It’s got the word health in it and so it must be the responsibility of the health department. And yet one of the things we have learned in the last 50 years is that the health department by itself can’t possibly produce health for all children without the active participation of many other partners. And that’s the nature of this first kind of accountability. It’s not about the health department. It’s about the kind of cross community partnerships necessary to make progress on quality of life for any population. Now the second kind of accountability, Performance Accountability, is about the health department. It’s about the programs and services we provide, and our role as managers, making sure our programs are working as well as possible. These are two profoundly different kinds of accountability. We going to talk about how to do each one well and then how they fit back together again. Performance Accountability is about the well-being of CUSTOMER POPULATIONS for Programs – Agencies – Service Systems

This list comes from the Georgia Policy Council for Children and Families and is used by the network of Family Connections Councils in Georgia. Georgia has gone one step further and identified 25 indicators to tell if these conditions are being achieved. And Georgia has produced a report card at the state level and for each of the 159 counties. Many other places in the United States have produced such report cards, including CALIFORNIA                 Contra Costa County: www.cccoe.k12.ca.us                 San Mateo County: www.pls.lib.ca.us/healthysmc/33/children.pdf                 Santa Cruz County: appliedsurveyresearch.org/cap_report.htm                 Silicon Valley Joint Venture: jointventure.org                                            GEORGIA                 Georgia Policy Council for Children and Families, and The Family Connection:                                                                      gpc-fc.org MINNESOTA                 Hennepin County: www.co.hennepin.mn.us/opd/opd.htm          OHIO                 Montgomery County Family and Children First Council:                                                                      http://www.fcfc.montco.org         OREGON                 Oregon Progress Board: econ.state.or.us/opb PENNSYLVANIA                 Philadelphia Safe and Sound: Children's Report Card and Children's Budget                                                                   www.philasafesound.org VERMONT                 Agency for Human Services: Community Profiles: ahs.state.vt.us                 . Links to the best of these sites can be found on www.raguide.org.

“All performance measures that have ever existed for any program in the history of the universe involve answering two sets of interlocking questions.”

How Much How Well did we do? did we do it? Performance Measures Quantity Quality How Much did we do? ( # ) How Well did we do it? ( % ) All performance measures can be derived from the cross between two sets of interlocking questions: How much did we do? And How well did we do it?

Effort Effect How hard did we try? Is anyone better off? Performance Measures Effort How hard did we try? Effect Is anyone better off? Vs. these two dimensions of the work itself: Effort and Effect

Effort Effect How Much How Well Performance Measures Effort Effect How Much How Well

How much service did we deliver? How well did we deliver it? Performance Measures Quantity Quality How much service did we deliver? How well did we deliver it? Effect Effort Output Input How much change / effect did we produce? What quality of change / effect did we produce? This leads to a four part or four quadrant way of describing the different types of performance measures.

How much did we do? How well did we do it? Is anyone better off? Performance Measures Quantity Quality How much did we do? How well did we do it? Effect Effort Is anyone better off? Or an even simpler construction: How much did we do? How well did we do it? Is anyone better off? # %

Student-teacher ratio Education Quantity Quality How much did we do? How well did we do it? Number of students Student-teacher ratio Effect Effort Is anyone better off? This illustrates the different types of measures for schools. Number of high school graduates Percent of high school graduates

Student-teacher ratio Education Quantity Quality How much did we do? How well did we do it? Number of students Student-teacher ratio Effect Effort Is anyone better off? Percent of 9th graders who graduate on time and enter college or employment after graduation Number of 9th graders who graduate on time and enter college or employment after graduation Here, with a tougher, more challenging measure in the lower right quadrant.

Number of children treated Pediatric Practice Quantity Quality How much did we do? How well did we do it? Percent of patients waiting less than 30 min in the waiting room Number of children treated Effect Effort Is anyone better off? # children fully immunized (in the practice) % children fully immunized (in the practice) Examples of measures for a typical health plan or practice.

Drug/Alcohol Treatment Program Quantity Quality How much did we do? How well did we do it? Percent of staff with training/ certification Number of persons treated Effect Effort Is anyone better off? Number of clients off of alcohol & drugs - at exit - 12 months after exit Percent of clients off of alcohol & drugs - at exit - 12 months after exit Examples of measures for a typical drug and alcohol treatment program.

# of fires kept to room of origin % of fires kept to room of origin Fire Department Quantity Quality How much did we do? How well did we do it? Number of responses Response Time Effect Effort Is anyone better off? Examples of measures for a fire department. # of fires kept to room of origin % of fires kept to room of origin

Employees per vehicle produced General Motors Quantity Quality How much did we do? How well did we do it? Employees per vehicle produced # of production hrs # tons of steel Effect Effort Is anyone better off? # of cars sold $ Amount of Profit $ Car value after 2 years % Market share Profit per share % Car value after 2 years Examples of measures for a private sector business, in this case the auto industry. These examples are taken from an article in USA Today from September 1998 Source: USA Today 9/28/98

Not All Performance Measures Are Created Equal Quantity Quality Quality How much did we do? How well did we do it? Least Least Important Also Very Important Effect Effort Is anyone better off? So why sort measure for your program into these categories? Simple. These categories are not equally important. The upper left is the least important. And yet we have some people who spend their whole careers living in this quadrant counting cases and activity. Somehow we have to push the discussion to the lower right quadrant, the one that measures whether our customers are better off. Most Important Most

Total Quality Mgmt (TQM) Admin overhead, Unit cost RBA Categories Account for All Performance Measures (in the history of the universe) Total Quality Mgmt (TQM) Efficiency, Admin overhead, Unit cost Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety Efficiency Admin overhead, Unit cost Cost Quantity Quality Process Input Effort Customer Satisfaction (quality service delivery & customer benefit) Product Output Impact Effect Cost / Benefit ratio Return on investment Client results or client outcomes This scheme accounts for all performance measures in the history of the universe and this chart is an attempt to back that claim up. A lot of us grew up with the terms “efficiency” and “effectiveness” as the terms of art in performance measurement. And you would think, considering their age and venerability, that they would somehow account for all performance measures. But interestingly enough they don’t. Efficiency is only one type of measure in the upper right quadrant. Effectiveness shares the stage with many other measures. Benefit value Effectiveness Effectiveness Value added Productivity

Total Quality Mgmt (TQM) RBA Categories Account for All Performance Measures (in the history of the universe) Total Quality Mgmt (TQM) Efficiency, Admin overhead, Unit cost Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety Cost Quantity Quality Process Input Effort Customer Satisfaction (quality service delivery & customer benefit) Product Output Impact Effect Cost / Benefit ratio Return on investment Client results or client outcomes Other measures, in addition to efficiency, in the upper right quadrant, that answer the question “How well did we deliver services.” Benefit value Effectiveness Value added Productivity

Total Quality Mgmt (TQM) RBA Categories Account for All Performance Measures (in the history of the universe) Total Quality Mgmt (TQM) Efficiency, Admin overhead, Unit cost Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety Cost Quantity Quality Process Input Effort 1. Did we treat you well? 2. Did we help you with your problems? * Customer Satisfaction (quality service delivery & customer benefit) Product Output Impact Effect Cost / Benefit ratio Return on investment Client results or client outcomes Customer satisfaction has two different dimensions which are often mixed up together. Customer satisfaction with how well service is delivered is different from customer satisfaction with whether the service helped with the customer’s problems. The world’s simplest, yet complete, customer satisfaction survey: “Did we treat you well?” and “Did we help you with your problems?” Benefit value Effectiveness Value added Productivity * World’s simplest complete customer satisfaction survey

Sometimes Customer Service can go too far.

Total Quality Mgmt (TQM) OBA Categories Account for All Performance Measures (in the history of the universe) Total Quality Mgmt (TQM) Efficiency, Admin overhead, Unit cost Staffing ratios, Staff turnover Staff morale, Access, Waiting time, Waiting lists, Worker safety Cost Quantity Quality Process Input Effort Customer Satisfaction (quality service delivery & customer benefit) Product Output Impact Effect Cost / Benefit ratio Return on investment Client results or client outcomes Customer satisfaction has two different dimensions which are often mixed up together. Customer satisfaction with how well service is delivered is different from customer satisfaction with whether the service helped with the customer’s problems. The world’s simplest, yet complete, customer satisfaction survey: “Did we treat you well?” and “Did we help you with your problems?” Benefit value Effectiveness Value added Productivity

Most Control Least Control The Matter of Control Quantity Quality How much did we do? How well did we do it? Most Control Effect Effort Is anyone better off? As you move from the least important measures to the most important measures, you go from having the most control to having the least control. And this is another reason why people spend their whole lives in the upper left quadrant. Fear. It can be scary to look at the data in the lower right quadrant. But ask people why they went into their profession and the answers all lie in the lower right, in the ways in which we try to make our customer’s lives better. Least Control PARTNERSHIPS

Advanced Baseline Display Create targets only when they are: FAIR & USEFUL Aspiration NOT Punishment Goal (line) ● Target or Standard Avoid publicly declaring targets by year if possible. x Your Baseline Note: You can use this slide here or after the discussion of standards in the Performance Accountability section. Here is a way to show all three comparisons on the same chart. Your baseline, A comparison baseline, And a goal, target or standard line, as a horizontal line – the idea being that you turn the curve and cross the goal line as soon as possible. Avoid publicly declaring year by year targets, if you can. Instead, count anything better than baseline as progress. Instead: Count anything better than baseline as progress. Comparison Baseline

" Bummer of a Birthmark Hal " © The Farside 1989 " Bummer of a Birthmark Hal "

Program Performance Measures Quantity Quality How much did we do? How well did we do it? Effect Effort Is anyone better off? Remember the three basic categories of performance measures? Now let’s look at what measures fall in each of these categories in more detail. # %

Performance Accountability Types of Measures found in each Quadrant How much did we do? How well did we do it? % Common measures e.g. client staff ratio, workload ratio, staff turnover rate, staff morale, % staff fully trained, % clients seen in their own language, worker safety, unit cost # Clients/customers served % Activity-specific measures e.g. % timely, % clients completing activity, % correct and complete, % meeting standard # Activities (by type of activity) Is anyone better off? % Skills / Knowledge (e.g. parenting skills) # This chart shows in detail the different types of measure we typically find in each quadrant, and the measures that go with the three basic categories of performance measurement: How much did we do? How well did we do it? Is anyone better off? In the upper left, How much did we do? Quadrant, we typically count customers and activities. In the upper right, How well did we do it? Quadrant, there are a set of common measures that apply to many different programs. And there is a set of activity specific measures. For each activity in the upper left, there is one or more measures that tell how well that particular activity was performed, usually having to do with timeliness or correctness. In the lower quadrants, Is anyone better off? We usually have # and % pairs of the same measure. And these measures usually have to do with one of these four dimensions of better-offness: Skills/knowledge, Attitude, Behavior and Circumstance. For each of these measures, we can use point in time measures or point to point improvement measures. Point in Time vs. 2 Point Comparison % Attitude / Opinion (e.g. toward drugs) # # % % Behavior (e.g.school attendance) # % Circumstance (e.g. working, in stable housing) #

If your service works really well, how are your customer's better off? School Hospital Job Training Fire Department How much did we do? How well did we do it? Primary customers Unit cost # students patients persons trained Workload ratio Primary activity % of ___x___ that happen on time # job courses alarms responded to diagnostic tests hours of instruction Is anyone better off? If your service works really well, how are your customer's better off? This chart shows in detail the different types of measure we typically find in each quadrant, and the measures that go with the three basic categories of performance measurement: How much did we do? How well did we do it? Is anyone better off? In the upper left, How much did we do? Quadrant, we typically count customers and activities. In the upper right, How well did we do it? Quadrant, there are a set of common measures that apply to many different programs. And there is a set of activity specific measures. For each activity in the upper left, there is one or more measures that tell how well that particular activity was performed, usually having to do with timeliness or correctness. In the lower quadrants, Is anyone better off? We usually have # and % pairs of the same measure. And these measures usually have to do with one of these four dimensions of better-offness: Skills/knowledge, Attitude, Behavior and Circumstance. For each of these measures, we can use point in time measures or point to point improvement measures. # % fires kept to room of origin persons who get jobs patients who fully recover students who graduate

Choosing Headline Measures and the Data Development Agenda Quantity Quality How much did we do? How well did we do it? #3 DDA # Measure 1 ---------------------------- # Measure 2 ---------------------------- # Measure 3 ---------------------------- # Measure 4 ---------------------------- # Measure 5 ---------------------------- # Measure 6 ---------------------------- # Measure 7 ---------------------------- % Measure 8 ---------------------------- % Measure 9 ----------------------------- % Measure 10 --------------------------- % Measure 11 --------------------------- % Measure 12 --------------------------- % Measure 13 --------------------------- % Measure 14 --------------------------- #2 Headline Effect Effort Is anyone better off? Examples of measures for the Jim Casey Youth Opportunity Initiative # Measure 15 ---------------------------- # Measure 16 ---------------------------- # Measure 17 ---------------------------- # Measure 18 ---------------------------- # Measure 19 ---------------------------- # Measure 20 ---------------------------- # Measure 21 ---------------------------- #2 DDA % Measure 15 ---------------------------- % Measure 16 ---------------------------- % Measure 17 ---------------------------- % Measure 18 ---------------------------- % Measure 19 ---------------------------- % Measure 20 ---------------------------- % Measure 21 ---------------------------- #3 Headline #1 Headline #1 DDA

Select 3 to 5 Performance Measures ACROSS THE BOTTOM OF THE ORG CHART 3 - 5 3 - 5 This section presents instructions and reporting formats for the two turn the curve exercises, one for population accountability and one for performance accountability. And other exercises 3 - 5 3 - 5 3 - 5 3 - 5 3 - 5 3 - 5 3 - 5 20 – 60 – 20 Rule

Primary v. Secondary Direct v. Indirect Internal v. External LR UR Baseline & Story I & R Agenda CAUSES Here is the thinking process in the form of 7 plain language common sense questions. These questions should be asked an answered periodically (monthly, quarterly) at every intersection of supervision from the top to the bottom of the organization. This is the most important take-away page for performance measurement. It can be used immediately without any further training. I & R Agenda SOLUTIONS

Welsh Epilepsy Unit case study was also posted on the same page and that's the one showing benefits such as:" • The average length of time from seizure to a confirmed diagnosis has decreased by 81 days from 111 days to 30 days • The number of patients who have been seen by a specialist within the NICE guideline of two weeks has increased from 35% to 61% • The average waiting time to see a specialist has decreased from 22 days to 11 days • The number of admissions following a seizure have decreased from 5 a month to 2 a month on average"

New Mexico Christus St. Vincents Medical Center

How Population & Performance Accountability FIT TOGETHER

THE LINKAGE Between POPULATION and PERFORMANCE POPULATION ACCOUNTABILITY Healthy Births Rate of low birth-weight babies Stable Families Rate of child abuse and neglect Children Ready for School Percent fully ready per K-entry assessment POPULATION RESULTS Contribution relationship PERFORMANCE ACCOUNTABILITY Alignment of measures Pre-natal Nutrition Program % attendance Unit cost # of women served The relationship is a “contribution” relationship, not a cause and effect relationship. What we do for our customers is our contribution to what we and our partners are trying to do across the community. Often the only difference between a population indicator and a lower right (Is anyone better off?) performance measure is the difference in scale between a client population and the total population. This allows us to think about how our work is aligned with what we are trying to accomplish across the community. It allows us to think about how the measures we use at the program level relate to those at the population level. And it allows us to avoid the trap of holding programs responsible for population level change. We can hold program responsible for what they do for their clients. We must hold ourselves, across the community, responsible for the well being of the population. Appropriate responsibility # low birth-weight births (for participating women) % low birth-weight births (for participating women) CUSTOMER RESULTS

Performance measure Population Indicator As the service population approaches the total population, the measures of client better-offness begin to play a double role. They are used as management measures for the service system and they also can be used as an indicator proxy for the well-being of the whole population. High School graduation rate is a good example. It is used by the school system as a measure of performance. And it is also used as an indicator by community collaboratives for the results “All Children Succeed in School.”

Every time you present your program, Use a two-part approach. Population Accountability Result: to which you contribute to most directly. Indicators: Story: Every time you present your program, Use a two-part approach. Partners: What would it take?: Your Role: as part of a larger strategy. Your Role Performance Accountability Program: Budgets of the future will have two parts: Volume I will present a picture of quality of life results and indicators and what is being done by government and its partners to improve. Volume II will present performance measures for departments and programs. Both will use the Baseline, Story, What Works and Strategy format shown above. This is a fractal… the same pattern at every level of magnification. Performance measures: Story: Partners: Action plan to get better:

Every time you present your program, Use a two-part approach. Population Accountability Result: to which you contribute to most directly. Indicators: Story: Every time you present your program, Use a two-part approach. Partners: What would it take?: Your Role: within the larger strategy. Your Role Performance Accountability Program: What happens when the population perspective is missing? Any questions about population quality of life get loaded onto the performance discussion. And the program is held unfairly responsible for producing population / community level change that it can not possibly deliver by itself. Both perspectives are needed to respond to the combination of population and performance questions you will inevitably get. Volume I will present a picture of quality of life results and indicators and what is being done by government and its partners to improve. Volume II will present performance measures for departments and programs. Both will use the Baseline, Story, What Works and Strategy format shown above. This is a fractal… the same pattern at every level of magnification. Performance measures: Story: Partners: Action plan to get better:

Every time you present your program, Use a two-part approach. Population Accountability Result: to which you contribute to most directly. Indicators: Shortcut Story: Every time you present your program, Use a two-part approach. Partners: What would it take?: Your Role: as part of a larger strategy. Your Role Performance Accountability Program: At a minimum, any program or agency ought to be able to articulate the role it plays in contributing to quality of life Budgets of the future will have two parts: Volume I will present a picture of quality of life results and indicators and what is being done by government and its partners to improve. Volume II will present performance measures for departments and programs. Both will use the Baseline, Story, What Works and Strategy format shown above. This is a fractal… the same pattern at every level of magnification. Performance measures: Story: Partners: Action plan to get better:

The Special Case of Prevention Programs (Read page 100)

New Mexico smoking rate has declined significantly between 2001 and 2010 However rates are still high among certain groups including lower income unemployed or uninsured Smoking among high school youth is higher than the national rate

Result: HEALTHY PEOPLE. Indicator: Population Accountability Result: HEALTHY PEOPLE. Indicator: Story: Partners: What would it take to significantly reduce this rate? Your Role Your Role = Smoking Prevention Program + Contribution Performance Accountability Program: Smoking Prevention Program Performance measures: Budgets of the future will have two parts: Volume I will present a picture of quality of life results and indicators and what is being done by government and its partners to improve. Volume II will present performance measures for departments and programs. Both will use the Baseline, Story, What Works and Strategy format shown above. This is a fractal… the same pattern at every level of magnification. - % youth in HS assembly who change attitudes about tobacco - Ad campaign effectiveness in hitting target audience (# reached by % target demographic) Story: Partners: Action plan to get better:

Different Kinds of Progress 1. Data a. Population indicators Actual turned curves: movement for the better away from the baseline. b. Program performance measures: customer progress and better service: How much did we do? How well did we do it? Is anyone better off? There are four kinds of progress which can be reported. The first at the population level, and 2 3 and 4 at the program agency or service system level. 2. Accomplishments: Positive activities, not included above. 3. Stories behind the statistics that show how individuals are better off.

Board of Directors Meeting AGENDA 1. New data 1. New data 2. New story behind the curves 2. New story behind the curves 3. New partners 3. New partners 4. New information on what works. 4. New information on what works. Community collaborative groups and programs and agencies could use this as the agenda for their meetings. The meeting would be aligned with the thinking process that produced the action plan. Each iteration of this thinking process will improve the action plan. 5. New information on financing 5. New information on financing 6. Changes to action plan and budget 6. Changes to action plan and budget 7. Adjourn 7. Adjourn

SUMMARY

RBA in a Nutshell 2 – 3 - 7 2 - kinds of accountability      Population accountability      Performance accountability plus language discipline      Results & Indicators      Performance measures  3 - kinds of performance measures.      How much did we do?      How well did we do it?      Is anyone better off? 7 - questions from ends to means in less than an hour. Baselines & Turning the Curve

“If you do what you always did, you will get what you always got.” Kenneth W. Jenkins President, Yonkers NY NAACP

“We will succeed together… or fail separately.” - Winston Churchill or - Sonny and Cher

Book - DVD Orders amazon.com resultsleadership.org THANK YOU ! Book - DVD Orders amazon.com resultsleadership.org

“Data is the unblinking eye of reform.” Barbary Curley, Area Director, Hyde Park Office, Boston

EXERCISES This section presents instructions and reporting formats for the two turn the curve exercises, one for population accountability and one for performance accountability. And other exercises

Creating a Working Baseline from Group Knowledge Indicator or Performance Measure Not OK? 65% Forecasting Backcasting Use this method to create a working baseline when only one point of data is available or no data is available. Step 1: Estimate where you are now. Step 2: Backcasting: Where have we been over the last few years – getting better worse or about the same. Draw an increasing, decreasing or flat history. Step 3: Forecasting: Where are we headed if we don't do something more or different? Not where we want to go, where we are going. Step 4. Ask yourself I this is OK or not. If it's not OK then you can talk about turning the curve. NOTE: If you draw the improvement line make sure people don't interpret that as a promise. Now

Turn the Curve Exercise: Program Performance 5 min: Starting Points - timekeeper and reporter - identify a program to work on - two hats (yours plus partner’s) 10 min: Performance measure baseline - choose 1 measure to work on – from the lower right quadrant - forecast (to 2017) – OK or not OK? 15 min: Story behind the baseline - causes/forces at work - information & research agenda part 1 - causes Two pointers to action 15 min: What works? (What would it take?) - what could work to do better? - each partners contribution - no-cost / low-cost ideas - information & research agenda part 2 – what works Participant instructions for the performance turn the curve exercise. 10 min: Report convert notes to one page

ONE PAGE Turn the Curve Report: Performance 4. --------- Off the Wall ONE PAGE Turn the Curve Report: Performance Program: _______________ Performance Measure (Lay definition) Performance Measure Baseline Story behind the baseline --------------------------- --------------------------- (List as many as needed) Partners --------------------------- --------------------------- (List as many as needed) Group report out format for the performance turn the curve exercise. Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost Sharp Edges 4. --------- Off the Wall

Turn the Curve Exercise – Lessons Talk to Action in an hour 1. How was this different from other processes? What worked and what didn’t work? 2. Why did we ask for: a. Results before indicators? b. Forecast? c. Story? d. No cost / low cost? e. Two hats? f. Crazy idea? g. Only 3 best ideas? 3. Do you think a lay audience could understand the reports? 4. How many think you could lead this exercise with a small group? (2+ curves at the same time)

Next Steps: 1. What’s one thing I could personally do with what I learned today? 2. What’s one thing I would like to ask someone else to do to support this work?

Book - DVD Orders amazon.com resultsleadership.org Resources www.raguide.org www.resultsaccountability.com RBA Facebook Group Book - DVD Orders amazon.com resultsleadership.org

My question is: Are we making a contribution? This list comes from the Georgia Policy Council for Children and Families and is used by the network of Family Connections Councils in Georgia. Georgia has gone one step further and identified 25 indicators to tell if these conditions are being achieved. And Georgia has produced a report card at the state level and for each of the 159 counties. Many other places in the United States have produced such report cards, including CALIFORNIA                 Contra Costa County: www.cccoe.k12.ca.us                 San Mateo County: www.pls.lib.ca.us/healthysmc/33/children.pdf                 Santa Cruz County: appliedsurveyresearch.org/cap_report.htm                 Silicon Valley Joint Venture: jointventure.org                                            GEORGIA                 Georgia Policy Council for Children and Families, and The Family Connection:                                                                      gpc-fc.org MINNESOTA                 Hennepin County: www.co.hennepin.mn.us/opd/opd.htm          OHIO                 Montgomery County Family and Children First Council:                                                                      http://www.fcfc.montco.org         OREGON                 Oregon Progress Board: econ.state.or.us/opb PENNSYLVANIA                 Philadelphia Safe and Sound: Children's Report Card and Children's Budget                                                                   www.philasafesound.org VERMONT                 Agency for Human Services: Community Profiles: ahs.state.vt.us                 . Links to the best of these sites can be found on www.raguide.org. My question is: Are we making a contribution? My question is: Are we making an impact?

A large group of professionals built the Titanic. Never be afraid to try something new. Remember that a  lone amateur built the Ark.  A large group of  professionals built the Titanic. - Dave Barry

Book - DVD Orders amazon.com resultsleadership.org THANK YOU ! Book - DVD Orders amazon.com resultsleadership.org

Communication Power Proxy Power Data Power Criteria for Choosing Indicators as Primary vs. Secondary Measures Communication Power Does the indicator communicate to a broad range of audiences? Proxy Power Does the indicator say something of central importance about the result? These are three criteria that have been used to choose indicators for a result. From www.raguide.org: Given a set of candidate indicators, it is then possible to use criteria to select the best indicators to represent the result. Using the best of what=s available necessarily means that this will be about approximation and compromise. If we had a thousand measures, we could still not fully capture the health and readiness of young children. We use data to approximate these conditions and to stand as proxies for them. There are three criteria which can be used to identify the best measures:   Communication Power: Does the indicator communicate to a broad range of audiences? It is possible to think of this in terms of the public square test. If you had to stand in a public square and explain to your neighbors "what we mean, in this community, by children healthy and ready for school," what two or three pieces of data would you use? Obviously you could bring a thick report to the square and begin a long recitation, but the crowd would thin quickly. It is hard for people to listen to, absorb or understand more than a few pieces of data at time. They must be common sense, and compelling, not arcane and bureaucratic. Communication power means that the data must have clarity with diverse audiences. Proxy Power: Does the indicator say something of central importance about the result? (Or is it peripheral?) Can this measure stand as a proxy for the plain English statement of well-being? What pieces of data really get at the heart of the matter? Another simple truth about indicators is that they run in herds. If one indicator is going in the right direction, often others are as well. You do not need 20 indicators telling you the same thing. Pick the indicators which have the greatest proxy power, i.e. those which are most likely to match the direction of the other indicators in the herd Data Power: Do we have quality data on a timely basis? We need data which is reliable and consistent. And we need timely data so we can see progress - or the lack thereof - on a regular and frequent basis. Problems with data availability, quality or timeliness can be addressed as part of the data development agenda ! Identify primary and secondary indicators, and a data development agenda. When you have assessed the candidate indicators using these criteria, you will have sorted indicators into three categories:   Primary indicators: those 3 or 4 most important measures which can be used as proxies in the public process for the result.  You could use 20 or 40, but peoples= eyes would glaze over. We need a handful of measures to tell us how we=re doing at the highest level.   Secondary indicators: All the other data that=s any good. We will use these measures in assessing the story behind the baselines, and in the Behind the scenes@ planning work. We do not throw away good data. We need every bit of information we can get our hands on to do this work well.   A data development agenda: It is essential that we include investments in new and better data as an active part of our work. This means the creation of a data development agenda - a set of priorities of where we need to get better. Does the indicator bring along the data HERD? Data Power Quality data available on a timely basis.

Choosing Indicators Worksheet Outcome or Result_______________________ Safe Community Communication Power Proxy Power Data Power Candidate Indicators Measure 1 Measure 2 Measure 3 Measure 4 Measure 5 Measure 6 Measure 7 Measure 8 H M L H M L H M L H H H Rate each candidate measure high, medium or low on each criteria. Those that score highest rise to the top. Those that score H, H, L are powerful measures for which we do not now have data. These form the basis for the data development agenda. H H L Data Development Agenda

Three Part Indicator List for each Result Part 1: Primary Indicators ● 3 to 5 “Headline” Indicators ● What this result “means” to the community ● Meets the Public Square Test Part 2: Secondary Indicators ● Everything else that’s any good (Nothing is wasted.) ● Used later in the Story behind the Curve This sorting process will create a three part list for each result. This list will change over time as new data is developed. Part 3: Data Development Agenda ● New data ● Data in need of repair (quality,timeliness etc.)

Alternative to Traditional Evaluation Methods: DEMONSTRATING a CONTRIBUTION to complex change efforts… requires 3 elements: A Curve to Turn 2 …and it had a timely relationship to…. 3 …. a turn in the curve. 1 We tried a bunch of stuff that had a credible chance of making a difference … c FPSI

The Matter of Use The first purpose of performance measurement is to IMPROVE PERFORMANCE. 2. Avoid the performance measurement equals punishment trap. ● Create a healthy organizational environment. ● Start small. ● Build bottom-up and top-down simultaneously. The first purpose of performance measurement is to improve performance. We lose this simple idea in all the fads that run through this field. We forget that the purpose of the work is to get better. For many people, their only experience with performance measurement involves punishment. We must create a healthy environment in our organizations where people can use the most important information about what they do to get better. There are three ways to compare performance: To ourselves, to others and to standards. The first order of business is comparing to ourselves. Using a baseline, we can try to do better than our own history. We can compare to others when it is a fair comparison. And we can compare to standards.

Comparing Performance 1. To Ourselves Can we do better than our own history? 2. To Others When it is a fair apples/apples comparison. 3. To Standards When we know what good performance is.

Comparing Performance 1. To Ourselves First Can we do better than our own history? Using a Baseline CHART ON THE WALL 2. To Others When it is a fair apples/apples comparison. 3. To Standards When we know what good performance is.

CHARTS on the WALL We’ve talked about two different kinds of accountability. Now let’s look at how they fit together.

CHARTS ON THE WALL

Comparing Performance 1. To Ourselves First Can we do better than our own history? Reward? 2. To Others When it is a fair apples/apples comparison. Punish? Comparing performance: To others, when it is a fair apples/apples comparison What happens when you compare different providers of the same service on a measure? You get a bunch of providers clustered in the middle (click). You get some outliers high (click). And you get some outliers low (click). What happens when you reward these people (click), and you punish these people (click)? Well, before you answer this question you have to know why are these people doing well and why are these people doing poorly. Maybe these (top) people have all the easy cases. And these (low) people have all the tough cases. So you reward one and punish the other, and what message do you send throughout the service system? “Skim the easy cases for yourself. Dump the tough cases on someone else.” So if you’re not careful you can actually do damage to the service system and the people you are trying to serve. We’ve got to make sure that we go behnd these numbers so that we can know, “Are these people doing something exemplary or do they have an easier caseload?” “Are these people screwing up or do they have a tougher caseload?” We have to know the answers to these questions. For those trying to implement a results-based contracting system, I recommend a 3 year moratorium on rewards and punishments associated with the use of data. Give people time to learn how to do it right, working to improve against their own baseline. Then at the end of the period you can add rewards and if necessary punishments. You never give up your right to cancel contracts, so you always preserve that bottom line safeguard. 3. To Standards When we know what good performance is.

Comparing Performance 1. To Ourselves First Can we do better than our own history? 2. To Others When it is a fair apples/apples comparison. 3. To Standards When we know what good performance is.

Next Generation Contracting Contract Provisions Provision 1. Specify the 3 to 5 most important performance measures (from the How well did we do it? and Is anyone better off? categories). Provision 2. Specify that the contractor will use a continuous improvement process (the RBA 7 Questions). Provision 3. Specify how the funder and contractor will work in partnership to maximize LR customer results (quarterly meetings using the 7 questions as the agenda). We have lots of examples of well-established standards in the upper right (How well did we do it?) quadrant, because we know what good service delivery looks like. But standards in the lower right (Is anyone better off?) quadrant are almost always experimental. This is partly because of the different mixes of easy and hard cases in different caseloads or workloads. Provision 4. Specify that the funder will work with the funding community to simplify and standardize contracting and performance reporting.

Next Generation Contracting Contract Provisions Provision 5: : Clear articulation of role in population/community well-being using the language of contribution not attribution. Provision 6: 10% for quality management and administration. Provision 7: Multi-year funding using 3 year rolling contracts Provision 8: Use of targets that are fair and useful. Provision 9: Fund flexibility and virtual funding pool: transfer of up to 10% across line items and program lines. Provision 10: Request for Results: Getting past the sometimes negative effects of competitive RFP contracting or tendering.