Developing Indicators of Success Tammy Horne, Ph.D. WellQuest Consulting Ltd. (780) 451-6145 thorne@wellquestconsulting.com www.wellquestconsulting.com

Indicators of Success/Progress Indicators are specific, observable, and measurable characteristics or changes that show the progress a program is making toward achieving a specified outcome. (CDC, 2001, page 52) Indicators need to be specific. A common error in writing indicators is to repeat the outcome without actually defining it: 8/23/2019

Indicators of Success/Progress Outcome: Stay-at-home parents will increase their social connections with each other over the next 3 months Indicator: % (or #) of parents at home full- time who report making new friends with other parents at home full-time who have children of similar ages NOT: % (or #) of stay-at-home parents who become socially connected 8/23/2019

Indicator Examples Outcome: Decreased exposure of adult nonsmokers to environmental tobacco smoke (ETS) Indicators: % of adult nonsmokers who report they have not been exposed to cigarette smoke during the previous 7 days 8/23/2019

Indicator Examples % of adults who report they are never exposed to cigarette smoke in restaurants   % of adults who report they are not exposed to cigarette smoke at work during a typical work day Example taken from www.cdc.gov/tobacco/evaluation_manual/Evaluation.pdf 8/23/2019

A Note on % Indicators Indicators involving %’s need a denominator Often the denominator is implied – such as all program participants or the total population of an area 8/23/2019

A Note on % Indicators We could also write the indicator to make the denominator, and thus our calculation of %, explicit: # of participants who report making new friends through the program ____________________________ All program participants 8/23/2019

Qualitative Indicators Indicators have traditionally been numerical (quantitative) Qualitative indicators give us more in-depth information on the meaning of change for people, rather than simply counting whether a change did or did not happen. 8/23/2019

Qualitative Indicators For example, in addition to looking at percentage of stay-at-home parents who make new friends, we can define what the friendships might look like (what people do together, how they involve children or not, etc) 8/23/2019

More Indicator Examples Outcomes: Young adults recovering from alcohol or drug addiction will have more connections with (a) non-using peers, (b) adults who focus on their strengths, and (c) supportive neighbourhood organizations, within three months of contact with an outreach worker 8/23/2019

More Indicator Examples Indicators: (a) % [or #] of recovering youth reached who do recreational or social activities with people their age who are not using alcohol or drugs; youth descriptions of types of activities they do with these other youth (qualitative) 8/23/2019

More Indicator Examples (b) % [or #] of youth reached who reporting spending time with at least one adult who recognizes what they are good at or have potential to learn; youth describe type of contact with such adult(s) – e.g., encouragement to pursue their talents, coaching or mentoring of some type (academic, sport, arts) 8/23/2019

More Indicator Examples (c) % [or #] of youth reached who make at least one visit a week to a neighbourhood organization that offers positive alternatives to alcohol and drug use; youth descriptions of the types of activities they do at the organization (e.g., arts, sports, recreational physical or social, activities, education in an area of interest to them, volunteer work with younger children, youth leadership 8/23/2019

More Indicator Examples Outcome: Children in our program achieve developmental milestones appropriate to their age and/or developmental ability (In this case, timelines may vary for each child) 8/23/2019

More Indicator Examples Indicators Number (%) of children will demonstrate age/developmentally appropriate: Communications skills (e.g. speaks, responses verbally to questions) Gross motor skills (e.g. throws a ball, climbs, hops   8/23/2019

More Indicator Examples Fine motor skills (e.g. draws with a pencil, does puzzles, uses scissors) Problem solving skills (e.g. understands size and direction) Personal/social skills (serves food, washes hands, knows names of others)  Above examples based on Ages and Stages Questionnaire (ASQ) 8/23/2019

Application # 2: Indicators Take one of the outcomes you developed in Application #1 What would be a good indicator for that outcome? Might there be more than one indicator for it? Would your indicator(s) be quantitative, qualitative, or both? 8/23/2019

Limits of Performance-Focused Evaluation Logic models and performance indicators have come to dominate evaluation of publicly funded projects/programs Fits with government accountability focus 8/23/2019

Limits of Performance-Focused Evaluation This is helpful for upfront planning, but can lead to an excessive focus on outputs and quantifiable outcomes, at the expense of an in-depth look at process and context, and of qualitative data Can encourage monitoring without more in-depth evaluation 8/23/2019

Limits of Performance-Focused Evaluation “Preoccupation with measuring a few indicators creates a kind of tunnel vision that shuts out the complexity of reality. Things appear simpler and more controllable than they are.” (Westley et al,. 2006, p. 240) From: Getting to Maybe (Random House) 8/23/2019

Limits of Performance-Focused Evaluation We get preoccupied with ‘did we achieve our outcomes?’ – without asking why or why not, what else happened or did not What gets measured gets done – can lead to narrow focus on pre-determined performance criteria, rather than being open to evolving needs/capacities and unexpected findings 8/23/2019