Using Evaluation Training to Create Change: The Influence of the Evaluation Fellows Program Amelia E. Maynard Jean A. King

Using Evaluation Training to Create Change: The Potential of the Evaluation Fellows Program Jean A. King Amelia E. Maynard

EFP partners at the U of MN Evaluation Fellows Program EFP partners at the U of MN The Evaluation Studies track in the Department of Organizational Leadership, Policy, and Development in the College of Education and Human Development The Consortium on Children, Youth, and Families (CCYF) King & Meyer 2011 AEA Presentation

Overview Brief description of the Evaluation Fellows Program (EFP) at the University of MN Analysis using three frameworks Evaluation influence Evaluation capacity building (ECB) Evaluation field building (EFB) Questions remaining/next steps

The evaluation fellows program (EFP) An innovative training experience in Minnesota’s Twin Cities The evaluation fellows program (EFP)

Innovative EFP features Focus on evaluation AND a specific field of practice Involvement of people from a variety of roles: Practitioners Evaluators Funders Policy makers

EFP operates in the “performance sweet spot” Evaluation knowledge Program knowledge PERFORMANCE SWEET SPOT D. Blyth

The intersection of people in four roles Practitioners Evaluators Policy Makers Funders

Overall EFP outcomes 1. Build the capacity of individual leaders to assess the impact of reform efforts Generate and apply knowledge about the special evaluation challenges inherent in attempting change in a specific field 3. Create knowledge about how to engage people in evaluating reform efforts in ways that will advance public dialogue, policy change, and improvement 4. Apply participants’ new knowledge to reform and policy development

EFP cohorts to date Cohort 1 (2007-2008) - Youth development program evaluation (out of school time) Cohort 2 (2009-2010) - Evaluation of educational reform Cohort 3 (proposed) - Evaluation of a public health program area

Frameworks for analy-zing the impact of efp Different ways to think about this innovative training Frameworks for analy-zing the impact of efp

Frameworks to shape our thinking about the EFP Evaluation influence Evaluation capacity building (ECB) Evaluation field building (EFB)

Evaluation influence Evaluation “influence” – integrated understandings of the broader consequences and impact of evaluations Kirkhart (2000) Henry & Mark (2003), Mark & Henry (2004) Influence is “the capacity or power of persons or things to produce effects on others by intangible or indirect means” (Kirkhart, 2000)

“Integrated theory” of influence Dimensions: Intention Unintentional Intentional Source Process Results Time Immediate End-of-cycle Long-term Kirkhart, 2000

Schematic “theory” of evaluation influence Intermediate and long-term evaluation outcomes leading to social betterment (Mark & Henry, 2004) Cognitive/affective Motivational Behavioral Salience Personal goals Individual practice Opinion valence Social reward Collaborative practice Descriptive norms Incentives Program future Agenda setting Market forces Policy adoption Program future is continuation, termination or expansion King & Meyer 2011 AEA Presentation

Applying influence to the EFP Traditional notions of influence are not quite right when applied to the EFP It’s not about the process of conducting an evaluation (Evaluation 101 or 304) It’s not (only) about wanting key people to use the evaluation’s results It IS about evaluation understanding (could lead to changes in thinking, motivation, behavior)

For thinking about the EFP Influence of the evaluation enterprise broadly framed Our question: Does collaborative training and understanding of the evaluation process and the use of evaluation lead to improvement in an area (both content and geography)?

Frameworks to shape our thinking about the EFP Evaluation influence Evaluation capacity building (ECB) Evaluation field building (EFB)

Evaluation capacity building “Intentional work to constantly co-create and co-sustain an overall process that makes quality evaluation and its uses routine in organizations and systems” Stockdill, Baizerman, & Compton (2002)

What, exactly, is evaluation capacity building (ECB)? Using an evaluation for (1) its results AND (2) the explicit purpose of building people’s capacity to evaluate again

For ECB practitioners, the focus “. . .[is] on responding to requests for evaluation services while simultaneously considering how today’s work will contribute to sustaining the unit in the longer term” (Compton, Glover-Kudon, Smith, & Avery, 2002, p. 55) King & Meyer 2011 AEA Presentation

Volkov & King (2007) Checklist for ECB Purpose- “To provide a set of guidelines for organizational evaluation capacity building (ECB), i.e., for incorporating evaluation routinely into the life of an organization” Available at http://www.wmich.edu/evalctr/checklists/evaluation-capacity-buildinginstitutionalization/

Broad categories of ECB content Category Overarching Activity Organizational context Be aware of the internal and external organizational context, power hierarchies, administrative culture, and decision-making processes ECB structures Purposefully create structures--mechanisms within the organization--that enable the development of evaluation capacity Resources Make evaluation resources available and use them

The EFP trainings are ECB Organizational context- Bringing together people from a variety of settings ECB structures- Developing capacity within the group to move a broader evaluation agenda forward Resources- Creating collective power so that they can garner resources for evaluation collectively

Adapting ECB structures for EFP Develop and implement a purposeful long- term ECB plan for the Twin Cities area Build and reinforce infrastructure to support continued training (MESI) Foster purposeful socialization into a growing evaluation network Build and expand a network of EFP alumni (peer learning structure)

Frameworks to shape our thinking about the EFP Evaluation influence Evaluation capacity building (ECB) Evaluation field building (EFB)

The concept of evaluation field building (EFB) Three phases of EFB in development evaluation in South Asia (Hay, 2010) Creating a centralized planning system (served planners) Improving evaluation quality, pushed by northern-based organizations, e.g., funding agencies (served donors) Increasing use (to serve government professionals, donors, and their constituents)

EFB applied to an urban center Program evaluation is alive and well in the Twin Cities, MN Many forms of evaluation practice Evaluation training opportunities (local, national)

EF [P +B] The Evaluation Fellows Program is a coordinated effort to build the field of evaluation in Minnesota Increasing evaluation capacity in people working in local agencies and organizations “Increasing use (to serve government professionals, donors, and their constituents)”

EFP’s field building intent Attending to overlapping roles Attending to the content “sweet spot”

Thoughts on future developments Where do we go from here?

Questions remaining Can EFP-style training actually lead to increased influence/evaluation capacity/a more developed field in a geographic area? What would be the outcomes and benefits of building geographic vs. individual/ organizational capacity?

Next steps Get the EFP re-funded- Why are funders reluctant to fund this work? Continue to build the informal network of EFP alumni Conduct research on the challenges/ benefits of geographic ECB King & Meyer 2011 AEA Presentation