Involving Teachers in Evaluation We first completed a large scale evaluation of an Early Childhood Education program in a comprehensive, urban school district in San Diego County. Our intent here is not to share the results of that yearlong investigation, but to describe the process we used to engage teachers in the analysis of findings and action planning. Involving Teachers in Evaluation

Teachers are often the object of an evaluation but frequently are not involved in any other part of it. Investment: Most educational problems today are too complex not to have stakeholders involved from multiple levels of the organization, even when these different stakes run counter to each other and possibly the purpose of the evaluation. In fact, evaluation research is inundated with caveats about not involving stakeholders and success stories about when stakeholders are a part of the process (Alkin, Daillak, & White, 1979; Cousins & Shulha, 2006; King, 2007; Patton, 2008). Additionally, involving teacher is both time-consuming and costly. In California, we have a perfect storm of issues that make involving teachers in evaluation challenging. First, the recession created an enormous shortage of new teachers, which seriously limits the number of teachers in a subpool. Currently, in San Diego, we, on average about 2/3rds of the teachers we need to fill current substitute teaching opportunities. We just don’t have enough subs to take the place of teachers so they can participate in the evaluation process fully. Furthermore, teachers are expensive. On average, a district pays anywhere from $400-$700 per teacher absence. The district we reference in this work has over 300 potential teachers who could participate in the work.

Broadening: The last issue of Educational Research, education’s most practical and subscribed to journal, devoted to the role of evaluation in schools was published in 1956 (Educational Leadership). Additionally, a review of federal and state educational grant stipulations require grantees to write to summative outcomes rather than measures focused on improving programs or organizational effectiveness (CaMSP and Institute of Education Evaluation). Finally, a review of the most read articles from the American Journal of Evaluation and Educational Evaluation and Policy Analysis (albeit these are not journal practitioners often read) devote much of their evaluation attention to performance evaluations. Essentially, if we get them involved, the next challenge is broadening their understanding of the purpose of evaluation, so they recognize that evaluation seeks to improve programs versus a thumbs up or down decision. With participation, we must broaden teachers’ views of the purpose of evaluation

Teachers will ultimately implement most actions that stem from an evaluation of an instructional program. Focus on implementation: Often failed improvement efforts are the results of using evaluations to manage or control program resources, which often leads to less supply of what programs need to succeed rather than using the results of an evaluation as the catalyst to improve the program or service. In addition to the challenges of involving stakeholders in educational evaluations and broadening their understanding of the purpose of evaluation, good educational evaluation work must also develop use methods that result in actions steps that have a reasonable chance of being implemented successfully.

What the Literature teaches us… Use stakeholders who are served by the program and use stakeholders in the decision-making process (Cai, 1996; Cousins, 1995; Johnson, 1993; McCormic, 1997; Preskill & Caracelli 1997; Sperlazza, 1995)

What the Literature teaches us… Use stakeholders as part of an organizational improvement process that results from participating in evaluation, or what might be described as process use in evaluation (Patton, 2003).

What the Literature teaches us… Use stakeholders as a way to build evaluation capacity. When stakeholders engage in a systematic process that identifies actions and outcomes, the individuals within the organization have a better understanding of what those actions are and how they lead to outcomes (Cousins, 2010).

We wrote a semi-fictional story that… Reflected the context and major findings from the evaluation. Allowed teachers to be “outside observers” in defining the problem. Encouraged authentic inquiry and discussion of the evidence.

We facilitated a dialogue on... Current Reality What does the evidence tell us? What barriers and beliefs impede us?

We facilitated a dialogue on... Current Reality Future State What does the evidence tell us? What barriers and beliefs impede us? What beliefs and behaviors must be affirmed? What strategies and structures are needed?

We empowered teachers by… Capturing their collective thinking, Synthesizing their ideas, Incorporating their input into the action plan

Attractiveness/Capacity Grid of Plan Implement Plan Capacity to No Yes 17% 46% 8% 29% Attractiveness is defined as the potential for the plan to improve literacy in the district and how well the planned is aligned to their organization’s goals. The horizontal axis represents capability. It examines whether teachers believe the district has the capacity to carry out the plan, whether they have the personnel to lead the work, and whether the organizations support exists

Lessons Learned & Questions Engage teachers in evaluation in culturally and professional sensitive ways Engaging teachers in evaluation is a mechanism for reflective practice Engaging teachers in evaluation develops shared ownership of a systemic problem Focus on implementation: Often failed improvement efforts are the results of using evaluations to manage or control program resources, which often leads to less supply of what programs need to succeed rather than using the results of an evaluation as the catalyst to improve the program or service. In addition to the challenges of involving stakeholders in educational evaluations and broadening their understanding of the purpose of evaluation, good educational evaluation work must also develop use methods that result in actions steps that have a reasonable chance of being implemented successfully.