Unit 1 Module 4 Explain: Roles of the Evaluator Introduction to Educational Evaluation Dr. Kristin Koskey

Multiple Personalities 2 “Detective”“Designer”“Judge” “Honest broker” “Creative consultant” “Organizational analyst” “Leader”“Devil’s advocate”“Change agent” “Counselor”“Follower”“Communicator” “Mediator”“Technical geek”“Diplomat” “Negotiator”“Problem solver”“Public relations agent” Grubbs, 2009; Mark, 2002; Patton, 2007; Skolits et al., 2009

3 Pre Evaluation Stage Detective Learn about the context Negotiator Agree on the parameters of the evaluation Designer Develop the evaluation plan Active Evaluation Stage Diplomat Build relationships and trust Researcher Implement the plan Judge Judge the merit, value, worth Reporter Communicate results to stakeholders Post Evaluation Stage Advocator Promote appropriate uses of evaluation findings Reflector Engage in self-reflection

Pre-Evaluation Stage Role of the detective  Learn about the context  Conduct an evaluability assessment Role of the negotiator  Reach an agreement on the evaluation Role of the designer  Develop the evaluation plan 4

Pre-Evaluation Stage: Role of the Detective Learn about the context 1.What is to be evaluated? 2.Who is the client? 3.Why was this program initiated? 4.What are the program goals? 5.What activities are believed to lead to these intended goals? 6.What is the theory behind these program goals 7.What is the political climate surrounding the evaluation? 8.Who are the most relevant stakeholders? 9.Who are the beneficiaries of the program? 10.Has this program been previously evaluated? If so, what have different stakeholders observed as a result of the program? 11.What data already exist on this program? 5

Pre-Evaluation Stage: Role of the Detective Continued Determine the evaluability of the project Identify whether the program model is clear. Check the alignment between the program activities and program model Determine if the program goals are achievable Determine if the stakeholders information needs are feasible Assess whether the stakeholders and you agree on the evaluation goals and uses 6

Pre-Evaluation Stage: Role of Detective Continued When and when not to get involved 7 Green Light Program goals are well defined Program objectives are feasible to achieve given the program activities and timeframe Relevant data can be collected or obtained at a reasonable cost (cost in terms of money and time) It is clear that the evaluation results will be used for decision-making and in what ways Sufficient funds are available to provide for a quality evaluation

Pre-Evaluation Stage: Role of Detective Continued When to and when not to get involved 8 Red Light Program goals are ill-definedThe request for the evaluation is too premature for the stage of the program Program objectives are not feasible to achieve A conflict of interest exists Data needed to achieve the evaluation goals will not be available or feasible to collect in the allotted timeframe or budget Key stakeholders cannot reach a consensus on the direction of the program and/or evaluation Evaluation would provide trivial information and the results would not be used for decision making Intended users of the results is not clear

Pre-Evaluation Stage: Role of Negotiator Reach an agreement on: – Evaluation to be conducted – Goals of the evaluation – How it will be conducted – Timelines – Budget – Who will do what and when – Use of evaluation findings 9

Pre-Evaluation Stage: Role of Designer Evaluation model(s) based on the evaluation goals Research design Methodology Data sources to be collected Formative and summative Strategies for data analysis and reporting Budget 10

Active-Evaluation Stage Role of the diplomat  Build good working relationships and trust with the stakeholders Role of the researcher  Implement the evaluation plan Role of the judge  Program strengths and weaknesses  Value, merit, worth of whatever is being evaluated Role of the reporter  Determine audience(s)  Communicate relevant evaluation results  Provide recommendations 11

Post-Evaluation Stage Role of the advocator  Educate stakeholders how to use the evaluation results  Promote stakeholders use of the evaluation results Role of the reflector  Engage in self-reflection as a professional What worked well and what didn’t work well? Is there anything I could have done differently? What skills should I strengthen? 12

References American Evaluation Association (2009). Guiding principles for evaluators. American Journal of Evaluation, 30(1), 5-6. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed). Boston, MA: Pearson. Grubbs, S. T. (2009). An evaluation of an alternative teacher certification program: A matter of trust. American Journal of Evaluation, 30(4), Mark, M. M. (2002). Toward better understanding of alternative evaluator roles. In K. E. Ryan & T. A. Schwandt (Eds.), Exploring evaluator role and identity (pp ). Greenwich, CT: Information Age Publishing. Patton, M. Q. (2007). Utlization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Skolitis, G. J., Morrow, J. A., & Burr, E. M. (2009). Reconceptualizing evaluator roles. American Journal of Evaluation, 30(3), doi: /