Note to evaluator… The overall purpose of this presentation is to guide evaluators through the completion of steps 4 to 6 of the UFE checklist. The main goal is to suggest a process that can help UFE evaluators facilitate the formulation of key evaluation questions. From the UFE checklist, this activity covers: Task 3 of Step 1;

INTRODUCTION TO UTILIZATION FOCUSED EVALUATION SLEVA Colombo June 6, 2011 Facilitators: Sonal Zaveri Chelladurai Solomon IDRC Consultants Assisted by Nilusha (LirneAsia) and Malathi (TESA)

Agenda Summary and brief discussion on the previous meeting and/or report. Group reflection on some of the discussed topics. Formulation of key evaluation questions Learn how to apply steps 1 through 6

Review – Basic UFE principles UFE is a PROCESS for helping primary intended users select the most appropriate content, model, methods, theory, and uses for their particular situation. UFE is a COLLABORATIVE APPROACH that seeks to generate learning. Evaluation should be JUDGED by its utility and USE in the real world. Evaluation plan needs to be part of the INITIAL DESIGN of project. The evaluator’s role is to COLLABORATE with those engaged in the design of the evaluation process.

Group exercise and reflection In a group use the selected evaluation and reformulate it using the UFE steps 1-3. For steps 4 to 5, follow the guidelines in this module. Identify the most limiting resources or factors of the evaluation process. Do you still think that the available resources are enough (or not enough) to carry out the evaluation process? Give 15 minutes

Group reflection – Stakeholders (1/4)… From the stakeholder groups that you identified in the previous meeting(s), what group(s) do you think you represent as primary intended users of the evaluation? From the UFE checklist, this question helps cover tasks 7 & 8 of Step 4.

Identified stakeholders (from previous meetings) Group Stakeholder Assigned role Stakeholder Group 1 Stakeholder 1 Stakeholder 2 Stakeholder 3 Stakeholder Group 2 Stakeholder 4 Stakeholder 5 Stakeholder 6 Stakeholder Group 3 Stakeholder 7 Primary intended user Stakeholder 8 Show this slide right after the previous one to help participants think through the question. The stakeholder table is supposed to come from previous meetings or from the reports of the meetings. From the UFE checklist, this slide helps cover tasks 7 & 8 of Step 4.

Group reflection – stakeholders (3/4)… What key stakeholder groups are not represented by the primary intended user? Would there be any implications on the use of the evaluation as a result of not having these stakeholder groups represented by the primary intended users? What would those implications be? From the UFE checklist, these questions help cover tasks 7 & 8 of Step 4.

Group reflection – Stakeholders (4/4)… Can you think of any political factors within the project that could affect the use of the evaluation? Nilusha to present her impressions for this and previous three slides – 6,

Discussion

FORMULATION OF KEY EVALUATION QUESTIONS - FIRST APPROACH

What is the purpose of the evaluation?

The trajectory of change… INPUT / RESOURCES ▼ ACTIVITIES OUPUTS OUTCOMES IMPACT / RESULTS CONTROL & PREDICTION This slide is to illustrate that the main challenge of the evaluation is having to identify and document impact and results of a project or program. While one usually has good and relatively constant ability to control and predict project input/resources, activities, outpouts and outcomes, one loses such ability when it comes to controlling and predicting the project’s impact and results because these can be generated by too many factors other than the project itself. ?

The evaluation’s purpose According to Patton (2008) evaluation can be oriented towards different purposes based on the findings’ primary intended USES.

MENU OF INTENDED USES (1/2) Evaluation Purposes Primary intended uses Typical primary users Overall summative judgment “To provide data for judging the overall value of a program and deciding whether it is worth continuing with it or not” (p. 114). Those charged with making major decisions: funders, directors, other adopters of model, etc. Program administrators, staff, those involved in the day-to-day management. Formative improvement & learning “To provide data for program improvement” (p. 116). The intent of the following slides is to suggest a process that can help the evaluator facilitate the formulation of key evaluation questions. From the UFE checklist, the suggested process helps cover Step 6. SUGGESTED PROCESS – I 1. Show and explain the menu of intended uses of findings proposed by Patton (2008) – make sure it is well understood by primary intended users. 2. From the six possible purpose options, ask primary intended user to select 2-3 evaluation purpose options. If deemed useful, from these 2-3 options, ask primary intended user to select the “dominant” or “main menu option”. 3. Invite primary intended users to comment on the meaning of the selected purpose options and, if necessary, to modify the wording of the options so that it makes sense to them and to the project’s context. Also invite primary intender user to set conditions or make clarifying comments on each of the selected purpose options. “To look across findings from different programs to identify patterns of effectiveness” (p. 131). Program designers, planners, modelers, theorists, scholars, policy-makers. Knowledge generating

MENU OF ITENDED USES (2/2) Purposes Intended primary uses Typical primary users Organization / Program Development “To provide data for adapting interventions to emergent conditions” (p. 137). Social innovators, those involved in bringing about major systems change in dynamic environments. “To provide data for describing and explaining achievements” (p. 121). Those with administrative and funding authority, responsible for resource use. Accountability Program managers responsible for internal accountability and information system management (Adaptaded from Patton p. 139 – Ch. .4). Monitoring “To provide information about key areas that require managerial attention” (Pg 126).

Formulation of key evaluation questions SUGGESTED PROCESS (II) 4. Invite primary intended user to formulate all the key evaluation questions that they can think of according the selected purpose menu. Post-it cards seem to work fine – one questions per card. 5. Collect the cards and read them out loud so that all questions are shared with the audience. 6. Ask primary intended user to classify each question into one or more options of the selected purpose menu.

What makes good KEQs? (adapted from Dart, 2007) Specific enough to be useful in guiding you through the evaluation Broad enough to be broken down - are not the same as a question in a survey Data (qualitative/quantitative) can be brought to bear on the KEQ KEQs are open questions (can’t answer yes or no!) Have meaning for those developing the plan Lead to useful, credible, evaluation There aren’t too many of them (2-4 is enough). SUGGESTED PROCESS (III) 7. Show and explain this slide to the primary intended user. Ask primary intended user to reflect on the last item of the slide. It is likely that there will be too many questions. 8. Invite primary intended user to reduce the number of questions. Follow one or more of these strategies to reduce the number of questions: (i) Identify duplicates among the preliminary questions and put them together as 1 question; (ii) regardless of the first grouping that was based on purpose; do a final grouping by common themes among preliminary questions (select a maximum of 4 top themes that are relevant to the evaluation). 9. Name the 4 (or less) top themes identified in previous step. 10. Invite primary intended user to formulate first draft of key evaluation questions by formulating a question for each of the 4 (or less) themes. Make sure that the proposed question for each theme represents well the group of questions that make up the theme. 11. Analyze the 4 (or less) “final” questions according to the features of good key evaluation questions on this slide.

Categories of key evaluation questions INPUT / RESOURCES IMPACT OUTCOMES APPROACH / MODEL PROCESS QUALITY COST- EFFECTIVENESS SUGGESTED PROCESS (IV) 12. Show and explain this slide to the primary intended user. Make sure that the user understands the slide. Invite user to propose alternative wording for the categories if deemed useful. 13. Invite primary intender user to assign one or more categories to each of the formulated 4 (or less) “final” questions.

Linking KEQ to the project objectives List the specific project objectives here… SUGGESTED PROCESS (V) 14. Invite primary intended user to link each of the 4 (or less) “final” questions to one or more specific objectives of the project. Make sure that the objectives of the project and the objectives of the evaluation, although related, are well differentiated.

Conclusion and next steps SUGGESTED PROCESS (VI) 15. Invite primary intended user to comment on the process and suggest possible next steps.

References Dart, J. 2007. “Key evaluation questions”. Presentation at the Evaluation in Practice Workshop. Kualal Lumpur, December. http://evaluationinpractice.files.wordpress.com/2008/01/keyquestionschoices.pdf Patton, M.Q. (2008) Utilization focused evaluation, 4th Edition. Sage.

LUNCH