1. Patricia Rogers, RMIT University, Melbourne Part 3 of 8 AEA Coffee Break Webinars 2013 FRAME what is to be evaluated

2. The Rainbow Framework

3. Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Make Decision

4. Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Frame DecisionMake Decision

5. Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Frame DecisionMake DecisionDesign Evaluation

6. Why do we need to start with a clear definition? Source: Hobbies on a Budget / Flickr Frame DecisionMake DecisionFrame EvaluationDesign Evaluation

7. Four evaluation tasks in FRAMING Identify primary intended users Decide purpose(s) (intended uses) Specify key evaluation questions Determine what ‘success’ looks like

8. Identify primary intended users 1.

9. Possible primary intended users Image source: IN 157s01 – Curt Carnemark/ World Bank

10. Decide purpose(s)(intended uses) 2.

11. Formative – improve it Summative – continue or stop it Broader evidence base Purposes (intended uses) Image source: CK-CO138 - Charlotte Kesl / World Bank Lobby and advocate

12. Purposes (intended uses) Build trust and legitimacy Accountability Provide voice

13. Interconnection between intended users and intended uses Identify primary intended users Decide purpose(s) (intended uses) FRAME

14. Specify the key evaluation questions 3.

15. Descriptive: How many children attend? What learning tools are used? Has learning improved? Descriptive: How many children attend? What learning tools are used? Has learning improved? Key evaluation questions Image source: ML030S09 - Curt Carnemark / World Bank

16. Causal: Has the program contributed to improved learning? Causal: Has the program contributed to improved learning? Key evaluation questions

17. Synthesis: Has the program been a success? Is it Value For Money compared to alternatives? Synthesis: Has the program been a success? Is it Value For Money compared to alternatives? Key evaluation questions

18. Action: How can the program be improved? Should it continue? Action: How can the program be improved? Should it continue? Key evaluation questions

19. MANAGE DEFINE FRAME DESCRIBE UNDERSTAND CAUSES SYNTHESIZE REPORT & SUPPORT USE Descriptive Questions- What were the activities, changes, context? Causal questions –What caused or contributed to the identified changes? Synthesis questions – Overall was it good? Value for money? Action questions- What should we do? Options for answering different types of questions

20. Determine what success looks like 4.

21. Processes: Are students fairly treated? What does success look like? Image source: LQ-CN7499 World Bank

22. What does success look like? Outcomes: Do students learn to read?

23. What does success look like? Distribution of costs and benefits: Who benefits from the program? Whose needs are met? Who is disadvantaged by it?

24. What does success look like? Criteria and standards: Reading levels – better than before, or better than national average, or national benchmark?

30. You can read about this case here : http://betterevaluation.org/blog /outcome_harvesting_bionet

31. Patricia Rogers RMIT University patricia.rogers@rmit.edu.au Further resources http://betterevaluation.org/plan/frame facebook.com/betterevaluation @bettereval