1. Representing Simple, Complicated and Complex Aspects in Logic Models for Evaluation Quality Presentation to the American Evaluation Association conference, San Antonio, TX November 2010 Patricia Rogers Patricia.Rogers@rmit.edu.au

2. 2 Truth The truth The whole truth And nothing but the truth

3. 3 Beauty “walking in beauty”/ harmony

4. 4 Justice Without fear or favor Balancing the scales

5. 5

6. 6 Funnell and Rogers Purposeful Program Theory 6 The material in this presentation is drawn from Purposeful Program Theory: Effective Use of Theories of Change and Logic Models, by Sue C. Funnell and Patricia J. Rogers, ISBN: 9780470478578, John Wiley/Jossey-Bass, February 2011.

7. 7 Funnell and Rogers Purposeful Program Theory 7 Program theory An explicit theory of how an intervention contributes to the intended or observed outcomes, which has 2 components: Theory of change The process by which change comes about (for an individual, organization or community) Theory of actionHow the intervention is constructed to activate the theory of change Logic model A visual representation of a program theory, usually in a diagram Program theory evaluation An evaluation that is at least partly guided by an explicit program theory. It is not necessarily ‘driven’ by the theory, since it should be driven by its intended purpose and the needs of its intended users What I mean by terms used today

8. 8 Funnell and Rogers Purposeful Program Theory 8 Aspects of complicated and complex situations and interventions with potentially important implications for evaluation 1)Focus 2)Governance 3)Consistency 4)Necessariness 5)Sufficiency 6)Change trajectory 7)Unintended outcomes

9. 9 Funnell and Rogers Purposeful Program Theory 9 Simple interventions

10. 10 Funnell and Rogers Purposeful Program Theory 10 Complicated interventions Intervention that produces different outcomes valued by different stakeholders

11. 11 Funnell and Rogers Purposeful Program Theory 11 Focus –Complicated - Objectives at multiple levels of a system

12. 12 Funnell and Rogers Purposeful Program Theory 12 Complex interventions Focus – Complex - Emergent outcomes

13. 13 Funnell and Rogers Purposeful Program Theory 13 (2) Governance SimpleSingle organization ComplicatedSpecific organizations with formalized requirements ComplexEmergent organizations working together in flexible ways

14. 14 Funnell and Rogers Purposeful Program Theory 14 (4) Necessariness Simple Only way to achieve the intended impacts Complicated One of several ways to achieve the intended impacts – which can be identified in advance Complex One of several ways to achieve the intended impacts – which are only evident in retrospect

15. 15 Funnell and Rogers Purposeful Program Theory 15 (5) Sufficiency Simple Sufficient to produce the intended impacts. Works the same for everyone Complicated Only works in conjunction with other interventions (previously, concurrently, or subsequently) and/or only works for some people and/or only works in some circumstances – which can be identified in advance Complex Only works in conjunction with other interventions (previously, concurrently, or subsequently) and/or only works for some people and/or only works in some circumstances – which is only evident in retrospect Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass)

16. 16 Funnell and Rogers Purposeful Program Theory 16 Ways in which an intervention can work with other interventions Stronger Families and Communities Strategy evaluation 2000-2004 Final Report http://www.rmit.edu.au/casr/sfcse

17. 17 Funnell and Rogers Purposeful Program Theory 17 Sufficiency – Complicated - Multi- stage intervention

18. 18 Funnell and Rogers Purposeful Program Theory 18 Sufficiency – Complicated – Intervention that works in combination with other interventions

19. 19 Funnell and Rogers Purposeful Program Theory 19 Sufficiency – complicated - intervention that works differently for different types of participants

20. 20 Funnell and Rogers Purposeful Program Theory 20 Change trajectory – complex - Tipping points

21. 21 Funnell and Rogers Purposeful Program Theory 21 (7) Unintended outcomes SimpleUnintended outcomes can be anticipated and monitored ComplicatedDifferent unintended outcomes are likely in particular combinations of circumstances – expertise is needed to anticipate them and identify them ComplexUnintended outcomes cannot be anticipated but only identified (and addressed) as they emerge or in retrospect Funnell and Rogers 2010 Purposeful Program Theory. Jossey-Bass)

22. 22 Funnell and Rogers Purposeful Program Theory 22 Generic theory of change with emergent theory of action

23. 23 Funnell and Rogers Purposeful Program Theory 23 What people sometimes assume you mean by logic models that address complexity

24. 24 Funnell and Rogers Purposeful Program Theory 24 Two framings of simple, complicated and complex Glouberman and Zimmerman 2002Kurtz and Snowden 2003 SimpleTested ‘recipes’ assure replicability Expertise is not needed The domain of the ‘known’, Cause and effect are well understood, Best practices can be confidently recommended, ComplicatedSuccess requires high level of expertise in many specialized fields + coordination The domain of the ‘knowable’ Expert knowledge is required, ComplexEvery situation is unique – previous success does not guarantee success Expertise can help but is not sufficient; relationships are key The domain of the ‘unknowable’, Patterns are only evident in retrospect. Kurtz, C. F. and D. J. Snowden (2003) ‘The New Dynamics of Strategy: Sense-making in a Complex and Complicated World’, IBM Systems Journal 42(3): 462–83.

25. 25 Funnell and Rogers Purposeful Program Theory 25 Some thoughts on how evaluation might help us to understand the complicated and the complex Issues that may need to be addressed 1.Focus 2.Governance 3.Consistency 4.Necessariness 5.Sufficiency 6.Change trajectory 7.Unintended outcomes Possible evaluation methods, approaches and methodologies Emergent evaluation design that can accommodate emergent program objectives and emergent evaluation issues Collaborative evaluation across different stakeholders and organisations Non-experimental approaches to causal attribution/contribution that don’t rely on a standardized ‘treatment’ Realist evaluation that pays attention to the contexts in which causal mechanisms operate Realist synthesis that can integrate diverse evidence (including credible single case studies) in different contexts ‘Butterfly nets’ to catch unanticipated results

26. 26 Resources and references Glouberman, S., and Zimmerman, B. Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Ottawa: Commission on the Future of Health Care in Canada, 2002. http://www.healthandeverything.org/fi les/Glouberman_E.pdf. http://www.healthandeverything.org/fi les/Glouberman_E.pdf Kurtz, C. F., and Snowden, D. F. “The New Dynamics of Strategy: Sense- Making in a Complex and Complicated World.” IBM Systems Journal, 2003, 42(3), 462–483. Rogers, P. J. “Using Programme Theory for Complicated and Complex Programmes.” Evaluation, 2008, 14 (1), 29–48. Rogers, P. J., Guijt, I., and Williams, B. “Thinking Systemically: Seeing from Simple to Complex in Impact Evaluation.” Presented at the 3IE/African Evaluation Association Impact Evaluation Conference, Cairo, Egypt, 2009. Snowden, D. J., and Boone, M. “A Leader’s Framework for Decision Making.” Harvard Business Review, Nov. 2007, pp. 69–76.