1. Revisiting the Relevance of Evaluation to Society: Challenging Assumptions About Social Interventions and Change Emily Gates & Thomas Schwandt University of Illinois, Urbana-Champaign AEA Annual Meeting, October 2012

3. What mindset?

4. Characteristics of the traditional mindset Social interventions Targeted at particular problems and groups of people Understand the intervention by examining its component parts Event-oriented view Implicit boundary between intervention and context Inputs and outputs are correlated

5. Characteristics of the traditional mindset Nature of change Predictable Linear Incremental and cumulative Logic of intervening Predict-and-act model Design – Implement – Evaluate

6. http://www.uwex.edu/ces/pdande/evalua tion/evallogicmodel.html

7. “Where the world is dynamic, evolving, and interconnected, we tend to make decisions using mental models that are static, narrow, and reductionist.” (Sternman, 2006)

8. Systems Thinking and Complexity Science Public Health Political Science Foreign Affairs Ecosystem Management Organizational Decision- making International Development Evaluation

9. Turn to thinking differently A different mindset is suggested by ideas from complexity science and systems thinking.

10. Characteristics of the different mindset Social interventions Understand system Relational/interacting view Re-define the problem and target impact Construct and re-construct boundary between intervention and context

11. Characteristics of the different mindset Nature of change Change is normative Uncertain – non-linear – emergent Multiplicity of change patterns Logic of intervening Iterative, cyclical process Maintaining system-wide conditions

12. Thinking differently Social Navigation in Public Health “Public craft of navigating health futures in an unstable and openly contested landscape.” (Milstein, 2008, p. 11) Adaptive Management in Ecosystem Management “Iterative process of predict, act, establish metrics of successful action, predict anew, establish metrics of successful action, act, gather data about consequences, predict anew…” (Mitchell, 2009, p. 97)

14. Thinking differently in evaluation? Margaret Hargreaves, Mathematica Michael Patton, Developmental Evaluation Beverly Parsons, InSites Bob Williams & Iraj Imam, Systems Concepts in Evaluation Bob Williams & Richard Hummelbrunner, Systems Concepts in Action Patricia Rogers, Purposeful Program Theory

15. So what? How does this different mindset challenge the ways we think about good evaluation theory: Social programming Knowledge Value Use Practice (Shadish, Cook, & Leviton, 1990)

16. References Jervis, R. (1997). System Effects: Complexity in Political and Social Life. Princeton, NJ: Princeton University Press. Leat, D. (2005). Theories of social change. International Network on Strategic Philanthropy. www.insp.efc.bewww.insp.efc.be Milstein, B. (2008). Hygeia’s Constellation: Navigating Health Futures in a Dynamic and Democratic World. Center for Disease Control and Prevention Report. Mitchell, S. D. (2009). Unsimple Truths: Science, Complexity, and Policy. Chicago, IL: University of Chicago Press. Shadish, W., Cook, T., & Leviton, L. (1990). Foundations of Program Theory: Theories of Practice. Newbury Park, CA: SAGE Publications, Inc. Sterman, J. D. (2006). Learning from Evidence in a Complex World. American Journal of Public Health, 96(3), 505-514.

17. Thank you! Emily Gates, egates3@illinois.eduegates3@illinois.edu Thomas Schwandt, tschwandt@illinois.edu tschwandt@illinois.edu Department of Educational Psychology University of Illinois, Urbana-Champaign