Unpacking the notion of program theory from a complexity lense: Can indigenous and cross-cultural perspectives help? Sanjeev Sridharan and Janet Smylie Centre for Research on Inner City Health University of Toronto Benita van Wyk, Feedback Research & Analytics

Outline 1. Why is program theory important and useful? 2. What are the limitations of the standard approach to program theory? The view from evaluation 3. Ideas for a potential research project

Conceptualizing and Critiquing Programs: Program Planning and Evaluation Evaluation Literature Indigenous/ Cross-Cultural Literature Identify concepts to enhance program planning and evaluation Implement Concepts in Two Programs Evaluate Programs

Basis for dialogue Sustainability Connectivity Dynamics Heterogeneity

The important questions What is a program? Does the nature of the causal assumptions make a difference in the way we plan programs and evaluations? Dealing with complexity

An Example of a complex intervenion Dealing with complexity

An Example: Primary Prevention Have a Heart Paisley

The nature of the causal problem Note that the causal assumptions are not clearly specified Most assumptions are implied and of the form: If (A and B and C and D and E and at time T) then good outcomes might result The problem: – Causal contingencies not clearly specified or understood – Dynamic complexity not well understood

Another example: The Smoking Ban in Scotland

The incompleteness of program theories  The logic model while complex is still incomplete  A policy such as the smoking ban might lead to proximal impacts through perhaps an uncomplicated pathway; however the logic of the causal chain for intermediate and long-term outcomes might be incomplete  Additional “programmatic” inputs might be needed.  Information on the nature of such additional interventions can perhaps come from a synthesis of literature plus additional dialogue with stakeholders with an intimate knowledge of the processes by which programmes works.

Defining a complex intervention (MRC, 2000) “The greater the difficulty in defining precisely what exactly are the ‘active ingredients’ of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention.”

System Dynamic Approaches (Sterman, 2006)  Constantly changing;  Governed by feedback;  Non-linear, History-dependent;  Adaptive and evolving;  Characterized by trade-offs;  Policy resistance: “The result is policy resistance, the tendency for interventions to be defeated by the system’s response to the intervention itself.”

Features of complex health interventions (Pawson et al., 2004) The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not linear, and involve negotiation and feedback at each stage. Interventions are embedded in social systems and how they work is shaped by this context. Interventions are prone to modification as they are implemented. Interventions are open systems and change through learning as stakeholders come to understand them.

An additional purpose of evaluation for complex interventions Explication  Why?  Multiple components– be explicit about the “active ingredients”  Timeline of impact unclear  Non-uniformity– Contextualizing  Stakeholder involvement: Co-construction of intervention

On causality and complexity  Does a cause operate the same way in simple systems as compared to complex systems?  Are there different kinds of causes?  E.g. Famines  Precipitating causes  Amplifying causes  Causes of vulnerability  Innovations in evaluation design and approaches

So what do programmes do?  A need for a richer vocabulary on programmes as causes– can we learn from other sciences such as evolution or chemical kinetics?  Different metaphors: Programmes as catalysts, programmes as foundations, programmes as operating systems  Understanding complexity in multiple light – The system-as-cause – Understanding dynamic complexity

System-as-Cause Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3): Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, Available at. Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.

“Solutions” Can Also Create New Problems Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936: Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3): Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself. -- Meadows, Richardson, Bruckman

Unpacking the causal contingencies of change If (A AND B AND C AND D AND E and at time T) then good outcomes might result What does this really mean? Unpacking the complexity of the change process What is missing in our understanding of how programs work?

Seeing and understanding connection “ You think you understand two because you understand one and one. You think you understand two because you understand one and one. But you must also understand ‘and’.” You think you understand two because you understand one and one. But you must also understand ‘and’.” -- Sufi Saying

1. The role of sustainability Why sustaining programs might be key to success Going further – sustainability of programs at the individual level – the critical importance of Continuity of Care

2. The limited understanding of heterogeneity Heterogeneity of needs Heterogeneity of mechanisms of change Heterogeneity of interventions

3. The role of connectivity between interventions Far too much focus of most evaluations on a single intervention as the unit of analysis Understanding connectivity between programs

Better understanding of ecology of interventions

4. Understanding the relationship between multiple levels of interventions Many complex interventions require programming at the community, neighborhood, school and individual level Multilevel alignment across the interventions

4. Multilevel alignment complexity

5. Understanding Dynamic Complexity Timeline of impact Developmental Synergies

Anticipated timeline of impact

Dynamic complexity (1)

Dynamic complexity (2)

5. Coordination complexity The advantages of partnerships are poorly understood More importantly, collaborative advantages of individual recipients rarely made

Coordination complexity

Cracks in a network complexity

6. Sequential Complexity Relationship between sequences of interventions poorly understood Most of our methodologies are extremely limited in understanding such relationships

6. Sequential intervention complexity

7. Structural complexity Policy resistance Translating programs to fit within the structures The system-as-cause

Next Steps Identify partners Identify concepts from Evaluation and Indigenous/Cross-Cultural Field/Literature that can help enhance the concepts Translate concepts into practical testable experiments

Other concepts and ideas?