Module 7 Key concepts Part 2: EVALUATING COMPLEX DEVELOPMENT PROGRAMS Technical Assistance on Evaluating SDGs: Leave No One Behind EvalGender+ Network together with UN Women, UNEG and EvalPartners

Presentation developed by Michael Bamberger and Asela Kalugampitiya based on Chapter 3 of ”Evaluating the Sustainable Development Goals within a “No-one left behind” lens through equity-focused and gender-responsive evaluations”

Outline The dimensions of complexity Complexity is a continuum Some programs are less complex than they seem Challenges of evaluating complex programmes Applying complexity analysis to equity focused and gender responsive evaluation

The dimensions of complexity

COMPLEXITY ANALYSIS FRAMEWORK EMERGENT OUTCOMES SYSTEMS DEFINING BOUNDARIES Causal levels and social processes Complex causality Dimensions of complexity Complexity dynamics

Dimensions of complexity in development evaluation EMBEDDEDNESS AND THE NATURE OF THE SYSTEM Historical, economic, political, socio-cultural, administrative and organizational, climatic and ecological, legal and regulatory context Norms and beliefs Interconnectedness, boundaries, dynamics (e.g. path dependence, system shock) INSTITUTIONS AND STAKEHOLDERS Governance, funding, coordination, implementation system Number and diversity of stakeholders (e.g. implementing agencies, donors, politicians, beneficiaries, evaluators) Stakeholder expectations, demands and ‘theories-in-use’ Conflict, cooperation, evaluation culture INTERVENTION Design and purpose (e.g. initial logframe, logic model, theory of change) Size and scope (e.g. number and types of intervention activities, levels of intervention) Data coverage, quality and accessibility CAUSALITY AND CHANGE Causality (e.g. non-linearity, emergence, feedback loops, multiple pathways) Attribution and contribution Theories, mechanisms, models of behavioral change Implementation Direct, indirect, intended, unintended, positive, negative effects challenges in delimitation, sense-making, consensus-seeking, design, implementation and use of evaluations EVALUATION Purpose Time, resources and data Methodology Participation and process Values and ethics

The dimensions of complexity The nature of the intervention (program) Interactions among stakeholders and participating agencies Causality and change The nature of the system within which the program is embedded Factors affecting the complexity of the evaluation

Dimension 1: The nature of the intervention Complexity of the objectives Size Stability of program design Implementation procedures Number of services/components Technical complexity Social complexity Duration Is program design well tested

Dimension 2: Interactions among institutions and stakeholders Budget: is use of funds well defined? Number of funding and implementing agencies Number of stakeholders and similarity of interests

Dimension 3: Causality and change The nature of causal pathways (see next slide) Degree of certainty of outcomes Agreement on appropriate actions to address problems

Simple, linear causal chain Intervening variables Input Outcome

Complex causal chains

Examples of complex, non-proportional change Tipping point Inertia e f c t e f c t Inputs Inputs

Dimension 4: Embededness and the nature of the system How independent is the program of the wider context Complexity of the processes of behavioral change

Some key concepts in complexity analysis Embededness Adaptability and self-organization Emergence

Complexity is a continuum A program can have a high or low level of complexity on each of the 4 dimensions For operational purposes an agency may use the Checklist for Assessing Levels of Complexity to define cut-off points: rating programs as having high, medium or low levels of complexity on each dimension

Interventions that look “complex” may not be “complex” interventions may comprise a number of “simple” components Many programs with complex coordination and management mechanisms may have relatively “simple” components and causal chains Some agencies may find it convenient to claim their programs are too complex to be rigorously evaluated – to avoid critical scrutiny.

Challenges relating to program design Program covers whole country Multiple components with complex interactions Interventions intended to cause multiple outcomes No logical linkages between interventions and high level program goals Program components operate differently in different settings Multiple stakeholders and implementing agencies

Data availability Agencies collect only limited monitoring data Difficult to compare data collected by different agencies No standard data platforms Different methodologies and reporting systems No data on process analysis Need real-time data to monitor change and detect problems

The complex nature of causality Multiple causal pathways Multiple outcomes caused by different combinations of inputs Influence of contextual variables Non-linear causality Emergent designs Long duration of programs

Real-world constraints Budget Time Data access Attitudes of stakeholders Clients have different values and methodological preferences Political time-tables and evaluation time-tables

5. Applying complexity analysis to equity focused and gender responsive evaluation Complexity dimensions of equity Equity outcomes are affected by all of the SDG so there are very complex pattern of interaction There are multiple stakeholders and implementing agencies often with competing objectives Social exclusion is determined by the interaction between objective factors such as income, housing and access to services and by political, cultural and psychological factor that maintain existing inequalities. Equity has a historical dimension which must be understood

Complexity dimensions of gender Gender inequalities and patterns of change are determined by interactions among all of the SDGs A model of social control is required to understand interactions among political, economic, socio-cultural, legal, ecological and historical factors. Social control mechanisms include social pressures, sanctions, force and threats of force, violence, legal and religious sanctions It is also necessary to study subtle patterns of behavioral change Social control also operates within the family and involves processes that are difficult to observe.

Complexity dimensions of gender [continued] Social media play an important role and must be studied Many of the processes of empowerment and change in gender roles are intergenerational so the evaluation must have a long time- perspective

Resources Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation & Use http://betterevaluation.org/resources/guides/developme ntal_evaluation/applying_complexity_concepts_patton

Thank you for attention For more information, please contact: Marco Segone, Director, Independent Evaluation Office -UN Women, Co-chair - EvalGender+, Chair – United Nations Evaluation Group marco.segone@unwomen.org Florencia Tateossian, Evaluation Specialist – UN Women florencia.tateossian@unwomen.org Asela Kalugampitiya, EvalPartners Executive Coordinator, coordinator@evalpartners.org