1. PERSPECTIVES ON IMPACT EVALUATION Approaches to Assessing Development Effectiveness CAIRO, March - April 2009 Ignacio Pardo Universidad de la República, Uruguay

2.  2005 - 2008  Mixed - Methods Impact Evaluation  2009 -2010

3.  2002 Crisis  Poverty and indigence  Long – term unemployment  Very few social programs for the most poor and unemployed in the past  Ministry of Social Development, created in 2005  Evaluation: accountability, feedback

4.  Conditional Cash Transfer program: “Ingreso Ciudadano”  Adult Literacy program  Educative interventions  Housing programs  …  Job training program: TRABAJO POR URUGUAY

5.  Increasing importance of mixed methods research in many disciplines  Evaluation: early use of mixed methods in evaluation, as in other “practical” fields (education, for example)  Recent systematization of mixed – methods research (Greene, 2007; Tashakkori and Teddlie, 2009)

6. (+ Monitoring) QUANTITATIVE + QUALITATIVE APPROACH (Mixed Methods? We know why… But… How? When?) QUAN (Survey) QUAL (Focus Groups, In-Depth Interviews) Plan Level (PANES) Impact (mainly, CCT) Program Level (Trabajo Por Uruguay) Outputs / Outcomes Process Outputs / Outcomes

7. “Qual” Objectives Meta Analysiis of data Survey “Quan” Objectives In depth Interviews + Focus Groups “Qual” Analysis“Quan” Analysis

8.  Triangulation  Complementarity  Development  Initiation  Expansion

9. ConcurrentSequential EqualQUAL + QUAN QUAL -> QUAN QUAN -> QUAL Dominant QUAL + quan QUAN + qual QUAL -> quan Qual -> QUAN QUAN -> qual Quan -> QUAL

10.  Usual approach:  Community self – analysis, implementation, PROCESS EVALUATION: Qualitative tools  Outputs, outcomes, IMPACT EVALUATION: Quantitative tools

11.  Plan level: PLAN DE EQUIDAD  Program level: URUGUAY TRABAJA  Similar to “Trabajo por Uruguay”  Evaluation design: NEW DESIGN? YES MIXED – METHODS APPROACH? YES IMPACT EVALUATION? LUCKILY QUALITATIVE IMPACT EVALUATION? PROBABLY

12. “RCTs are not always best for determining causality and can be misleading. RCTs examine a limited number of isolated factors that are neither limited nor isolated in natural settings. The complex nature of causality and the multitude of actual influences on outcomes render RCTs less capable of discovering causality than designs sensitive to local culture and conditions and open to unanticipated causal factors.” (American Evaluation Association response to US Department of Education, 2003)

13. “RCTs are not always best for determining causality and can be misleading. RCTs examine a limited number of isolated factors that are neither limited nor isolated in natural settings. The complex nature of causality and the multitude of actual influences on outcomes render RCTs sometimes less capable of discovering causality than designs sensitive to local culture and conditions and open to unanticipated causal factors.” (American Evaluation Association response to US Department of Education, 2003, and me)

14.  Increasing number of experts answering yes and trying to develop a more complete answer on how  RCT is not the ideal approach in every case. It is approapiate in certain cases (few variables, stable situation, context not affecting variables, clear expected outcome…) (Perrin, Tuesday).  Lipsey – Scriven debate  Qualitative Impact Evaluation analysis techniques suggested, as the Qualitative Comparative Analysis (QCA) (O’Reilly, 2008)

15.  “In many instances, it is difficult to know in advance the various factors impinging on outcomes and in the absence of that knowledge, one cannot empirically isolate the role of the intervention relative to other (extraneous) factors. In this context, information on processes, institutions, and on perspectives of participants (and non-participants) themselves can be very helpful. These approaches are also well suited to explain in-depth the reasons for, and character of, critical incidents and events, and the dynamics or causality of such events when they are linked into sequences or processes” (Ezemenari et al,1999)  Physical causality will often have a much more inmediate and compelling “presence” than relations that are only of the “if not X” variety. (…) And it is eminently available to the qualitative research approach (that is) well suited to establish causality (Mohr, 1995)

16.  Interpretation of causality > causal logic  Synthesis of quantitative and qualitative methods will be the best approach to causality (Davidson, 2005)  “Unpacking the complexity of the change process” (Sridharam, yesterday), looking as precipitating causes, amplyfing causes, causes of vulnerability to change. And finding the best approach to each of this kind of causation. Thinking of evidence as in a trial, looking for traces…  QUAL methods can answer the important question: What Makes What Works Work? (Scriven, 2008)

17.  Impact evaluation lies on counterfactual causality. This is usually regarded as the most robust approach. For good reasons. But it is only one.  At the same time, we can take into account other types of causality. Specifically, physical cause (modus operandi technique, forensic analysis, direct observation)  Reasons as causes (Berge, 2007)  “The qualitative method do not rely on any inference about the counterfactual: it relies on establishing with high probability a physical cause” (Berge, 2007)  Causal relations can be observed (realistic framework) and depend on context, which allows qualitative explanation to get into attribution analysis (Maxell, 2004)

18.  QUAN methods can tell us what and where meanwhile QUAL methods are better tools to answer why and how. That is why the latter can be maybe the main tool in Impact Evaluation, in a process of “creative experimentalism” (Prowse, Tuesday) Life History method can be a powerful QUAL tool, generating QUAN and QUAL data, besides visual data on trajectories. And providing a specific approach on causality

19.  In the next months, an evaluation design will be elaborated for “Uruguay Trabaja”  Mixed – methods evaluation may be taked one step further Mixing methods at earlier steps; using the mixing for more purposes  The concept of Impact Evaluation can probably be expanded in orden to give a better picture of the causal relations that will take place in the context of the program  Work in progress…