1. Ann M. Doucette, PhD Bringing Rigor to Qualitative Data Qualitative Comparative Analysis Ann M. Doucette, PhD 25 October 2013 Midge Smith Center for Evaluation Effectiveness The George Washington University

2. Ann M. Doucette, PhD Development as a Complex Adaptive System (CAS) ● Non-linear, dynamic, adaptive, emerging ● Outcomes better understood through dynamic analysis ─ Individual action changes the context for other individuals Individuals include: beneficiaries (direct and indirect) other community members, organization members (NGOs), etc. Diversity and Individuality: beneficiaries have unique and shared perspectives, values, norms ─ Interconnections between individuals and systems Systems embedded within systems  Example: beneficiaries, within communities, within sectors, within countries, within national policies (not always in agreement) ─ Study of the interaction of individuals and systems  evolving behavior – adaptation to novelty and the unexpected ● Predictability is not expected at the detail level ● Broader-focused predictability is possible - patterns ● Order is natural and not imposed 2

3. Ann M. Doucette, PhD Causal Density - Intervention Complexity Extent of complexity o Number of person-to-person transactions o Diversity of front-line implementing agent preferences o Influence of distracting conditions on implementing agents and program quality o Solution addressing concerns and challenges Known – best practices, prior experience, etc. Need for innovation, adaptation, modification Implementation capacity o Level of fidelity that can potentially be reached given resources Expectations o Potential impact understood within a framework of what can be achieved – time to expectancy 3 Adherence, integrity, quality of program/intervention implementation – protocol

4. Ann M. Doucette, PhD Analytic Approach Qualitative Comparative Analysis (QCA) Theory-based Qualitative Comparative Analysis (QCA): analytic technique using Boolean logic as a method of quantitative analysis of macro (cross-case) phenomena yielded from a case study Different paths lead to the outcome of interest ( coverage – equifinality ) o Counter to traditional statistics where factors are assumed to have the same incremental effect on outcome across cases ( additivity ) Cases are sorted in terms of causal and outcome conditions o Small “N” is addressed through the maximization of comparison causal and outcome variables – cases either having or not having (“0” – “1”) the condition o Combination of “A” + “B”  outcome “X” “A” + “B” + “C”  outcome “X” Cluster analysis conducted to examine emerging shared properties ( consistency - proportion of cases characterized by pattern ) 4 estimate of a dependent variable is obtained by adding together the appropriately computed effects (independent variables)

5. Ann M. Doucette, PhD QCA Approach Certain aspects of cases co-occur o Not deterministic (e.g., avoid infection by using hand sanitizer, reduce corruption via transparency, etc.) Necessity : almost always present if outcome occurs, but does not produce the outcome (e.g., micro-finance  community economic development, HIV infection  AIDS) o Outcome is a subset of condition o The frequency to which a condition occurs relative to outcome Consistency : degree to which subset relationship (configuration) is linked to outcome conditions o Relevance of program aspects to outcomes o Participation in program activities to awareness building Sufficiency : causal complexity – the degree to which a condition is present relative to other conditions and the outcome o Many solutions are possible – several sufficiency conditions o Use of theory is helpful 5

6. Ann M. Doucette, PhD Necessity – Sufficiency Example: Addressing Corruption Outcome: decreased corruption Conditions o “A” Transparency o “B” Enforcement of sanctions o “C” Written procedures Path 1: A + B  sufficient in leading to decreased corruption Path 2: A + C  sufficient in leading to decreased corruption A is a necessary condition o Linked with both “B” and “C” o It is not sufficient as it does not yield the outcome in the absence of “B” and/or “C” 6

7. Ann M. Doucette, PhD Configuration Sets 7 Crisp (csQCA) o Each case is assigned membership 1 = yes - present 0 = no – absent Fuzzy (fsQCA) o Each case is assigned the degree to which the case fits 1.00 = fully in 0.80 = mostly in 0.60 = more in than out 0.40 = more out than in 0.20 = mostly out 0.00 = fully out 1.00 = fully in.75 = more in than out.50 = neither in nor out.25 = more out than in 0.00 = fully out

8. Ann M. Doucette, PhD Truth Tables 8 Rows: logical combination of independent variables (input, suspected causes) o 0 = does not occur o 1 = occurs Each row is assigned an output value in terms of the dependent variable o 0 = does not occur o 1 = occurs Each row represents multiple cases (summary) having certain combinations Combinations are reduced, representing sufficient causal configurations

9. Ann M. Doucette, PhD Example: Cases (Crisp Set) 9

10. Ann M. Doucette, PhD Looking for Matches 10

11. Ann M. Doucette, PhD QCA - Example 11 Four conditions were thought to be linked to the outcome (protest versus no protest) yielding 16 possible configurations (4 x 4 – selected data shown) o Condition candidates established using qualitative/quantitative data o Conditions identified using cluster analysis Consistency is shown for the first seven configurations Estimation of the extent to which a configuration is associated with outcome (protest versus no protest) *Ragin, C., 1987, The Comparative Method. Moving beyond qualitative and quantitative strategies, Berkeley/Los Angeles/London: Univ. of California Press Case study: Examining imposed economic austerity measures*

12. Ann M. Doucette, PhD QCA and Its Advantages 12 Addresses challenges of small-n studies o Number of comparisons - insufficient for probabilistic statistics o QCA maximizes number of comparisons within/across cases Addresses complexity of issues found in development studies o Additivity not assumed – each causal condition has an independent association with the case outcome o Uniformity is not assumed – conditions may be positively or negatively associated with outcome in combination with other conditions o Outcome asymmetry is assumed – presence or absence of outcome may have different explanations No assumption of linearity Permanency of outcomes is not assumed Parsimony Case-oriented approach facilitates a form of counterfactual analysis – logical analysis

13. Ann M. Doucette, PhD Addressing the Counterfactual 13 Exploration of plausible counterfactual configurations # cases need to explore conditions geometrically increases according to function 2 k o 10 condition = 1024 cases to express all configurations o Many configurations would likely not be empirically supported  counterfactual configurations Possible configurations, but not observed in the data o Differentiation of strong versus weak counterfactual configurations Theoretical consistency Logical consistency (each outcome is explained by the configurations of conditions Theory-informed simulations

14. Ann M. Doucette, PhD QCA - Disadvantages 14 Need for a strong theory – prior causal knowledge is essential Small-n study o QCA does not seek to identify central tendencies, but to identify causal pathways linked to individuals cases Emphasis on dichotomizing variables (crisp set)  loss of information o Fuzzy set approach allows measurement at set intervals between 0 and 1 More flexibility, but can be subjective and not standardized appropriately Coding rules must be transparent Measurement error can erode conclusion based on deterministic methods Variable selection bias o Relevant variable selection is not particular to QCA, but to all analytic approaches Average effects of conditions are not estimated Outside mainstream science -- approach differs from traditional additive causation associated with probabilistic statistics Does not allow for time dimension  does not address process o Temporal sequencing is a separate exercise – could be incorporated in case selection

15. Ann M. Doucette, PhDLinks 15 Software http://www.u.arizona.edu/~cragin/fsQCA/software.shtmlWebsites http://www.u.arizona.edu/~cragin/fsQCA/ http://poli.haifa.ac.il/~levi/method.html

16. Ann M. Doucette, PhD Ann Doucette, Ph.D. The George Washington University 2147 F Street NW, Suite B-01 Washington, DC 20052 Tele: 202.994-8112 Email: doucette@gwu.edu