Allie Davis Dr. Amy Javernick-Will and Dr. Sherri Cook Avoiding Failure: The Use of Qualitative Comparative Analysis to Identify Pathways to Successful Sanitation Interventions UNC Water & Health October 16 – 20, 2017 Allie Davis Dr. Amy Javernick-Will and Dr. Sherri Cook Department of Civil, Environmental, and Architectural Engineering Civil Systems and Engineering for Developing Communities University of Colorado Boulder

Acknowledgments Sridar Selvaraj Cook Research Group Sherri.Cook@Colorado.edu Amy.Javernick@Colorado.edu 2

Overview Failed sanitation systems impede SDGs Qualitative Comparative Analysis can help identify conditions for success Pathways to success can refine project planning Observed Problem QCA Method Future Work Summary Pathways to Success 3

Overview Failed sanitation systems impede SDGs Qualitative Comparative Analysis can help identify conditions for success Pathways to success can refine project planning Observed Problem QCA Method Future Work Summary Pathways to Success 3

What combinations of factors lead to successful sanitation systems? Understanding factors that lead to success is important to achieving SDGs Social? Successful Failed Institutional? 30% Economic? Technical? 70% What combinations of factors lead to successful sanitation systems? Davis, S. (2015). “Statistics on Sanitation Failures.” Improve International. Observed Problem QCA Method Future Work Summary Pathways to Success 4

India’s ambitious national initiatives and large infrastructure gaps motivate the context selection Only 21 percent of wastewater is treated1 23 percent lack access to electricity2 50 percent of shared toilets are unused3 60 percent still go for open defecation3 67 percent use solid fuels for cooking2 1Kamyotra, J. S., and Bhardwaj, R. M. (2011). “Municipal wastewater management in India.” India Infrastructure Report, 299. 2World Health Organization, and UNICEF. (2015). Progress on sanitation and drinking water: 2015 Update and MDG Report. Joint Monitoring Programme, World Health Organization, [S.l.]. 3Davis, S. (2015). “Statistics on Sanitation Failures.” Improve International. Observed Problem QCA Method Future Work Summary Pathways to Success 5

Overview Failed sanitation systems impede SDGs Qualitative Comparative Analysis can help identify conditions for success Pathways to success can refine project planning QCA Method Observed Problem Future Work Summary Pathways to Success 6

Fuzzy-set Qualitative Comparative Analysis uses set theory and Boolean algebra to analyze pathways Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Validity Tests QCA Method Observed Problem Future Work Summary Pathways to Success 7

Success is defined as the presence of use, maintenance, and performance Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Success Use Maintenance Performance > 75% of community uses correctly & daily 90% of maintenance tasks performed correctly & on time Treated water meets all local regulations QCA Method Observed Problem Future Work Summary Pathways to Success 8

Hypothesized causal conditions come from literature and pilot studies Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Community Participation Training Priorities Skilled Operator Priority Assessment Community Participation in Planning Technical Training Addressed Sanitation Priorities Community Contributions Behavior Change Education Community Participation in Construction Institutional Support Income Generation Community Participation in O&M Infrastructure Lack of Vandalism Ongoing External Support Water Supply Lack of Government Barriers QCA Method Observed Problem Future Work Summary Pathways to Success 9

Resource recovery sanitation systems were selected to ensure variation within conditions Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success 5 Successful 5 Failed 800 – 1200 users Low-income Karnataka Dalit caste Tamil Nadu Peri-urban settlements QCA Method Observed Problem Future Work Summary Pathways to Success 10

In-depth case studies were completed to understand each system’s story Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Interviews Documentation Observations Technical Evaluations QCA Method Observed Problem Future Work Summary Pathways to Success 11

Conditions were calibrated to define in-set and out-of-set membership Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Skilled Operator Fuzzy-Set Score Condition No operator exists. 0.33 Unskilled operator exists but is not present daily at the system. Operator cannot diagnose and recognize system problems and is unreliable. 0.67 Operator has some technical knowledge, but needs external assistance to diagnose and recognize system problems. Operator is not present daily at the system. 1 Skilled operator regularly present; demonstrates extensive technical system knowledge; capable of recognizing and rectifying system problems independently; proactive. QCA Method Observed Problem Future Work Summary Pathways to Success 12

Ongoing External Support Set values were assigned for all cases and conditions and assembled in a truth table Skilled Operator Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Fuzzy-Set Score Condition No operator exists. 0.33 Unskilled operator exists but is not present daily at the system. Operator cannot diagnose and recognize system problems and is unreliable. 0.67 Operator has some technical knowledge, but needs external assistance to diagnose and recognize system problems. Operator is not present daily at the system. 1 Skilled operator regularly present; demonstrates extensive technical system knowledge; capable of recognizing and rectifying system problems independently; proactive. Case Number Skilled Operator Ongoing External Support N Conditions Outcome: Success 1 2 N Cases QCA Method Observed Problem Future Work Summary Pathways to Success 13

Ongoing External Support Set values were assigned for all cases and conditions and assembled in a truth table Skilled Operator Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Fuzzy-Set Score Condition No operator exists. 0.33 Unskilled operator exists but is not present daily at the system. Operator cannot diagnose and recognize system problems and is unreliable. 0.67 Operator has some technical knowledge, but needs external assistance to diagnose and recognize system problems. Operator is not present daily at the system. 1 Skilled operator regularly present; demonstrates extensive technical system knowledge; capable of recognizing and rectifying system problems independently; proactive. Case Number Skilled Operator Ongoing External Support N Conditions Outcome: Success 1 2 N Cases QCA Method Observed Problem Future Work Summary Pathways to Success 13

Ongoing External Support Set values were assigned for all cases and conditions and assembled in a truth table Skilled Operator Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Fuzzy-Set Score Condition No operator exists. 0.33 Unskilled operator exists but is not present daily at the system. Operator cannot diagnose and recognize system problems and is unreliable. 0.67 Operator has some technical knowledge, but needs external assistance to diagnose and recognize system problems. Operator is not present daily at the system. 1 Skilled operator regularly present; demonstrates extensive technical system knowledge; capable of recognizing and rectifying system problems independently; proactive. Case Number Skilled Operator Ongoing External Support N Conditions Outcome: Success 1 2 0.33 N Cases QCA Method Observed Problem Future Work Summary Pathways to Success 13

Ongoing External Support Set values were assigned for all cases and conditions and assembled in a truth table Skilled Operator Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Fuzzy-Set Score Condition No operator exists. 0.33 Unskilled operator exists but is not present daily at the system. Operator cannot diagnose and recognize system problems and is unreliable. 0.67 Operator has some technical knowledge, but needs external assistance to diagnose and recognize system problems. Operator is not present daily at the system. 1 Skilled operator regularly present; demonstrates extensive technical system knowledge; capable of recognizing and rectifying system problems independently; proactive. Case Number Skilled Operator Ongoing External Support N Conditions Outcome: Success 1 0.67 2 0.33 N Cases QCA Method Observed Problem Future Work Summary Pathways to Success 13

Conditions that were found to be inconsistent were removed and allowed us to minimize analyzed conditions Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Community Participation Training Priorities Skilled Operator Priority Assessment Community Participation in Planning Technical Training Addressed Sanitation Priorities Community Contributions Behavior Change Education Community Participation in Construction Institutional Support Income Generation Community Participation in O&M Infrastructure Lack of Vandalism Ongoing External Support Water Supply Lack of Government Barriers QCA Method Observed Problem Future Work Summary Pathways to Success 14

Conditions that were found to be inconsistent were removed and allowed us to minimize analyzed conditions Define Outcomes Select Causal Conditions Select Cases Data Collection Calibrate Conditions Analysis & Minimization Pathways to Success Community Participation Training Priorities Skilled Operator Priority Assessment Community Participation in Planning Technical Training Addressed Sanitation Priorities Community Contributions Behavior Change Education Community Participation in Construction Institutional Support Income Generation Community Participation in O&M Infrastructure Lack of Vandalism Ongoing External Support Water Supply Lack of Government Barriers QCA Method Observed Problem Future Work Summary Pathways to Success 14

Pathways reflect the most simplified solution that still captures important case complexity Community Participation Training Priorities Skilled Operator Priority Assessment Community Participation in Planning Technical Training Addressed Sanitation Priorities Community Contributions ? Behavior Change Education Community Participation in Construction Success Institutional Support Income Generation Community Participation in O&M Infrastructure Lack of Vandalism Ongoing External Support Water Supply Lack of Government Barriers QCA Method Observed Problem Future Work Summary Pathways to Success 15

The combination of six conditions led to success Skilled Operator Lack of Government Barriers Ongoing External Support Water Supply Lack of Vandalism Addressed Sanitation Priorities * Condition Necessity Skilled Operator 1 Lack of Government Barriers Ongoing External Support 0.93 Water Supply 0.87 Lack of Vandalism 0.80 Addressed Sanitation Priorities 0.73 Consistency = 0.93 Coverage = 0.66 Pathways to Success QCA Method Observed Problem Future Work Summary 16

The presence of a skilled operator was important for maintenance and use Lack of Government Barriers Ongoing External Support Water Supply Lack of Vandalism Addressed Sanitation Priorities * Condition Necessity Skilled Operator 1 Lack of Government Barriers Ongoing External Support 0.93 Water Supply 0.87 Lack of Vandalism 0.80 Addressed Sanitation Priorities 0.73 Consistency = 0.93 Coverage = 0.66 Pathways to Success QCA Method Observed Problem Future Work Summary 17

The absence of government barriers was important for continued operation of the systems Skilled Operator Lack of Government Barriers Ongoing External Support Water Supply Lack of Vandalism Addressed Sanitation Priorities * Condition Necessity Skilled Operator 1 Lack of Government Barriers Ongoing External Support 0.93 Water Supply 0.87 Lack of Vandalism 0.80 Addressed Sanitation Priorities 0.73 Consistency = 0.93 Coverage = 0.66 Pathways to Success QCA Method Observed Problem Future Work Summary 18

The types and quality of external support was more important than the source of external support Skilled Operator Lack of Government Barriers Ongoing External Support Water Supply Lack of Vandalism Addressed Sanitation Priorities * Condition Necessity Skilled Operator 1 Lack of Government Barriers Ongoing External Support 0.93 Water Supply 0.87 Lack of Vandalism 0.80 Addressed Sanitation Priorities 0.73 Consistency = 0.93 Coverage = 0.66 Pathways to Success QCA Method Observed Problem Future Work Summary 19

Addressed Sanitation Priorities Higher percentages of weighted sanitation priorities are associated with successful systems Case Number Addressed Sanitation Priorities 1 0.78 2 0.60 6 0.86 8 0.74 5 0.67 3 0.40 10 0.32 9 0.23 7 0.19 4 0.14 Skilled Operator Lack of Government Barriers Ongoing External Support Water Supply Lack of Vandalism Addressed Sanitation Priorities * Condition Necessity Skilled Operator 1 Lack of Government Barriers Ongoing External Support 0.93 Water Supply 0.87 Lack of Vandalism 0.80 Addressed Sanitation Priorities 0.73 Consistency = 0.93 Coverage = 0.66 Pathways to Success QCA Method Observed Problem Future Work Summary 20

Future Work Analyze pathways to failure Perform fsQCA with 20 cases: 10 resource recovery and 10 non-resource recovery systems Design trainings and practical tools for implementing organizations Pathways to Success Future Work QCA Method Observed Problem Summary 21

Summary Institution building should be a focus of project planning since ongoing external support and lack of government barriers are necessary for success Addressed sanitation priorities suggests that participation and planning must ensure projects meet community priorities The combination of Skilled Operator, Lack of Government Barriers, Ongoing External Support, Water Supply, Lack of Vandalism, & Addressed Sanitation Priorities lead to successful sanitation systems Questions? Allison.L.Davis@Colorado.edu Pathways to Success QCA Method Observed Problem Future Work Summary 22

EXTRA SLIDES

Fuzzy-Set Qualitative Comparative Analysis Statistical Analyses Case Studies Quantitative Methods Qualitative Methods Adapted from Gross and Garvin 2011

Understanding Pathways Adapted from Kaminsky and Jordan 2017 Cases w/ Factor X Cases w/ Outcome Cases w/ Factor X Cases w/ Outcome = = Necessary Condition ≥ 0.9 All (or nearly all) instances of the outcome show the condition Pathway Coverage ≥ ? The percent of cases with the outcome explained by a given pathway Pathway Consistency ≥ 0.8 The pathway consistently leads to the outcome

Failure * * * * * * Communities 3, 9, 10 Consistency = 1.0 Lack of Skilled Operator Lack of Income Generation * 3, 9, 10 Government Barriers Lack of Ongoing External Support Vandalism * 4 Lack of Addressed Sanitation Priorities Lack of Skilled Operator Lack of Ongoing External Support Vandalism Lack of Income Generation Lack of Water Supply * * * * 7 Condition Necessity Lack of Addressed Sanitation Priorities 0.74 Government Barriers 0.68 Lack of Skilled Operator 0.67 Lack of Ongoing External Support Vandalism Lack of Income Generation 0.60 Lack of Water Supply 0.47 Consistency = 1.0 Coverage = 0.74

Use * Use Communities Community Participation in Planning 10 Ongoing External Support Water Supply Community Participation in Planning Addressed Sanitation Priorities * 10 1, 2, 5, 6, 8 Condition Necessity Ongoing External Support 0.90 Water Supply 0.85 Community Participation in Planning 0.75 Addressed Sanitation Priorities 0.68 Consistency = 0.95 Coverage = 0.94

Maintenance and Performance Maintenance and System Performance Communities Lack of Government Barriers Skilled Operator Water Supply * * 1, 2, 5, 6, 20 Condition Necessity Lack of Government Barriers 1.0 Skilled Operator 0.94 Water Supply 0.83 Consistency = 0.95 Coverage = 0.94