1. Embedded Evaluation in Rule of Law programs

2. Informal Rule of Law Where to find it – State absence – Barriers – Tribal culture What it does – Resolves conflicts – Establishes harmony – Human rights?

3. Complexity and ROL evaluation Open conflict External/internal Programming fails IP risk appetite is low Weak partners $$ for M&E is tight Zero-sum Minimal room for error High donor expectations Interests

4. Recommendations External evaluation Context information Study arms Internalize it Internal evaluation M&E at the proposal stage Have a plan and budget Use the ToC Chatham House rules Adaptation

5. Rule of Law Stabilization Informal Component (RLS-I) Findings from a quasi-experimental evaluation October 16, 2014

6. Perception differential between Jirga and State 6 Jirgas / district courts are…2007 200820092010201120122013 Accessible+5% +8%+11%+13%+10% Fair and trusted+20% +22%+20% +18%+19% Follow local norms and values+19% +21%+19%+20%+19% Effective at delivering justice+18% +17%+18%+15%+17%+19%+16% Resolve cases promptly+21% +24% +26%+28%21%

7. Custom, Islam, and the State … But if traditional dispute resolution (TDR) carries inherent legitimacy, it also has problems: – Persistence of harmful social practices (baad, badal, no formal female voice or participation) – Informal self-governance practices that are locally adaptive but complicate modern state-building – Conflation of custom with Islamic values 7

8. Networking meetings Learning workshops (30-35 hours) -Constitutional law -Criminal law -Family law (2 days) -Inheritance law (2 days) -Property -Deeds Discussion sessions - State-TDR coordination- Best practices in TDR - Alternatives to baad- Overcoming influence of powerbrokers - Women’s participation in jirga RLS-I Activities

9. Where to measure ? – District selection Province DistrictStatus Kunar ChawkayTreatment NarangComparison Logar Mohammad AgaTreatment KhoshiComparison Kandahar ZhariTreatment Shah Wali KotComparison

10. Difference-in-Differences (D-i-D) with Spillover group Treatment effect Spillover effect

11. Hypothesis 1: TDR decisions better reflect Afghan law, Islamic Shari’ah RLS-I elders recognize jurisdiction of the state RLS-I increases recording of decisions and registration with a government entity RLS-I elders gain knowledge of Afghan law and Islamic Shari’ah

12. RLS-I elders recognize jurisdiction of the state Elder jurisdiction Treatment gain Spillover gain Comparison gain D-i-DSpillover Elders resolve criminal aspects-6.6%14%4.5%-11.1%3.7% Elders resolve civil aspects36%-1%-2.6%38%1.6% Elders resolve both-15%38%23%-38%2.3%

13. RLS-I decision documentation and registration Measure Treatment gain Spillover gain Comparison gain D-i-DSpillover Documented71%57%45%26%12% Percent documented14%5.9%4.1%9.7%1.8% Registered*21%18%7.7%13%10% Percent registered*6.7%-6.4%0.4%6.3%-6.8%

14. RLS-I elder knowledge of Afghan law and Shari’ah TopicItems Treatment gain Spillover gain Comparison gain D-i-DSpillover Constitutional and Criminal1611.5%-7.1%7.5%4%-15% Family and Inheritance1411.5%-6.1%-8.3%19.8%2.2% Property45.8%-2.5%-12%18%9.3% All knowledge items3411.3%-6.1%-1.8%12.3%-4.6%

15. An elder’s capacity to learn affects outcomes Percentile rank at baseline 10 th percentile 25 th percentile 50 th percentile 75 th percentile 90 th percentile Gain score at endline 3.8%6.1%14%20%21% Highlights necessary tradeoffs between program objectives Higher capacity elders show strong gains (Access to Justice) Lower capacity elders show weak gains, but will reside in more rural, contested terrain (Counterinsurgency, Stability)

16. Hypothesis 2: Disputants perceive Jirga members and their decision making as impartial RLS-I improves disputant perception of procedural fairness in the process of resolution RLS-I improves disputant perception of overall justice in the decision RLS-I increases disputant perception of undue influence within and outside jirga 16

17. Impact measureD-i-DEffect sizep-value Procedural justice11%0.20.014 Subversion of decision7.5%.018.037 Justice of outcome21%0.18.293

18. RLS-I graduates and disputant assessment Each RLS-I graduate is predicted to increase disputant satisfaction by 4-7% In some specifications, 3 RLS-I graduates increases disputant satisfaction by as much as 30%

19. Elder knowledge and disputant assessment Average elder knowledge gain of 12-14% – increases procedural fairness by 9% – decreases perception of undue influence by 4% – increases overall justice of outcome by 6%

20. Link between arranged marriage and disputes between spouses or families Measure Treatment gain Comparison gain D-i-DSpillover Is it possible that parents’ marriage arrangements could be against wishes of children? 2.3%-6.3%8.5%30.8% If a marriage was against the wishes of one or both spouses, do you think this might lead to more disputes between the spouses or families? 3.1%-2.9%6.0%-3.8% Are sons in your community able to choose their life partner? 19.4%8.2%11.2%-4.1% Are daughters in your community able to choose their life partner? 10.9%4.6%6.3%33.2% 20

21. Does perceived jurisprudence interact with RLS-I treatment? Adding controls does not affect treatment effect estimate, except for jurisprudence 21 Procedural justice Subversion of decision Justice of outcome D-i-D11%21%7.5% D-i-D with jurisprudence controls16%23%11% D-i-D with jurisprudence instrumented28%73%21%

24. Recommendations for program M&E Prepare administrative data systems for impact evaluation! Recognize program M&E unit as a department for structured, experiential learning about what works and what doesn’t – Not just monitoring outputs and identifying outcomes Fundamental reference: It’s All About MeE – Lant Pritchett, Salimah Samji, Jeffrey Hammer 24

25. THANK YOU!!! Checchi Consulting

26. Performance Evaluation of Rule of Law Stabilization Informal Component (RLS-I) Program Implemented by Sayara Research May-July 2014

27. ADS 203: Performance Evaluation: Focus on descriptive and normative questions: What a particular project or program has achieved; How it is being implemented; How it is perceived and valued; Whether expected results are occurring; and Other questions pertinent to program design, management, and operational decision-making. No counterfactual analysis, unlike impact evaluation Performance and impact evaluations are complementary for programmatic learning

28. RLS-I Performance Evaluation Questions 1.Were the methods used to train elders effective in increasing their knowledge and skills to resolve disputes in accordance with Afghan law? 2.What role did the Community Cultural Centers (CCCs) play in disseminating informal justice knowledge to the local communities? To what extent do they plan to continue that role after RLS-I ends? 3.To what extent did RLS-I reduce harmful social practices that violate women’s rights, such as baad? 4.To what extent are there increased formal-informal justice sector linkages in RLS-I target districts? For example, to what extent are RLS-I trained elders more likely to refer criminal cases to the formal justice system and register decisions with government actors?

29. Mixed Qualitative and Quantitative Methods: Desk review of programmatic documents 40 key informant interviews (KIIs) with local government officials, implementing agency staff, and program beneficiaries 9 focus group discussions (FGDs) with male and female traditional dispute resolution (TDR) actors, key parties to disputes, and indirect beneficiaries N = 600 quantitative survey: Population proportionate to size representative sampling of the general population (indirect beneficiaries) across six targeted districts

30. Evaluation Question 1: Effectiveness of TDR Training 58% of the population reported that TDR actor’s knowledge increased (survey) 75% trainees reported high satisfaction with training (KIIs) 85% of female trainees reported greater responsibility and empowerment for TDR and respect from male TDR actors (KIIs) Female trainings were poorer quality (KIIs, FGDs)  Key recommendation: Training effectiveness would be improved by better quality control and more knowledge review

31. Awareness of Training for Local TDR Actors

32. Evaluation Question 2: Role of Community Cultural Centers (CCCs) RLS-I was not designed to provide direct support to CCCs, which were created under a previous USAID rule of law program that ended in 2009 (desk review) CCCs no longer existed or were not recognized by the population as institutions mandated to address rule of law issues (survey, KIIs, FGDs)  Key recommendation: Future programming should establish CCCs as stand-alone institutions with premises dedicated to TDR training and knowledge dissemination, certifications, record keeping of disputes, and other potentially valuable means of institutionalizing best practices in TDR.

33. Awareness of Radio and Television Programming on TDR

34. Evaluation Question 3: Reduction of Harmful Social Practices Such As Baad 55% of the population reported that baad and similar practices had occurred less often over the past two years (survey) 51% reported that general violations of women’s rights had had occurred less often over the past two years (survey) 40% of the population indicated that women’s involvement in community decision-making had increased (survey)  Key recommendation: Implement a gender strategy based on the different roles and responsibilities of male versus female TDR actors. Improve the delivery of training to women in insecure areas and foster greater cooperation between male and female TDR actors.

35. Change in Violations Committed Against Women

36. Frequency that Baad is Disputed

37. Evaluation Question 4: Improved Linkages between Informal and Formal Justice Sectors Referrals from the formal to the informal justice sector were most common (KIIs, FGDs) 46% of the population said they would go to a government court to resolve a serious crime No difference in levels of trust for government versus traditional forms of justice. Availability, convenience, reliability is more of a deciding factor (survey, KIIs, FGDs) Women more aware of, trusting, and more likely to seek justice in formal courts compared to men (survey, KIIs, FGDs)  Key recommendation: Directly facilitate greater contact and cooperation between formal and informal justice actors

38. Provider of Justice Trusted the Most by Men and Women

39. Conclusions The performance evaluation corroborated the findings of the impact evaluation while allowing USAID to explore questions relevant to future programming Answering the evaluation questions led to clear recommendations for future programming TDR is an important area of focus for increasing gender equality and women’s empowerment in Afghanistan

40. Thank you for your attention!