1. LESSONS LEARNED 1: IMPORTANCE OF HAVING CLEAR CONCEPTUAL FRAMEWORK TO GUIDE EVALUATION

2. SASA! Intervention Uganda Clear conceptual framework for intervention about process of change trying to achieve in community Stages of intervention focus and associated activities clearly articulated and translated into programming Used to inform M&E design Power Within Pre- contempl ation Phase 1: Start Power Over Contemplati on Phase 2: Awarene ss Power With Preparatio n for Action Phase 3: Support Power To Action / Maintenan ce Phase 4: Action SASA! Phases / Process of Change

3. Power Within Pre- contemplation Phase 1: Start Power Over Contemplation Phase 2: Aware- ness Power With Preparation for Action Phase 3: Support Power To Action / Maintenanc e Phase 4: Action Individual & Collective Capacity Supportive environment Enhanced ability to prevent and respond to VAW Sustained Action Changed policies Organized groups Changed practice in relationships, community, institutions SASA! Logic Model Action Intention to act Personal change: balancing power Public change: sanctions ag VAW Acceptability of expanded gender roles Behaviors Balancing power Communication with partner Decreased risk behaviors Community activism Acceptance and Influence Attitudes toward power, gender, human rights Silence broken Connection Activists/leaders/profe ssionals increasingly connected and active Skills Response to women experiencing violence Hold men accountable Promote balanced power Support to activists Context Societal Natl policy makers, media Community Police, LCs, HCPs, NGOs, Ssengas, RLs, Relationship Relatives, elders, neighbors Levels of SASA! Activities reaching each Circle of Influence Initial Outcomes Knowledge Recognizing VAW a problem Types of VAW Consequences of VAW VAW/HIV linkage Awareness Root cause as imbalance of power between women and men Change can happen Participation Activists at grassroots, in leadership, in institutions Intermediate Outcomes Individual Women, men, youth, CAs Longer term outcomes Impact Socio- demographic Factors Sex Age Income Education Employment Religion Place of residence Socio-cultural Factors Family Characteristics Social Support Alcohol Use Reduced social acceptance of VAW Decrease in experience / perpetration of VAW Improved response to women experiencin g violence Decrease in HIV/RH risk behaviors Critical Thinking and Dialogue Public debate and discussion Personal reflection SASA! Phases / Process of Change

4. Key elements in conceptual framework What are the range of potential exposures people may have to intervention? What is expected process of change? What are likely pathways of change? Among individuals receiving the intervention In the broader community What is a realistic timeframe for this change to occur? How do we measure this?

5. LESSONS LEARNED 2: POWER OF MULTIPLE FORMS OF EVIDENCE (IMAGE STUDY)

6. Did IMAGE affect MFIs financial performance? 50% reduction in centre vulnerability = attendance, arrears, savings Impact persisted 2 years after intervention completed Similar impact on drop out rates.

7. Past year experience of intimate partner violence reduced by 55% (aRR 0.45 95% CI 0.23-0.91) Past year experience of physical/sexual partner violence - pushed you - hit you - forced sex - scared to say no to sex Adjusted risk ratio 0.11 10 0.45 (0.23-0.91) Pronyk et al. The Lancet Dec. 2006

8. How was violence reduced? Changes in womens relationships Now that we have money we are able to say how we feel without fearing that your husband will stop supporting you. You can buy him cigarettes from your profit. Because of SEFs money we are experiencing fewer problems in our households Women supporting women We are able to overcome abuse when we are in SEF because we get support from the women in the groups. When you engage yourself with other women and listen to their problems that will help you to cope

9. LESSON LEARNED 3: COSTING IMPORTANT PART OF INTERVENTION EVALUATION

10. What does costing do? Document all inputs into intervention Capital & recurrent expenditure Ingredients approach to costing puts a financial value to all inputs: Monetary Non-monetary Can be used to: Look at break down of costs by activity Estimate unit costs Estimate cost-effectiveness

11. 2001-2004 2005-2007 2008-2010 430 households 4500 households (30,000) 15 000 households (80,000) Scaling up IMAGE in South Africa Pilot Study Additional cost = US $43/client Scale-up Additional cost = US $13/client

12. CHALLENGES 1. REPORTING BIAS

13. Physical and/or sexual partner violence against women at baseline – SASA! study in past 12 months INTERVENTIONCONTROL Violent BehaviourWomenMenWomenMen Slapped her or thrown something at her that could hurt her 23.4%20.5%17.6%23.1% Pushed her or shoved her or pulled her hair 11.5%7.1%9.5%7.2% Hit her with his fist or with something else that could hurt her 11.2%3.8%8.4%4.5% Kicked her, dragged her or beat her up 10.2%1.6%8.4%1.5% Choked or burnt her on purpose 3.9%0.0%0.7%0.0% Threatened or actually used a gun, knife or other weapon 2.0%0.0%2.6%0.0% Threatened or intimidated her into having sexual intercourse even when she did not want to 9.5%0.6%7.7%0.9% Physically forced her to have sexual when she did not want to 10.5%2.6%9.5%1.2% Physical and/or sexual violence 29.1%23.1%24.5%25.4%

14. The potential for reporting bias about gender norms & violence Women tend to under-report experiencing violence May be more willing to disclose following exposure to intervention Could find higher disclosure in intervention arms Men may not disclose perpetrating less socially condoned acts of violence May be even less willing to report disclosure following contact with intervention Could find higher disclosure in intervention arms

15. Implications for evaluation Importance of choosing quantitative outcome measures carefully Rely on womens reports on experiencing of violence, rather than mens reports of perpetration If find reduction, effect less likely to be due to reporting bias Use qualitative research to triangulate

16. CHALLENGE 2: CAPTURING SOCIAL MOBILISATION DIFFICULT AND UNPREDICTABLE

17. IMAGE: community mobilisation led to a diverse range of activities Many focused on violence and HIV Also focused on other issues of concern to women Difficult to fully document all of activities Community mobilization: 40 village workshops 16 meetings with local leaders 5 public marches 2 partnerships with local institutions 2 new village committees target Crime and Rape

18. Timelines as part of M&E activities in Uganda Periodically using timeline with key informants Discuss key actions and key events in the community Aim to map out scale and geographical diversity of activities over time

19. CHALLENGE 3: EVALUATION DIFFICULT

20. Getting reliable evidence on impact is not easy Getting strong quantitative data on impact difficult: RCTs gold standard study design, but expensive and technically difficult Require partnership between intervention agencies & research organisations May be rare that can show intervention impact on HIV or VAW directly Seeking to evaluate complex interventions on complex issue but: Trial design generally focuses on limited number of impacts – need to identify primary outcomes Forms of social change may have multiple benefits Limited evidence about what may be reasonable timeframes for different forms of change May be lack of recognition of importance of change in pathway variables

21. The challenge of building a strong evidence base Data from control communities important to help attribute impact Challenge of getting sufficient number of study and control communities Difficult to identify and maintain control populations May not be feasible if intervention national level Difficult to identify and maintain control populations Difficult to fund evaluation research: IMAGE had 9 + donors SASA! funding from 3 donors for baseline alone GBV not primary focus of most research donors Leads to under-powered studies Less focus on statistical significance

22. CONCLUSIONS

23. The value of good M&E evidence Growing body of experience, but extremely limited evidence about intervention impact on VAW & IPV Importance of M&E to provide insights about: Process & scale of project implementation Impact on recipients and broader community Resources / costs required Multiple forms of evidence important: Quantitative - scale of activity, magnitude of impacts Qualitative – acceptability, what happened, what means, quality, role of key players, unexpected events Economic - forms of input, economic value, potential costs if replicate Lessons from M&E important to inform ongoing programming replication locally, nationally & internationally

24. The challenge of building a strong evidence base Fundamental challenge of how to move from documenting levels of activity to level of change Getting strong data on impact difficult: RCTs expensive and technically difficult Require partnership between intervention agencies & research organisations May be rare that can show intervention impact on HIV or VAW directly Seeking to evaluate complex interventions on complex issue: Timeframe for change my be longer than evaluation funding Importance of capturing broad benefits of social empowerment interventions Mixed methods likely to provide most reliable evidence Draws on strengths of each approach Provides opportunities for triangulation

25. The way forward? Document experience and lessons learned From broader field of evaluation research – community interventions, service delivery, media interventions From current M&E of different models of GBV intervention Experience combining quantitative and qualitative methods About realistic timeframes and process of change Lobby for strategic investment in multi-disciplinary evaluation research: Womens empowerment School based programmes Community mobilisation Masculinities Alcohol Service provision Role of legal and policy reform Recognise need for methodological work Use evaluation research to learn about pathways of change, and broader lessons for M&E How to conceptualise, define and measure key variables Integrating participatory evaluation methods with quant evaluation frameworks