1. Evaluating Work with Men and Boys Gary Barker Instituto Promundo Rio de Janeiro, Brazil

2. Why evaluate? 1.) Improve our programmes: the process  How did the intervention go?  Did we reach men and boys?  Did they find it interesting and useful? 2.) Assess impact: the effectiveness  What impact can we achieve in engaging men and boys with our specific intervention?  How long does it take? What is the “dose” required to achieve that impact?  How much does it cost?

3. What are we evaluating? The What has to do with our specific goals for engaging men and boys, and our approach What can work with men and boys accomplish? What do we propose with the work? Examples: Knowledge change, attitude change, behavior change, increase in users of a service, increase in condom use or condom sales, etc. PITFALL 1 PITFALL 1: We too often seek to measure things that are outside or not contemplated in our intervention

4. A Typology of Programmes for Engaging Boys and Men Gender nuetral: No distinction between men & women; men just another target group Gender sensitive: Recognize gender norms but little attempt to transform them Gender norms transformative: Seek to promote equitable relationships, somehow change gender norms How do we evaluate interventions that seek to change gender norms?

5. What is engaging men and boys from a gender norms transformative perspective?  Include discussion of gender norms and how these influence behavior;  Include deliberate public debate + critical reflection about these norms; and  Measure changes in attitudes toward gender norms + relevant behaviors.

6. Evaluating a Gender Norms Approach 1.) Formative research – mapping of masculinities, gender norms. Define: What gender norms need to be transformed? How best to promote change? What are the “cracks” or existing gender-equitable attitudes that can be built on? 2.) Intervention formulation – Logical framework, setting objectives + outcome indicators. What interventions will transform those norms? 3.) Implementation – Process evaluation: How many men/boys? How did it work? 4.) Impact evalualtion: Did behavior + norms change?

7. 1.) Formative research Qualitative and quantitative What are the prevailing norms related to masculinities/gender in a given setting? How do these norms affect relations between men and women? Where are the “cracks”, “voices of resistance” or opportunities for change? Interview: men, women, boys, girls, gatekeepers

8. 2.) Intervention Formulation: Components to Change Gender Norms

9. 3.) Implementation: Process evaluation Number of sessions, participants (attendance forms) Quality of sessions (interviews with facilitators, field notes, simple post-session questionnaire) Number of participants at community activities (observation, field notes) Condoms distributed

10. 4.) Impact Evaluation: Measuring Gender Norms: the Gender Equitable Men Scale – GEM Scale The domains or major categories of themes:  Home & child-care  Sexual relationship  Health & STI prevention  Violence  Homophobia & relations with other men

11. Examples of Items from the GEM Scale: Men are always ready to have sex. There are times when a woman deserves to be beaten. A woman’s most important role is to take care of her home and cook for her family. I would never have a gay friend.

12. Association between traditional norms and STI Symptoms* *p < 0.001 – Mantel-Haenszel chi-square test, logistic regression Percent w STI Symptoms Baseline:Change in 1 year: More equitable young men reported less STI symptoms: In Bangu, 4 times less In Maré, 8 times less

13. Traditional Views Regarding Gender by Self- Reported Use of GBV – Young Men in Brazil (n=225, 15-24, RJ) *p < 0.001 - Chi-square test Percent of Partner Violence

14. Pre-test N = 258 Post-test 1 N = 230 Post-test 2 N = 217 6 months Pre-test N = 250 Post-test 1 N = 217 Post-test 2 N = 172 6 months BANGUBANGU MARÉMARÉ MACACOSMACACOS Pre-test N = 250 Post-test 1 (contr.) Pre-test (int.) N = 180 Post-test 1 (delayed int.) N =89 6 months3 months

15. Change in Selected GEM Scale Items *p < 0.05 – t- test

16. Behavior Change: Condom Use at Last Sex *p < 0.05 - Chi-square test Percent of Use w Primary Partner

17. Qualitative Results: Program H Evaluation Study (Brazil)  Reported changes in styles of interacting with other men: movement toward more cooperative, less aggressive interactions  Ability to discuss sexuality openly (male and female)  Recognition of women as having sexual rights, sexual agency  Worry about own health needs, HIV testing (male and female)  Delayed initiation of sexual activity with current partner

18. Pre Test Post Test BehaviorAttitudes % Agree % harassed any girl A man can hit a woman if she cheats on him A man can hit a woman if she refuses sex A woman should tolerate violence to keep family together There are times when women deserve to be beaten Harassment of Girls

19. Other Outcome Indicators: Interventions Engaging Men and Boys Descreased self-reported use of physical, sexual and psychological violence in intimate relationship (Safe Dates Programme, US; Soul City, South Africa) Increased contraceptive use (Young Dads Programme, US; Together for a Happy Family, Jordan; Male Motivation Campaign, Zimbabwe and Guinea; Involving Men in Contraceptive Use, Ethiopia) Increased communication with spouse or partner about MCH, RH(Involving Men in Maternity Care, India; Together for a Happy Family, Jordan; Male Motivation Campaign, Guinea; Soul City, South Africa)

20. Other Outcome Indicators: Interventions Engaging Men and Boys - 2 More equitable treatment of children (Together for a Happy Family, Jordan) Increased use of SRH services by men (Integration of Men RH Services in Family Welfare Centers, Bangladesh) Increased condom use (Sexto Sentido, Nicaragua; Program H, Brazil) Decreased rates of STIs (Program H, Brazil) Increased social support of spouse (Soul City, South Africa).

21. How much evaluation do you need? Limited resources: Some formative + process + post-only focus groups Modest resources for evaluation.: Formative + process + simple pre- and post-test (quantitative) with no control group PITFALL 2: PITFALL 2: Although qualitative research may be less costly, analysis is more complicated and should be treated as such. Collecting pre- and post-test frequencies is easier although sometimes more costly.

22. Higher level of resources:  Extensive formative  Multiple sites + control group (or delayed intervention group)  Triangulation with partners/women  Qualitative and Quantitative throughout

23. Conclusions More + longer-term evaluation often difficult, but necessary Be realistic in selecting indicators; use methods that are appropriate for the intervention Gather cost data: Need to show cost of achieving change in gender norms Move from the individual to the structural level – from gender norms to gender structures Use evaluation for advocacy but recognize that we can’t prove it all.....