Methods for the evaluation of the Abstinence and Faithfulness among Youth (ABY) interventions in South Africa Dr Geoff Setswe, Dr Mark Ottenweller & Roger Dickinson SAMEA Evaluation in Action conference 30 March 2007

2 Structure of the presentation Introduction to the ABY project ABY interventions at Hope Worldwide Evaluation methods  Survey methods  Qualitative methods – Focus Group Discussions Some baseline demographic results Recommendations

3 Introduction The goal of the ABY project is to enhance local responses to prevent HIV infection through encouraging abstinence, faithfulness and avoidance of unhealthy sexual behaviors among youth. ABY interventions are implemented by Hope Worldwide in selected sites in 5 cities in South Africa. Project is funded by PEPFAR/USAID over five years ( ) in 5 African countries (SA, Botswana, Kenya, Nigeria and Ivory Coast). We present methods used for the evaluation of ABY interventions in South Africa.

4 ABY interventions at Hope Worldwide Abstinence interventions encourage unmarried individuals to abstain or delay sex as the best way to prevent exposure to HIV and STIs. Targeted at primary school learners aged years. Be faithful interventions encourage individuals to practice fidelity in marriage and in other sexual relationships as a critical way to reduce the risk of exposure to HIV. Targeted at youth aged years. Both interventions could be part of ABC or DRC programs

5 ABY interventions at Hope Worldwide Interventions are designed and implemented by local people and adapted to fit local conditions; Implemented in age-appropriate and culturally relevant groups, in targeted areas. Include life skills programs with components of Men As Partners (MAP) and VCT for youth; For and 15 – 19 year-olds, interventions are school and/or church-based. For youth aged 20-24, interventions are church or community-based. Parents and adult church members have interventions specifically targeted to them.

6 Methods ABY interventions are evaluated using mixed methodologies - quasi-experimental and qualitative designs with 3 strategies: 1. Pre-intervention or baseline surveys for learners and youth; just before the interventions were implemented in 2005/6. 2. Post-intervention surveys for learners and youth (who participated in the interventions); when the interventions have been completed in 2008/9. 3. Focus Group Discussions (FGDs) for parents and church members who participated in the interventions. FGDs will be conducted once – during the middle of the project, in 2007.

7 Pre- and Post intervention surveys One-group, Pretest/Posttest design A single group i.e. learners and youth, is studied before and after the ABY intervention. No control group. Layout O 1 X O 2 O 1 = Baseline or pre-intervention survey in 2005/6 X = ABY intervention in 2006/7 to selected participants O 2 = Post-test survey in 2008/2009

8 Sampling and sample size estimation CityNo. of sites No. of sampled sites No. of evaluation points No. of participants Johannesburg823 x 2 = 660 x 6 = 360 Cape Town423 x 2 = 660 x 6 = 360 Durban323 x 2 = 660 x 6 = 360 Mthatha323 x 2 = 660 x 6 = 360 Port Elizabeth213 x 1 = 360 x 3 = 180 Total points1,620

9 Sample size estimation 11 sites were randomly selected from 20 intervention sites. Then 27 evaluation points were identified. The evaluation points comprised:  primary schools (for learners aged years),  high schools (for learners aged years)  sites for out-of-school youth (aged years). Data were collected from 60 randomly selected learners or youth at each of the 27 evaluation points = 1,620. Grades 11 and 12 were not evaluated as they cannot be followed up in post-intervention surveys Data were collected by trained fieldworkers.

10 Survey questionnaires Two types of questionnaires were used: 1. for learners aged years and 2. for learners/youth aged years. Key questions were on:  Awareness/knowledge of HIV/AIDS and attitudes towards PLWHA  Views on dating, relationships, sexuality and pregnancy  Views on abstinence and faithfulness  The role of the media, role models and workshops in encouraging abstinence and faithfulness  The role of peer-group pressure and violence on abstinence and faithfulness.

% (740) 43.5% (570) n = Response rate = 81% Some baseline demographic results Gender distribution of respondents

12 Some baseline demographic results Distribution of respondents according to cities 20% 33.1% 13.1% 25.6% 8.2%

% (505) 46.6% (610) 12.5% (164) 2.4% (31) n = 1310 Some baseline demographic results

14 Qualitative methods - FGDs 15 FGDs planned – 3 in each of the 5 cities:  5 FGDs - adults who have participated in the ABY interventions provided through FBOs in the 5 cities  5 FGDs - adults or parents of children at ABY intervention schools in the five cities.  5 FGDs - persons aged 25 years and above from the general population in each of the 5 cities. FGDs will be used to gather in-depth information on parents and church members' opinions and views on abstinence and faithfulness in relationships. Each FGD will have participants who will be asked to provide verbal consent to participate and permission to record the interviews. The training for facilitators of FGDs has been scheduled and the FGD guide is ready.

15 Schedule of ABY FGDs CitySitesTarget group Cape TownTo be confirmed (TBC)FBO adults Parents at schools 25 years + general population DurbanTBCFBO adults Parents at schools 25 years + general population JohannesburgTBCFBO adults Parents at schools 25 years + general population MthathaTBCFBO adults Parents at schools 25 years + general population Port ElizabethTBCFBO adults Parents at schools 25 years + general population

16 Recommendations To ensure credibility:  evaluation should be integrated into the initial planning of the project;  all evaluation preparations and training should be done before the interventions start. To ensure effective evaluation of ABY interventions, it is essential to:  use various designs and strategies in different target groups;  involve team members in the design of the evaluation;  train team members on the methods of evaluation, data capturing and the importance of data quality and management.

17 Acknowledgements ABY intervention and evaluation teams at Hope Worldwide South Africa Learners, youth, parents and adults who participated in the evaluations and FGDs Funding received from USAID/PEPFAR