EMPOWERMENT EVALUATION: Khulisa’s Experience in Using Project Staff as Data Collectors Peter Njaramba, Mary Pat Selvaggio and Josie Mangxaba Khulisa Management Services Presented at SAMEA Conference March 2007

What is Empowerment Evaluation? Developed by David Fetterman at Stanford University in 1994 Empowerment Evaluation is the “use of evaluation concepts, techniques, and findings to foster improvement and self-determination” (Fetterman et al. 1996) 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

What is Empowerment Evaluation?(2) Based on 10 Principles Evaluation Process Principles Evaluation Outcomes Principles Community ownership Inclusion Democratic participation Community knowledge Accountability Evidence-based strategies Improvement Organizational learning Social justice Capacity-building Improvement: A key aim is to improve people, programs, organisations and communities and to help them achieve results. 2. Community ownership: Program stakeholders, with the assistance of evaluators, take responsibility for designing and conducting the evaluation and putting the findings to use. 3. Inclusion: Participants, staff from all levels of a program or organisation, funders, and community members are invited to participate. 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Traditional vs. Empowerment Evaluation In the evolution toward empowerment evaluation, there has been increasing use of participatory and collaborative approaches to evaluation Evaluation Generation First Second Third Fourth Evaluation Activity Measurement Description Judgment Responsive Constructivist Role of Evaluator Technical Describer Judge Collaborator Change agent 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Traditional vs. Empowerment Evaluation (2) Traditional Evaluation Empowerment Evaluation External Internal Outsiders are evaluators Outsiders are facilitators Stakeholders often don't participate Broad range of stakeholders participate Dependency Self-determination, capacity-bldg Donor focus and ownership of evaluation Participant focus and ownership of evaluation Focus on accountability Focus on learning Predetermined design Flexible design Formal methods Rapid appraisal methods 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Traditional vs. Empowerment Evaluation (3) Distinguishing feature from other participatory evaluation methods: …….Its acknowledgement and deep respect for people's capacity to create knowledge about, and solutions to, their own experiences (AEA, 2005). 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Traditional vs. Empowerment Evaluation (4) The role of evaluation expert in EE: Facilitator and teacher Program supporter rather than being impartial Change agent Facilitate the development of a rigorous and organized approach to evaluation Help establish baseline data, monitor change over time Ensure everyone is heard 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors Khulisa Evaluations Khulisa establishes a participative working relationship with clients Participation takes place throughout all phases of the evaluation Degree and breadth of participation differs by: Nature of the program Contractual agreement with funders Requests by clients 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s use of EE Principles One of the principles of Empowerment evaluations is to facilitate organizations’ use of data to learn and their ability to sustain their evaluation efforts (capacity-building principle). Khulisa has incorporated this EE principle into several programme evaluations by encouraging programme staff to: Be involved in design of the data collection tools, and To serve as data collectors …..In order to build greater understanding of evaluation among programme staff and build buy-in to the findings. 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Projects that used Programme Staff as Data Collectors HIV and AIDS Audit: Interventions in South African Higher Education Institutions District Development and Support Programme for Schools in South Africa Evaluation of the Life Skills-HIV/AIDS Programme in Primary and Secondary Schools of Northwest Province AIDS Prevention, Positive Living and Empowerment (APPLE) Project in Mozambique and Malawi 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors Did the use of Programme Staff increase knowledge and understanding of Evaluation? We recently contacted 20 individuals who were involved in data collection, and we asked them about their experience in collecting the data. A few of the respondents indicated that they already had good M&E understanding at the time of the evaluation. No. of who reported that participating in the evaluation: Facilitated learning = 14 of 20 Taught them general evaluation issues and techniques = 11 of 20 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors Did the use of Programme Staff increase knowledge and understanding of Evaluation? (2) “I found the training beneficial. I also was able to know actual issues on the ground, not what is reported as differences in program implementation from school to school”. “I learned skills on management of data collection” “I learnt about questionnaire design. I learnt what to include in tools” “I developed skills for data collection for monitoring” “I found the workshop we had with Khulisa to look at Terms of Reference and to get together basic questions useful-data collection process and trying to get people to return questionnaires was a lot of hard work and frustrating rather than a learning process” 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors Were programme staff able to apply what they learned about evaluation to other work situations? Number who reported having applied what they learned about evaluation to other work situations = 10 out of 20 “I used the experience in data collection and analysis in my Masters Research. In another program, the experience increased my awareness and ability to determine program problems and solutions” “I made changes to the monitoring tool used in managing those who are involved in OVC, and also to include community in programs not just rely on Government” “Based on this experience, I realized the internal audits that are done in my workplace are not formulated correctly. We came up with new instruments” 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors Did the use of programme staff as data collectors increase their buy-in to the evaluation findings? No. who report that the experience increased their: No. Ownership of the overall evaluation activity and the results (N=20) 8 Capacity to use evaluation data (N=20) 11 Capacity to interpret evidence (N=20) 10 Capacity to draw conclusions (N=20) 9 Capacity to make judgments (N=20) Capacity to disseminate findings (N=20) Capacity to consider inconsistencies and contradictions (N=20) 7 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Other Feedback from Project Staff on their participation No. reporting ability to: No. Identify programme strengths (N=20) 9 Identify programme problems (N=20) Identify programme solutions (N=20) 10 Implement programme solutions (N=20) 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors Conclusions Many (but not all) respondents report having benefited from participating in data collection. Benefits were felt more strongly by those who had little prior M&E experience before participating in Khulisa’s evaluation Perceived benefits include understanding of general issues and ultimate ownership of collected data. 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Empowerment Evaluation Resources Robin Lin Miller and Rebecca Campbell: Taking Stock of Empowerment Evaluation: An Empirical Review, American Journal of Evaluation 2006; 27; 296 USAID Center for Development Information and Evaluation: Conducting A Participatory Evaluation, 1996 David Fetterman: Empowerment evaluation in Brazil: building capacity and facilitating self-determination, Dec 2004 David Fetterman: Empowerment Evaluation: Building a Learning Organization, March 3, 2006 Wayne Miller and June Lennie: Challenges & issues in applying empowerment evaluation principles in practice: Case study of the evaluation of a national school breakfast program, 2006 Empowerment Evaluation website: www.stanford.edu/~davidf/empowermentevaluation.html 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors

Khulisa’s Experience in Using Project Staff as Data Collectors The End Thank You 13 January 2019 Khulisa’s Experience in Using Project Staff as Data Collectors