1. EMPOWERMENT EVALUATION: Khulisa’s Experience in Using Project Staff as Data Collectors
Peter Njaramba, Mary Pat Selvaggio and Josie Mangxaba
Khulisa Management Services
Presented at SAMEA Conference
March 2007

2. What is Empowerment Evaluation?
Developed by David Fetterman at Stanford University in 1994
Empowerment Evaluation is the “use of evaluation concepts, techniques, and findings to foster improvement and self-determination” (Fetterman et al. 1996)
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

3. What is Empowerment Evaluation?(2)
Based on 10 Principles
Evaluation Process Principles
Evaluation Outcomes Principles
Community ownership
Inclusion
Democratic participation
Community knowledge
Accountability
Evidence-based strategies
Improvement
Organizational learning
Social justice
Capacity-building
Improvement: A key aim is to improve people, programs, organisations and communities and to help them achieve results.
2. Community ownership: Program stakeholders, with the assistance of evaluators, take responsibility for designing and conducting the evaluation and putting the findings to use.
3. Inclusion: Participants, staff from all levels of a program or organisation, funders, and community members are invited to participate.
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

4. Traditional vs. Empowerment Evaluation
In the evolution toward empowerment evaluation, there has been increasing use of participatory and collaborative approaches to evaluation
Evaluation Generation
First
Second
Third
Fourth
Evaluation Activity
Measurement
Description
Judgment
Responsive
Constructivist
Role of Evaluator
Technical
Describer
Judge
Collaborator
Change agent
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

5. Traditional vs. Empowerment Evaluation (2)
Traditional Evaluation
Empowerment Evaluation
External
Internal
Outsiders are evaluators
Outsiders are facilitators
Stakeholders often don't participate
Broad range of stakeholders participate
Dependency
Self-determination, capacity-bldg
Donor focus and ownership of evaluation
Participant focus and ownership of evaluation
Focus on accountability
Focus on learning
Predetermined design
Flexible design
Formal methods
Rapid appraisal methods
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

6. Traditional vs. Empowerment Evaluation (3)
Distinguishing feature from other participatory evaluation methods:
…….Its acknowledgement and deep respect for people's capacity to create knowledge about, and solutions to, their own experiences (AEA, 2005).
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

7. Traditional vs. Empowerment Evaluation (4)
The role of evaluation expert in EE:
Facilitator and teacher
Program supporter rather than being impartial
Change agent
Facilitate the development of a rigorous and organized approach to evaluation
Help establish baseline data, monitor change over time
Ensure everyone is heard
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

8. Khulisa’s Experience in Using Project Staff as Data Collectors
Khulisa Evaluations
Khulisa establishes a participative working relationship with clients
Participation takes place throughout all phases of the evaluation
Degree and breadth of participation differs by:
Nature of the program
Contractual agreement with funders
Requests by clients
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

9. Khulisa’s use of EE Principles
One of the principles of Empowerment evaluations is to facilitate organizations’ use of data to learn and their ability to sustain their evaluation efforts (capacity-building principle).
Khulisa has incorporated this EE principle into several programme evaluations by encouraging programme staff to:
Be involved in design of the data collection tools, and
To serve as data collectors
…..In order to build greater understanding of evaluation among programme staff and build buy-in to the findings.
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

10. Projects that used Programme Staff as Data Collectors
HIV and AIDS Audit: Interventions in South African Higher Education Institutions
District Development and Support Programme for Schools in South Africa
Evaluation of the Life Skills-HIV/AIDS Programme in Primary and Secondary Schools of Northwest Province
AIDS Prevention, Positive Living and Empowerment (APPLE) Project in Mozambique and Malawi
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

11. Khulisa’s Experience in Using Project Staff as Data Collectors
Did the use of Programme Staff increase knowledge and understanding of Evaluation?
We recently contacted 20 individuals who were involved in data collection, and we asked them about their experience in collecting the data.
A few of the respondents indicated that they already had good M&E understanding at the time of the evaluation.
No. of who reported that participating in the evaluation:
Facilitated learning = 14 of 20
Taught them general evaluation issues and techniques = 11 of 20
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

12. Khulisa’s Experience in Using Project Staff as Data Collectors
Did the use of Programme Staff increase knowledge and understanding of Evaluation? (2)
“I found the training beneficial. I also was able to know actual issues on the ground, not what is reported as differences in program implementation from school to school”.
“I learned skills on management of data collection”
“I learnt about questionnaire design. I learnt what to include in tools”
“I developed skills for data collection for monitoring”
“I found the workshop we had with Khulisa to look at Terms of Reference and to get together basic questions useful-data collection process and trying to get people to return questionnaires was a lot of hard work and frustrating rather than a learning process”
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

13. Khulisa’s Experience in Using Project Staff as Data Collectors
Were programme staff able to apply what they learned about evaluation to other work situations?
Number who reported having applied what they learned about evaluation to other work situations = 10 out of 20
“I used the experience in data collection and analysis in my Masters Research. In another program, the experience increased my awareness and ability to determine program problems and solutions”
“I made changes to the monitoring tool used in managing those who are involved in OVC, and also to include community in programs not just rely on Government”
“Based on this experience, I realized the internal audits that are done in my workplace are not formulated correctly. We came up with new instruments”
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

14. Khulisa’s Experience in Using Project Staff as Data Collectors
Did the use of programme staff as data collectors increase their buy-in to the evaluation findings?
No. who report that the experience increased their:
No.
Ownership of the overall evaluation activity and the results (N=20) 8
Capacity to use evaluation data (N=20) 11
Capacity to interpret evidence (N=20) 10
Capacity to draw conclusions (N=20) 9
Capacity to make judgments (N=20)
Capacity to disseminate findings (N=20)
Capacity to consider inconsistencies and contradictions (N=20) 7
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

15. Other Feedback from Project Staff on their participation
No. reporting ability to:
No.
Identify programme strengths (N=20) 9
Identify programme problems (N=20)
Identify programme solutions (N=20) 10
Implement programme solutions (N=20)
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

16. Khulisa’s Experience in Using Project Staff as Data Collectors
Conclusions
Many (but not all) respondents report having benefited from participating in data collection.
Benefits were felt more strongly by those who had little prior M&E experience before participating in Khulisa’s evaluation
Perceived benefits include understanding of general issues and ultimate ownership of collected data.
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

17. Empowerment Evaluation Resources
Robin Lin Miller and Rebecca Campbell: Taking Stock of Empowerment Evaluation: An Empirical Review, American Journal of Evaluation 2006; 27; 296
USAID Center for Development Information and Evaluation: Conducting A Participatory Evaluation, 1996
David Fetterman: Empowerment evaluation in Brazil: building capacity and facilitating self-determination, Dec 2004
David Fetterman: Empowerment Evaluation: Building a Learning Organization, March 3, 2006
Wayne Miller and June Lennie: Challenges & issues in applying empowerment evaluation principles in practice: Case study of the evaluation of a national school breakfast program, 2006
Empowerment Evaluation website:
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors

18. Khulisa’s Experience in Using Project Staff as Data Collectors
The End
Thank You
13 January 2019
Khulisa’s Experience in Using Project Staff as Data Collectors