Evaluation on a Shoestring Participatory Techniques for the Evaluator with Minimal Funding

Overall goals for the morning Learn how to use 11 interactive techniques Study frameworks for participatory evaluation practice Discuss how to overcome barriers to participatory evaluation Reflect on how to apply these ideas to your own evaluation practice

The mornings agenda Seven techniques before the break Lecturette: Frameworks and psychological principles Four more techniques before lunch Throughout the morning: Continuing reflection on your own practice

Types of techniques For responding to set content (#1-5) For generating information (#6-10) For organizing information (#11)

The basic tasks of inquiry 1. Framing questions 2. Determining an appropriate design 3. Identifying a sample 4. Collecting data 5. Analyzing data and presenting results 6. Interpreting results 7. Reporting

Important point You can use these techniques THROUGHOUT the evaluation process [Not just at the beginning...]

Why use participatory techniques? How can participatory techniques help evaluators?

My participatory principles 1. Building peoples capacity to think evaluatively matters. 2. Participation in evaluations should be a learning experience. 3. It is essential to involve people actively in evaluations.

How participation helps People invest in the evaluation process and outcomes It makes evaluation less scary Some will learn evaluation skills It is fun!

Technique #3- Statement 1 State specialists should include an evaluation tool for each program they introduce to county staff. Strongly DisagreeAgree Strongly Disagree Agree

Technique #3- Statement 2 State specialists should be responsible for compiling overall evaluation results that will lead to the creation of impact statements for ND reporting. Strongly DisagreeAgree Strongly Disagree Agree

Technique #3- Statement 3 County staff should be required to enter all county survey results into a system that will allow state staff to aggregate data and produce impact reports. Strongly DisagreeAgree Strongly Disagree Agree

Debrief strategies for responding to set content #1- Voicing variables #2- Fist-to-five #3- Belief sheet #4- Dot voting #5- Corners

Strategies for generating information #6- Three-step interview #7- Data dialogue #8- Making metaphors #9- Check-in #10- Graffiti/carousel

Technique #6- Three-step interview Three roles create three steps: Interviewer Interviewee/respondent Recorder The interview process is structured to build on social psychological principles of cooperation

Technique #7- Data dialogue A process to use when you cannot afford focus groups It takes advantage of some of the processes of the three-step interview Can be useful in community settings

Analysis exercise In what ways is a data dialogue like a three-step interview? In what ways is it different?

Conceptual frameworks for participatory techniques Useful for planning Helpful for analysis

What roles can evaluators play? A relationship exists between the evaluator and the client, the program staff, and other evaluation stakeholders The evaluation decision-making and implementation relationship may shift during the study

Interactive Evaluation Quotient LOW HIGH Evalu- ator Program leaders, staff Involvement in decision making and implementation Participant- directed Collabor- ative Evaluator- directed ZONES

Examples of evaluator roles Technical expert on research design, measurement, statistics Facilitator of group interaction Coach of others doing their own evaluations Others?

Types of participant involvement Mere awareness Passive support or minimal participation Active participation in the evaluation process Commitment to consider and ultimately use the evaluation results

Social interdependence theory Goal Structures Interactions Outcomes

Social interdependence theory At best Cooperative goal structures (positive interdependence) Promotive (responsive) interaction Constructive outcomes At worst Competitive goal structures (negative interdependence) Oppositional (obstructive) interaction Destructive outcomes

Structuring cooperative participation Positive interdependence Individual accountability Promotive (face-to-face) interaction Social skills Group processing (Adapted from Johnson & Johnson, 2000)

Structuring positive interdependence 1. Identify a common purpose 2. Create shared benefits or consequences 3. Provide one set of materials 4. Assign complementary and interconnected roles... plus

Structuring positive interdependence 5. Designate an outside force to motivate people to coordinate efforts 6. Arrange the workspace purposefully 7. Have the group establish a shared identity

Strategies for generating information #6- Three-step interview #7- Data dialogue #8- Making metaphors #9- Check-in #10- Graffiti/carousel

Strategy for organizing information #11- Concept formation

Techniques #10 and #11 Can be done on the wall, informally, and is then called graffiti Can be done on flipchart paper passed among groups and is then called carousel Always coupled with Technique #11- Concept formation

Technique #8- Making metaphors A is worth 1000 words tall impressive BIG magnificent terrifying High Not like home awesome sun struck

Technique #9- Check-in What is the most important idea you have learned this morning?

Thank you, and good luck! Jean A. King