Chapter 8: Participant-Oriented Evaluation Approaches By: Jason Artz, Andrea Hogentogler, Stacey Knerr, and Robert Shaw SAK

Orienting Question #1: Who were the historic leaders in establishing participant-oriented approaches and what did they contribute? Robert Stake: Responsive Approach 1967 – The Countenance of Educational Evaluation Two basic acts of evaluation are description and judgment Created evaluation framework on p. 192 1973 – Responsive Evaluation “What people do naturally in evaluating things. They observe and react” (p. 193) 1975a, 1975b, 1978, 1980 – subsequent conceptions of responsive evaluation Flexible, changing methods & approaches Multiple realities Local is key SAK

Orienting Question #1 cont. Egon Guba and Yvonna Lincoln: Naturalistic and Fourth-Generation Evaluation 1985 – Naturalistic Inquiry Consider qualitative, naturalistic methods, including interviews, observations, and case studies “The naturalistic evaluator…observes the program and its actions, its participants and staff, in its natural environment and, through observations, documents and records, interviews, and unobtrusive measures, comes to understand and describe the program.” (p. 198) 1989 – Fourth-Generation Evaluation Evaluators no longer simply measuring, describing, judging Argued for more active roles for stakeholders & less neutral roles for the evaluator SAK

Orienting Question #2: What prompted their work? 1967 – Several evaluation theorists began to react to what they considered to be the dominance of mechanistic and insensitive approaches to evaluation in the field of education Argued that human element was missing 1970s – Stressed first hand experience with program activities and settings and involvement of program participants, staff, and managers in evaluation SAK

Orienting Question #3: How do the practical participative approaches differ from the transformative ones? Orienting Question #4: Contrast the Practical Participatory Evaluation, Empowerment Evaluation, Development, and Deliberative Democratic Evaluation approaches. How do their purposes and methods differ? SAK

Practical participative approaches involve stakeholders in an attempt to increase use of the evaluation results based on the view that involvement gives stakeholders, the potential users, a sense of ownership and understanding that increases the likelihood of use. Stakeholders: managers, staff, and policymakers Balanced control Practical participative approaches include: Stakeholder-based evaluation (evaluator retains control, and seeks input from a variety of stakeholders Practical Participatory Evaluation (Shared decision making, or partnerships between evaluators and key stakeholders) Developmental Evaluation (Shared decision making, or partnerships between evaluators and key stakeholders) SAK

Transformative participatory approaches involve stakeholders for political reasons: to empower those stakeholders by providing them with tools in evaluation and self-determination or insights into power arrangements in their community or society, and tools to change those power arrangements. Stakeholders: Program managers and staff Involve stakeholders for political reasons To empower stakeholders Transformative participatory approaches include: Empowerment Evaluation (Primary decision-making power given to the stakeholders) Deliberative Democratic Evaluation (Uses democratic principles to empower each stakeholder group, and encourages full participation to help the evaluator make informed decisions) SAK

Orienting Question #5: Under what conditions would you use each? Conditions to use Practical Participatory Evaluation To achieve goals, may be used: Primarily at the beginning and ending stages of evaluation Intensively throughout entire process Helps with formative decision making through balanced control SAK

Orienting Question #5 cont. Conditions to use Empowerment Evaluation To increase the probability of program success Helps to assess the planning and implementation of the program Allows for self-evaluation of the program Used for mainstreaming evaluation as part of planning and management SAK

Orienting Question #5 cont. Conditions to use development evaluation Development activities Changing the environment SAK

Orienting Question #5 cont. Conditions to use deliberative democratic evaluation When there is a need for moderation need for dialogue among stakeholders power differentials among participants many stakeholders SAK

Orienting Question #6: How are participatory approaches used in practice? Used to: empower stakeholders Build evaluation capacity in the organization Increase organizational learning and data-based decision making Can be used at beginning and end, or throughout the process SAK

Orienting Question #7: What are the major strengths of participant-oriented approaches? Involving stakeholders has advantages: They have knowledge and perspectives that evaluators do not have: Policy makers know what programs they are considering, the budgetary and time constraints involved, and the factors (political or otherwise) that may affect their decisions Program managers and staff know the details of the program and some of the struggles and successes of the students or clients. They know what they have tried before and why it did not work. They know the logic behind the current program, why they adopted it, and what elements of it they think may work. They have expertise in working with these types of clients or problems. The clients know the details of their own lives, the problems with which they struggle, as well as the solutions they find, and how they respond to and think about the program or policy Evaluators’ communication with stakeholders increases the evaluators’ understanding of the concept and ability to develop more valid measures for evaluation Involving stakeholders helps them gain trust in the evaluation SAK

Orienting Question #8: What are the limitations of participant-oriented approaches? The feasibility, or manageability, of implementing a successful participative study Increased time and costs Difficulty in obtaining participation from the most disadvantaged stakeholders Few evaluators have extensive training or skills in facilitation, particularly with groups that may have political differences (Need to develop “people skills” ) SAK

Orienting Question #8 cont. The credibility of the results to those who do not participate in the process Other audiences wonder if stakeholders are able to be objective about their own programs (potential for bias). Evaluators can even become biased as they work with stakeholders. To avoid bias, many organizations hire external evaluators for important, controversial, or costly programs Competence of stakeholders to perform tasks that some approaches require Encouragement of organizations to improve their performance and evaluative ways of thinking is positive; however, Peter Dahler-Larson “bemoans the ‘expectation that lay persons will become good evaluators based on a short introduction and rudimentary training in methodology’” (p. 226). SAK