1. MICHAEL A. HARNAR DOCTORAL CANDIDATE, CLAREMONT GRADUATE UNIVERSITY AMERICAN EVALUATION ASSOCIATION ANNUAL CONFERENCE NOV 5, 2011; SESSION 842 ANAHEIM, CA Theory Building Through Praxis Discourse: A Theory- And Practice-Informed Model Of A Transformative Participatory Approach To Evaluation

2. Today’s Agenda Discipline Building/Background The Method  Survey  Modeling Practice  Webinars Preliminary Findings  Key T-PE Variables  T-PE Questions Next Steps

3. Evaluation Discipline Development Practitioner Surveys Practice Reflections Theoretical Writings Categorization Efforts Practice Studies

4. Subjects of Inquiry in RoE* ConceptSubcategoriesExample ContextThe circumstances within which evaluation occurs Societal Organizational Evaluation-specific Evaluator-specific Segerholm (2003) ActivitiesThe procedures used in planning, carrying out, and disseminating evaluation Approaches Components (within an evaluation model) Practices Christie (2003) ConsequencesChanges that do (or do not) occur as a result of evaluation Use/Influence Participants Evaluation context, context, activities, activities, and professional issues Henry (2004) Professional Issues Issues involving the structure, norms, and continuation of the field of evaluation Training Standards Etc. Jones & Worthen (1999) *Adapted from Mark, M. (2008). Building a better evidence base for evaluation theory. In Fundamental Issues in Evaluation, N. L. Smith and P. R. Brandon (Eds.) (p. 117).

5. Cousins and Whitmore (1998)

6. P-PE vs. T-PE Practical Participatory Evaluation (P-PE)  Involves key stakeholders to  Increase evaluation’s usefulness,  Increase its use  Grounded data & findings  Organizational decision-making Transformative Participatory Evaluation (T-PE)  Involves diverse stakeholders to  Grounded data & findings  Empower participants  Increase social action

7. Research Questions 1. What are the key variables in a T-PE prescriptive theory of action? 2. What are the key variables in a descriptive theory of T-PE practice? a) How can T-PE evaluators be identified so that they can select the key variables in T-PE practice? b) Which of a set of theoretically prescribed variables do these T-PE evaluators identify as key to their preferred practice? 3. How do T-PE evaluators model their preferred practice with a set of theoretically prescribed variables? a) How is this descriptive theory of T-PE practice different from other participatory forms of evaluation? 4. How does a descriptive T-PE model reflect practice?

8. Modeling Evaluation Theories LeBaron Wallace, Hansen, & Alkin (2008)

9. Four-Phase Mixed Method Study Phase 1  Develop list of important T-PE variables  Develop questions to identify T-PE evaluators Phase 2  Survey AEA membership to identify T-PE evaluators  Rank importance of variables Phase 3  Model building – practice mapping Phase 4  Webinars to gain practical insight into endorsed model Mixed Method  Purposes: Development & Convergence  Design: Component & Integration

10. Variables And Question Development Cousins, Whitmore, Mertens  Wiki-based dialogue  Produced 30 variables  Produced 8 questions

11. Questions Depth of Stakeholder Involvement 1. I always try to involve stakeholders in my evaluations. 2. I prefer not to take on an evaluation unless it has a strong participatory component. Selection of Stakeholders 3. Program beneficiaries should participate in carrying out evaluation. 4. People representing all important perspectives should be involved in any evaluation. 5. Evaluators should help train all legitimate groups to do evaluation.

12. Questions Control of the Evaluation 6. Evaluators should share technical decision-making with stakeholders. Philosophical Reasons for Stakeholder Involvement 7. Evaluation should focus on bringing about individual empowerment, emancipation and self-determination. 8. Evaluation should focus on bringing about social justice.

13. Instructions In answering these questions, please think about how you prefer to practice evaluation. I know that answers to these questions are almost always context dependent, and "it depends" might be your answer choice. But, I would like you to think of your ideal evaluation situation. The term "stakeholder" is used here to mean anyone, other than the evaluator, with a vested interest in the entity (evaluand) being evaluated. "Participants" are those stakeholders who take an active role in the evaluation. "Participation" is any active role and may vary widely in breadth and depth.

14. Questions Strongly Disagree Slightly Disagree Somewhat Agree Strongly Agree ΟΟΟΟ

15. Survey AEA membership; N=6,614 May 23-June 18 1 invitation, 2 reminders 1,228 completed survey 323 responded positively to 8 T-PE questions  Invited to model their practice  142 completed a model 238 also invited  98 completed a model

16. Modeling Filter Questions I always try to involve stakeholders in my evaluations. I prefer not to take on an evaluation unless it has a strong participatory component. I prefer to involve stakeholders in every possible stage of the evaluation.

17. Principles Importance Ranking (n=142) 1 Multiple Method Perspective Evaluator applies multiple methods as appropriate to the evaluation context. 2 Community Trust Evaluator works to build trust by developing working relationships with a broad range of stakeholders. 3 Diverse Perspectives Evaluator ensures representation of diverse perspectives by including concerns, values, and interests of stakeholders. 4 Engage Intended Beneficiaries Evaluator engages intended program beneficiaries in meaningful participation. 5 Negotiable Purpose The purpose of the evaluation is negotiated with stakeholders. 6 Engage Marginalized Stakeholders Evaluator engages marginalized program stakeholders (e.g., those who might otherwise lack representation) in meaningful participation. 7 Negotiable Participation Scope of stakeholder participation in evaluation is not decided ahead of time. Barriers to and supports necessary for participation are identified and negotiated. 8 Negotiable Decision Making Technical decision-making (e.g., survey instrument selection, statistical analyses, data presentation) is negotiated with stakeholders. 9 Community-Sensitive Sampling Sampling procedures account for community diversity.

18. Activities Importance Ranking (n=142) 1 Develop Questions Evaluator collaborates with stakeholders in defining evaluation purpose and evaluation questions. 2 Build Capacity Evaluator trains stakeholders in the necessary skills to participate in the evaluation. 3 EducateEvaluator educates stakeholders on the value of evaluation. 4 Use Local Program Knowledge Evaluation decisions are made using local program knowledge. 5 Collect & Analyze Data Evaluator collaborates with stakeholders in data collection and analysis. 6 Develop Judgments & Recommendations Evaluator collaborates with stakeholders in interpreting findings, and formulating judgments and recommendations from the data. 7 Report & Disseminate Evaluator collaborates with stakeholders in reporting and disseminating the findings. 8 Share Control Evaluator negotiates the giving of control of the evaluation to program stakeholders.

19. Outcomes Importance Ranking (n=142) 1 Shared Understanding All participants develop shared understanding of program functions and processes. 2 Credible FindingsParticipants see evaluation findings as credible. 3 Increase Systematic Inquiry Increase capacity for participants to engage in and use systematic inquiry. 4 Outcomes Change Program outcome expectations change as a result of the process. 5 LearningAll participants learn new skills. 6 Increase Self- Determination Increase individual self-determination, emancipation and empowerment. 7 Increase Self-CritiqueIncrease participants’ capacity for self-critique. 8 Increase Social JusticeEnhance social justice. 9 Increase Social ActionIncrease social action.

20. Modeling Participants Participatory evaluators  Based on 3 questions 240 completed models 142 T-PE evaluators  Based on 8 questions 98 other PE evaluators

21. Model Building Process

22. Model Building Variable around which model is built: Stakeholder Involvement Question: “How do you ensure stakeholder involvement and what outcomes do you intend to create?”

23. Model Building Process

24. Modeling Data Range of links 2-59 Mean links 23.4 Median links 21 Mode 18 Skew 0.7 Kurtosis.28

25. Modeling Data Range of links 2-59 Mean links 23.4 Median links 21 Mode 18 Skew 0.7 Kurtosis.28

26. Develop Judgments & Recommendations Report & Disseminate Shared Understanding Collect & Analyze Data Community Trust T-PE Model (top 21 links; n=142 participants) (25-37 endorsers per link, or 18%-26% of 142 participants) Credible Findings Increase Systematic Inquiry Multiple Method Perspective Build Capacity Use Local Program Knowledge Engage Intended Beneficiaries Educate Diverse Perspectives Develop Questions Increase Self-Critique Stakeholder Involvement Lowest Endorsed Middle Endorsed Highest Endorsed Michael A. Harnar, Claremont Graduate University, 2011

27. Variable Practice Ranking (n=142) 1 Community Trust 14 Multiple Methods Perspective 2 Build Capacity 15 Share Control 3 Shared Understanding 16 Increase Systematic Inquiry 4 Develop Questions 17 Engage Marginalized Stakeholders 5 Collect & Analyze Data 18 Negotiable Purpose 6 Credible Findings 19 Outcomes Change 7 Develop Judgments & Recommendations 20 Negotiable Participation 8 Engage Intended Beneficiaries 21 Negotiable Decision Making 9 Report & Disseminate 22 Increase Social Action 10 Diverse Perspectives 23 Increase Self-Critique 11 Learning 24 Increase Self-Determination 12 Use Local Program Knowledge 25 Increase Social Justice 13 Educate 26 Community-Sensitive Sampling

28. Webinars Three (3) 1-hr webinars (n=10) Guided conversation  How is your practice reflected in this model?  Explain some critical variables. (e.g., community trust; increased self-critique) Preliminary Themes:  Model rarely matches practice.  Social Action, though not highly endorsed, underlies choices.  Because involving stakeholders is often inconsistent, those embracing T-PE value involvement at every possible stage.  Highest endorsed “causal links” look much like any PE.

29. Preliminary Conclusions Mixed method converges understanding  Different rankings provide greater understanding Participant reflections  Modeling  Webinar Publish findings as a visual framework to discuss practice Provide more accessible research Provide a preliminary list of key variables for PE & T- PE

30. MICHAEL.HARNAR@CGU.EDU Questions?