1. Evaluating Collaboration National Extension Family Life Specialists Conference April 28, 2005 Ellen Taylor-Powell, Ph.D. Evaluation Specialist University of Wisconsin-Extension

2. Types of evaluation questions The outcomes are broad and complex. How do we get started? The outcomes are broad and complex. How do we get started? Is evaluating process good enough or do we have to evaluate outcomes? Is evaluating process good enough or do we have to evaluate outcomes? Who should be involved in evaluating a collaborative program? Who should be involved in evaluating a collaborative program? I’m not in charge. How do I evaluate it? I’m not in charge. How do I evaluate it? How do I take credit for something that we’ve done together? How do I take credit for something that we’ve done together?

3. Issues and challenges Power – control Power – control Process of the evaluation Process of the evaluation Data Data Standards and quality of the evaluation Standards and quality of the evaluation Cross-cultural issues Cross-cultural issues Measurements issues Measurements issues Attribution Attribution Taking credit Taking credit

4. Collaborative evaluation (not evaluation of collaboration) Since mid – 1970’s, new paradigm of participatory evaluation Since mid – 1970’s, new paradigm of participatory evaluation “applied social research that involves trained evaluation personnel…and practice-based decision makers working in partnership” ( Cousins and Earl, 1992) “applied social research that involves trained evaluation personnel…and practice-based decision makers working in partnership” ( Cousins and Earl, 1992) Multiple approaches -from broadening decision making (practical) to emancipation and social change (transformation) Multiple approaches -from broadening decision making (practical) to emancipation and social change (transformation) Emphasis on using data collection and feedback to strengthen and monitor collaboration and thus increase overall effectiveness and efficiency Emphasis on using data collection and feedback to strengthen and monitor collaboration and thus increase overall effectiveness and efficiency value in the process of evaluation, process use (Patton, 1997), as much as product value in the process of evaluation, process use (Patton, 1997), as much as product

5. Who controls? Who participates? How much? Researcher control Practitioner/participant control Primary users Deep participation All legitimate groups Consultation Adapted from Cousins and Whitmore, 1998

6. First… Who wants to know what? Who wants to know what? For what purpose? For what purpose? How will information be used? How will information be used?

7. OUTCOMESINPUTSOUTPUTS Collaborative Product Collaborative Relationship AssumptionsExternal factors Building a logic model of collaboration EVALUATION SITUATION Collaborative Effectiveness

8. Partners Funding Research- based Key stake holders Change in behaviors Value- added Clientele Users Policy makers Publics Policy changes Change in Knowldge Attitudes Skills Motivation Intent Self- efficacy Implement activities – action plan Monitor and evaluate Communicate Advocacy/ Policy Collaborative Relationship building Individual members Group Change in behaviors decision making Change in KAS Self- efficacy Intent Effective functioning partnership Member satisfaction Changes in conditions Collaboration: Theory of change WHAT DO YOU WANT TO KNOW? System changes Community changes Capacity building - TA

9. Evaluating the Collaborative Relationship 1. Process evaluation How is it functioning? How effective is the group work? Are we likely to achieve our desired results? How is it functioning? How effective is the group work? Are we likely to achieve our desired results? How satisfied are members? How satisfied are members? Questions about capacities, operations, climate, context Questions about capacities, operations, climate, context Factors influencing success Factors influencing success Projected tasks/activities relative to stages of development Projected tasks/activities relative to stages of development Milestones and Critical Events (journey) Milestones and Critical Events (journey)

10. MILESTONES Significant points along the way Significant points along the way Examples Examples Key stakeholders on board Key stakeholders on board Vision statement established Vision statement established Grant secured Grant secured Action plan formulated – plan of work Action plan formulated – plan of work Project implemented/service provided Project implemented/service provided Project evaluated Project evaluated CRITICAL EVENTS Unexpected events, positive and negative Progress markers Evidence of accomplishments Disruptions or obstacles Examples Change in membership Policy change New donor added

11. 2. Outcomes (Process outcomes): What difference has being a part of this group made for the individual? What difference has being a part of this group made for the individual? Knowledge, skills, motivations, behaviors, etc. Knowledge, skills, motivations, behaviors, etc. Human capital development Human capital development What difference is their for the group? What difference is their for the group? Group functioning, identify, resource pooling, etc Group functioning, identify, resource pooling, etc Note: Outcomes can be positive, negative or neutral

12. Methods Informal feedback Informal feedback Member (partner) Survey Member (partner) Survey Member (partner) interviews Member (partner) interviews Group discussions Group discussions Key informant interviews Key informant interviews Observation Observation Identification and use of indicators Identification and use of indicators Network analysis ; sociogram Network analysis ; sociogram Use existing materials (integrate into ongoing operations) Use existing materials (integrate into ongoing operations) Minutes of meetings Minutes of meetings Logs: telephone, event, registration forms Logs: telephone, event, registration forms Management charts Management charts WHEN? Periodic Review Points of particular concern

13. Tools - Techniques Community Group Member Survey Community Group Member Survey Collaborative Relationship scales Collaborative Relationship scales Internal collaborative functioning scales Internal collaborative functioning scales Plan Quality Index Plan Quality Index Meeting effectiveness inventories Meeting effectiveness inventories Stage of readiness Stage of readiness On-line Wilder Collaboration Factors Inventory (Amherst H. Wilder Foundation) On-line Wilder Collaboration Factors Inventory (Amherst H. Wilder Foundation) On-line Partnership self-assessment tool (Center for Advancement of Collaborative Strategies in Health) On-line Partnership self-assessment tool (Center for Advancement of Collaborative Strategies in Health)

14. Evaluating Programs/Products created/implemented by the collaboration 1. Process or implementation evaluation (Focus: program delivery vs. coordination or support role) (Focus: program delivery vs. coordination or support role) How is program being implemented? Fidelity to plan? Extent of delivery? Participation? What is/has happened that wasn’t planned? How is program being implemented? Fidelity to plan? Extent of delivery? Participation? What is/has happened that wasn’t planned?

15. Outcome evaluation Outcome evaluation What is different? For whom? How? To what extent? What is different? For whom? How? To what extent? For: Individuals, Groups/Families, Agencies, Systems, Communities For: Individuals, Groups/Families, Agencies, Systems, Communities Changes in … Changes in …

16. Individuals Attitudes, perceptions, knowledge, competence, skills, abilities, behaviors, actions, lifestyles Groups/families Interactions, behaviors, actions, values, culture Agency, organization #/type of services/programs delivered, access, practices, resource generation, resource use, policies Systems Relationships, interaction patterns, linkages, networks, practices, policies, resource use, institutionalization of changes Communities Values, attitudes, relations, support systems, civic action, social norms, policies, laws, practices, conditions Change in :

17. Tools - Techniques Monitor implementation Monitor implementation Logs, management charts, Logs, management charts, Interviews Interviews Observations Observations Achievement of outcomes Achievement of outcomes Clientele surveys Clientele surveys Clientele interviews Clientele interviews Observations Observations  Mixed Methods

18. Evaluating self - Taking credit Mutual (reciprocal) accountability Mutual (reciprocal) accountability How do I take credit for my part? How does Extension gain visibility, recognition? How do I take credit for my part? How does Extension gain visibility, recognition? What is your contribution? What role did you play? What value did you bring? What is your contribution? What role did you play? What value did you bring? Document role you play, your activities and contributions, inputs you bring, resources you make available, niche, value… Document role you play, your activities and contributions, inputs you bring, resources you make available, niche, value…

19. Your contribution Log of activities, roles played Log of activities, roles played Record inputs, resources contributed Record inputs, resources contributed Management chart; analysis of minutes Management chart; analysis of minutes Independent assessment Independent assessment Survey Survey Interviews Interviews

20. Your (partner) performance: Your (partner) performance: Most important indicator: other partners’ satisfaction with your performance (Brinkerhoff, 2002) Most important indicator: other partners’ satisfaction with your performance (Brinkerhoff, 2002) Mutual assessment among partners of each partner’s performance. Resulting discussion re. Discrepancies = powerful information sharing and trust building. Mutual assessment among partners of each partner’s performance. Resulting discussion re. Discrepancies = powerful information sharing and trust building. (We aren’t very good at this type of thing) (We aren’t very good at this type of thing)

21. Web address http://www.uwex.edu/ces/pdande http://www.uwex.edu/ces/pdande