1. Evaluation Communities of Practice: Building capacity from the inside out Ben Silliman, Department of Youth, Family, & Community Sciences NC State University

2. Communities of Practice Informal networks (online) Focused on improvement, growth Characterized by… –Structured and spontaneous dialogue –Strong facilitation –Flexible levels of participation –Public and private spaces –Balance of routine and novelty –Flexible, interactive technology

3. Sample Communities of Practice NC Extension/Youth Development –Learning Circle 1 (2008-11) –Learning Circle 2 (2015- + E-Basics online training Multi-state Extension System –eXtension/webinar (2009-15) –Evaluation Champions (2013-15) –Youth Development Evaluation (2015-

4. NC 4-H Evaluation Learning Circles (E-LC) Short-term projects (2006-07) –Evaluation research and publication on Conferences, Public Speaking, Camps Long-term projects (2009-2012) –Evaluation of clubs, camp, science fair New Learning Circle (2015- –Monthly training webinars –Small group projects and teaching tools

5. NC 4-H E-LC1 Learning of E-LC Members (2009-2012) (based on Evaluation Skills Self Assessment) –Pre-test: most E-LC Unfamiliar/Minimal Familiarity (Self-ratings 1.15-1.44 for phases) –Post-test: > 50% Implement with Assistance on most items; 25-49% on some items

6. Results: Perceived Skill Improvement Cohort 1, Y1-2: ^p <.05 Cohort 1, Y1-3: +p <.05, * p <.01 (N = 7) Evaluation Skills (ESSA)Year 1Year 2Year 3 Planning (9)2.863.86 5.67* Focusing (18)6.3311.83^12.60* Design (21)8.1412.4212.84* Selecting Methods (27)7.1415.29^15.33* Collecting Data (18)4.868.86^9.00+ Analyzing/Interpreting (18)4.296.298.33+ Communicating Results (18)4.006.28^7.50*

7. Results: Perceived Skill Improvement Cohorts 1 & 2, Y1-2: +p <.05, *p <.01 (N = 15) Evaluation Skills (ESSA)Year 1Year 2 Planning (9)3.605.20+ Focusing (18)7.5012.07* Design (21)8.6712.67* Selecting Methods (27)10.0015.53* Collecting Data (18)5.538.00+ Analyzing and Interpreting (18)5.677.29 Communicating Results (18)4.676.53+

8. Evaluation Champions Study eXtension Evaluation CoP Extension ECB Framework –Professional Development (training, TA, collaborative projects, mentoring/coaching, CoPs) –Resources and Supports (expertise, materials, champions, assets, financing, technology, time) –Organizational Environment (leadership, demand, incentives, structures, policies/procedures) Taylor-Powell, E., & Boyd, H.H. (2008). Evaluation capacity- building in complex organizations. In M.T. Braverman, M.Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55-69.

9. Evaluation Champions Study eXtension Evaluation CoP Extension ECB Framework –Professional Development (training, TA, collaborative projects, mentoring/coaching, CoPs) –Resources and Supports (expertise, materials, champions, assets, financing, technology, time) –Organizational Environment (leadership, demand, incentives, structures, policies/procedures) Taylor-Powell, E., & Boyd, H.H. (2008). Evaluation capacity- building in complex organizations. In M.T. Braverman, M.Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55-69.

10. Evaluation Champions Study eXtension Evaluation CoP Evaluation Champions (2013-15) –4 states, purposive sample –10/state; all disciplines, levels, settings –30-40 min. phone interview Part 1 What do you do? What motivates you? (then and now) Part 2 What do you need? (esp. technology) How can your organization help?

11. Evaluation Champions Part 1 eXtension Evaluation CoP The reason I work with Extension is that I want to have an impact on my community and improve it, so measuring it and knowing that I have an impact is important to me. —county agent

12. Evaluation Champions Part 1 eXtension Evaluation CoP Champions’ Roles –Advocate: embedded, strategic: talking to peers, administrators –Practitioner: exemplar, innovator engaging clients, managing resourcefully, measuring precisely but pragmatically, using data creatively…esp. to learn –Mentor/Trainer: investor, expert capturing ‘teachable moments’ in project work, professional development, personal contacts

13. Evaluation Champions Part 1 eXtension Evaluation CoP Advocate I meet with other supervisors and remind them of the importance of evaluation and advocate for evaluation when we choose staff training priorities.—county director

14. Evaluation Champions Part 1 eXtension Evaluation CoP Practitioner: Thinking I tend to be a science person and think about how and why things work, so evaluation is a natural part of science and Extension. —county agent

15. Evaluation Champions Part 1 eXtension Evaluation CoP Practitioner: Managing I have partnered with faculty and their students in applied research fields at the university on several projects. —community development agent

16. Evaluation Champions Part 1 eXtension Evaluation CoP Practitioner: Measuring In 4-H livestock with novice learners, I evaluated what they knew before, during, and after…then tracked 3-10 years and measured. —county agent

17. Evaluation Champions Part 1 eXtension Evaluation CoP Practitioner: Using That [program] value is recognized in the individual stories, testimonies, quotes, and capturing themes from focus groups. —county agent

18. Evaluation Champions Part 1 eXtension Evaluation CoP Mentor I supervise and mentor staff and require them to do projects. We start with logic models and identify appropriate points in a program to conduct evaluation. --county director

19. Evaluation Champions Part 1 eXtension Evaluation CoP How Champions Roll –Personal commitment to field, people, CES –Few with formal, extensive training –More with personal, practical experience Also… –Early, intensive, relevant training, program/ evaluation success provides intrinsic rewards –Organization rules, positive feedback from stakeholders offers extrinsic support VS… –Inconsistent process, unused results

20. Evaluation Champions Part 1 eXtension Evaluation CoP Looking at evaluation from a broader perspective I could see that it was not just ‘bean counting,’ but could help me, my supervisor, and the system to understand stakeholder needs and Extension impacts better and decide what programs should be continued or modified.—county agent

21. Evaluation Champions Part 2 eXtension Evaluation CoP What Champions Want (and see as needs for everyone) –Support and Resources Basic  Advanced training Team support, innovation Tools for all phases of the evaluation cycle Rewards for advancement, innovation –Policies and Practices Value and reward

22. Evaluation Champions Part 2 eXtension Evaluation CoP What Champions Want … maybe a video that discusses a specific topic like how to analyze data in Excel and then how to effectively present the results… We need to do a better job in thinking about how to include evaluation before we begin a program. If there was a tool that helped us do that, it would be a great help.—county agent

23. Evaluation Champions Part 2 eXtension Evaluation CoP What Champions Want (and see as needs for everyone) –Technology Tools and equipment for doing evaluation Efficient/effective data and reporting systems Tools for learning evaluation, resource materials and people

24. Evaluation Champions Part 2 eXtension Evaluation CoP What Champions Want Last year I hired a videographer to capture the message in a way that stakeholders get excited about.—county agent

25. Evaluation Champions Part 2 eXtension Evaluation CoP What Organizations can do –Talk the Talk Clear, consistent emphasis –Walk the Walk Consistent training, implement, rewards –Get out of the way Remove barriers, inconsistencies Support advocacy, practice, training/mentoring (informal, formal)

26. Evaluation Champions Part 2 eXtension Evaluation CoP What Champions Can Do Maybe it’s a matter of changing perceptions that it’s everybody’s job and everybody can do it. —county agent

27. Literature Review: Extension Capacity-building Extension ECB Framework –Professional Development (training, TA, collaborative projects, mentoring/coaching, CoPs) –Resources and Supports (expertise, materials, champions, assets, financing, technology, time) –Organizational Environment (leadership, demand, incentives, structures, policies/procedures) Taylor-Powell, E., & Boyd, H.H. (2008). Evaluation capacity- building in complex organizations. In M.T. Braverman, M.Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55-69.