1. Systems Level Evaluation of Communities of Practice
AEA/CDC Summer Evaluation Institute
June 15 & 16, 2010
12/4/2018

2. Introductions Andrea M. Hegedus, Ph.D., M.P.A.
Northrop Grumman Corporation
Jan C. Jernigan, Ph.D.
Centers for Disease Control and Prevention
12/4/2018

3. Today’s Objectives
Define and describe communities of practice (CoPs) and the benefits they provide to individuals and organizations
Think through a conceptual approach to a systems level evaluation
Discuss evaluation questions and related activities
Describe how different evaluation methods can be applied
Many ways to do theory driven evaluation
Don’t have to define an explicit model
Learn to look at the system level approach
Ask questions based on evidence
Improve evaluation practice as well as evaluation science
Think outside the box
NEW WAY OF DOING BUSINESS
12/4/2018
12/4/2018

4. Think Outside The Box: Problems
Convert the following figure into a six by adding only one line. IX
2. Leave two squares in the following figure by removing just two lines.
12/4/2018
12/4/2018 3

5. Think Outside The Box: Solutions
Convert the following figure into a six by adding only one line.
SIX
2. Leave two squares in the following figure by removing just two lines.
12/4/2018
12/4/2018 4

6. Today’s Schedule Welcome and Introductions 9:25
Who, What, Where, Why for Communities of Practice 9:35
Create Your Own CoP (Group Activity) :55
Unanswered Questions and Constructs 10:15
Conceptualizing a Systems Approach to the Evaluation
Develop Your Own Questions (Group Activity) 10:30
Break :55
Methods to Evaluate CoPs :15
Apply Methods to Research Questions (Group Activity) 11:45
Feedback :35
Adjourn :45

7. Communities of Practice Overview
12/4/2018

8. A shared domain of interest (e.g., vocabulary standards)
Community of Practice (CoP) Definition
A group of people who share a concern, a set of problems, or a passion about a topic, and who deepen their knowledge and expertise by interacting on an ongoing basis (Wenger, 1998)
A CoP has three crucial characteristics:
Domain
A shared domain of interest (e.g., vocabulary standards)
Community
A community that enables interaction (discussions, collaborative activities, relationship-building)
CoP
Domain = Topical Area
Community = Who you are
Practice = What you do
Practice
A shared practice of experiences, stories, tools, ways of addressing recurring problems
12/4/2018 7

9. Why Communities of Practice?
More democratic - shared decision-making
No longer agency-centric
Includes a wide range of partners
All partners determine priorities and work
More equitably resourced - shared work and resources
Communities determine work and use of resources
Do not have to recreate the wheel
Economies of scale
Improved collaborations - shared information exchange
Share existing ideas
Create novel approaches to existing issues
Disseminate knowledge
Democratic – all partners have an equal say – large organizations like CDC, state health departments – NEW WAY OF DOING BUSINESS
Resources – help each other
Collaborations – cross traditional boundaries and means of social learning
12/4/2018 8

10. Examples of Successful CoP Models
Government:
NCI: CaBIG
CDC: PHIN CoPs
Federal Highway Administration
Private Sector:
Caterpillar
Hewlett-Packard
Shell
12/4/2018
12/4/2018 9

11. FHWA: Rumble Strips CoP
Problem: Run-off road crashes cause one-third of all traffic fatalities
Innovative Solution: Noise and vibration produced by shoulder rumble strips are effective alarms for drivers who drift off the roadway
Effectiveness in states that participated in the FHWA Rumble Strips Community that developed and implemented this safety feature:
State
% Crash Reduction
Pennsylvania
70%
New Jersey
34%
New York
72%
Massachusetts
42%
California
49%
12/4/2018
12/4/2018

12. Characteristics of CoPs (that make them so interesting)
Members
Organizations
Public Health
Continual learning & professional development
Reduced time/cost to retrieve information and reduced learning curves
Consistent communication & reporting
Access to expertise
Knowledge sharing & distribution
Improved analytic capability
Communication with peers improved
Coordination, standards, & synergies across organizational units
Promotion of standards
Increased productivity & quality of work
Reduced rework & reinvention
Support and promotion of key national initiatives
Network for keeping current in field
Innovation
Advancement of domain- specific capabilities
Sense of professional identity
Benchmarking against & influencing industry standards
Link geographically dispersed practitioners
Enhanced professional reputation
Alliance building
Increased government efficiency
Wenger, E. (1998). Communities of Practice: Learning, Meaning, and Identity. New York: Cambridge University Press.

13. CoPs Are Different Than…..
What’s the purpose?
Who belongs?
What holds it together?
How long does it last?
Community of Practice
To develop members’ capabilities; to build and exchange knowledge
Members who select themselves
Passion, commitment, and identification with the group’s expertise
As long as there is an interest in maintaining the group
Formal work group
To deliver a product or service
Everyone who reports to the group’s manager
Job requirements and common goals
Until the next reorganization
Project team
Employees assigned to the group’s manager
The project’s milestones and goals
Until the project has been completed
Informal network
To collect and pass on business information
Friends and business acquaintances
Mutual needs
As long as people have a reason to connect
Wenger & Snyder. (2000). Communities of Practice: The Organizational Frontier. Harvard Business Review, 78, 139–145.

14. RESOURCE: PHIN CoPs
Visit to access the CoP Resource Kit and lots more about CoPs!
Plan a community
Launch a community
Sustain and Evolve a community
(Evaluate) a community
12/4/2018
12/4/2018 13 13

15. Group Exercise 1 Create a CoP
12/4/2018

16. Evaluation of Communities of Practice
12/4/2018

17. Steps Taken to Develop the Evaluation Framework
Work with team members/stakeholders to develop goals of the evaluation and generate evaluation questions
Agree on outcome/indicators that address evaluation questions
Increased collaboration among partners
Improved use of expertise and resources
Creation of novel ideas, tools, and products related to CoP’s goals and objectives
Dissemination of innovations
Improved use of program tools and processes
Decide on approaches that unify these different goals/questions/ outcomes
Goals: more than just evaluating CoPs – what are stakeholders/partners are interested in seeing in terms of outcomes
Primary users
Primary uses
Add this information to summary of the literature to start thinking about how to approach the evaluation
12/4/2018

18. Communities as a System
CoPs can be viewed as a system:
How do individuals change as they participate in a CoP?
How do CoPs evolve?
What impact do CoPs have on the larger program?
Refer to socio-ecologic model
Looking at only one level really is an incomplete evaluation.
12/4/2018

19. What is a Systems Approach to Evaluation?
Paradigm or perspective that considers:
connections among different components,
plans for the implications of their interaction,
requires transdisciplinary thinking, as well as
active engagement of those who have a stake in the outcome to govern the course of change
(Leischow and Milstein, 2006)

20. System Level Evaluation Framework
Goal: Evaluate how individuals change over time as they participate in CoPs
Individual Level
Goal: Evaluate how CoPs change over time to accomplish their goals
Organizational Level
Group activity 2 after this slide
Goal: Evaluate how CoPs influence the larger program
Programmatic Level
12/4/2018 19

21. How Will We Evaluate CoPs?
How do you approach a systems level evaluation?
Experience-based – you or someone else you know has done similar work
Evidence-tested – examine already available evidence
Theory-based – use valid constructs from social and behavioral science theories

22. Discussion What are the evaluation questions?
What approach best addresses your questions?
Experience-based
Practice-tested
Theory-based

23. Questions (Koliba & Gajda, 2009)
Knowledge/Theory Testing
What are the primary characteristics of CoPs?
CoP Development
What affect does CoP participation have on an individual?
What conditions foster individual participation in CoPs?
What is the connection between administrative leadership and the wider environment, and CoP development?
How are relationships constructed between and among CoPs?
Merit/Worth of CoPs
How do we assess the quality of CoPs?
What is the relationship between CoPs and organizational effectiveness?
How can formal evaluation and applied research be used to align CoPs and organizational goals and outcomes
This information comes from a summary of the literature.
We will refer back to these questions in as you build your models.
12/4/2018

24. Define the System and Develop
Group Exercise 2
Define the System and Develop
Evaluation Questions
12/4/2018

25. Use of Theory in Evaluation
Implicit Use – assumptions that underlie the program
Explicit Use – program activities are linked to theories in which there is usually a body of evidence to support their inclusion
AKA – espoused theory or theory in use
12/4/2018

26. What Exactly is a Theory of Change Approach?
A theory of change approach to evaluation has been defined as:
a theory of how and why an initiative works (Weiss, 1995)
a systematic and cumulative study of the links between activities, outcomes, and contexts of the initiative (Connell & Kubisch, 1998).
In a theory of change approach, compare the desired or expected outcomes, to what actually happens.
Using a theory of change approach allows:
evaluation of the process of how CoPs develop over time
a better understanding of the context of change
improved prediction of outcomes
ToCIt also reduces problems associated with causal attribution of impact. By specifying up front, how activities will lead to interim and longer-term outcomes and identifying the contextual conditions that may affect them, the scientific case for attributing subsequent change in these outcomes (from initial levels) is improved significantly.
Fits well with systems thinking/evaluation
Goes beyond more traditional approaches (formative, summative, process) to combine their activities
Comprehensive approach to thinking about CoPs
12/4/2018

27. Next Steps
Select appropriate theories and identify relevant constructs
Map theories/constructs to questions to chosen methods (e.g., survey or interview questions, focus groups, etc.)
Develop measures and collect data
12/4/2018

28. Using Relevant Theories
Collaborate with a social or behavioral scientist with a broad background in theoretical approaches
Review theories at different levels of analysis in light of unanswered questions that are of interest to the community
Select theoretical constructs from theory
Map theory and constructs to questions

29. Social Capital Theory (Individual Level) - 1
Adapted from many sources: specifically Lesser & Storck (2001) – Communities of practice and organizational performance, IBM Systems Journal,
Social Capital is the “glue that holds a community together”
Three relevant constructs:
Structural – connections made to others in the organization
Relational – interpersonal relationships that reinforce those connections
Cognitive – shared context among partners
Example Evaluation Question:
As CoP members build social capital there will be an increase in their ability to solve common problems and share resources
12/4/2018

30. Social Capital Theory (Individual Level) - 2
Example Data Collection Questions:
Structural - What mechanisms do you use to get information about ….? (list types)
Relational - Currently, do you identify yourself as being part of the larger community? (Y/N)
Cognitive - Currently, how much do your interactions around your community improve your knowledge base? (scale of 1-5)
Expected Outcomes:
Increased buy-in of community
Broader connections among partners with similar issues
Improved sharing of relevant knowledge, resources, and best practices
12/4/2018

31. Moving Theory to Outcomes - 1
According to Lipsey (1993) the use of a theory-driven approach, in settings where complex causal processes cannot be construed with empirical experimentation, can improve the strength of assessing the effectiveness of causal outcomes.
Theory plays a role as a:
basis for planning prior to conducting the evaluation,
means for organization and interpretation of results, and
target for revision or rejection as outcomes become evident
Use data from all sources to triangulate on results
Compare predicted outcomes to actual outcomes
Don’t forget about cross-theory synergy within the system

32. Moving Theory to Outcomes - Example
Using social network analysis:
Evaluation Questions: As the community evolves, do members’ social networks expand?
Methods: Survey Questions (Quantitative):
What are the most important issues that impact your work?
Whom do you go to for help for each of these issues?
Triangulation: Use focus groups or interviews (Qualitative) to discuss network linkages based on responses of survey questions
Predicted Outcomes: Members’ social networks will expand across disciplines
Compare: Actual outcomes to predicted outcomes and track evolution of network over time

33. Evaluation Framework Organizational Level Programmatic Level
Expected Outcomes: Increased social capital, social learning, sharing of information, knowledge base
Relevant Theories: Social capital, social network, social cognitive
Individual Level
Relevant Theories: Organizational learning, social network theory, action theory
Expected Outcomes: Increased number of relationships, CoP processes and evolution, CoPs that meet their goals
Organizational Level
Expected Outcomes: Improvements in function, structure and operation; increased
collaboration; creation and dissemination of best practices
Relevant Theories: Diffusion of innovations, organizational change, systems theory
Programmatic Level

34. Evaluation: CoP Outcomes
Individual Outcomes
CoP Outcomes
Increased interactions with others
More information channels
Increased learning
Improved social networks
Shared resources
More collaborations
Expanded social networks
Improved co-learning
More productive partnerships
Increased use of promising practices
Achieved goals
Long-term
Outcomes
Increased buy-in and capacity of partners
Increased participation of partners
Greater creation, adoption, dissemination of standards and best practices
Program Outcomes
Expanded social networks
Improved structure and operations
Improved functional outcomes
Improved adoption and dissemination of products
12/4/2018

35. What Methods Will We Use to Evaluate CoPs?
Mixed methods approach
Uses both quantitative and qualitative data
Multiple Data Sources
Questionnaires completed on a regular basis by CoP members
Focus groups/Interviews
Quantitative data: Interim progress reports, relevant program data, CoP meeting logs, social software statistics, etc.
Time phased measurement and analysis
Looks at change over time
12/4/2018

36. Develop Evaluation Plan
Group Exercise 3
Develop Evaluation Plan
12/4/2018

37. Questions?
Comments!
12/4/2018