1. How do we know that two heads are better
than one? Evaluating Collaborative Networks
American Evaluation Association Meeting
October 27, 2016
Lisa LeRoy, MBA, Ph.D.
Learning collaboratives (LC), learning networks (LN), and research networks (RN) have proliferated in recent years as important methods to improve the quality of health and community services, to expand and share knowledge, and to change practice in public health and healthcare settings. While these methods have many similarities, they also have distinct purposes and participants. How do we differentiate these approaches in health care? What does each method offer for advancing knowledge and improving practice?

2. Overview What do we mean by Collaborative Networks?
What evaluation frameworks are currently being used to evaluate networks?
What are possible future directions?

3. What do we mean by Collaborative Networks (in health care)?
Terminology is broad
Learning Collaboratives
Learning Networks
Research Networks
Community of Practice
Peer-to-Peer Networks
Team Science
Collective Impact
There are definitions for each of these which I didn’t include

4. Definition of collaborative network
“Three or more organizations working together for a common purpose” -IBM Center for the Business of Government

5. Why create a collaborative network?
Enhance performance by sharing best practices
Lessons learned (qualitative)
Benchmarking (quantitative)
Solve complex problems
Accelerate research to practice
Bring more resources to bear on a problem or project
Recruit study subjects from multiple sites
Work across/break down “siloes” or disciplines
The benefits of collaborative networks are clear: they stimulate creativity and the identification of innovative approaches to solve complex problems; they align organizational objectives and activities to achieve efficient and high quality results; they enhance sharing of individual and collective assets (e.g., lessons learned, tools, funding); and they foster trust, teamwork, reciprocity, and mutuality (Camarinha-Matos and Afsarmanesh, 2006; Sorgenfrei and Smolnik, 2014).

6. Evaluation of collaboration is nascent
Search
Terms
PubMed 1
evaluate1
“collaborative network” 36 2
evaluate
“research network”
1,085 3
“learning network” 18 4
“quality improvement collaborative” 50 5
“community of practice” 76
1 “Evalu*” was used to capture all forms.

7. Evaluation of collaboration is nascent
Compared to what? Comparison groups of non- collaborating individuals or organizations may not exist
What is the level of analysis? Individual, organization, network
What is expected from collaboration? Vague or poorly defined outcomes
What did the collaboration produce? Outcomes that are difficult to measure or change over time
How long is collaboration supported? Outcomes that may take a long time to emerge (innovation, collective impact, joint products)
Collaboration is time and resource intensive
Often facilitation is needed to foster collaboration

8. Evaluation Frameworks
Few evaluation frameworks because the process and outcome desired from collaboration is not explicit
Collective Impact model and evaluation framework collective-impact
QI evaluation focuses on the results of the changes to service delivery
Research networks have focused on collaborative publications
Paper by Hall et al compared a transdisciplinary research collaborative with individual grants and found that initially the individual grants published more but over time the collaborative superceded the individual pubs and had more co-authors on the papers and similar impact factors

9. Example Logic Model for a Collaborative Network
Outcomes
Network Level
Organization or Team
Individual Member
Other
(e.g. Community)
Inputs
Establish Preliminary Vision
Recruit Members
Establish Governance Structure
Create Culture of Share Leadership
Relationships & Report Facilitate
Etc.
Network Activities
In Person Meeting
Affinity Groups
Etc.
Collaborative Products

10. Collaborative Network: Evaluation Tools
Surveys of participants
Perceptions, satisfaction, readiness to collaborate
Collaboration Scales
Wilder Collaboration Factors Inventory
Levels of Collaboration Scale
Provider Collaboration Scale
Organizational Trust Scale
Social network analysis (individual and organizational)
Some of the collaboration scales have been validated and others have not. Broadleaf consulting group has pulled together a sample of tools online

11. Collaborative Network: Evaluation Tools
Case studies (qualitative and mixed methods)
Process measures
Participation rates in group teleconferences, collaborative web portals, action groups
Stability/strength of infrastructure, e.g. governance structures, membership, sustainability after initial funding
Bibliometrics (co-authorships, co-inventors, collaborations, references, citations and co-citations)
Paper by Hall et al compared a transdisciplinary research collaborative with individual grants and found that initially the individual grants published more but over time the collaborative superceded the individual pubs and had more co-authors on the papers and similar impact factors

12. Evaluating collaborative networks
There are many gaps in our knowledge that network evaluation could begin to fill
Two network evaluation case studies:
Baby Friendly – a health services improvement network
Asthma Evidence to Action Network (E2AN) – a research network

13. Contact Information
Lisa LeRoy, MBA, PhD

14. Future directions in evaluating collaborative networks
Build the evidence base for benefits/contributions of collaborative networks Develop explicit conceptual models of change for collaborative networks Explore potential comparison groups Explore what kinds of networks are most effective for specific outcomes Examine the types of collaboration that are most productive in each network