How do we know that two heads are better than one? Evaluating Collaborative Networks American Evaluation Association Meeting October 27, 2016 Lisa LeRoy, MBA, Ph.D. Learning collaboratives (LC), learning networks (LN), and research networks (RN) have proliferated in recent years as important methods to improve the quality of health and community services, to expand and share knowledge, and to change practice in public health and healthcare settings. While these methods have many similarities, they also have distinct purposes and participants. How do we differentiate these approaches in health care? What does each method offer for advancing knowledge and improving practice?

Overview What do we mean by Collaborative Networks? What evaluation frameworks are currently being used to evaluate networks? What are possible future directions?

What do we mean by Collaborative Networks (in health care)? Terminology is broad Learning Collaboratives Learning Networks Research Networks Community of Practice Peer-to-Peer Networks Team Science Collective Impact There are definitions for each of these which I didn’t include

Definition of collaborative network “Three or more organizations working together for a common purpose” -IBM Center for the Business of Government

Why create a collaborative network? Enhance performance by sharing best practices Lessons learned (qualitative) Benchmarking (quantitative) Solve complex problems Accelerate research to practice Bring more resources to bear on a problem or project Recruit study subjects from multiple sites Work across/break down “siloes” or disciplines The benefits of collaborative networks are clear: they stimulate creativity and the identification of innovative approaches to solve complex problems; they align organizational objectives and activities to achieve efficient and high quality results; they enhance sharing of individual and collective assets (e.g., lessons learned, tools, funding); and they foster trust, teamwork, reciprocity, and mutuality (Camarinha-Matos and Afsarmanesh, 2006; Sorgenfrei and Smolnik, 2014).

Evaluation of collaboration is nascent Search Terms   PubMed 1 evaluate1 “collaborative network” 36 2 evaluate “research network” 1,085 3 “learning network” 18 4 “quality improvement collaborative” 50 5 “community of practice” 76 1 “Evalu*” was used to capture all forms.

Evaluation of collaboration is nascent Compared to what? Comparison groups of non- collaborating individuals or organizations may not exist What is the level of analysis? Individual, organization, network What is expected from collaboration? Vague or poorly defined outcomes What did the collaboration produce? Outcomes that are difficult to measure or change over time How long is collaboration supported? Outcomes that may take a long time to emerge (innovation, collective impact, joint products) Collaboration is time and resource intensive Often facilitation is needed to foster collaboration

Evaluation Frameworks Few evaluation frameworks because the process and outcome desired from collaboration is not explicit Collective Impact model and evaluation framework http://www.fsg.org/publications/guide-evaluating- collective-impact QI evaluation focuses on the results of the changes to service delivery Research networks have focused on collaborative publications Paper by Hall et al compared a transdisciplinary research collaborative with individual grants and found that initially the individual grants published more but over time the collaborative superceded the individual pubs and had more co-authors on the papers and similar impact factors

Example Logic Model for a Collaborative Network Outcomes Network Level Organization or Team Individual Member Other (e.g. Community) Inputs Establish Preliminary Vision Recruit Members Establish Governance Structure Create Culture of Share Leadership Relationships & Report Facilitate Etc. Network Activities In Person Meeting Affinity Groups Etc. Collaborative Products

Collaborative Network: Evaluation Tools Surveys of participants Perceptions, satisfaction, readiness to collaborate Collaboration Scales Wilder Collaboration Factors Inventory Levels of Collaboration Scale Provider Collaboration Scale Organizational Trust Scale Social network analysis (individual and organizational) Some of the collaboration scales have been validated and others have not. Broadleaf consulting group has pulled together a sample of tools online http://broadleafconsulting.ca/uploads/3/4/0/8/3408103/tools_for_measuring_collaboration.pdf

Collaborative Network: Evaluation Tools Case studies (qualitative and mixed methods) Process measures Participation rates in group teleconferences, collaborative web portals, action groups Stability/strength of infrastructure, e.g. governance structures, membership, sustainability after initial funding Bibliometrics (co-authorships, co-inventors, collaborations, references, citations and co-citations) Paper by Hall et al compared a transdisciplinary research collaborative with individual grants and found that initially the individual grants published more but over time the collaborative superceded the individual pubs and had more co-authors on the papers and similar impact factors

Evaluating collaborative networks There are many gaps in our knowledge that network evaluation could begin to fill Two network evaluation case studies: Baby Friendly – a health services improvement network Asthma Evidence to Action Network (E2AN) – a research network

Contact Information Lisa LeRoy, MBA, PhD1 Lisa_LeRoy@abtassoc.com 617-349-2723

Future directions in evaluating collaborative networks Build the evidence base for benefits/contributions of collaborative networks Develop explicit conceptual models of change for collaborative networks Explore potential comparison groups Explore what kinds of networks are most effective for specific outcomes Examine the types of collaboration that are most productive in each network