Strengthening KM Measurement and Process Saori Ohkubo & Tara Sullivan Knowledge for Health (K4Health) Project GHKC Meeting October 22, 2015 So for the next 45 minutes or so, we will look at the issues surrounding the future of monitoring and evaluating knowledge management. While one of the focus areas will be the session title, “improving and measuring collaboration” we will be also covering other related issues, in particular, different components in building KM capacity and nurturing KM culture.

GHKC Achievement to Date Guide to Monitoring and Evaluating Knowledge Management in Global Health Programs KM for Global Health Logic Model GHKC M&E Task Team I want to start with recognizing the accomplishment that this group, GHKC, have made in terms of advancing M&E of KM for global health. KM M&E guide It includes an unique KM logic model, and 42 common KM indicators mapped to key elements of the model, as presented by Tara. Each indicator is described in detail, including its definition, data requirement, data source, purpose and issues, as well as illustrative examples where applicable In covers emerging themes such as social media, interactive usability assessment, web analytics, and community of practice, presenting proven or best practices, and making recommendations for appropriate metrics to consider. It describes a variety of data collection techniques and methods It includes useful and practical tools and sample instruments

Proposed Next Steps Reconvene M&E Task Team Work together on KM M&E tool(s) KM capacity assessment tool Continue our dialogue on emerging topics such as Using stronger research designs Measuring the “social” aspects of KM

Assessing KM Capacity and Culture K4Health Knowledge Management Capacity Assessment Tool in Development Center piece of the processes in the KM logic model is “strengthening of culture and capacity.”

Knowledge is not static nor permanent. Knowledge grows. Knowledge management needs to be adaptive and flexible to continuing changes within the individuals, organization, and surrounding environment. How to foster collaboration and become a learning organization?

K4Health KM CAT: Background It draws from and builds upon various resources (esp. APQC’s KM CAT and MSH’s PROGRES) It is a participatory self-assessment tool It aims to measure KM maturity and competency of an individual organization and/or a group of organizations It includes both quantitative and qualitative data collection approaches Its findings should be used to identify concrete action items to advance through the capacity continuum Resources: Capacity building/strengthening assessment tools in the field of social and behavior change communication and KM developed by CCP Organizational capacity assessment guidance and tools developed by other international development agencies (e.g., World Bank, PACT) KM capacity assessment guidance and tools developed by private sector companies (e.g., APQC) Literature review of KM maturity models Other relevant journal articles KM CAT is a participatory self-assessment tool to identify existing skills and assets as well as areas for improvement to manage KM It aims to measure KM maturity and competency of an individual organizations and/or a group of organization (e.g., professional networks, technical working groups) It includes both quantitative and qualitative data collection

Five Stages: Capacity Continuum Not present Not started Stage 2: In-development Limited On ad-hoc basis Stage 3: Standardized Formalized Not applied Stage 4: Applied by some staff/unit Stage 5: Applied completely Improved continuously KM function, procedure, activities, etc. are… Changes in KM capacity and culture happen in continuum, moving from one stage to the next, from lower to higher stages. Stage 3 is the most important milestone in an organization's journey toward KM maturity/competency because it denotes standardization. Standardizing and implementing KM approaches and processes will build the foundation for a knowledge-sharing culture. (Adapted from MSH’ PROGRES and APQC’s KM CAT)

Five Ps: Core Domains People covers 1) resources/human capital required for KM, 2) leadership, which is the organization’s senior management support to KM, and 3) organizational culture supporting knowledge sharing and networking. Process refers to 1) KM strategy that is aligned with the broader mission of the organization, 2) knowledge flow/cycle of assessing, capturing, generating, adapting, and sharing knowledge within organization, and 3) measurement such as M&E system and indicators. Platform includes the organization’s use of 1) KM systems, 2) KM approaches, and 3) information technologies. Partnership refers to the organization’s involvement in collaborating with key stakeholders and partners for 1) knowledge exchange, 2) coordination, and 3) networking purposes. Problem solving is a vehicle for connecting knowledge and performance and covers skills such as 1) knowledge seeking, 2) identification of new ideas and problems, and 3) critical thinking.

Resource (staff and training) Leadership (leaders/managers, coordinating body) Culture (support to networking and sharing) People covers 1) resources/human capital required for KM, 2) leadership, which is the organization’s senior management support to KM, and 3) organizational culture supporting knowledge sharing and networking.

Strategy (common vision, objectives, budgets) Knowledge flow (knowledge cycle elements) Measurement (indicators, systems, data use) Process refers to 1) KM strategy that is aligned with the broader mission of the organization, 2) knowledge flow/cycle of assessing, capturing, generating, adapting, and sharing knowledge within organization, and 3) measurement such as M&E system and indicators.

KM system (repository/sharing, content management) KM approach (tools and techniques) Information technology (IT team, use of IT) Platform includes the organization’s use of 1) KM systems, 2) KM approaches, and 3) information technologies.

External knowledge (gathering and promotion) Coordination (partnership and structure) Networks (participation to professional networking) Partnership refers to the organization’s involvement in collaborating with key stakeholders and partners for 1) knowledge exchange, 2) coordination, and 3) networking purposes.

Knowledge seeking (initiative, locating knowledge) Identification (of new ideas and problems) Critical thinking (representation and group solution) Problem solving is a vehicle for connecting knowledge and performance and covers skills such as 1) knowledge seeking, 2) identification of new ideas and problems, and 3) critical thinking.

Glance at KM CAT in Action K4Health KM CAT in Google Sheet Example of scoring sheet Example of reporting

Discussion Questions Thinking about your work environment/context, when, how, and with whom do you think you might use a tool like the KM CAT with? How does it support your work? How does it benefit your audiences? Do main domains (5Ps) resonate with you? Is there anything you would suggest changing, adding, or deleting?

Using Stronger Research Designs Measuring the “Social” Aspects of KM

Case Study for Research Design Bangladesh Knowledge Management Initiative (BKMI) eHealth pilot in 2 divisions (Sylhet and Chittagong) with eToolkit and eLearning videos for field workers to use on netbook computers when counseling mothers Baseline Survey April 2013 Field workers N= 304 Mothers with children < 2 years N = 674 Endline Survey September 2013 N = 675

Quasi-experimental Pre-Post Design Tier 1 Field workers Tier 2 Mothers with < 2 yrs child O X O April 2013 Sept 2013 O X O April 2013 Sept 2013

BKMI-II Research Design Cluster Randomized Control Trial with 3 arms: Randomization of districts into either Arm1, Arm 2 or Arm 3 Arm 1: eToolkit + IPCC training (2 districts) Arm 2: eToolkit (2 districts) Arm 3: Control (2 districts) Baseline Survey January 2015 All HAs (field workers) in 6 study districts Mothers with children < 2 years N = 800 per district Endline Survey January 2016 All HAs (field workers) in the 6 study districts

Malawi Net-map

2010 Baseline/Endline Net-Map Information Flows with District Level Stakeholders in Nkotakota

2010 Baseline/Endline Net-Map Information Flows with District Level Stakeholders in Salima

Social Knowledge Management Social benefit Social systems Social networks Social learning Social capital Social software/media

Resources Guide to Monitoring and Evaluating Knowledge Management in Global Health Programs https://www.k4health.org/sites/default/files/km-monitoring-and-eval-guide.pdf Leveraging Knowledge Management for Public Health http://www.ghspjournal.org/content/3/2/150.full.pdf+html Conceptualizing a New Knowledge Management Logic Model for Global Health: A Case-Study Approach http://www.popline.org/sites/default/files/384001.pdf Taking Knowledge for Health the Extra Mile: Participatory Evaluation of a Mobile Phone Intervention for Community Health Workers in Malawi http://www.ghspjournal.org/content/2/1/23.full.pdf+html The Intersection of Knowledge Management and Health Systems Strengthening: Implication from the Malawi Knowledge Management for Health Demonstration Project https://www.k4health.org/sites/default/files/KM%20HSS%20Technical%20Brief_%20October%2024%202012_final.pdf

Thank you and stay tuned for a follow-up message about the KM M&E TT via GHKC email