Evaluating R&D Programs: R&D Evaluation Planning Concept Mapping and Logic Modeling Presentation to the Washington Research Evaluation Network June 2008 William Trochim Cornell University wmt1@cornell.edu

Questions What new useful and applicable approaches and tools for ex-ante evaluations have been uncovered? Which agencies have used these tools? How can these tools be matched with different levels of decision making? Can systems-level analysis address gaps and assist in science and technology decision making at the federal level?

Overview of Evaluation Planning Process Concept Map Logic Model Evaluation Questions Measurement Development Evaluation Design  Implement Evaluation

Where Has Approach Been Used NIH/NCI Transdisciplinary Tobacco Use Research Centers (TTURCs) CDC, Prevention Research Centers (PRCs) NIH/NIAID/DMID, Research Centers of Excellence (RCEs) NIH/NIAID/DAIDS, AIDS Clinical Research Networks NIH/NCRR/CTSA, Weill Cornell Clinical and Translational Science Center

DMID Research Centers of Excellence (RCEs) PLANNING: Identified the focus prompt and participants “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...” IDEA GENERATION: Eliciting knowledge and opinion Brainstorming : 279 stakeholders were invited to participate. Participants generated 248 statements with 117 unique visitors hitting the site. 100 statements were derived from the Literature Review. The planning group edited statement list for relevance and representativeness. Planning Group approved final set of 94 ideas. STRUCTURING: Data collection to build the conceptual framework Sorting and Rating of Ideas 279 participants were invited to rate the 94 ideas on relative Importance and Expected Progress from 2003-present 74 people (27%) contributed their input on the ratings. 33 people invited to organize the 94 ideas into conceptually similar categories. 24 people (72%) participated in the conceptual sorting of the ideas. ANALYSIS: Series of multivariate analyses

The point map shows all the elements in relation to one another. value and reward original ideas and innovative technologies (4) “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”

Conceptually similar ideas are in close proximity value and reward original ideas and innovative technologies (4) demonstrate its flexibility in addressing emerging infectious disease research needs. (19) have the capacity to rapidly move into areas of growing importance. (56)

Conceptually different ideas are further apart define who the stakeholders are once product development moves into the pipeline (72) value and reward original ideas and innovative technologies (4) Foundation for understanding the relationship among and between points across the territory/ map create training that is relevant to and integrated with the goals of the strategic plans. (21) “In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”

The cluster map organizes ideas into groups 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 Translational Activities Research Mission Innovation and Flexibility Emergency Response Training and Recruitment Funds Management Management Integration and Synergy Capacity and Infrastructure Collaboration and Communication Each cluster is comprised of ideas related to that theme.

Each cluster retains the details: For example, Training and Recruitment promote and encourage opportunities for mentoring junior faculty/researchers in the field of biodefense and emerging infectious disease research. (7) encourage the participation of post-docs, students, and other non-PI scientists at each RCE's regional meeting. (8) recruit qualified students, trainees, women and minorities. (10) help new investigators obtain funding. (16) create training that is relevant to and integrated with the goals of the strategic plans. (21) develop more standardized training programs across RCEs by looking at successful programs at each RCE and developing some best practices. (26) develop an expanded cadre of new researchers, clinicians, and technical personnel who can help lead the national biodefense mission. (45) supplement RCE activities by applying for direct training supports through T, K and F awards. (55) establish training programs with pharmaceutical company partners to allow training of scientists in drug discovery and non-clinical development. (68) develop policies, criteria and processes for selecting career development candidates, including special efforts to recruit qualified women and minorities. (76) bring new people into the area of biodefense and emerging infectious disease research. (80)

DMID RCE Final Conceptual Framework Research Mission Innovation and Flexibility Translational Activities Collaboration and Communication Capacity and Infrastructure Emergency Response Integration and Synergy Training and Recruitment Management Funds Management …contains all the details and provides a conceptual framework.

Concept Map with Regions Training and Recruitment Integration and Synergy Funds Management Management Innovation and Flexibility Capacity and Infrastructure Collaboration and Communication Research Mission Translational Activities Emergency Response Outcomes Outputs Concept Map with Regions

Logic Model for RCE Program Funding Manage Funds flexibly and efficiently Establish Management structures & processes Innovative & Flexible response to changing needs and opportunities Synergy between Centers (Integration & Synergy) Capacity to serve as a regional resource Expanded cadre of investigators (Recruit and Train staff) Leveraged funds from elsewhere Readiness to Respond in an Emergency Better response to infectious diseases that threaten humans Translate and Apply Science to Practice Scientific Knowledge on Category A,B,C organisms (Research ) Leadership with expertise (Integration & Synergy) Biodefense Priorities Build Infrastructure Recruit & Train staff Inputs Outputs Outcomes Activities Impact Collaboration & Communication Conduct Research on Category A,B,C organisms Research results on Category Product Development (Translational Activities) Blue, italicized text relates to a final cluster on the cluster map.

TTURC Concept Map Health Impacts Outcome Communication Structure Recognition Collaboration Training Transdisciplinary Integration TD Research Institutionalization Health Outcomes Improved Interventions Translation to Practice Policy Implications Science & Models Methods Publications Communications Communication Structure Scientific Integration Professional Validation Process Collaboration

Scientific Integration TTURC Logic Model Intermediate Markers Communication Professional Validation External Recognition And Support Communication Internal Recognition And Support Translation To Practice Policy Implications Collaboration Health Impacts Immediate Markers Long-Term Outcomes Collaboration Training Transdisciplinary Integration Publications Methods Science & Models Scientific Integration Interventions Health Outcomes

CDC PRC Concept Map Engage the Community Diversity & Sensitivity Relationships & Recognition Active Dissemination Technical Assistance Training Research Methods Research Agenda Core Expertise & Resources The results were then subjected to analytical methods of multi-dimensional scaling and hierarchical cluster analysis that resulted in the ideas being sorted into similar groups or “concepts.” As shown in this slide, 9 concepts resulted for the national level map. With stakeholder input these concepts were labeled as shown.

From Map to Logic Model Inputs Outputs Outcomes Activities Core Expertise & Resources Engage the Community Diversity & Sensitivity Research Methods Relationships & Recognition IN THIS SLIDE you can see an example of how the pieces of the concept mapping was then abstracted into a logic model framework. Using a simple logic model framework of inputs-activities-outputs-outcomes, you can see that one piece of the concept map, “core expertise and resources”, was considered an input into the PRC program, so it informed the input section of the logic model. Each of these concepts or pieces could be abstracted to inform differ components of the logic model, which we will not show because of time. Training Research Agenda Technical Assistance Core Expertise & Resources Active Dissemination

Effective Interventions Draft logic model Inputs Outputs Outcomes Activities Community Advisory Board Engage the Community Translation of Research To Practice Widespread Use of Effective Interventions Conduct Core & Other Research Using Sound Research Methods PRC Capacity Core expertise & resources Diverse faculty/ staff Faculty/staff sensitive to community issues Facility Communication & data systems Research and Other Publications National, Regional or Local Health Priorities and Health Disparities Test Innovative Strategies Improved Community Health and Reductions in Health Disparities Widespread Knowledge of Effective Interventions Disseminate Research Findings Here you can see how that same piece of the concept map, “core expertise and resources” is in one of the input boxes of this draft logic model. The items in black come from other sources Establish Research Agenda Recognition Conduct Training Trust Relationships With Community Partners, Other PRCs, and CDC Skilled Workforce Provide Technical Assistance

Logic Model INPUTS PROGRAM ACTIVITIES OUTPUTS OUTCOMES Evaluation National, Local, or Regional Health Priorities And Health Disparities Engage the Community Conduct Core and Other Research Using Sound Research Methods Improved Community and PopulationHealth and Elimination of Health Disparities Programs/ Interventions Translation of Research to Practice and Policy PRC Community Committee PRC Capacity Human resources: core expertise, diversity, sensitivity Facility Communication & data systems Administrative capacity Evaluation expertise Research/ Evaluation Findings Communicated & Disseminated in: Publications Presentations Media Reports This slide shows the final logic model, which is also in your packet. You will see some of the labels from the concept map on the logic model – for example, “Engage the Community” and “Research Agenda”, two shapes on the concept map, appear as boxes in the Program Activities column of the logic model. “Core Expertise and Resources,” another shape on the concept map, appears on the logic model in the Input column. “Recognition,” also on the concept map, appears as an Outcome on the logic model, and so on. If anyone is interested in learning more about the Concept Mapping process and how concept maps have been used in this project, including how they have been helpful in developing the logic model for the PRC Program, please consider going to the APHA session giving on Monday at 10:30 am. Now, let me turn to the way we used the comments from the Workbook to finalize the logic model. On the slide, we’ve highlighted the changes that were made in the logic model after reviewing comments from the Workbook in RED. Because some comments were more relevant to a logic model that an individual PRC might develop for its own Center than to the logic model for the national PRC Program, we did not incorporate all Workbook suggestions in the final version of logic model. This logic model is a national framework under which each PRC would want to develop an individual Logic Model. Widespread Use of Effective Programs and Policies Establish Research Agenda Train/Mentor/Provide Technical Assistance to: Researchers Practitioners Students Community Members Enhanced Community Capacity for Prevention Relationships with State & Local Health Departments, Community Partners, University Partners, Other PRCs, and CDC Trainees and Technical Assistance Recipients Skilled Public Health Professionals Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust) Expanded Resources Recognition

AIDS Clinical Research Networks Map DAIDS Policies and Procedures Operations and Management Resource Utilization Community Involvement Communication, Collaboration, Harmonization Scientific Agenda-setting Biomedical objectives Relevance to Participants CSI (MK): Stable CM that’s reflective of input across areas of science and broad array of stakeholders. Celebrate. Then move on to how we use this information to develop evaluation system. Next step is development of LM. What is LM and its importanceNote colors 20

Community Involvement DAIDS Policies and Procedures Inputs Activities Outputs Short Term Outcomes Longer Term Outcomes/ Impact Recruitment and retention goals met (Community Involvement) Community Involvement Increased capacity of developing world sites (Resource Utilization) Community input (Community Involvement) Scientific Agenda Setting processes High quality scientific results and increased knowledge (Biomedical Objectives) Treatment and prevention measures (Biomedical Objectives) Integrated use of developing world sites (Resource Utilization) Collaboration and Communication (Communication, Collaboration, Harmonization) Appropriate human and infrastructure Resources are in place Scientific research plan and priorities (Scientific Agenda Setting) Results Published (Biomedical Objectives) Biomedical Objectives: HIV/AIDS mortality and morbidity reduced CSI (MK): colors and bold text correspond to the clusters. Operations and Management processes are in place Harmonized systems and procedures (Communication, Collaboration, Harmonization) DAIDS Policies and Procedures Scientific agenda is Relevant to Participants 21

Conclusions Participatory and Collaborative Integrative Mixed Methods Systems-Based Approach Comprehensive Conceptual Model Foundation for eEvaluation

Relevant Literature Trochim. W., Marcus, S.E., Mâsse, L.C., Moser, R.P., Weld, P. (2008). The Evaluation of Large Research Initiatives: A Participatory Integrative Mixed-Methods Approach, American Journal of Evaluation, 29, 1, 8-28. Anderson, L. A., Gwaltney, M. K., Sundra, D. L., Brownson, R. C., Kane, M., Cross, A. W., et al. (2006). Using concept mapping to develop a logic model for the prevention research centers program. Preventing Chronic Disease: Public Health Research, Practice and Policy, 3(1), 1-9. Trochim, W. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1-16. Trochim, W. and Kane, M (2005). Concept mapping: An introduction to structured conceptualization in health care. International Journal for Quality in Health Care, 17, 3, June 2005, 187-191. http://www.socialresearchmethods.net/research/cm.htm

Further Information William Trochim Concept Systems Incorporated wmt1@cornell.edu http://www.socialresearchmethods.net/ Concept Systems Incorporated http://www.conceptsystems.com/