1. Evaluating R&D Programs: R&D Evaluation Planning Concept Mapping and Logic Modeling
Presentation to the
Washington Research Evaluation Network
June 2008
William Trochim
Cornell University

2. Questions
What new useful and applicable approaches and tools for ex-ante evaluations have been uncovered?
Which agencies have used these tools?
How can these tools be matched with different levels of decision making?
Can systems-level analysis address gaps and assist in science and technology decision making at the federal level?

3. Overview of Evaluation Planning Process
Concept Map
Logic Model
Evaluation Questions
Measurement Development
Evaluation Design
 Implement Evaluation

4. Where Has Approach Been Used
NIH/NCI Transdisciplinary Tobacco Use Research Centers (TTURCs)
CDC, Prevention Research Centers (PRCs)
NIH/NIAID/DMID, Research Centers of Excellence (RCEs)
NIH/NIAID/DAIDS, AIDS Clinical Research Networks
NIH/NCRR/CTSA, Weill Cornell Clinical and Translational Science Center

5. DMID Research Centers of Excellence (RCEs)
PLANNING: Identified the focus prompt and participants
“In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”
IDEA GENERATION: Eliciting knowledge and opinion
Brainstorming :
279 stakeholders were invited to participate.
Participants generated 248 statements with 117 unique visitors hitting the site.
100 statements were derived from the Literature Review.
The planning group edited statement list for relevance and representativeness.
Planning Group approved final set of 94 ideas.
STRUCTURING: Data collection to build the conceptual framework
Sorting and Rating of Ideas
279 participants were invited to rate the 94 ideas on relative Importance and Expected Progress from 2003-present
74 people (27%) contributed their input on the ratings.
33 people invited to organize the 94 ideas into conceptually similar categories.
24 people (72%) participated in the conceptual sorting of the ideas.
ANALYSIS: Series of multivariate analyses

6. The point map shows all the elements in relation to one another.
value and reward original ideas and innovative technologies (4)
“In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”

7. Conceptually similar ideas are in close proximity
value and reward original ideas and innovative technologies (4)
demonstrate its flexibility in addressing emerging infectious disease research needs. (19)
have the capacity to rapidly move into areas of growing importance. (56)

8. Conceptually different ideas are further apart
define who the stakeholders are once product development moves into the pipeline (72)
value and reward original ideas and innovative technologies (4)
Foundation for understanding the relationship among and between points across the territory/ map
create training that is relevant to and integrated with the goals of the strategic plans. (21)
“In order to foster excellence in advancing the biodefense and infectious disease research enterprise, the RCE program and its Centers should...”

9. The cluster map organizes ideas into groups1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94
Translational
Activities
Research Mission
Innovation and Flexibility
Emergency
Response
Training and
Recruitment
Funds Management
Management
Integration
and Synergy
Capacity and
Infrastructure
Collaboration and
Communication
Each cluster is comprised of ideas related to that theme.

10. Each cluster retains the details: For example, Training and Recruitment
promote and encourage opportunities for mentoring junior faculty/researchers in the field of biodefense and emerging infectious disease research. (7)
encourage the participation of post-docs, students, and other non-PI scientists at each RCE's regional meeting. (8)
recruit qualified students, trainees, women and minorities. (10)
help new investigators obtain funding. (16)
create training that is relevant to and integrated with the goals of the strategic plans. (21)
develop more standardized training programs across RCEs by looking at successful programs at each RCE and developing some best practices. (26)
develop an expanded cadre of new researchers, clinicians, and technical personnel who can help lead the national biodefense mission. (45)
supplement RCE activities by applying for direct training supports through T, K and F awards. (55)
establish training programs with pharmaceutical company partners to allow training of scientists in drug discovery and non-clinical development. (68)
develop policies, criteria and processes for selecting career development candidates, including special efforts to recruit qualified women and minorities. (76)
bring new people into the area of biodefense and emerging infectious disease research. (80)

11. DMID RCE Final Conceptual Framework
Research Mission
Innovation and Flexibility
Translational
Activities
Collaboration and
Communication
Capacity and
Infrastructure
Emergency
Response
Integration
and Synergy
Training and
Recruitment
Management
Funds Management
…contains all the details and provides a conceptual framework.

12. Concept Map with Regions
Training and
Recruitment
Integration
and Synergy
Funds Management
Management
Innovation and
Flexibility
Capacity and
Infrastructure
Collaboration and
Communication
Research Mission
Translational
Activities
Emergency
Response
Outcomes
Outputs
Concept Map with Regions

13. Logic Model for RCE Program
Funding
Manage Funds
flexibly and efficiently
Establish Management
structures & processes
Innovative & Flexible
response to changing
needs and opportunities
Synergy between
Centers
(Integration & Synergy)
Capacity to serve as
a regional resource
Expanded cadre of
investigators
(Recruit and Train staff)
Leveraged funds
from elsewhere
Readiness to Respond
in an Emergency
Better response
to infectious
diseases
that threaten
humans
Translate and
Apply Science
to Practice
Scientific Knowledge
on Category
A,B,C organisms
(Research )
Leadership
with expertise
(Integration
& Synergy)
Biodefense
Priorities
Build Infrastructure
Recruit & Train
staff
Inputs
Outputs
Outcomes
Activities
Impact
Collaboration &
Communication
Conduct Research on
Category A,B,C organisms
Research results on
Category
Product
Development
(Translational
Activities)
Blue, italicized text relates to a final cluster on the cluster map.

14. TTURC Concept Map Health Impacts Outcome Communication Structure
Recognition
Collaboration
Training
Transdisciplinary Integration
TD Research Institutionalization
Health Outcomes
Improved Interventions
Translation to Practice
Policy Implications
Science & Models
Methods
Publications
Communications
Communication
Structure
Scientific Integration
Professional Validation
Process
Collaboration

15. Scientific Integration
TTURC Logic Model
Intermediate Markers
Communication
Professional Validation
External Recognition
And Support
Communication
Internal Recognition
And Support
Translation
To Practice
Policy
Implications
Collaboration
Health Impacts
Immediate Markers
Long-Term Outcomes
Collaboration
Training
Transdisciplinary
Integration
Publications
Methods
Science &
Models
Scientific Integration
Interventions
Health
Outcomes

16. CDC PRC Concept Map Engage the Community Diversity & Sensitivity
Relationships & Recognition
Active Dissemination
Technical Assistance
Training
Research Methods
Research Agenda
Core Expertise & Resources
The results were then subjected to analytical methods of multi-dimensional scaling and hierarchical cluster analysis that resulted in the ideas being sorted into similar groups or “concepts.” As shown in this slide, 9 concepts resulted for the national level map. With stakeholder input these concepts were labeled as shown.

17. From Map to Logic Model Inputs Outputs Outcomes Activities
Core Expertise & Resources
Engage the Community
Diversity & Sensitivity
Research Methods
Relationships & Recognition
IN THIS SLIDE you can see an example of how the pieces of the concept mapping was then abstracted into a logic model framework.
Using a simple logic model framework of inputs-activities-outputs-outcomes, you can see that one piece of the concept map, “core expertise and resources”, was considered an input into the PRC program, so it informed the input section of the logic model.
Each of these concepts or pieces could be abstracted to inform differ components of the logic model, which we will not show because of time.
Training
Research Agenda
Technical Assistance
Core Expertise & Resources
Active Dissemination

18. Effective Interventions
Draft logic model
Inputs
Outputs
Outcomes
Activities
Community
Advisory Board
Engage
the
Community
Translation of
Research
To Practice
Widespread
Use of
Effective
Interventions
Conduct
Core & Other
Research
Using Sound Research
Methods
PRC
Capacity
Core expertise &
resources
Diverse faculty/
staff
Faculty/staff
sensitive to
community issues
Facility
Communication &
data systems
Research
and Other
Publications
National,
Regional or
Local Health
Priorities and
Health Disparities
Test Innovative
Strategies
Improved
Community
Health
and
Reductions in
Health Disparities
Widespread
Knowledge of
Effective Interventions
Disseminate
Research Findings
Here you can see how that same piece of the concept map, “core expertise and resources” is in one of the input boxes of this draft logic model. The items in black come from other sources
Establish
Research
Agenda
Recognition
Conduct Training
Trust
Relationships
With Community
Partners, Other PRCs,
and CDC
Skilled
Workforce
Provide
Technical Assistance

19. Logic Model
INPUTS
PROGRAM ACTIVITIES
OUTPUTS
OUTCOMES
Evaluation
National, Local, or Regional Health
Priorities And Health Disparities
Engage
the
Community
Conduct Core and
Other Research
Using Sound
Research Methods
Improved Community and
PopulationHealth
and Elimination of Health Disparities
Programs/
Interventions
Translation of
Research
to Practice and Policy
PRC
Community
Committee
PRC Capacity
Human resources:
core expertise,
diversity,
sensitivity
Facility
Communication &
data systems
Administrative capacity
Evaluation expertise
Research/
Evaluation Findings
Communicated &
Disseminated in:
Publications
Presentations
Media
Reports
This slide shows the final logic model, which is also in your packet.
You will see some of the labels from the concept map on the logic model – for example, “Engage the Community” and “Research Agenda”, two shapes on the concept map, appear as boxes in the Program Activities column of the logic model. “Core Expertise and Resources,” another shape on the concept map, appears on the logic model in the Input column. “Recognition,” also on the concept map, appears as an Outcome on the logic model, and so on.
If anyone is interested in learning more about the Concept Mapping process and how concept maps have been used in this project, including how they have been helpful in developing the logic model for the PRC Program, please consider going to the APHA session giving on Monday at 10:30 am.
Now, let me turn to the way we used the comments from the Workbook to finalize the logic model.
On the slide, we’ve highlighted the changes that were made in the logic model after reviewing comments from the Workbook in RED.
Because some comments were more relevant to a logic model that an individual PRC might develop for its own Center than to the logic model for the national PRC Program, we did not incorporate all Workbook suggestions in the final version of logic model.
This logic model is a national framework under which each PRC would want to develop an individual Logic Model.
Widespread Use of Effective Programs and Policies
Establish
Research
Agenda
Train/Mentor/Provide Technical Assistance to:
Researchers
Practitioners
Students
Community Members
Enhanced
Community
Capacity
for
Prevention
Relationships with
State & Local Health Departments,
Community Partners, University
Partners, Other PRCs, and CDC
Trainees and
Technical Assistance
Recipients
Skilled Public Health
Professionals
Motivating Conditions for Developing and Maintaining Relationships (e.g., Trust)
Expanded
Resources
Recognition

20. AIDS Clinical Research Networks Map
DAIDS Policies and Procedures
Operations and
Management
Resource
Utilization
Community Involvement
Communication,
Collaboration,
Harmonization
Scientific
Agenda-setting
Biomedical objectives
Relevance to Participants
CSI (MK): Stable CM that’s reflective of input across areas of science and broad array of stakeholders. Celebrate. Then move on to how we use this information to develop evaluation system. Next step is development of LM. What is LM and its importanceNote colors 20

21. Community Involvement DAIDS Policies and Procedures
Inputs
Activities
Outputs
Short Term Outcomes
Longer Term Outcomes/ Impact
Recruitment and retention goals met (Community Involvement)
Community Involvement
Increased capacity of developing world sites (Resource Utilization)
Community input (Community Involvement)
Scientific Agenda Setting processes
High quality scientific results and increased knowledge (Biomedical Objectives)
Treatment and prevention measures (Biomedical Objectives)
Integrated use of developing world sites (Resource Utilization)
Collaboration and Communication (Communication, Collaboration, Harmonization)
Appropriate human and infrastructure Resources are in place
Scientific research plan and priorities (Scientific Agenda Setting)
Results Published (Biomedical Objectives)
Biomedical Objectives: HIV/AIDS mortality and morbidity reduced
CSI (MK): colors and bold text correspond to the clusters.
Operations and Management processes are in place
Harmonized systems and procedures (Communication, Collaboration, Harmonization)
DAIDS Policies and Procedures
Scientific agenda is Relevant to Participants 21

22. Conclusions Participatory and Collaborative Integrative Mixed Methods
Systems-Based Approach
Comprehensive Conceptual Model
Foundation for eEvaluation

23. Relevant Literature
Trochim. W., Marcus, S.E., Mâsse, L.C., Moser, R.P., Weld, P. (2008). The Evaluation of Large Research Initiatives: A Participatory Integrative Mixed-Methods Approach, American Journal of Evaluation, 29, 1, 8-28.
Anderson, L. A., Gwaltney, M. K., Sundra, D. L., Brownson, R. C., Kane, M., Cross, A. W., et al. (2006). Using concept mapping to develop a logic model for the prevention research centers program. Preventing Chronic Disease: Public Health Research, Practice and Policy, 3(1), 1-9.
Trochim, W. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12(1), 1-16.
Trochim, W. and Kane, M (2005). Concept mapping: An introduction to structured conceptualization in health care. International Journal for Quality in Health Care, 17, 3, June 2005,

24. Further Information William Trochim Concept Systems Incorporated
Concept Systems Incorporated