1 National Indicators and Qualitative Studies to Measure the Activities and Outcomes of CDC’s PRC Program Demia L. Sundra, MPH Prevention Research Centers Program AEA 2006, Portland, OR The findings of this presentation are the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention
PRC National Evaluation Contributors 2004-Present Collaborative Evaluation Design Team (CEDT) Geri Dino (co-chair), West Virginia University Cynthia Boddie-Willis, MA Dept. of Health Josefina Dhungana, UCLA community representative Katie Barnes, UNC Chapel Hill community representative Kenneth McLeroy, Texas A & M Thomas Reischl, Univ. of Michigan Delores Pluto, Univ. of S. Carolina Michelle Kegler, Emory University David Collins, Morehouse School of Medicine PRC Program Office at CDC Eduardo Simoes (co-chair), Demia Sundra, Jo Anne Grunbaum, Margaret Kaniewski ORC Macro Nicola Dawkins, Doryn Chervin, Stephanie Kamin, Amee Bhalakia, Gayle Payne and Jesse Gerwig-Moore
Outline of Presentation Overview of National PRC Program Evaluation Purpose Indicators Qualitative studies Activities supplementing evaluation Lessons Learned
Levels of PRC Evaluation Evaluation of PRC research Vs. Local evaluation of single research center Vs. Evaluation of national program/ network of 33 centers
Project DEFINE 2004-Present Delivering Evidence and Findings: Implementing the National Evaluation Developing a systematic approach that will allow the PRC program to describe and assess program inputs, activities, outputs and outcomes over time Guided by CDC Framework for Program Evaluation in Public Health 1 1 Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11). 03/05 from:
Purposes of the PRC National Evaluation The national evaluation will accomplish the following purposes: Primary purpose: To demonstrate national program accountability Secondary purpose: To facilitate program improvement
Evaluation Approach Quantitative Indicators 33 PRCs Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs NATIONAL EVALUATION ACTIVITIES Supplements Helps Identify Case Studies & Stories from single PRCs COMMUNICATION/DISSEMINATION ACTIVITIES
Program Indicators Quantitative Indicators 33 PRCs NATIONAL EVALUATION ACTIVITIES
Program Indicators Addressing national accountability Looking for the overarching critical components of PRCs that are feasible and meaningful to count and summarize Balancing burden and highlighting variety Collecting data through PRC Information System
Development of Indicators CEDT rated indicators: Useful, feasible, meaningful, relevant Ongoing work with CEDT to refine Developed corresponding questions for PRC Information System Sent out for feedback from all PRCs Finalizing changes/beginning Information System development
Current Status of Indicators 23 program indicators cover inputs, activities, outputs, and outcomes of a national prevention research program Research Training Partnerships Dissemination Reflect variety of program perspectives Information System is in development
Examples of Research Indicator Concepts ACTIVITIES Research projects INPUTS Project funding Academic institution support OUTCOMES Distribution of products Dissemination and adoption of interventions Policy and environmental changes OUTPUTS Effectiveness of interventions Publications
Sample Reporting – Fictitious Examples of Research Indicator Data Across 33 PRCs, there are 287 projects, totaling $147 million in funding Funding: 32% CDC, 15% NIH, 52% Other 82% of projects are research In the past 5 years of research at 33 PRCs, 29 interventions were funded out of core PRC Program dollars: 10 interventions found promising 4 found effective 1 found adoptable
Examples of Training Indicator Concepts INPUTS Project funding OUTPUTS Number of people trained Number of students working with PRCs ACTIVITIES Trainings, by topic, audience, format, duration
Sample Reporting – Fictitious Examples of Training Indicator Data Nationally, 1,484 individuals attended training programs sponsored by PRCs in The numbers and types of individuals trained include the following: 105 Policy Makers 541 Students 287 Public Health and Health Care Practitioners 311 Community Members 151 Researchers 89 Project Staff
Evaluation Approach Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs NATIONAL EVALUATION ACTIVITIES
Qualitative Study Development Used for concepts critical to program priority for some stakeholders suitable for qualitative work Reserved resources from beginning
Four Descriptive Studies Interaction between PRC researchers and their communities around core prevention research project PRCs’ characteristics related to organizational and partnership structures Variety, goals, and contextual factors of PRCs’ core prevention research Diversity of PRC training, technical assistance, and mentoring with communities and partners
Data Collection Methods Document Review – example topics Staffing counts and structures Community Committee Guidelines Interviews – example topics Participation of communities in research Benefits of being in the PRC network
Supplementing Evaluation Data Supplements Helps Identify Case Studies & Stories from single PRCs COMMUNICATION/DISSEMINATION ACTIVITIES
Case Studies and Stories
Case Study – WVU/Not-On-Tobacco
Story – WVU/Not-On-Tobacco
Evaluation Approach Quantitative Indicators 33 PRCs Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs NATIONAL EVALUATION ACTIVITIES Supplements Helps Identify Case Studies & Stories from single PRCs COMMUNICATION/DISSEMINATION ACTIVITIES
Lessons Learned Expect to use mixed methods Local feedback was critically important to national evaluation strategy Advisory group (CEDT) feedback Broader PRC Program feedback Keep your eye on the prize Accountability of national program helps everyone
Demia L. Sundra, MPH Health Education Specialist, PRC Program, Division of Adult and Community Health, National Center for Chronic Disease Prevention and Health Promotion