1. 1 National Indicators and Qualitative Studies to Measure the Activities and Outcomes of CDC’s PRC Program Demia L. Sundra, MPH Prevention Research Centers Program AEA 2006, Portland, OR The findings of this presentation are the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention

2. PRC National Evaluation Contributors 2004-Present Collaborative Evaluation Design Team (CEDT) Geri Dino (co-chair), West Virginia University Cynthia Boddie-Willis, MA Dept. of Health Josefina Dhungana, UCLA community representative Katie Barnes, UNC Chapel Hill community representative Kenneth McLeroy, Texas A & M Thomas Reischl, Univ. of Michigan Delores Pluto, Univ. of S. Carolina Michelle Kegler, Emory University David Collins, Morehouse School of Medicine PRC Program Office at CDC Eduardo Simoes (co-chair), Demia Sundra, Jo Anne Grunbaum, Margaret Kaniewski ORC Macro Nicola Dawkins, Doryn Chervin, Stephanie Kamin, Amee Bhalakia, Gayle Payne and Jesse Gerwig-Moore

3. Outline of Presentation Overview of National PRC Program Evaluation  Purpose  Indicators  Qualitative studies  Activities supplementing evaluation Lessons Learned

4. Levels of PRC Evaluation Evaluation of PRC research Vs. Local evaluation of single research center Vs. Evaluation of national program/ network of 33 centers

5. Project DEFINE 2004-Present Delivering Evidence and Findings: Implementing the National Evaluation Developing a systematic approach that will allow the PRC program to describe and assess program inputs, activities, outputs and outcomes over time Guided by CDC Framework for Program Evaluation in Public Health 1 1 Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No. RR-11). 03/05 from: http://www.cdc.gov/eval/framework.htmhttp://www.cdc.gov/eval/framework.htm

6. Purposes of the PRC National Evaluation The national evaluation will accomplish the following purposes:  Primary purpose: To demonstrate national program accountability  Secondary purpose: To facilitate program improvement

7. Evaluation Approach Quantitative Indicators 33 PRCs Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs NATIONAL EVALUATION ACTIVITIES Supplements Helps Identify Case Studies & Stories from single PRCs COMMUNICATION/DISSEMINATION ACTIVITIES

8. Program Indicators Quantitative Indicators 33 PRCs NATIONAL EVALUATION ACTIVITIES

9. Program Indicators Addressing national accountability Looking for the overarching critical components of PRCs that are feasible and meaningful to count and summarize Balancing burden and highlighting variety Collecting data through PRC Information System

10. Development of Indicators CEDT rated indicators: Useful, feasible, meaningful, relevant  Ongoing work with CEDT to refine Developed corresponding questions for PRC Information System Sent out for feedback from all PRCs Finalizing changes/beginning Information System development

11. Current Status of Indicators 23 program indicators cover inputs, activities, outputs, and outcomes of a national prevention research program  Research  Training  Partnerships  Dissemination Reflect variety of program perspectives Information System is in development

12. Examples of Research Indicator Concepts ACTIVITIES Research projects INPUTS Project funding Academic institution support OUTCOMES Distribution of products Dissemination and adoption of interventions Policy and environmental changes OUTPUTS Effectiveness of interventions Publications

13. Sample Reporting – Fictitious Examples of Research Indicator Data Across 33 PRCs, there are 287 projects, totaling $147 million in funding  Funding: 32% CDC, 15% NIH, 52% Other  82% of projects are research In the past 5 years of research at 33 PRCs, 29 interventions were funded out of core PRC Program dollars:  10 interventions found promising  4 found effective  1 found adoptable

14. Examples of Training Indicator Concepts INPUTS Project funding OUTPUTS Number of people trained Number of students working with PRCs ACTIVITIES Trainings, by topic, audience, format, duration

15. Sample Reporting – Fictitious Examples of Training Indicator Data Nationally, 1,484 individuals attended training programs sponsored by PRCs in 2006. The numbers and types of individuals trained include the following:  105 Policy Makers  541 Students  287 Public Health and Health Care Practitioners  311 Community Members  151 Researchers  89 Project Staff

16. Evaluation Approach Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs NATIONAL EVALUATION ACTIVITIES

17. Qualitative Study Development Used for concepts  critical to program  priority for some stakeholders  suitable for qualitative work Reserved resources from beginning

18. Four Descriptive Studies Interaction between PRC researchers and their communities around core prevention research project PRCs’ characteristics related to organizational and partnership structures Variety, goals, and contextual factors of PRCs’ core prevention research Diversity of PRC training, technical assistance, and mentoring with communities and partners

19. Data Collection Methods Document Review – example topics  Staffing counts and structures  Community Committee Guidelines Interviews – example topics  Participation of communities in research  Benefits of being in the PRC network

20. Supplementing Evaluation Data Supplements Helps Identify Case Studies & Stories from single PRCs COMMUNICATION/DISSEMINATION ACTIVITIES

21. Case Studies and Stories www.cdc.gov/prc

22. Case Study – WVU/Not-On-Tobacco

23. Story – WVU/Not-On-Tobacco

24. Evaluation Approach Quantitative Indicators 33 PRCs Qualitative Studies Document Review – 33 PRCs Interviews – Sample of PRCs NATIONAL EVALUATION ACTIVITIES Supplements Helps Identify Case Studies & Stories from single PRCs COMMUNICATION/DISSEMINATION ACTIVITIES

25. Lessons Learned Expect to use mixed methods Local feedback was critically important to national evaluation strategy  Advisory group (CEDT) feedback  Broader PRC Program feedback Keep your eye on the prize  Accountability of national program helps everyone

26. www.cdc.gov/prc Demia L. Sundra, MPH Health Education Specialist, PRC Program, Division of Adult and Community Health, National Center for Chronic Disease Prevention and Health Promotion 770-488-5506 dsundra@cdc.gov