1. Sue Lin Yee, MA, MPH Senior Evaluation Scientist Center for Disease Control and Prevention (CDC) American Evaluation Association Meeting October 25, 2012 Planning and Conducting Internal Evaluations: Portfolio Review Guidance from CDC National Center for Injury Prevention and Control Division of Analysis, Research, and Practice Integration

2. Acknowledgements  Mary Bovenzi, CDC PHPS Fellow assigned to the Boston Public Health Commission  Silvia Trigoso, CDC PHPS Fellow assigned to the Department of Health in American Samoa  Dr. Arlene Greenspan, Associate Director for Science, National Center for Injury Prevention and Control (NCIPC)

3. Think Tank Overview  Clarify CDC policy for “Peer Review of Scientific Programs”  Discuss the purpose and background of NCIPC’s portfolio reviews  Discuss context and need for the “NCIPC Portfolio Review Policy and Practice” guide  Highlight primary stakeholders and proposed roles  Solicit and discuss participants’ experiences with internal evaluations  Obtain feedback on NCIPC’s approach to planning and implementing portfolio reviews

4. CDC Peer Review for Scientific Programs  All research and scientific programs conducted or funded by CDC subject to periodic external review Research and non-research Intramural and extramural research Major research topics Organizational units Core service activities (e.g., laboratories, statistical support, health communications) Public health practice activities (e.g., surveillance, state/local programs)

5. Purpose of Policy “... External experts continues to be an integral part of how CDC ensures the quality, relevance, and impact of its public health science and programs. Center, Institute, and Office (CIO) leadership should identify the highest priority programs for external peer review, and should ensure that peer reviews within their CIO are meaningful and produce actionable recommendations. CIO leadership has primary responsibility for ensuring that appropriate improvements are made to programs based on the recommendations from completed reviews.” (July 2011) CDC Peer Review of Research and Scientific Programs, 2008

6. Background: NCIPC Portfolio Reviews  What are they?  Internal evaluations of NCIPC’s research and scientific program portfolios on a specific injury topic or center function  Why does NCIPC do them?  Describe contribution to public health, guide scientific and programming direction, and justify use of limited financial resources

7. NCIPC Portfolio Reviews, 2005-2012  Youth Violence Prevention 2005  Falls Among Older Adults2006  Traumatic Brain Injury 2007  Biomechanics2007  Injury Control Research Centers* 2009  Motor Vehicle Injuries 2010  National Violent Death Reporting System 2010  Core State Injury Program*2012  Sexual Violence Prevention*2012  Health Comm Research & Activities*In progress *Co-led by NCIPC, Office of the Associate Director for Science

8. Division of Analysis, Research and Practice Integration Office of the Director Acting Director Lisa C Barrios DrPH MS Deputy Director James A Enders MPH Acting Policy Lead Tochukwu E Igbo JD Communications Lead Vacant Health Communications Graham Kirkland Association Director for Science Vacant Principal Management Official William Ramsey Secretary Mary Stuckey CAP-OM Sr. Advisor for Injury Surveillance Lee Annest PhD Statistics Programming and Economics Branch (SPEB) Acting Branch Chief Lee Annest PhD Practice Integrations & Evaluation Branch (PIEB) Acting Branch Chief Angela Marr September 2012 Statistics Team Marci-jo Kresnow Lead Programming Team Kevin Webb Lead Health Economics & Policy Research Team Curtis Florence Lead Violence & Injury Prevention Program Team Lead - Vacant Evaluation & Integration Team Chris Jones Acting Lead

9. What’s different... What’s the same  Each new portfolio review is a new evaluation project  Different key stakeholders  Different evaluation questions  Different scientific programs  Different ideas about evaluation  Different interpretations of the CDC policy  Different fears, concerns, and levels of interest  Different ideas about use of findings and recommendations, but  Essentially same process for planning and implementation

10. NCIPC Portfolio Reviews: Key Phases & Time-line

11. NCIPC Portfolio Review Policy and Practice  Establish expectations for future portfolio reviews  Build capacity to participate and implement the project  Communicate expectations across the Injury Center  Increase buy-in and accountability in the portfolio review process at all organizational levels  Build in efficiencies where possible (i.e. contracts, staff resources)

12. Key Portfolio Review Stakeholders  Evaluation Lead from Division of Analysis, Research, and Program Integration (DARPI)  Portfolio Review Lead  Contracting Officers Representative (COR) from Office of the Associate Director for Science (OADS)  Program or Content Workgroup Members  External Evaluation Contractor  External Peer Review Panel  NCIPC Board of Scientific Counselors (BSC)

13. Portfolio Review Lead  Takes a leadership role in planning and implementing the portfolio review  Convenes and facilitates portfolio review workgroup  Collaborates with COR to ensure on course and within budget  Engages the DARPI evaluator and external contractor to ensure quality and feasibility of evaluation questions and evaluation design  Supports other workgroup members to identify and collect relevant information to inform evaluation questions  Provides a process for collating NCIPC’s feedback to contractor  Presents the portfolio review updates or findings to the Board of Scientific Counsellors (BSC)

14. DARPI Evaluator  Provides overall evaluation perspective and feedback as member of workgroup in all phases of project  Guides the workgroup through the portfolio review process  Collaborates with COR to ensure SOW, budget, and time- line are sufficient to support evaluation  Works with Portfolio Review Lead and OADS to identify external review panel members, chair person, and provide guidance on panel charge  In conjunction with program and OADS, may provide advice on implementation of actionable recommendations

15. Contracting Officer’s Rep (COR)  Manages the portfolio review’s external contractor  Develops a scope of work, budget, and time-line with evaluator and Portfolio Review Lead  Intervenes and communicates with Program and Grants Office  Provides history and background of past portfolio reviews  Participates in workgroup meetings and meetings with external contractor, as needed, to ensure portfolio review on course and within budget  Serves as the logistics lead and meeting planner for workgroup, external contractor, and external panel meeting

16. Porfolio Review Workgroup Members  3-4 members from program who attend weekly meetings  Provides secondary data sources and participates in primary data collection  Provides feedback on materials during each phase (20- 30% time)  Ensures internal stakeholders (i.e., management and ADS) are aware of purpose and status of portfolio review  Ensures external stakeholders (i.e., partner groups, grantees) are aware portfolio review, time-line, and intended use of the evaluation findings and recommendations  Attends portfolio review kickoff meeting, external panel meeting, and NCIPC BSC meeting

17. External Evaluation Contractor  Works closely with all NCIPC stakeholders to plan and implement the portfolio review  Designs the evaluation  Analyzes secondary data  Develops additional data collection instruments  Draft and revises multiple drafts of report  Participates in meetings with workgroup and COR  Communicates with external stakeholders  Plans and participates in external peer review panel  Works with external panel to solidify recommendations  Integrates final recommendations to finalize the portfolio review report

18. Chair Person for External Peer Review Panel  Works closely with DARPI evaluator, Portfolio Review Lead, and OADS to plan panel meeting  Leads and facilitates the panel meeting according to the rules and guidelines of the BSC or FACA meetings  Present panel’s preliminary findings and recommendations to NCIPC leadership at midpoint of panel meeting  Works with the external contractor, portfolio review lead, and DARPI evaluator capture the spirit and intent of the panel’s recommendations to NCIPC  Presents panel’s feedback and actionable recommendations to NCIPC and BSC

19. External Peer Review Panel Members  Declare any conflicts of interest  Review draft report and provide preliminary feedback and draft recommendations to CDC before panel meeting  Actively participate in the panel meeting in Atlanta, GA  Maintain full confidentiality regarding discussions  Work with chair person and external evaluation contractor after meeting to finalize actionable recommendations  Agree to utilized technology for sharing materials and communicating remotely to facilitate development of recommendations

20. NCIPC Board of Scientific Counselors (BSC)  Reviews and discusses report findings and proposed recommendations  Raises questions and proposed additions to the report  Vote to accept the findings and recommendations  Recommendations not binding but may inquires about status of implementation  NCIPC reports annual progress to the Office of the Associate Director for Science at the agency level

21. Discussion Questions  What do internal evaluations look like in your environment?  How do you plan and carry out internal evaluations?  Stakeholders  Key steps  Time-line  Use of findings and recommendations  Feedback on NCIPC’s approach  Lessons Learned  Aha moments  Strategies for planning and implementing

22. National Center for Injury Prevention and Control Office of the Director, Office of the Associate Director for Science For more information please contact Centers for Disease Control and Prevention 1600 Clifton Road NE, Atlanta, GA 30333 Telephone: 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348 E-mail: cdcinfo@cdc.gov Web: http://www.cdc.gov The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. THANK YOU! Sue Lin Yee, MA, MPH National Center for Injury Prevention and Control Division of Analysis, Research, and Program Integration Program Integrationand Evaluation Branch, Evaluation and Integration Team Email: sby9@cdc.gov