1. TM PEN: CDC Guidance for Developing Plans and Reports 2011 TB ETN/PEN Conference Awal Khan, PhD Lead, Program Evaluation Team Field Services and Evaluation Branch (FSEB) Division of Tuberculosis Elimination (DTBE) OID/NCHHSTP

2. TM This talk – a quick overview:  Program evaluation (PE) activities  DTBE guidance on PE  DTBE guidance on cohort review  Developing PE plan & reporting results  Current and on-going PE initiatives  Questions

3. TM Program Evaluation (PE) Activities

4. TM Program Evaluation (PE) Plans Submitted - 2010 Annual Report

5. TM PE Reports Following Application Guidance - 2010 Annual Report

6. TM Program Using NTIP to Select Evaluation Focus Area, 2010

7. TM Focus Area of Program Evaluation Focus AreaInterimAnnual Contact Investigation2422 Known HIV Status1310 Sputum Culture Conversion 88 Completion of Treatment 66 Evaluation of Immigrants and Refugees47 TB Case Rate42 Sputum Culture Reported31 Recommended Initial Therapy20 Universal Genotyping21 Program Evaluation/ Evaluation Focal Point 11

8. TM CDC/DTBE Guidance on Program Evaluation

9. TM CDC Funding Opportunity Announcement (FOA) Requirements, 2012 Monitor their progress toward meeting national TB objectives and performance target using NTIP reports and other data sources When applicable, use NTIP to report their performance on national objectives in the annual and interim progress reports for the TB cooperative agreement Where appropriate, grantees should continue to report and describe progress toward project- specific objectives and activities using their own data and/or NTIP.

10. TM Guidance for Program Evaluation - CDC 2012 FOA A systematic and in-depth study of priority program areas to enhance a program’s ability to meet its objectives and performance targets The focus of evaluation should be on where the program needs improvement Collect information that can be used to inform, enhance, and facilitate programmatic decision making process.

11. TM Guidance for Program Evaluation Programs should monitor their progress toward meeting national TB objectives and performance target using NTIP reports and other data sources Identify a focus area and provide a justification Clearly defined objectives, purposes, and methods of the evaluation Uses and dissemination of findings including strategy and plan to work with local programs.

12. TM Program Evaluation Focal Points Serve as the point of contact in their respective jurisdiction for evaluation activities Build program evaluation capacity Work closely with TB program staff Engage in TB PEN activities Share PE experiences and lessons learned with stakeholders and other programs.

13. TM CDC/DTBE Guidance on Cohort Review

14. TM Programs Implementing Cohort Review

15. TM Guidance for Conducting Cohort Review Programs will be required to conduct at least one cohort review per year at the state or local levels or both Programs should continue ongoing case reviews as usual Programs are recommended to conduct cohort reviews Semi-annually, if 0-50 cases reported Quarterly, if over 50 cases reported

16. TM Developing PE Plan & Reporting Results

17. TM Format for Developing Evaluation Plan A.Purpose of the evaluation: 3 purposes/ goals including focus area B.Objectives: What do you want to do? should be “SMART”: Specific, Measurable, Achievable, Realistic, and Time-bound C. Activities: An objective may have multiple activities

18. TM Format for Developing Evaluation Plan – Contd. D. Measures of activities: Specific indicators for each activity Criteria for success/failure in each activity E.Expected outcomes / impact E.Potential challenges and opportunities E.Use of results.

19. TM Program Evaluation Report I. Introduction: Describe Evaluation Objectives Describe the purpose of your evaluation, rationale for selecting evaluation focus area, and clearly define the evaluation objectives II. Describe data collection and analyses how the evaluation was conducted (activities and timeline) data sources, data collected, and methods

20. TM Program Evaluation Report – Contd. III. Conclusion and Discussion of results. Provide results of the objectives. If not achieved, why not? What were the barriers to and facilitators? IV. Recommendations to Program Administrators for Future Program Focus and Evaluation Efforts Describe the best and/or evidence-based practices including identifying the next focus area.

21. TM PET’s Feedback Process for Interim/Annual Reports PET provides feedback reports only for program evaluation Timeline – usually May and October Feedback is reviewed by program consultant and evaluation representative TB Control Officer and PE Focal Point receive the feedback Program are encouraged to contact the program consultant and PE representative to discuss the feedback.

22. TM Feedback Format

23. TM Conceptual model for Evaluation Focus Area “Implementation Gap” TB Elimination The “Implementation Gap” Some goals and objectives are achieved and implemented and some or not The “Implementation Gap” Use of new Tools or Technology for evaluation activities

24. TM Current and on-going PE initiatives

25. TM Current and Ongoing PE Initiatives Identify PE technical assistance needs collaboratively with TB control programs, PE focal points, and PET representatives Program develops PE plan and discusses with designated PET representative for additional support and shares with the respective program consultant for collaboratively implementing the PE activities Field Operations and Program Evaluation Teams meet quarterly to share evaluation & programmatic perspectives for meeting the grantee’s needs

26. TM Current and Ongoing PE Initiatives – Contd. Identify ways to provide technical assistance for sustaining PE efforts Developed template for contact investigation evaluation plan and report in response to requests from the program evaluation focal point Plan to develop evaluation plan and report templates for all national TB objectives.

27. TM Number of Abstracts Received, 2009-2011

28. TM Program Evaluation Focal Points Serve as the point of contact in their respective jurisdiction for evaluation activities Build program evaluation capacity Work closely with TB program staff Engage in TB PEN activities Share PE experiences and lessons learned with stakeholders and other programs.

29. TM Acknowledgements Al ForbesVictor Balaban Gail Grant Judy Gibson Regina GoreLakshmy Menon Bruce HealthBrandy Peterson Andy HeetderksSylvia Trigoso (PHPS Fellow) Mark MinerChristina Dahlstrom (PHPS) Vic TomlinsonLauren Polansky Awal Khan Paul TribbleGreg Andrews Dawn TuckeyDan Ruggiero Joe ScavottoTerry Chorba TB PEN Focal Points and TB Control Officers

30. TM Thank you

31. TM Good Program Evaluation Shifts Our Focus From Did our efforts work? to Are our efforts working?

32. TM Additional Slides on Cohort Review

33. TM What is Cohort Review ? Cohort review is a systematic review of patients with TB disease and their contacts The “Cohort” is a group of TB cases counted over a specific time and the review occurs after the cases are counted Case reviews are not cohort reviews.

34. TM Purposes of Cohort Review A useful tool for ensuring accountability, informing and education program staff about TB protocol and national objectives Review of comprehensive TB case management procedures for all cases Opportunity for reviewing TB data for reliability and quality checking, and identifying deviations Opportunity for reviewing completeness of key data variables of RVCT and making corrections.

35. TM Purposes of Cohort Review – Contd. Help program staff Understand the challenges patients face as they try to complete TB treatment Undertake activities required for program improvement Enable programs to verify data for national program objectives and understand activities most likely to impact program targets.