1. TM Low Cost and No Cost Tools for Program Evaluation Victor Balaban, PhD Program Evaluation Team (PET) Field Services and Evaluation Branch (FSEB) Division of Tuberculosis Elimination (DTBE) NCHHSTP/CDC

2. TM Disclaimer The contents and conclusions in this presentation have not been formally disseminated by CDC and should not be construed to represent any agency determination or policy.

3. TM Evaluation Evaluation is: An activity that assists in planning and measuring programs a way of managing, improving and being accountable for: resources activities results Evaluation answers the question- “Is the program doing what we intend it to do?”

4. TM What Can Be Evaluated? Direct service interventions Community mobilization efforts Research initiatives Surveillance systems Policy development activities Outbreak investigations Laboratory diagnostics Communication campaigns Infrastructure-building projects Training and educational services Administrative systems MMWR, 1999, Framework for Program Evaluation in Public Health Source: MMWR, 1999, Framework for Program Evaluation in Public Health

5. TM Why is Evaluation Important? Improve knowledge of program design Improve program implementation Reporting Ensure that a program reaches those who need it most Give visibility to work Demonstrate accountability Share information Enhance understanding of what works best and what does not work – and why

6. TM Low Cost and No Cost Tools for Program Evaluation When designing an evaluation, there are often existing tools that can be used Many were not designed for evaluating TB programs and so will need to be modified Focus on evaluation instruments and tools for data collection and analysis, not guidelines and trainings

7. TM CDC Resources

8. TM Find TB Resources http://www.findtbresources.org/programevalu ation.aspxhttp://www.findtbresources.org/programevalu ation.aspx Resources: Links to tools and resources related to program evaluation

9. TM CDC - Office of the Associate Director for Program (OADPG) http://www.cdc.gov/eval/resources/index.htm #datacollectionhttp://www.cdc.gov/eval/resources/index.htm #datacollection Resources: Step by Step Manuals Manuals/Assistance with Specific Evaluation Steps: Logic models Data Collection Methods and Sources Websites Offering Comprehensive Evaluation Resources and Assistance

10. TM Evaluation Guidance Handbook: Strategies for Implementing the Evaluation Guidance for CDC-Funded HIV Prevention Programs http://www.cdc.gov/hiv/topics/evaluation/healt h_depts/guidance/strat- handbook/appendix.htmhttp://www.cdc.gov/hiv/topics/evaluation/healt h_depts/guidance/strat- handbook/appendix.htm Instruments: Intervention plan worksheets Data collection forms

11. TM CDC - Healthy Youth! Program Evaluation Strategic Planning Tools & Resources http://www.cdc.gov/healthyyouth/evaluation/s p_toolkit.htm#3http://www.cdc.gov/healthyyouth/evaluation/s p_toolkit.htm#3 Strategic Planning Kit for School Health Programs: Data Sources Matrix Identifying Stakeholders Worksheet SWOT Worksheet for School Health Programs Program Strategy Checklist Program Communication Tracking Tool Data Collection Framework

12. TM CDC - Healthy Youth! Handbook for Evaluating HIV Education http://www.cdc.gov/HealthyYouth/publications/hiv _handbook/index.htmhttp://www.cdc.gov/HealthyYouth/publications/hiv _handbook/index.htm Instruments: HIV and AIDS knowledge related to HIV-risk behaviors Attitudes toward people with AIDS Attitudinal dimensions related to HIV-risk behaviors Confidence in one's ability to resist peer pressures Perceptions of peers' behaviors and values Intentions regarding HIV-risk behaviors and their precursors HIV-risk behaviors

13. TM TB-PEN Resources

14. TM TB-PEN – Tools Committee http://tbpen.pbworks.com/w/page/4124488/FrontPage Instruments: (organized by National TB Objectives) http://tbpen.pbworks.com/w/page/4124488/FrontPage Completion of Treatment TB Case Rates Contact Investigations Laboratory Reporting Treatment Initiation Sputum Culture Conversion Sputum Culture Reporting Data Reporting Recommended Initial Therapy Universal Genotyping Known HIV Status Evaluation of Immigrants and Refugees Program Evaluation Human Resource Development Plan TB Training Focal Points Cohort Review

15. TM American Evaluation Association (AEA) Resources

16. TM AEA – Online Resources http://www.eval.org/Resources/instruments.asp Links to collections of evaluation instruments http://www.eval.org/Resources/surveylinks.asp Links to companies that provide software for the development, distribution, scanning, and analysis of surveys/questionnaires

17. TM AEA 365 http://www.aea365.org/blog/ A blog dedicated to highlighting Tips, Tricks, Resources, and Lessons Learned for evaluators

18. TM Other Resources

19. TM Children, Youth and Families Education and Research Network (CYFER) http://www.cyfernet.org/index.php?c=6 Instruments for: Process Evaluation - populations served, satisfaction, program accountability, etc. Program Sustainability Organizational Capacity

20. TM UCSF - Center for AIDS Prevention Studies (CAPS) http://caps.ucsf.edu/research/survey- instruments/http://caps.ucsf.edu/research/survey- instruments/ Instruments for: Evaluation Treatment adherence Focus group guides

21. TM Innovation Network – Point K Learning Center http://www.innonet.org/index.php?secti on_id=4&content_id=16http://www.innonet.org/index.php?secti on_id=4&content_id=16 Instruments: Organizational Assessment Tool Logic Model Builder Evaluation Plan Builder

22. TM Penn State Extension: Program Evaluation http://extension.psu.edu/evaluation/examples.htmlhttp://extension.psu.edu/evaluation/examples.html Resources: Evaluation tipsheets that answer questions about program evaluation Evaluation examples, including selections from instruments

23. TM University of Wisconsin - Extension http://www.uwex.edu/ces/pdande/evaluation/ evalinstruments.htmlhttp://www.uwex.edu/ces/pdande/evaluation/ evalinstruments.html Instruments: Community Organization Assessment Tool Program Planning Questionnaire

24. TM The Measurement Group http://www.themeasurementgroup.com/evaluationt ools/evaluation_tools.htmhttp://www.themeasurementgroup.com/evaluationt ools/evaluation_tools.htm Instruments General Purpose Instruments and Evaluation Modules Evaluation of the HRSA SPNS Innovative Models of HIV/AIDS Care Evaluation Report Forms for the JAHF Institute for Geriatric Nursing General Purpose Instruments and Specialized Evaluation Materials for Evaluation of the HRSA SPNS Adolescent HIV/AIDS Care Models Metro DC Collaborative for Women with HIV/AIDS Needs Assessment Instruments Evaluation Instruments for the HIFY Empowered Youth Evaluation (EYE) Model

25. TM Wilderdom http://www.wilderdom.com/tools/ToolsIndex.html Instruments List of 30+ tools for assessing the psycho-social impacts of intervention programs e.g. program quality surveys Summary reviews of the main features of each instrument, with overall rating and links to further information. Links to the major sites on the web for finding more research instruments.

26. TM DoView http://www.doview.com/ DoView is a software program that enables evaluators to create logic models

27. TM Data Collection and Analysis

28. TM Data Collection and Analysis CDC Resources Epi-Info EZ-Text Google Resources Google Analytics Google Refine

29. TM Epi-Info http://wwwn.cdc.gov/epiinfo/index.htm Software for the rapid creation of quantitative data collection instruments, analyzing and visualizing data, and generating reports Creates questionnaires and data entry forms Automatically creates a database from the questionnaire Reads and analyzes data into statistics, tables, graphs, and maps Generates reports

30. TM EZ-Text http://www.cdc.gov/hiv/topics/surveillance/res ources/software/ez-text/index.htmhttp://www.cdc.gov/hiv/topics/surveillance/res ources/software/ez-text/index.htm Software program to assist in creating, managing, and analyzing qualitative data Interview responses are entered into EZ-Text as a verbatim transcript (e.g., from a tape recording), or a summary generated from the interviewer's notes

31. TM EZ-Text (cont.) http://www.cdc.gov/hiv/topics/surveillance/res ources/software/ez-text/index.htmhttp://www.cdc.gov/hiv/topics/surveillance/res ources/software/ez-text/index.htm Users can investigators can interactively create on-line codebooks, apply codes to specific response passages, develop case studies, conduct database searches to identify text passages, and export data for further analysis with other qualitative or statistical analysis software programs

32. TM Google Google Analytics http://www.google.com/analytics/index.html A free program that help to analyze traffic to websites and measure the success of social media programs Google Refine http://code.google.com/p/google-refine/ for data cleanup and transformation to other formats to help clean messy data

33. TM Conclusion Using and adapting existing instruments can save time and resources when conducting evaluations DTBE/FSEB/PET is available to provide information to help you identify and use evaluation tools

34. TM Thank you

35. TM Acknowledgements DTBE/FSEB/Program Evaluation Team (PET) Awal Khan Judy Gibson Lakshmy Menon Brandy Peterson DTBE/FSEB/Field Services Teams I & II Dan Ruggiero Gail Burns-Grant Vincent Fears Alstead Forbes Regina Gore Vernard Green Andy Heetderks Mark Miner Dawn Tuckey