TM Low Cost and No Cost Tools for Program Evaluation Victor Balaban, PhD Program Evaluation Team (PET) Field Services and Evaluation Branch (FSEB) Division of Tuberculosis Elimination (DTBE) NCHHSTP/CDC

TM Disclaimer The contents and conclusions in this presentation have not been formally disseminated by CDC and should not be construed to represent any agency determination or policy.

TM Evaluation Evaluation is: An activity that assists in planning and measuring programs a way of managing, improving and being accountable for: resources activities results Evaluation answers the question- “Is the program doing what we intend it to do?”

TM What Can Be Evaluated? Direct service interventions Community mobilization efforts Research initiatives Surveillance systems Policy development activities Outbreak investigations Laboratory diagnostics Communication campaigns Infrastructure-building projects Training and educational services Administrative systems MMWR, 1999, Framework for Program Evaluation in Public Health Source: MMWR, 1999, Framework for Program Evaluation in Public Health

TM Why is Evaluation Important? Improve knowledge of program design Improve program implementation Reporting Ensure that a program reaches those who need it most Give visibility to work Demonstrate accountability Share information Enhance understanding of what works best and what does not work – and why

TM Low Cost and No Cost Tools for Program Evaluation When designing an evaluation, there are often existing tools that can be used Many were not designed for evaluating TB programs and so will need to be modified Focus on evaluation instruments and tools for data collection and analysis, not guidelines and trainings

TM CDC Resources

TM Find TB Resources ation.aspxhttp:// ation.aspx Resources: Links to tools and resources related to program evaluation

TM CDC - Office of the Associate Director for Program (OADPG) #datacollectionhttp:// #datacollection Resources: Step by Step Manuals Manuals/Assistance with Specific Evaluation Steps: Logic models Data Collection Methods and Sources Websites Offering Comprehensive Evaluation Resources and Assistance

TM Evaluation Guidance Handbook: Strategies for Implementing the Evaluation Guidance for CDC-Funded HIV Prevention Programs h_depts/guidance/strat- handbook/appendix.htmhttp:// h_depts/guidance/strat- handbook/appendix.htm Instruments: Intervention plan worksheets Data collection forms

TM CDC - Healthy Youth! Program Evaluation Strategic Planning Tools & Resources p_toolkit.htm#3http:// p_toolkit.htm#3 Strategic Planning Kit for School Health Programs: Data Sources Matrix Identifying Stakeholders Worksheet SWOT Worksheet for School Health Programs Program Strategy Checklist Program Communication Tracking Tool Data Collection Framework

TM CDC - Healthy Youth! Handbook for Evaluating HIV Education _handbook/index.htmhttp:// _handbook/index.htm Instruments: HIV and AIDS knowledge related to HIV-risk behaviors Attitudes toward people with AIDS Attitudinal dimensions related to HIV-risk behaviors Confidence in one's ability to resist peer pressures Perceptions of peers' behaviors and values Intentions regarding HIV-risk behaviors and their precursors HIV-risk behaviors

TM TB-PEN Resources

TM TB-PEN – Tools Committee Instruments: (organized by National TB Objectives) Completion of Treatment TB Case Rates Contact Investigations Laboratory Reporting Treatment Initiation Sputum Culture Conversion Sputum Culture Reporting Data Reporting Recommended Initial Therapy Universal Genotyping Known HIV Status Evaluation of Immigrants and Refugees Program Evaluation Human Resource Development Plan TB Training Focal Points Cohort Review

TM American Evaluation Association (AEA) Resources

TM AEA – Online Resources Links to collections of evaluation instruments Links to companies that provide software for the development, distribution, scanning, and analysis of surveys/questionnaires

TM AEA A blog dedicated to highlighting Tips, Tricks, Resources, and Lessons Learned for evaluators

TM Other Resources

TM Children, Youth and Families Education and Research Network (CYFER) Instruments for: Process Evaluation - populations served, satisfaction, program accountability, etc. Program Sustainability Organizational Capacity

TM UCSF - Center for AIDS Prevention Studies (CAPS) instruments/ instruments/ Instruments for: Evaluation Treatment adherence Focus group guides

TM Innovation Network – Point K Learning Center on_id=4&content_id=16http:// on_id=4&content_id=16 Instruments: Organizational Assessment Tool Logic Model Builder Evaluation Plan Builder

TM Penn State Extension: Program Evaluation Resources: Evaluation tipsheets that answer questions about program evaluation Evaluation examples, including selections from instruments

TM University of Wisconsin - Extension evalinstruments.htmlhttp:// evalinstruments.html Instruments: Community Organization Assessment Tool Program Planning Questionnaire

TM The Measurement Group ools/evaluation_tools.htmhttp:// ools/evaluation_tools.htm Instruments General Purpose Instruments and Evaluation Modules Evaluation of the HRSA SPNS Innovative Models of HIV/AIDS Care Evaluation Report Forms for the JAHF Institute for Geriatric Nursing General Purpose Instruments and Specialized Evaluation Materials for Evaluation of the HRSA SPNS Adolescent HIV/AIDS Care Models Metro DC Collaborative for Women with HIV/AIDS Needs Assessment Instruments Evaluation Instruments for the HIFY Empowered Youth Evaluation (EYE) Model

TM Wilderdom Instruments List of 30+ tools for assessing the psycho-social impacts of intervention programs e.g. program quality surveys Summary reviews of the main features of each instrument, with overall rating and links to further information. Links to the major sites on the web for finding more research instruments.

TM DoView DoView is a software program that enables evaluators to create logic models

TM Data Collection and Analysis

TM Data Collection and Analysis CDC Resources Epi-Info EZ-Text Google Resources Google Analytics Google Refine

TM Epi-Info Software for the rapid creation of quantitative data collection instruments, analyzing and visualizing data, and generating reports Creates questionnaires and data entry forms Automatically creates a database from the questionnaire Reads and analyzes data into statistics, tables, graphs, and maps Generates reports

TM EZ-Text ources/software/ez-text/index.htmhttp:// ources/software/ez-text/index.htm Software program to assist in creating, managing, and analyzing qualitative data Interview responses are entered into EZ-Text as a verbatim transcript (e.g., from a tape recording), or a summary generated from the interviewer's notes

TM EZ-Text (cont.) ources/software/ez-text/index.htmhttp:// ources/software/ez-text/index.htm Users can investigators can interactively create on-line codebooks, apply codes to specific response passages, develop case studies, conduct database searches to identify text passages, and export data for further analysis with other qualitative or statistical analysis software programs

TM Google Google Analytics A free program that help to analyze traffic to websites and measure the success of social media programs Google Refine for data cleanup and transformation to other formats to help clean messy data

TM Conclusion Using and adapting existing instruments can save time and resources when conducting evaluations DTBE/FSEB/PET is available to provide information to help you identify and use evaluation tools

TM Thank you

TM Acknowledgements DTBE/FSEB/Program Evaluation Team (PET) Awal Khan Judy Gibson Lakshmy Menon Brandy Peterson DTBE/FSEB/Field Services Teams I & II Dan Ruggiero Gail Burns-Grant Vincent Fears Alstead Forbes Regina Gore Vernard Green Andy Heetderks Mark Miner Dawn Tuckey