1. Evaluating Your Program: Developing Successful and Sustainable Programs OJJDP New Grantee Orientation April 6–7, 2010 Washington, DC CSR, Incorporated

2. Outline What is Program Evaluation? The Logic Model Evaluation Questions Types of Program Evaluation Data Collection and Analysis Evidence-Based Programs

3. Program Evaluation Objective measurement and systematic analysis to determine the manner and extent to which a program achieves its intended goal. Definitions adapted from the OMB Circular no. A-11 Part 6, Section 200 http://www.ojjdp.ncjrs.gov/grantees/pm/glossary.html

4. What Is Program Evaluation? There are many ways to define program evaluation Evaluation can be defined as the systematic process of collecting and analyzing data in order to: –Determine whether program objectives have been achieved; –Make a decision about the program; –Improve the delivery of services. Boulmetis, J., & Dutwin, P. (2002). The ABC’s of Evaluation. San Francisco: Jossey Bass.

5. Why Evaluate Your Program? Satisfy funding agency’s grant requirements Document program accomplishments Justify program Improve program activities and services

6. Why Evaluate a Program? Evaluation results provide useful feedback to: –Sponsors and Funders –Program Staff –Clients –Donors –The Larger Community

7. First Steps in Program Evaluation Construct a logic model Identify performance measures/outputs Identify outcomes

8. Logic Model: Definition “Tool used to visually describe the linkages between program goals, activities, and expected outcomes.” Definition from: http://www.jrsainfo.org/jjec/resources/definitions.html

9. Logic Model Conditions (Needs or Problems) ActivitiesOutputsOutcomes What community need will the project fill? (e.g., youth need for positive social outlets) What services will the project offer? (e.g., tutoring, after school sports, mentoring) How will you measure project services? (e.g., number of youth served) How will you know that the community need has decreased? (e.g., youth have improved attitudes, increased self-esteem; youth delinquency or substance use rates have decreased; youth school attendance improved)

10. Developing a Framework Conditions (Needs or Problems) ActivitiesOutputsOutcomes What community need will the project fill? (e.g., youth need for positive social outlets) What services will the project offer? (e.g., tutoring, after school sports, mentoring) How will you measure project services? (e.g., number of youth served) How will you know that the community need has decreased? (e.g., youth have improved attitudes, increased self-esteem; youth delinquency or substance use rates have decreased; youth school attendance improved) High rates of truancy among elementary students (K to 5th grade) After-school tutoring Services of a family support coordinator 1.Hours of tutoring 2.Number of youth tutored 3.Number of support sessions provided 1.Improved attitudes toward school 2.Improved self-esteem 3.Improved school attendance

11. ADVANCED MODEL Developing a Framework Advance Model will go here ASSUMPTIONS: Principles, evidence-based approaches, and theories that guide the program ENVIRONMENT: External and contextual factors that influence the program Sources: GAO-02-923 –Strategies for Assessing How information Dissemination Contributes to Agency Goals. GAO-03-9 –Efforts to strengthen the link between resources and results at the administration of children and families. GAO/GGD-00-10 Managing for Results: Strengthening Regulatory Agencies' Performance Management. Ellen Taylor-Powell, 2000. “A logic model: A program performance framework.“ University of Wisconsin-Cooperative Extension Program Evaluation Conference. Situation Analysis Priority Setting INPUTACTIVITIESOUTPUTSOUTCOME IMPACTS Problem Identification Mission Goals Resources and Contributions. Tasks conducted by grantee’s staff, subcontractor, or volunteers. Activities are directly linked to outputs. Products and services delivered. Changes in individuals, agencies, systems, and communities. Outcomes may be intended or unintended. Initial Learning Awareness Knowledge Attitude Skills Opinions Aspirations Motivations Intermediate Action Behavior Practice Policies Social Action Decision- Making Long-Term Conditions Social Economic Civic Environmental ADVANCED MODEL

12. Why Develop a Program Logic Model? Clearly identifies program elements — goals, inputs, activities, outputs, and outcomes Shows how program elements fit together Illustrates the logic of how your program works Helps specify what to measure in an evaluation

13. Why Develop a Program Logic Model? Builds a common and shared understanding of the program’s goals, structure, need for resources, and expected results among program stakeholders; Provides a valuable and concise tool when presenting on the program to other groups; Provides a record of your program.

14. How to Construct a Program Logic Model There are several ways to do so: Convene a meeting of your program’s stakeholders –Develop the model as a group. (Building the model helps build consensus and support for your program.) Program staff develop the logic model –Meet separately with various stakeholder groups to elicit their perceptions and expectations. Different stakeholder groups may hold different views about the program’s activities and expected results.

15. Where to Begin in Constructing a Logic Model A program logic model is a type of flowchart –One may begin constructing the model at either end of the chart. Try to construct a program logic model as early as possible during program planning. A program may be up and running before a logic model is developed. A logic model is a living document –Revisions may be necessary as the program changes.

16. Next Steps in Program Evaluation Once you have the logic model: Decide on your evaluation questions (They are related to the goals of your program) How effective is your program ?

17. Moving From Goals to Evaluation Questions Project: After-school recreation program designed to promote self- esteem and reduce delinquency Evaluation Questions: Do youth who complete the program show increased self- esteem and decreased likelihood of delinquency? Program Goals: Increase youth protective factor of self- esteem. Lower arrest rates among program youth. Evaluatio n Goal: Determine whether youth served increase their self-esteem and decrease delinquency

18. Types of Program Evaluation Process Evaluation –How does the program work? –What services are provided? –Is the program implemented as planned? –What are the outputs? What works or does not work in the program?

19. Types of Program Evaluation Outcome Evaluation –Assess program effectiveness –What are the results? Short term Long term

20. Next Steps in Program Evaluation Choose the type of evaluation to conduct: Decide on the data to be collected to monitor performance measures/outputs and outcomes Create instruments needed to collect data Decide how to analyze and use data

21. Data Collection Need up-front planning Need a sense of what you are trying to accomplish What data will you collect and why? What data sources are available and which will you use?

22. www.evaluationtools.org/tools_main.asp Types of Data Collection Tools Activity Logs/ Skill Sheets Written documentation of participant's attendance, achievement, or acquisition of skills Good for “What?” and “How many” questions Document/Records review Review of written documents such as performance ratings, program logs, tally sheets, and other existing indicators Good for “What?” and “How many?” questions Focus groupsModerated discussions on a particular topic or issue Good for “What?” “How?” and “Why?” questions InterviewsData collection through oral conversationsGood for “What?” and “Why?” questions ObservationWatching people engaged in activities and recording what occurs Good for “How?” “What?” and “How many?” questions Questionnaires Written responses to clearly defined questions Good for “What?” and “How many?” questions

23. Create Instruments Construct your own instruments –Attendance sheets –Activity logs –Surveys for clients Use existing instruments –Templates available online –Modify to suit your particular program

24. Ways to Collect Data Paper ̶ Internal forms/checklists Electronic ̶ Excel, Access, SPSS Online ̶ Web-based data collection tools ̶ OJJDP’s Data Collection & Technical Assistance Tool (DCTAT)

25. Analyze and Use Data Analyze Data –How will the data be analyzed? –Who will be responsible for data analysis? Use Data –How will the data be used? –Will the data be made available in reports? –Who will receive the reports?

26. Evidence-Based Programs A program is evidence-based if: –evaluation research shows that it produces the expected positive results; –the results can be attributed to the program itself, not to other factors or events; –the evaluation is peer-reviewed by experts in the field; –the program is “endorsed” by a federal agency or respected research organization and included in its list of effective programs. From: Evidence-based Programs: An Overview http://www.uwex.edu/ces/flp/families/whatworks_06.pdf

27. The Importance of Evidence-Based Practices Government-wide move toward accountability Government Performance and Results Act –Shift from accountability for process to accountability for results –Programs must show effectiveness to justify funding President’s Management Agenda Evidence-based practices are cost effective –It is important to provide programs that work

28. Program Evaluation: Success and Sustainability Use data to create interest and support for program continuation –From funders –From the community Program evaluation helps identify the most successful components of your program –Maintain effective staff, procedures, or activities

29. Conclusion Program evaluation is an ongoing process Start with a logic model to identify the main elements of the program Choose the type of evaluation needed Plan data collection and analysis activities in advance Use program evaluation results to justify and market the program

30. Resources The Program Manager’s Guide to Evaluation Office of Planning, Research and Evaluation Administration for Children and Families U.S. Department of Health and Human Services http://www.acf.hhs.gov/programs/opre/other_resrch/pm_guide_eval Planning a Program Evaluation University of Wisconsin Cooperative Extension http://learningstore.uwex.edu/assets/pdfs/G3658-1.pdf The Global Social Change Research Project http://gsociology.icaap.org/methods/evaluationbeginnersguide.pdf Boulmetis, J., & Dutwin, P. (2002). The ABC’s of Evaluation. San Francisco: Jossey Bass.

31. Contact Information Monica Robbers, Ph.D. Patricia San Antonio, Ph.D. CSR, Incorporated 703.312.5220 mrobbers@csrincorporated.com psanantonio@csrincorporated.com