1. PP450: Program Evaluation Faculty Sheila Toppin, DPA (abd), MPA

2. Welcome to the Class Greetings Availability ◦ Office hours: Monday & Thursday 6:00 - 8:00 pm EST ◦ AIM ID: Sheila Toppin ◦ Telephone ◦ Appointments: send an email message Faculty & Student Introductions ◦ Personal Triumphs

3. Course & Syllabus Info Units (10 weeks), Wednesday to Tuesday Discussion Board – 1 to 2 discussions/Unit Seminar meets every Monday at 8 PM EST Assignments Announcements: Informational & Reminders Last day to request an Incomplete: 8/6/12 Last day of class: Tuesday, 8/14/12

4. Course & Syllabus Info Assignments: ◦ Unit 7, Needs Assessment ◦ Unit 9, Final Project Discussion Board (25 pts ea) Seminars (10 pts ea)

5. Unit 7 Assignment: Needs Assessment Doc on Wheels Case Study Develop a program logic model for the “Doc on Wheels” program. Use the background material given for the “Doc on Wheels” program and design a flow chart to illustrate a program logic model. Include program inputs, activities, outputs and outcomes (initial, intermediate and long-term). Analyze the elements of the flow chart: How does the logic model inform contribute to a needs assessment of the “Doc on Wheels” program? What are the specific inputs, activities and outputs of the program? In what ways can this logic model assist the Hospital? If the Hospital chooses to share it with the City Council, in what ways can it assist the local government? What is the intended casual linkages of the “Doc on Wheels” program as illustrated by your program logic model.

6. Unit 9 Final Project Drug Court Case Study You are to design an evaluation plan to address the following: Qualitative concerns: program effectiveness, appropriate allocation of resources, necessary implementation improvements Quantitative concerns: impact on public safety, impact on recidivism, impact on employment outcomes. The design of your evaluation plan should have at a minimum these three parts: a clear articulation of the question(s) the evaluation seeks to answer, a section on the methods you propose to get the answers to the questions, a section that clearly outlines what you would propose to do with the answers to the questions. Your evaluation plan should include baseline information about the effectiveness of drug court programs in other localities, graphical data, and prediction of the outcomes of your evaluation and the possible policy changes that would result from your evaluation.

7. Course & Syllabus Info Con’t. Plan B Seminar: Live (Chat) in classroom Reading assignments are VITAL (eBook Chapters & Web Articles located via Reading Icon or DocSharing tab in classroom.) Assignments: Upload to DropBox in classroom Assignment Feedback: Grading Rubrics located in DocSharing tab in classroom. Late Policy (assignments are accepted up to 3 weeks from the due date.)

8. Unit 6 To-Do List Readings (text & web sources) Discussion Board Seminar (See the To-Do Checklist PDF that is available in the classroom to keep track each Unit) Questions??

9. What is Program Evaluation?

11. Key Concepts Program: a group of related purposive activities that is intended to achieve one or several related objectives. Program evaluation: a systematic process for gathering and interpreting information intended to answer questions about a program. Stakeholders: Individuals, groups, or organizations having a significant interest in how well a program functions (decision-makers, funders, sponsors, administrators and personnel, and clients or intended beneficiaries).

12. Stakeholder Engagement Stakeholder involvement is based upon the belief that expertise does not lie solely with program professionals. Stakeholders are persons or organizations that have investments in the content of a program, or in the dissemination and evaluation of a program (Centers for Disease Control and Prevention, 1999). Over the last several years the interpretation of stakeholder involvement has changed as programs have focused not just on individuals and families, but the broader ecology including neighborhood, workplace, schools, places of worship, communities and the society.

13. Stakeholder Engagement Stakeholders offer important insight into each phase of program planning, implementation and evaluation. Stakeholders are most commonly involved at the beginning stages of program planning. They are able to provide insight for the various needs that a program or curriculum should meet. Experience shows that once the goals have been set in the first part of program development, stakeholders are sometimes not consulted in latter stages of program implementation and evaluation. This is unfortunate because stakeholders have the potential to illuminate issues and needs during the course of program implementation (Banach & Gregory, 2001).

14. Stakeholder Engagement Frequently stakeholders who participate in an initial needs assessment may not be the same stakeholders who ultimately sustain the program. Consequently, eliciting on-going feedback and keeping lines of communication open are crucial to program success. This is particularly true in community-based youth development programs. A broad range of stakeholders has the knowledge, daily life experiences and expertise that can contribute to program success.

15. Stakeholder Engagement For example, work in the area of teen pregnancy prevention, for example, as in the general public health, social service and education fields, involves partnerships. Consequently, decisions regarding programs should include the considerations and perspectives of multiple stakeholders. Stakeholders include funders and administrators of programs, but also include staff, program participants and their peers, family members, and the wider community.

16. Stakeholder Engagement Program Evaluation should cast a big net to include diverse stakeholders: ◦ Race/ethnicity ◦ Ability/Experience ◦ Income ◦ Sexual orientation ◦ Education ◦ Politics ◦ Family constellation ◦ Marital Status

17. Stakeholder Engagement Robert Wood Johnson Foundation’s Guide for Engaging Stakeholders The RWJF is committed to engaging stakeholders based on the following: RWJF represents a public trust, thus the perceived impact of its work affects parties. Philanthropy is isolated from stakeholders’ diverse viewpoints and experiences, so it is important to seek out those perspectives systematically. RWJF seeks social change so it seeks the input of those with the power/resources to guide such change. Consulting stakeholders generally makes evaluations more relevant, credible and useful.

18. Step 1: Prepare for stakeholder engagement This step includes collecting information about the program or initiative being evaluated— its history why it came into being what it is trying to accomplish, and what success would look like.

19. Step 1: Prepare for stakeholder engagement Knowing the Background when the program or initiative started and who sponsored its development. the underlying assumptions about why the program or initiative exists. the resources allocated to the program or initiative. the activities the program or initiative undertakes to achieve its goals. the expected outputs and short- and long-term goals or outcomes of the program or initiative. the external forces that have affected or currently influence the program’s design and implementation.

20. Step 2: Identify potential stakeholders This step involves identifying all of the potential stakeholders whom you might engage in the evaluation question development process. Expertise Diverse perspectives and/or experiences Level of responsibility Position of influence Intensely interested in issue Proponents of the evaluation/ level of buy-in and support

21. Step 2: Identify potential stakeholders Stakeholders may be both internal and external to the program. Ex. Railroad Retirement Board

22. Step 3: Prioritize the list of stakeholders Due to feasibility or financial constraints it is necessary to rank or prioritization stakeholders: Going through a prioritization process should also be helpful in determining if any individuals or groups have been inadvertently overlooked. Reviewing the list of those stakeholders that are vital, important, or nice to include may reveal certain points of view that are well represented, and others that are not.

23. Step 3: Prioritize the list of stakeholders Rank Order: 1. Vital to the evaluation’s success and resulting use of findings 2. Important to include for practical or political reasons 3. Nice to include if possible given time and resources

24. Step 4: Consider potential stakeholders’ motivations for participating This step involves considering stakeholders’ motivations for participating in the question development process. Knowing this will help you select an engagement strategy.

25. Step 4: Consider potential stakeholders’ motivations for participating Issues Commitment to the Goals of the Program or Initiative Personal Stake in the Program or Initiative Professional Development Opportunity to Earn Additional Income

26. Step 5: Select a stakeholder engagement strategy Issues to consider to facilitate the identification and development of the evaluation’s questions: ◦ Time ◦ Budget ◦ Geographical locations ◦ Range of perspectives ◦ Existing relationships ◦ Stakeholder availability ◦ Number of Stakeholders ◦ Stakeholder familiarity with the evaluation ◦ Degree of complexity of the program and evaluation

27. Step 5: Select a stakeholder engagement strategy Methods for soliciting feedback from stakeholders: In-person interviews Telephone interviews Questionnaires (paper/online) Focus groups

28. Class Activity – Stakeholder Engagement Develop a Stakeholder Engagement Strategy for a Youth After School Program Stakeholders Elected city, county and state officials Religious leaders Business owners Neighborhood association members, Sports figures Coaches Students/out-of-school youth Parents Health and social service providers Educators Representatives of the art community, and any other interested person or group.

29. Class Activity – Stakeholder Engagement Youth After School Program Step 1: Prepare (Know the background) Step 2: Identify potential stakeholders Step 3: Prioritize the list of stakeholders Step 4: Consider potential stakeholders’ motivations for participating Step 5: Select a stakeholder engagement strategy Questions?

30. Unit 6 Seminar Summary Reviewed key concepts Defined stakeholder engagement Discussed stakeholder engagement purpose, strategic process, and challenges Class Activity – Stakeholder Engagement Youth After School Program Final Questions??? Dismissal