 How do we begin?  What are the different types of evaluation research?  How do these different types fit together?  What purpose do they serve?

Step 1 – Start with Program Model Step 2 – Identify Resources Step 3 – Identify Inputs Step 4 Identify Activities Step 5 – Identify Outputs Step 6 – Identify Outcomes

Step 1 – Identify Program Model Step 2 – Identify Inputs Step 3 - Activities Step 4 - Outputs Step 5 - Outcomes Step 1 – Identify Program Model Step 2 – Identify Resources Step 3 – Identify Program Events Step 4 – Identify Resulting Products Step 5 – Identify Changes Success Other Common Terms Used to Identify Steps

Program Model InputsActivitiesOutputsOutcomes Program Model Teachers Grant Money Service Learning Activities Service Learning Projects Completed Measure Leadership Skills

Program Model # of Teacher Amount of Money for Building Materials Number of Service Learning Projects Planned by Students Number of Projects Completed Perception of Leadership Skills Developed Program Model Teachers Grant Money Service Learning Activities Service Learning Projects Completed Leadership Skills

Resources Step One – Identify and count resources used to instigate program Activities Step Two – Identify and count activities that took place Outputs Step Three – Identify and count events or products that resulted Short-Term Outcomes Step Four – Identify and count short-term changes that took place (i.e., positive changes or accomplishments immediately after participation) Long Term Outcomes Step Five – Identify and count long-term impacts that occurred (i.e., positive changes or accomplishments at specified future date)

Resources Count number of staff and amount of money used to instigate program. Count number of networks established. Activities Count number of planning meetings and number of service learning activities planned by students. Count number of hours students spent in activities. Outputs Count number of projects completed by students. This could be subdivided into things such as number of picnic tables built for a park etc. Short-Term Outcomes Use survey to assess positive changes or accomplishments as perceived by service learning participants. Compare to control group. Long Term Outcomes At a specified future time, use survey to assess positive changes in the lives of students (i.e., number of hours spent in volunteer work etc.) Compare to control group

Resources Count number of staff and amount of money used to instigate 4-H program. Count number of networks established. Activities Number of 4-H activities that occurred, and/or number of students who participated in planning activities Outputs Number of students who participated in workshops and projects. Number of educational packets distributed. Short-Term Outcomes Assessment of their sense of accomplishment using survey. Assessment of increase in knowledge using survey. Long Term Outcomes Number of 4-H students who attend college after attending educational workshops. Compare to control group.

Outcome Evaluation Process Evaluation Formative Evaluation Needs Assessment

Example - Surveys, intensive interviews and focus groups are used to determine (1) Long-term impact on USU students and (2)Long term - impact on high school Outcome Evaluation Process Evaluation An ongoing evaluation is used to monitor the program Example -Grantee comes from time to time to “see how things are going”. Formative Evaluation Information is gathered to formulate a plan to establish a program. Example-Planning meetings with high school administrators etc. Needs Assessment A need for a program is identified Example –need for mentoring program with public school students An Example of Each Type of Evaluation

 Purpose  Evaluate the need for a program/intervention/ project  Basic Steps  Use secondary data (i.e., census data, community reports, etc.) and/or surveys, etc. to create a profile of the community that assess existing resources and current needs  Determine if current resources are adequately meeting needs  Identify one or more specific needs that are not currently being met

 Purpose  Determine the percent of local post secondary students that are from an underrepresented population. Compare this to the percent of students in post secondary educational institutions. Based on this and programs already available determine the need to provide additional opportunities for this group.  Basic Steps  Use secondary data to determine percent of population that is Latino. Use USU data to compute the percent of USU that is Latino. Compare differences in percentages. Note any discrepancy.  Utilize local webpages and phone books, identify programs/organizations that focus on the Latino population and meeting their needs.

 Purpose  Plan (form) a program that will meet the needs of the community, keeping in mind the need to develop indicators that can be used to evaluate the success of the program  Basic Steps  Locate other similar programs to use as a model  Meet with “experts” in this area and get their advice  Organize a program and apply for a grant  Form an “ad hoc” evaluation committee to evaluate program

 Purpose  Plan a mentoring program where USU students mentor high school students. Identify objectives (i.e., increase high school graduation rates, increase college retention rates of mentees).  Basic Steps  Identify other mentoring programs  Meet with directors of programs, professors, etc. who have been/are involved in mentoring programs  Organize your mentoring program and apply for a grant to implement program  Select members of your committee to engage in evaluation research

 Purpose  To monitor the implementation of your program on an ongoing basis. It is a type of program monitoring.  Steps  Specifically describe your program  Decide what products to count  Consider objectives (verb) (target) (date)  Decide on valid ways to count products  Identify process indicators  Quality assurance  Identify process indicators

 Purpose  Monitor the mentoring program and address two questions: (1) Is the mentoring occurring as planned? (2) How are the mentees responding?  Steps  Describe program  The mentoring program connects high school mentees with USU mentors through a monitored blog  Decide what products to count  The number of hours that mentors spent in contact with their mentee, etc.  Decide on valid way to count products  Have program administrator count hours spent on website  Quality assurance  High school counselors will be contacted periodically to determine how mentees are responding

 Purpose  To determine how well you meet the objectives of your program  Steps  Identify short-term and long-term outcomes  Construct indicators to measure outcomes  Decide on data collection and sampling techniques  Collect data  Analyze data  Write Report

 Purpose  To determine how well you meet the objectives of your program  Steps  Short-term outcomes are to establish relationship between mentors and mentees  Long-term outcomes are higher graduation and retention rates for mentees.  Mentees will be asked questions about their experience, and then about their graduation/retention  Surveys will be administered to all mentees after completion of program and then again in three years  Surveys are administered  Using SAS, data is entered and analyzed  Report is written

Inform Administrators and Clients Rely on “Expert” Opinion Inform AdministratorsRely on Statistics Quality Assurance Legislative Mandate Client Satisfaction is Focus Outcome Evaluation Not Mandated Does Not Necessarily Focus on Client Satisfaction

Questions or comments? Please contact: Carol Albrecht Assessment Specialist USU Extension