CAMP Evaluation: Measurement of Strategy Implementation Office of Migrant Education United States Department of Education CAMP Technical Assistance Webinar 5 October

Background for the Webinar 2 We have some variation in our results.

Background for the Webinar 3  2010 GPRA Measure 1 Target = Not Met  National Target = 86% of students complete their first year in college  National Results = 85%  Performance results excluded first year (2009) projects  2010 GPRA Measure 2 Target – MET!  National Target = 85% of first year completers continue into a second year  National Results = 88%  Performance results excluded first year (2009) projects

Purpose of the Webinar 4  SHARE PRACTICES from Improving or High Performing Projects, so that…  IMPROVE CAMP results nationwide, so that we may effectively and efficiently assist Migrant students to successfully complete their first year, and continue into their second year.  CONTROL VARIATION so that our CAMP projects are all successful.

Today’s Objectives Review the meaning of program evaluation. Review the four components of program evaluation. Individual CAMP project directors present effective ways to measure strategy implementation. Questions and answers for our presenters. Leave our webinar with ideas to improve measurement of strategy implementation! 5

The Meaning of Program Evaluation Evaluation means systematically and methodically collecting information about a program in order to improve the program or make decisions about the merit or worth of the program. 6

Components of Program Evaluation 1. To Determine Overall Effectiveness in Meeting Program Goals/Objectives (Are we on track?) 2. To Determine the Level Program Activities Are Being Implemented (Did we really deliver what we agreed to deliver in the application? How do we know?) We need both! Now let’s hear from the audience why we need program implementation… 7

Components of Program Evaluation (cont.) 3. To Identify Strengths and Weaknesses in Program Implementation and Program Effectiveness (What strategies are working, and which ones are not?) 4. To Provide Recommendations for Change in Program, in Order to Improve Results (What can we do differently to change the results?) 8

Components of Program Evaluation 1. To Determine Overall Effectiveness in Meeting Program Goals/Objectives  Are program activities effective, leading to the achievement of strong GPRA 1 and GPRA 2 results and reaching these targets?  What evidence is available that supports the conclusions?  Are program activities cost efficient? 9

Components of Program Evaluation 2. To Determine the Level Program Activities Are Being Implemented (Prerequisite to ALL Evaluation Practices)  “With a level of quality, and all of the time!”  “With a level of quality, most of the time ”  “With a level of quality, some of the time …”  “We really didn’t have the time  ” 10

Components of Program Evaluation Why study level of implementation? It can help answer the questions: 1. Did our project fulfill its responsibilities in our application? 2. Did our project’s strategies make a difference in meeting our GPRA targets? 11

Components of Program Evaluation 3. To Identify Strengths and Weaknesses in Program Implementation and Program Effectiveness  Through customer satisfaction surveys (plus and delta, all services)  Through observations (e.g., tutoring, instruction, placement)  Through documentation (e.g., logs of recruitment, counseling, tutoring)  Through exit interviews  Through research, by finding correlations between practices and results (using student level data, discovering relationships between practices and achievement) 12

Components of Program Evaluation 4. To Provide Recommendations for Change in Program, in Order to Improve Results. Examples:  Tutoring targeted to specific skills (teachers/mentors/students)  Mentors with regular and structured meetings, specific supports  Individualized Education Plans with specific learning objectives and learning plans  Structured recruiting plan 13

Measurement of Strategy Implementation… How Do We Know We Implemented Each Service? CAMP Directors Present Ideas…  Viridiana Diaz, CSU – Sacramento  Carolina Gonzalez-Lujan, CSU – Monterey Bay  Gypsy Hall – Boise State University  Doris Roundtree – ABAC  Minerva Gonzalez, CSU – San Marcos  Josue Estrada – Washington State University 14

Measurement of Strategy Implementation… CSU-San Marcos, Minerva Gonzales  Mid-Semester Evaluations: Communication with Instructors/Advisor  Appropriate Interventions  Recruitment  Communication with Students: Support  Counseling 15

Measurement of Strategy Implementation Questions for the presenters, from the audience… 16

Summary Good evaluation will tell you:  If program activities are delivering results.  If program activities are being highly implemented.  Program strengths and weaknesses.  Suggestions for program improvement. 17

Program Evaluation THANK YOU! In order for OME to better serve your needs, please provide an evaluation of today’s webinar! Also, if you’d like to discuss your program’s evaluation process… Ed Monaghan