Evaluating the Impact of MEP Programs and Services Presented by: John Wight Program Manager Georgia Department of Education Migrant Education Program & Omar Lopez-Nunez State Research & Development Coordinator Migrant Education program Title I Conference June 19-20, 2014 11/22/2018

Evaluating the Impact of MEP Programs and Services Overview: Evaluation Definition & Background Information Statewide Service Delivery Plan Evaluation of Services Current Impact of MEP Program and Services as Measured by Evaluations 11/22/2018

Program Purpose The purpose of the MEP in Georgia (and the United States) is to ensure that migrant children fully benefit from the same free public education provided to all children and that the unmet education-related needs resulting from their migrant lifestyle are met. 7/26/2011

Legal Reference for Program Evaluation Statute: Part C of Title I of the Elementary and Secondary Education Act (ESEA) of 1965 – Sections 1304 (b) and 1306 (a) Code of Federal Regulations: 34 CFR 200.83 Guidance: Non-Regulatory Guidance, October 2012, p. 55-64 11/22/2018

Legal Reference for Program Evaluation Under Title I, Part C, Education of Migratory Children of the Elementary and Secondary Education Act, State educational agencies (SEA) must deliver and evaluate Migrant Education Program (MEP)-funded services to migratory children based on a State plan that reflects the results of a current statewide comprehensive needs assessment. 11/22/2018

Georgia MEP Program Evaluation Section 1304(c)(5) of the Elementary and Secondary Education Act (ESEA) requires states to conduct a program evaluation for the Migrant Education Program. The purpose of conducting an evaluation of the Georgia Migrant Education Program is to examine program effectiveness and results of implemented program activities. 11/22/2018

Georgia MEP Program Evaluation The Georgia MEP will conduct a statewide summary evaluation of all program projects and services based on the guidance and suggestions delineated in the U.S. Department of Education’s MEP Program Evaluation Toolkit at the end of every academic year. In order to prepare for, conduct, and report on a statewide evaluation plan of the Georgia Title I, Part C Migrant Education Program, the Georgia MEP will: 11/22/2018

Georgia MEP Program Evaluation Conduct a review of current and existing data, data sources and related reports generated by the state, regional, and or local migrant projects; Disaggregate statewide assessment data and compare the progress of migrant students with non-migrant students; disaggregate statewide assessment data and compare migrant PFS students with non-PFS and non-migrant students; Review and compare the performance of migrant PFS students and migrant non-PFS students within the national Government Performance and Results Act (GPRA) measures; Conduct on-site visits at the local project levels for the purpose of conducting implementation plan observations and gathering additional facts and information relative to project plans and implementation fidelity; and Analyze the information gathered and create a written evaluation incorporating implications and recommendations for overall program improvement, marking the close of the Continuous Improvement Cycle (CIC) for the Georgia MEP. 11/22/2018

MEP Evaluation Cycle: An Ongoing Process Georgia Continuous Improvement Cycle (GCIC): 11/22/2018

Georgia MEP Continuous Improvement Cycle (GCIC) The strategies for delivery of services have been determined by setting a three-step project planning process that every LEA must follow in order to ensure fidelity in the Continuous Improvement Cycle (CIC) of the Georgia MEP. The three steps in the project planning process for LEAs involve: Submission of a CNA profile that captures the needs of the migrant population at the LEA level; Submission of implementation plan(s) in order to establish academic support services to be provided with projected measurable outcome(s); Observations of implementation plans and services to monitor fidelity of implementation (GaDOE MEP and LEA staff); completion of surveys by SSPs to monitor fidelity of implementation; and Completion of implementation plan evaluation(s) for each implementation plan submitted at the end of project cycle in order to validate actual measurable outcome(s) as projected on original implementation plan(s) submitted. 11/22/2018

Georgia MEP Continuous Improvement Cycle (GCIC) - Continued 11/22/2018

Program Evaluation Defined Evaluation is the systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program (Rossi & Freeman, 1993; Short, Hennessy, & Campbell, 1996). 11/22/2018

Evaluation Consideration Effective evaluation is not an "event" that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors. W.K. Kellogg Foundation Evaluation Handbook, p. 3 11/22/2018

Why do we conduct evaluations? Establish model programs and best practices by providing feedback about what worked and what failed. Tool of good management and quality improvement - gain insight into effective strategies on how to improve performance. Measures impact the program is making. 11/22/2018

Service Delivery Plan (SDP) 2013 Statewide Service Delivery Plan (SDP) Continuous Improvement Cycle (CIC) for the Georgia MEP Service Delivery Evaluation 2013 Statewide CNA SDP Report

Service Delivery Plan (SDP) – Overview As required under Section 1306 of the reauthorized ESEA, the Georgia Migrant Education Program (Georgia MEP) has developed a statewide Service Delivery Plan (SDP) to be implemented during the 2013-2016 school years. This state plan is a current and comprehensive plan for how the services provided by the Georgia MEP and local educational agencies (LEAs) are to be delivered in order to meet the needs of the migrant children and youth throughout the state. 11/22/2018

Service Delivery Plan (SDP) – Purpose (Continued) Design Georgia programs to help migrant children and youth overcome educational disruption, cultural and language barriers, social isolation, various health-related problems, and other factors that inhibit their ability to do well in school, and to prepare them to make a successful transition to postsecondary education or employment; and Ensure that migrant children and youth benefit from the state of Georgia and local systemic reforms. 11/22/2018

MEP Service Delivery Plan Goals Goal 1: Migrant students …will improve their writing proficiency ……. Goal 2: Migrant students … will improve their math proficiency …. Goal 3: The Georgia Migrant Education Program will improve school readiness … Goal 4: The MEP …will continue to support … OSY/DO at the district level….to foster English language acquisition, health and other relevant supplemental services… Goal 5: Migrant students…will continue to meet and/or exceed their proficiency in reading …. Goal 6: Georgia MEP SSP staff …will improve their professional competencies when working with migrant participants for short periods of time … 11/22/2018

Performance Targets ESEA Flexibility Waiver Migrant is not a subgroup College and Career Ready Performance Index (CCRPI) State performance targets in elementary, middle and high school 11/22/2018

Migrant Students Performance Targets and Results The SDP will set the state performance targets for migrant students on par with the performance targets set for all students in CRCT Reading, English/Language Arts and Math (elementary and middle school level), EOCT in 9th Grade Literature, American Literature, Mathematics I and Mathematics II (high school level) and graduation rate performance targets in Georgia. So, as the all-student population makes progress (as established in the performance targets), so will migrant students statewide. The State MEP and LEA MEPs will analyze the progress of migrant students as compared with non-migrant, and migrant Priority for Service (PFS) students as compared with migrant non-PFS and non-migrant students on local and state formative and summative assessments. 11/22/2018

Sample CRCT Assessment Migrant vs. Non-Migrant 3rd Grade 2011   Migrant Meets and Exceeds Non-Migrant Meets and Exceeds GAP CRCT Reading 90.85% 94.43% 3.57% CRCT ELA 82.62% 89.13% 6.51% CRCT Math 75.68% 81.21% 5.54% 4th Grade 80.19% 87.92% 7.72% 76.85% 88.03% 11.18% 73.44% 81.10% 7.66% 11/22/2018

Migrant Students Performance Targets and Results State-level academic performance data for migrant students is used by the Georgia MEP to ensure migrant student academic progress follows that of the all students’ performance targets in the state. These data are also used by the Georgia MEP to develop program policy and to target specific migrant education projects and interventions that will increase the academic achievement and success of migrant children and youth statewide. 11/22/2018

Current Impact of MEP Program and Services as Measured by Evaluations from 2012-2013 School Readiness and Preschool K-12 Enrolled Children Out of School Youth 11/22/2018

Measuring Fidelity 2013-2014 Completed by SSPs and Tutors After Observation Planning and Delivery Student Engagement 11/22/2018

Measuring Fidelity 2013-2014 Application of Professional Development: Provided by SSPs and Tutors After Observation Application of Professional Development: PDNow!, RESA, school level Conference or workshop MEP staff Identifying the Instructional Strategy Guided reading, direct instruction, modeling Manipulatives, visual aids, hands on activities Teacher guided questioning 11/22/2018

Reporting Evaluation Results The level and scope of content depends on to whom the report is intended (OME, State, LEAs staff, Parent Groups – PAC and CNA) Be sure internal staff have a chance to carefully review and discuss the report to ensure results and activities are followed. Translate recommendations to action plans, including who is going to do what about the program and by when. Include professional development to address MEP staff instructional duties and responsibilities Create a specific plan to observe MEP funded services in an ongoing manner; making changes to plans based on observations (fidelity of implementation). 11/22/2018

Reporting Evaluation Results Summarize the results in an executive summary; summary should include a an interpretation of migrant student performance on local and State formative and summative assessments (a list of test scores or grades is not helpful without this interpretation); Be sure to record and store the evaluation plans and results in a location where it can referenced when a similar program evaluation is needed in the future. 11/22/2018

Pitfalls to Avoid in Evaluations There is no "perfect" evaluation design. Don't worry about the plan being perfect. It's far more important to do something, than to wait until every last detail has been tested. Don't report just the successes. You'll learn a great deal about the program by understanding its failures too! Don't throw away evaluation results once a report has been generated. Results don't take up much room, and they can provide precious information later when trying to understand changes in the program. 11/22/2018

Questions? 11/22/2018

Georgia MEP on Twitter @GeorgiaMEP 8/2/2013

THANK YOU! Contact Information John Wight Program Manager Georgia Department of Education Migrant Education Program jwight@doe.k12.ga.us 404-463-1857 Omar Lopez-Nunez State Research & Development Coordinator Georgia Department of Education Migrant Education Program olopez@doe.k12.ga.us 404-463-1775 11/22/2018