Supplemental Educational Services Evaluations Data Collection Process Allison Potter Steven M. Ross Center for Research in Educational Policy The University of Memphis

Evaluation Questions for Data Collection What are the activities and experiences of State Educational Agencies (SEAs) and Local Educational Agencies (LEAs) with SES implementation? What are the perceptions and experiences of principals/site coordinators, teachers, and parents with SES interventions? Are provider services raising student achievement in mathematics and reading/language arts?

SEA/LEA Data Collection Conference calls with SEA during preliminary evaluation process Non-survey In-person meeting with SEA at State Department of Education (SDE) Conference calls and WebEx meetings with LEAs and phone correspondence with SEAs and LEAs throughout evaluation year End-of-evaluation meeting at SDE with SES coordinators, legal counsel, assistant superintendent, and data management department heads

Surveys District/Local Data Collection Common core set of questions from all groups to permit triangulation Online survey system for LEAs, principals/site coordinators, and teachers Login information ed to LEAs Login information mailed to schools with parent surveys Paper-based surveys for parents Mailed (FedEx) to schools for distribution

Selected survey questions: Data Collection Tools What was the start date of provider services? In which subjects did your students receive services from this provider? Are you employed by the provider for which you are completing this survey? How often does the provider: Communicate with you during the school year? Meet the obligations for conducting tutoring sessions?

Selected survey questions: Data Collection Tools The provider: Adapted the tutoring services to this school’s curriculum Aligned their services with state and local standards Offered services to Special Education and ESL students Complied with applicable federal, state, and local laws

Selected survey questions: Data Collection Tools Overall assessment: I believe the services offered by this provider positively impacted student achievement Overall, I am satisfied with the services of this provider

Sample questionnaire responses to open ended question The program began much too late in the school year (after testing) to impact learning this year. I have never spoken to the instructors. I have no knowledge as to the structure of the program. Data Collection Tools Teachers: [Provider] never called his classroom teacher, never looked at student records, or coordinated efforts until finally his classroom teacher got through and spoke of learning problems. I saw great gains with the kids who were served by this provider – they benefited from this program.

Sample questionnaire responses to open ended question  I am so happy that he has this provider. She has done wonderful for him. He is so smart! Data Collection Tools Parents:  Please let me know my child’s progress. I haven’t received any progress reports or anything.  The provider tutored my child 14 hrs. instead of 27 hrs. Somebody has to follow up with the services.  [Provider name] program was the best for my child. I found out about the program on my own. School system didn’t help me in any way.

Provider survey selected questions: Data Collection Tools Describe the format of your services: Program duration Setting Format (small groups, individual) What is your general instructional plan? Describe qualifications of tutors (including data on background checks) List information regarding students served, goals achieved, and tutoring sessions attended

States will need to collect a large amount of data to evaluate SES providers, which may require a regional database that connects: Technology and Database Considerations Achievement data and related characteristics for all students who are eligible for SES Each student served by SES with a specific SES provider Details about the services offered by each SES provider

Student Data Collection Request SES student names, identifiers (SSN), and demographic information from LEAs Student achievement analysis State department data division adds test scores to SES student files (pretest and current year) LEAs send confidential student data to SEA Comparison group test scores are pulled and added to data file Student identifiers are removed (or encrypted) and sent to CREP for analysis

Sample Results by Rubric Outcome Category Free to Learn Achievement Analysis and Survey Results

Rubric of Overall Evaluation of Provider Effectiveness – Free to Learn Outcome Insufficient Information Below Standards Marginal Quality AcceptableAbove Standards 1. Student Achievement There is insufficient information available to determine student achievement outcomes. Students have not shown gains related to tutoring received from service providers. About half of the students have made some gain related to tutoring received from service providers. There has been some gain for the majority (over 60%) of students related to tutoring received from service providers. The effect size for students in the provider’s program is in the top one-third of all the effect sizes demonstrated by providers meeting standards for student achievement. 2. CommunicationThere is insufficient information available to determine communication outcomes. Provider has not communicated with the principals, teachers, and parents of students served. There has been limited communication throughout the year between the provider and at least two of the following: principals, teachers, and parents. There has been some regular communication throughout the year between the provider and the principals, teachers, and parents of students served. There is an ongoing and sustained system of communication between the provider and the school- level educators as well as parents of students served. 3. Instructional Plans There is insufficient information available to determine instructional plans of the provider. Provider does not plan instruction explicitly geared to student needs or to reinforce their regular academic program. Provider is in the planning stages of gearing instruction to student needs, and reinforcing the regular academic program. Provider has made some attempt with the majority of students to plan instruction explicitly geared to student needs and to reinforce the regular academic program. Provider instructional plans are explicitly geared to the needs of most or all students and reinforce the regular academic program. 4. Local and State Standards There is insufficient information available to determine alignment with local and state standards. None of the instructional plans used by the provider are aligned with local and state academic standards for students. Provider is in the process of aligning instructional plans with local and state academic standards for students. Some of the instructional plans used by the provider are presently aligned with local and state academic standards for students. Most or all of the instructional plans are presently aligned with local and state academic standards for students. 5. Special Ed/ELL students There is insufficient information available to determine special ed/ELL student outcomes. Provider does not offer accommodations for addressing the needs of special ed or ELL students. Provider has made limited accommodations for addressing the needs of special ed and ELL students. Provider has made some accommodations for addressing the needs of special ed and ELL students. Provider offers appropriate services, if needed, to special education and ELL students. 6. Provider Overall There is insufficient information available to determine provider overall outcomes. There is overall dissatisfaction with the provider at the district and school levels. There is more dissatisfaction than satisfaction with the provider at the district and school levels. There are mixed but mostly positive reactions about the provider at the school and district levels. There is overall satisfaction with the provider at the district and school levels.