1. Supplemental Educational Services (SES) Data Collection Process: Roles and Responsibilities of LEAs GaDOE Data Collections Conference August 17, 2011 Athens, GA Jessica Johnson-Operations Analyst

2. Agenda SES Evaluation Overview SES Data Categories Parent and Student Surveys Provider Surveys District Surveys SES Student Data Collection Data Collection Reports

3. SES Overview Supplemental Educational Services (SES) include academic assistance such as tutoring and remediation which are provided outside of the regular school day and designed to increase the academic achievement of students in low-performing schools. Title I Schools NI2 or higher, or NI1 schools exercising the Differentiated Accountability option, must offer SES.

4. SES Evaluation Overview GaDOE is required by Title I, Part A to: – Develop, implement, and publicly report on standards and techniques for monitoring the quality and effectiveness of services offered by approved providers. – Withdrawing approval from providers that fail, for two (2) consecutive years, to contribute to increasing the academic proficiency of students served. – Title I Section 1116(e)(4)(B)

5. Purpose of Data Collection GaDOE collects information from Title I Directors, parents, and students to evaluate SES providers, the SES program, and Choice implementation. Use data for SES provider report cards. Allows GaDOE to make program modifications based on survey data and remove SES providers that have not demonstrated effectiveness in increasing student achievement.

6. SES Data Categories Effectiveness Customer Service Customer Service Delivery Service Delivery Has the SES provider contributed to increasing student achievement? Has the SES provider contributed to increasing student achievement? Are stakeholders pleased with the program? Are stakeholders pleased with the program? Are LEAs and SES providers implementing and complying with SES rules and regulations? Are LEAs and SES providers implementing and complying with SES rules and regulations?

7. Service Delivery An evaluation of SES provider program implementation and compliance as defined by: An evaluation of SES provider program implementation and compliance as defined by: – Monitoring score – Survey data Surveys administered and collected from the following stakeholders: LEAs (Title I Directors). Parent(s) / Legal Guardian(s). Middle school students. High school students. SES providers. Monitoring Scores provided by: GaDOE Monitors LEAs

8. Customer Satisfaction Customer Satisfaction measures stakeholder perception of: Communication & interaction with the school system. Service delivery of a SES provider’s program. Satisfaction with a SES provider’s program. Perceived impact on student achievement. LEA monitoring reports. Surveys administered and collected from the following stakeholders: LEAs (Title I Directors). Parent(s) / Legal Guardian(s). Middle school students. High school students. SES providers. 8

9. Effectiveness A matched samples comparison group model* is conducted by an external evaluator, University of Georgia (UGA), for examining SES student test results. *According to the Center on Innovation and Improvement, this approach is one of the most rigorous methods for determining the effect of SES services on student achievement (Evaluating Supplemental Educational Service Providers: Suggested Strategies for States, 2nd Edition). Analysis of effectiveness will be based upon results from: Criterion Reference Competency Test (CRCT). Georgia High School Graduation Test (GHSGT). End of Course Tests (EOCT). 9

10. Categories for SES Customer Service Survey Items DirectorsParentsStudents Communication & Interaction with School System (5 Questions) 1.The provider responds to requests to participate in district fairs, town halls, and SES-related parent meetings. 6. The provider submits invoices only for services rendered. 7. The provider submits invoices for services rendered in a timely manner. 8. The provider is easy to contact. 9. The provider works collaboratively with the district to resolve any issues that arise. Compliance/ Service Delivery (14 Questions) 2. The provider begins serving students in a timely manner. 3. The provider develops goals for each student receiving services. 4. The provider furnishes a written description of how each student’s progress will be measured. 5. The provider submits monthly progress reports for each student. 6. Did the tutor ask your opinion about your child’s learning needs before beginning the tutoring sessions? 7. Were you given a copy of your child’s learning plan? 8. Did the tutor give you a copy of a progress report for your child’s work at least once a month? 9. Did your tutor keep you informed about your child’s progress? 10. Were you able to ask the tutor questions about your child’s lessons and progress? 6. Did your tutor give you a test before beginning your after-school lessons? 7. Did your tutor share with you a plan for your after-school lessons? 8. Did your tutor give you a copy of a progress report on your work at least once a month? 9. Did your tutor keep you informed about your progress? 16. Did your parent or guardian ever meet with your tutor to discuss your progress?

11. Categories for SES Customer Service Survey Items DirectorsParentsStudents Satisfaction (17 Questions) 10. Overall, this provider offers quality instructional services to students. 11. Overall, it is easy for our LEA to work with this provider. 12. I would recommend that this provider continue offering SES to students in Georgia. 8a. If you got reports at least once a month, were the reports easy to understand? 10a. If you asked questions, did the tutor answer these questions to your satisfaction? 11. Were the sessions the right length of time for your child? 10. Would you sign up your child to receive tutoring from this tutor again? 12. Overall, were you satisfied with the tutoring your child received? 14. If your child missed a tutoring session, was it easy to reschedule? 15. Would you recommend this program to other parents? 11. Were you able to ask your tutor questions about your lessons and progress? 11a. If you asked questions, did the tutor answer the questions to your satisfaction? 12. Were the sessions the right length of time for you? 13. Would you sign up to receive tutoring from this tutor again? 14. Overall, were you satisfied with the tutoring you received? 15. If you could, would you like to receive more help from this tutor? 17. Would you recommend this program to another student? Impact (2 Questions) 13. Did your child do better academically in school after working with this tutor? 10. Did your grades in school improve after you started after- school tutoring?

12. Categories for SES Customer Service Survey Items DirectorsParentsStudents Monitoring (9 Questions) a. The provider’s instruction reinforces the LEA’s instructional program. b. The provider's instructional program is aligned with the Georgia Performance Standards. c. The provider develops a learning plan for each student. d. The provider's instruction is individualized for each student. e. The provider gives positive reinforcement to each student. f. The provider gives ongoing feedback to each student. g. Provider’s instructional materials are appropriate for student skill levels. h. The provider's instructional program is appropriate for students with limited English proficiency, if applicable. i. The provider offers appropriate SES instruction for students with disabilities (students with an IEP or 504 plan), if applicable.

13. Parent and Student Surveys Paper surveys Distributed by District Title I Directors All parents of students for which providers billed hours of service Middle and High School students for which providers billed hours of service English and Spanish versions Two week window in April Returned to UGA Program Evaluation Group

14. Parent and Student Surveys Parents of SES Students – 2,187 surveys (16.6% return rate) – Responded to questions about 80 SES providers across 55 school systems – 133 parents (6.1% returned a survey in Spanish) Middle and High School SES Students – 3,305 surveys (29.2% return rate) – Responded to questions about 77 different SES providers across 51 school systems.

15. Parent and Student Surveys Effective LEA practices: Maintain list of students receiving SES throughout the school year Develop clearly defined system for distribution and collection that works for your district Recruit one person to serve as SES contact for each SES school for collection purposes Include name of provider on parent/student surveys before distribution

16. Provider Surveys Providers surveyed about overall experience with LEAs SES Providers – 342 surveys from 57 of 92 providers (62% of SES providers) – 74 school systems represented

17. Findings from Provider SES Surveys According to Providers, Systems do a good job: – Clearly outlining contract obligations – Entering into contracts with providers in a timely manner – Treating providers fairly – Having clear policies about the use of facilities 57.6% of providers say they are allowed to use school facilities

18. District Surveys Districts serving SES in one or more schools complete the following: – Provider Satisfaction Surveys Questionnaire about overall experience with each provider – System survey Report district level data about SES and Choice – SES Student Data Collection Collect individual student level data for students receiving SES

19. District Survey Response Rate Title I Directors – 465 surveys from 61 school systems – 95% of systems with schools that used SES providers – 90 different SES providers represented

20. Provider Satisfaction Surveys What information is requested? – Complete a survey for each contracted provider in the system. Why is this information requested? – Provides GaDOE with information on the district perspective of SES provider compliance and interaction with the district, service delivery, and satisfaction with the SES provider. Results in program modifications and/or follow-up with individual providers. – Data is used in the SES provider report cards.

21. Provider Satisfaction Surveys How are surveys disseminated & collected? UGA Program Evaluation Group provides Title I Directors with a website link to access surveys. Title I Directors complete a survey for each applicable provider on the website. Title I Directors submit each completed survey directly to UGA.

22. System Survey What information is requested? Provide aggregate information on district and school level eligibility and participation. Why is this information requested? Provides GaDOE with information to track trends as well as information to provide on US Department of Education reports.

23. System Survey How are surveys disseminated & collected? UGA provides Title I Directors with a website link to access surveys. Title I Directors complete the survey on the website. Title I Directors submit the completed survey on the website.

24. System Survey Number of Title I schools required to offer Choice Number of students eligible for Choice Number of students whose parents requested a transfer Number of students who were offered the opportunity to transfer Number of students transferring Number of public and charter schools to which students transferred Amount spent on transportation for transferred students Number of Title I schools in your system required to offer SES Number of students eligible for SES (district and by school) Number of students whose parents requested SES (district and by school) Number of students who received SES (district and by school) Number of SES providers serving students Amount of Title I funds paid to all SES providers Per-pupil allotment Amount of the 20% obligation for SES and Choice Amount actually set-aside for SES and Choice

25. SES Student Data Collection What information is requested? Information on SES participation for every participating student. Why is this information requested? Allows GaDOE to determine which providers are effective in increasing student achievement. After the UGA evaluation, data is used for the SES provider report cards.

26. SES Student Data Collection How are surveys disseminated & collected? Title I Directors must access the GaDOE Portal and receive approval to receive the survey. Once approval is provided, Title I Directors will enter the required information for each student participating in SES. Once all student information is entered, Title I Directors submit the data on the GaDOE Portal.

27. SES Student Data Collection The following information is pre-populated by GaDOE: System name & code School name & code Student last & first name Student GTID Student grade The LEA provides the following information: Provider name Provider code Date services began (general date) Date services ended (general date) Subjects of service: ELA, reading, math, science, or social studies Provider hourly rate Total expended on the student What information is requested?

28. Data Collection Reports GaDOE publishes the following reports based on data collected from LEAs: – SES Provider Report Cards – SES Study of Academic Achievement – Survey Results for SES Providers http://www.gadoe.org/tss_title_parent.aspx? PageReq=TSSTitleSES http://www.gadoe.org/tss_title_parent.aspx? PageReq=TSSTitleSES

29. QUESTIONS?

30. Contact Information Jessica Johnson Operations Analyst Office of School Improvement Georgia Department of Education jejohnson@doe.k12.ga.us (404) 657-9864