Evaluating SES Providers Steven M. Ross Allison Potter Center for Research in Educational Policy The University of Memphis

Effectiveness: Increased student achievement in reading/language arts or mathematics. Customer satisfaction: Positive perceptions by parents of SES students. Service delivery and compliance: Positive perceptions by principals, teachers, LEA staff, etc. Determining Evaluation Measures

Effectiveness (Student Achievement) Service Delivery and Compliance Customer Satisfaction Provider Survey District Coordinator Survey Principal/Liaison Survey Teacher Survey Parent Survey Additional Tests State Tests Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan Overall Provider Assessment

1. Student-level test scores from state-mandated assessments Effectiveness Measures Considerations: availability only for certain grades (e.g., 3-higher)? Lack of pretest scores prevents gains from being determined

2. Supplementary individualized assessments in reading/language arts or math Effectiveness Measures Considerations: Without pretest scores and comparison students, SES gain cannot be determined Validity may be suspect if assessments not administered by trained independent testers

3.Provider-developed assessments in reading/language arts or math Effectiveness Measures Considerations: Test results may not be valid or suitable for state’s evaluation purpose Tests may favor provider’s strategies

1. Parent and family perceptions Customer Satisfaction Measures Considerations: Parent respondents may not be representative of the population served by provider Samples sizes will vary due to provider size Comparisons limited due to parent familiarity with only one provider

2. Student perceptions Customer Satisfaction Measures Considerations: Young students may have difficulty judging quality of services and communicating impressions Time consuming and may require parent permission to obtain

1. Records of services provided, student attendance rates, and costs Service Delivery and Compliance Measures Considerations: States may obtain data from a variety of sources, including providers, teachers, principals, and district staff Corroborating data from multiple sources can increase accuracy of evaluation conclusions

2. Feedback from SES customers Service Delivery and Compliance Measures Considerations: First-hand impressions or observations may be lacking Translation may be needed to reach parents who do not speak English Obtaining representative samples may be difficult

3. Feedback from district staff Service Delivery and Compliance Measures Considerations: Districts may lack firsthand impressions or observations of tutoring services Some districts may also be SES providers

4. Feedback from school staff Service Delivery and Compliance Measures Considerations: Teachers may also be SES instructors or lack first- hand impressions of providers Teachers may need to provide information on multiple providers, which may be confusing and time consuming Identifying teachers to solicit responses may be difficult

States will need to collect a large amount of data to evaluate SES providers, which may require a regional database that connects: Technology and Database Considerations Achievement data and related characteristics for all students who are eligible for SES Each student served by SES with a specific SES provider Details about the services offered by each SES provider

A. Benchmark Comparison Rating = ++ (Low to Moderate rigor) Percentage of SES students by provider attaining “proficiency” on state assessment Evaluation Designs: Student Achievement

A. Benchmark Comparison Evaluation Designs: Student Achievement Upgrades Percentage of SES in all performance categories (“Below Basic”, “Basic”, etc.) Comparison of performance relative to prior year and to state norms Comparison to a “control” sample

A. Benchmark Comparison Evaluation Designs: Student Achievement Advantages Inexpensive and less demanding Easily understood by practitioners and public Linked directly to NCLB accountability Disadvantages Doesn’t control for student characteristics Doesn’t control for schools Uses broad achievement indices

B. Multiple Linear Regression Design Rating = +++ (Moderate rigor) Compares actual gains to predicted gains for students enrolled in SES, using district data to control for student variables (e.g., income, ethnicity, gender, ELL, special education status, etc.). Evaluation Designs: Student Achievement

B. Multiple Linear Regression Design Evaluation Designs: Student Achievement Advantages More costly than Benchmark, but relatively economical Student characteristics are statistically controlled Disadvantages Doesn’t control for school effects Less understandable to practitioners and public Effect sizes may be less stable than for Model C.

C. Matched Samples Design Rating = ++++ (High Moderate to Strong rigor) Match and compare SES students to similar students attending same school (or, if not feasible, similar school) Use multiple matches if possible Evaluation Designs: Student Achievement

C. Matched Samples Design Evaluation Designs: Student Achievement Advantages Some control over school effects Easily understood by practitioners and public Highest potential rigor of all designs Disadvantages More costly and time consuming Within-school matches may be difficult to achieve

Surveys for LEAs, principals/site coordinators, teachers, parents, and providers. Common core set of questions from all groups to permit triangulation. Choice of “frequently”, “occasionally”, “not at all”, and “don’t know” Open-ended question, “Additional comments” Data Collection Tools

Selected survey questions: Data Collection Tools What was the start date of provider services? In which subjects did your students receive services from this provider? Are you employed by the provider for which you are completing this survey? How often does the provider: Communicate with you during the school year? Meet the obligations for conducting tutoring sessions?

Selected survey questions: Data Collection Tools The provider: Adapted the tutoring services to this school’s curriculum Aligned their services with state and local standards Offered services to Special Education and ESL students Complied with applicable federal, state, and local laws

Selected survey questions: Data Collection Tools Overall assessment: I believe the services offered by this provider positively impacted student achievement Overall, I am satisfied with the services of this provider

Sample questionnaire responses to open ended question The program began much too late in the school year (after testing) to impact learning this year. I have never spoken to the instructors. I have no knowledge as to the structure of the program. [Provider] never called his classroom teacher, never looked at student records, or coordinated efforts until finally his classroom teacher got through and spoke of learning problems. I saw great gains with the kids who were served by this provider – they benefited from this program. Data Collection Tools Teachers:

Provider survey selected questions: Data Collection Tools Describe the format of your services: Program duration Setting Format (small groups, individual) What is your general instructional plan? Describe qualifications of tutors (including data on background checks) List information regarding students served, goals achieved, and tutoring sessions attended

Rubric of Overall Evaluation of Provider Effectiveness Outcome Insufficient Information Below Standards Marginal Quality AcceptableAbove Standards 1. Student Achievement There is insufficient information available to determine student achievement outcomes. Students have not shown gains related to tutoring received from service providers. About half of the students have made some gain related to tutoring received from service providers. There has been some gain for the majority (over 60%) of students related to tutoring received from service providers. The effect size for students in the provider’s program is in the top one-third of all the effect sizes demonstrated by providers meeting standards for student achievement. 2. CommunicationThere is insufficient information available to determine communication outcomes. Provider has not communicated with the principals, teachers, and parents of students served. There has been limited communication throughout the year between the provider and at least two of the following: principals, teachers, and parents. There has been some regular communication throughout the year between the provider and the principals, teachers, and parents of students served. There is an ongoing and sustained system of communication between the provider and the school- level educators as well as parents of students served. 3. Instructional Plans There is insufficient information available to determine instructional plans of the provider. Provider does not plan instruction explicitly geared to student needs or to reinforce their regular academic program. Provider is in the planning stages of gearing instruction to student needs, and reinforcing the regular academic program. Provider has made some attempt with the majority of students to plan instruction explicitly geared to student needs and to reinforce the regular academic program. Provider instructional plans are explicitly geared to the needs of most or all students and reinforce the regular academic program. 4. Local and State Standards There is insufficient information available to determine alignment with local and state standards. None of the instructional plans used by the provider are aligned with local and state academic standards for students. Provider is in the process of aligning instructional plans with local and state academic standards for students. Some of the instructional plans used by the provider are presently aligned with local and state academic standards for students. Most or all of the instructional plans are presently aligned with local and state academic standards for students. 5. Special Ed/ELL students There is insufficient information available to determine special ed/ELL student outcomes. Provider does not offer accommodations for addressing the needs of special ed or ELL students. Provider has made limited accommodations for addressing the needs of special ed and ELL students. Provider has made some accommodations for addressing the needs of special ed and ELL students. Provider offers appropriate services, if needed, to special education and ELL students. 6. Provider Overall There is insufficient information available to determine provider overall outcomes. There is overall dissatisfaction with the provider at the district and school levels. There is more dissatisfaction than satisfaction with the provider at the district and school levels. There are mixed but mostly positive reactions about the provider at the school and district levels. There is overall satisfaction with the provider at the district and school levels.

Decision Tree for SES Providers Probation I

CONCLUSION SES Evaluation models that are both suitably rigorous and practical for states to employ are still evolving. Each state has unique needs, priorities, access to resources, and procedures used to implement SES. States may face a trade-off between practicality concerns (cost and time) and rigor (the reliability and accuracy of findings).

CONCLUSION Each state should begin its SES evaluation planning process by identifying a)the specific questions that its SES evaluation needs to answer, and b)the resources that can be allocated reasonably to support further evaluation planning, data collection, analysis, reporting, and dissemination.

CONCLUSION Work through the hierarchy of evaluation designs presented here and select the design that allows the highest level of rigor. States may wish to engage third-party evaluation experts in helping to plan and conduct these evaluations.