1. An Update on Florida’s Charter Schools Program Grant: CAPES External Evaluation 2014 Florida Charter Schools Conference: Sharing Responsibility November 19 th, 2014

2. Introduction of Presenters Principal Investigator: M. David Miller, Ph. D. Project Manager: Nan Thornqvist, Ed. D. Researcher: Wei Xu

3. The Charter Schools Program Grant and External Evaluation  The purpose of the USDE Charter School Program Grant (CSPG) is to increase the national understanding of the charter school model by: 1. expanding the number of high-quality charter schools by providing financial assistance for the planning, program design, and initial implementation of charter schools, and 2. by evaluating the effects of charter schools, including their effects on student academic achievement, staff, and parents

4. FL DOE IEPC Office: Objectives of the CSPG Objective 1: Increase access to high-quality charter schools for educationally disadvantaged students. Objective 2: Improve the authorizing practices and capacity of Local Education Agency Authorizers. Objective 3: Increase the number of high-quality charter schools in Florida. Objective 4: Increase the academic achievement of charter school students in Florida.

5. Objectives of the External Evaluation of the CSPG Objective 1: The external evaluation team will review IEPC activities and documents and report as to the annual progress regarding their process and outcome performance measures as established to obtain each of the four CSP grant objectives.

6. Objectives of the External Evaluation of the CSPG Objective 2: The CAPES evaluation team will help determine if the IEPC Office has the appropriate systems in place to sustain and grow a high-quality charter school sector in Florida by determining: 1) What are the strengths and weaknesses in the IEPC Office’s grant application and review processes? 2) How can the IEPC Office improve its CSP grant application and review process to better screen for quality? 3) What is the impact of CSP funding? 4) What is the impact of each of the dissemination grants funded by the grant?

7. Two Main Goals of Evaluation Examining, improving, and monitoring the grant application and application process Examining CSP funded schools

8. Selection of CSP Grant Awardees Large numbers of applicants each year for approximately 50-60 grant awards (152 applicants in 2011; 112 in 2012; 89 last year) All-encompassing online application; 2 grant stages (see handout #2) 2 grant stages (see handout #2)

9. Selection of CSP Grant Awardees (Cont’d) Ratings/scores based on multiple criteria Ratings/scores from a panel of five peer reviewers The high and low scores are dropped— average of the three scores is the final score

10. Evaluation Criteria 1) Validity of scoring criteria 2) Reliability of scores

11. Validity Content review of grant sections Consistency with models used in other states including grant sections and weighting/scoring

12. Reliability Dependability index from Generalizability Theory Dependability index from Generalizability Theory Absolute measurement Absolute measurement Allows complex data collection (5 raters with each application) Allows complex data collection (5 raters with each application) Allows dropping highest and lowest score Allows dropping highest and lowest score Consistency of ratings of participants (mean and variance of discrepancy from average) Consistency of ratings of participants (mean and variance of discrepancy from average)

13. Reliability 2013-14 AreaDependability – 3 (drop high and low) Project Need 0.607 Project Design and Implementation Founding/Governance 0.540 School Leadership 0.694 Business, Finance and Accounting 0.340 Special Populations 0.475 Curriculum and Instruction 0.443 Evaluation 0.596 Dissemination 0.520 Total 0.785

14. Implications for Grant Application Process Validity of process is high Reliability for decision (total) is high Continued monitoring and suggestions will continue to be made (regarding reviewers, grant sections, etc.)

15. CSP Grant Awardee Data Surveys (IEPC Office personnel, grant reviewers, grant applicants, site visited schools’ governing board members, principals, teachers, parents, community members) District interviews (five different districts each year) School site visits (25 schools selected each year; document review; direct observation of instruction and tour of facilities) Analysis of achievement data and additional research in discovered areas of concern (special populations, teacher attrition)

16. Factors Affecting Teacher Turnover in Florida Charter schools

17. Literature Review  Teacher attrition rates in charter schools across the country is between 20% and 25%. The teacher attrition rate for new teachers reaches 40% annually (Miron & Applegate, 2007).  Charter schools are negatively impacted by high teacher turnover:  Has a significant negative impact on academic achievement (Ronfeldt, Loeb,& Wyckoff, 2012).  presents a challenge for cultivating high-quality instructional environment (Stuit & Smith, in press).

18. Literature Review Factors associated with teacher turnover include:  Teacher background characteristics-e.g. age, gender, race/ethnicity (Miron & Applegate, 2007; Stuit & Smith, 2012);  School characteristics-e.g. geographical location, institution size (Stuit & Smith, 2012);  School personnel policies-e.g. union membership (Stuit & Smith, 2012);  Teacher quality-e.g. teaching experiences (Ingle, 2007).

19. Research Question  In Florida, CSP school site visits have revealed a concern about teacher turnover where many teachers are leaving charter schools midyear. Thus, there is a need to get updated information regarding factors that are associated with teacher turnover in charter schools including turnover within years as well as annual turnover.  To what extent are teacher salary, academic credentials, as well as years of experience associated with teacher turnover in Florida charter schools?

20. Data Source  Data pertaining to teacher salary, teaching experience, and attrition (2011-12, 2012-13) have been obtained from Florida Department of Education EIS (Education Information Services).  Teacher numbers were taken from Survey 2 (early fall) and were compared with Survey 3 numbers (late spring) to examine teacher attrition in Florida charter schools for those two years.

21. Descriptive Statistics 2011-12

22. Descriptive Statistics 2012-13

23. Descriptive Statistics (start of year) Teacher Salaries –Mean $33,175 in 2011-2012 –Mean $32,875 in 2012-2013 Teacher Experience –Mean 4.93 in 2011-2012 –Mean 4.97 in 2012-2012

24. Research Findings - Salary  Salary: 2011-12  Teachers who stayed had significantly higher salaries than those who left and those who replaced the teachers who left.  Replacement teachers had higher salaries than those who leave but not significantly.  Salary: 2012-13  Teachers who stayed had significantly higher salaries than those who left and those who replaced teachers who left.  Replacement teachers had significantly higher salaries than those who left.

25. Research Findings – Teacher Experience  Teaching Experience: 2011-12  Teachers who stayed had significantly higher experience than those who left.  New teachers had significantly less experience than teachers who stayed or teachers who left.  Teaching Experience: 2012-13  Similar conclusions could be drawn.

26. Research Findings – All variables  Combining analyses to examine the joint effect of salary, educational level, and years of experience.  For 2011-12, teachers with more years of working experience, a bachelor degree, and a higher salary are more likely to stay in charter schools  For 2012-13, teachers with more years of working experience and higher salaries are more likely to keep working in charter schools

27. Summary IEPC Office has a working CSPG application and review process in place Schools are new-need time for programs to show effects Data collection and monitoring will continue

28. Contact Information M. David Miller The University of Florida and CAPES dmiller@coe.ufl.edu Nan Thornqvist The University of Florida and CAPES nanthorn@coe.ufl.edu (352) 538-5187 Wei Xu Wei Xu The University of Florida and CAPES xuweiufl@ufl.edu