1. Arkansas’ K-12 Achievement & NSLA Funding Arkansas Association of Educational Administrators (AAEA) Fall Conference September 4, 2013 1

2. Agenda 1.Introduction to OEP 2.Overview of Student Achievement … or … Are We 5 th or 49 th ? 3.NSLA Funding a.Has it been working? b.OEP thoughts on modifying? 4.How to help struggling schools – one unique strategy 2

3. 3 1. Introduction to the Office for Education Policy

4. The OEP is a research center within the College of Education and Health Professions at the University of Arkansas that specializes in education research and policy. www.officeforeducationpolicy.org Publications Databases Blog OEP Awards Program Evaluation and Assistance 4 Office for Education Policy

5. 5 Office for Education Policy Publications Arkansas Education Policy Briefs Report Cards Reports (AER) Remember to sign up for our weekly e-mail, OEP Web Links (OWL), to get updated on current education news across the state and nation. Please e-mail oep@uark.edu to sign up.oep@uark.edu Also, sign up for the OEP Blog at www.officeforedpolicy.com to receive alerts when the latest OEP Blog posts are published.www.officeforedpolicy.com

6. 6 Office for Education Policy Databases Benchmark End of Course Norm Referenced (ITBS) Demographics Finances School- and District-Level Growth over Time Similar Districts

7. OEP Similar Schools Database 7 Allows for comparisons to districts with similar or the same SES characteristics, including % FRL, % household bachelor degrees, median income, and district enrollment growth. Find on our website (Officeforeducationpolicy.org), under Arkansas Schools Data

8. 8 www.officeforedpolicy.com Topical Issues in Arkansas Ed Policy Summaries of Interesting Education Research Summaries of OEP Publications These posts can be interesting to school leaders and teachers and parents.

9. 9 Outstanding Educational Performance (OEP) Awards We think it’s important to highlight when schools do an excellent job. We give OEP awards to the top 25 schools in different categories:  High-Achieving Elementary Schools in Arkansas  High-Achieving Middle Schools in Arkansas  Beating the Odds – High-Achieving Schools Serving Low-Income Communities  High-Achieving Elementary Schools by Region  High-Achieving Middle Schools by Region  High-Achieving High Schools Across Arkansas  Most Improved Schools The top winner in each category is invited to our spring OEP conference to accept their award in person.

10. 10 Program Evaluation and Assistance Trying a specific strategy in your school and want to know if it’s effective? Let us know before you start, and we can help! Bentonville Technology RazorCoaches Northwest Arkansas Wild cards Helped with recommendations for 2 nd year implementation Merit Pay Program Design and Assistance

11. 11 Creation of School Achievement Reports and Presentations to School Communities

12. 12 Are We 5 th or 49 th ? An Overview of Student Achievement in Arkansas

13. Overall AR Achievement: How was Arkansas’ performance on the Benchmark and End-of-Course Exams in 2012-13? Over time? 13

14. Benchmark Performance Growth over time, until slight decrease in 2012-13 in literacy and math Slight decrease can be attributed to many factors, including ceiling effects and CCSS “implementation dip” Grade-level trends: lower grades perform at higher levels than upper grades Benchmark, Grade 3 – 8, % Proficient/Advanced, Over time 14

15. EOC Performance In 2012-13, slight decreases in Algebra & Geometry scores Steady increases in Grade 11 Literacy and Biology scores over time 15

16. How was Arkansas’ performance on the NAEP in 2011? Over time? 16

17. NAEP National Assessment of Education Progress – Nation’s Report Card Administered to random sample of 4 th and 8 th grade students Most recent data from 2011 – New 2013 NAEP data to be released this fall 17

18. NAEP Math, 2011 Grade 4 Grade 4 in math: Slightly below national average 18

19. NAEP Math, 2011 Grade 8 Grade 8 in math: Below national average 19

20. NAEP Reading, 2011 Grade 4 Grade 4 in reading: Below national average 20

21. NAEP Reading, 2011 Grade 8 Grade 8 in reading: Below national average 21

22. NAEP Performance, 2011 AR % Prof US % Prof DiffSurrounding States Grade 4 Math37%39%-2%AR > TN, OKA, LA, MS Grade 4 Reading30%32%-2%AR > TN, OK, LA, MS Grade 8 Math29%34%-5%AR > OK, TN, LA, MS Grade 8 Reading28%32%-4%AR > TN, LA, MS 22

23. NAEP Performance, Over time In math, in grades 4 and 8, Arkansas’s students have decreased the gap between Arkansas and the nation on the NAEP. However, Arkansas still performs less well than the nation in math and grades 4 and 8 on the NAEP. (Closer in Grade 4) Math, Grade 4Math, Grade 8 23

24. NAEP Performance, Over time In literacy, in grades 4 and 8, Arkansas’s students have decreased the gap between Arkansas and the nation on the NAEP. However, Arkansas still performs less well than the nation in literacy and grades 4 and 8 on the NAEP. (Closer in Grade 4) Reading, Grade 4Reading, Grade 8 24

25. 5 th or 49 th ? Two stories are out there today: 1.AR is backwards … “Thank goodness for Mississippi” … falling way behind in school quality 2.AR is rapidly climbing … 6 th in national rankings on the 2012 Quality Counts report and now 5 th in 2013!! AR has better schools than in Connecticut, Florida, and Texas. Let’s look at comparable data to do a fair comparison of AR scores to US totals. 25

26. NAEP Math, 2011 “Apples to Apples” Comparisons – Positive Results for AR In Grade 4, Arkansas’ FRL students were slightly ahead of the nation’s average. In Grade 8, Arkansas’ FRL students were slightly below the nation’s average. 26

27. NAEP Reading, 2011 “Apples to Apples” Comparisons – Positive Results for AR In Grade 4, Arkansas’ FRL students were slightly ahead of the nation’s average. In Grade 8, Arkansas’ FRL students were on par with the nation’s average. 27

28. NAEP v. Region, 2011 Arkansas compares well to surrounding states and to the nation when scores are compared by poverty level. Our state suffers in the overall category because more of our students are in the low income group than in other states. 28 Math and Reading, Grade 4: Comparison to Region/US by Income

29. Careful with these results… When comparing performance of FRL students across states, it is important to keep in mind cost of living. – Income level of for a family of four at ~$30,000 (free lunch threshold) looks different in Little Rock than in Los Angeles Therefore, FRL is an imperfect measure when examining poverty levels and comparing data across states. This might generate a positive BIAS for AR E.G. LR FRL = $30K ~= $22K in Seattle; thus comparing a “wealthier” set of AR kids to WA kids. 29

30. Back to the Question at Hand … What do we think we know so far? AR students have been improving: -Benchmark and EOC growth over time (until 2012- 13) -But test scores generally increase with time due to test familiarity...so it’s important to compare AR to the US -Slight NAEP overall growth over time -Slight decrease in AR/US gap in 4 th grade math/reading Relates to question: Has NSLA funding for FRL students helped? 30

31. 31 NSLA Funding Is It Effective?

32. The NSLA Question 1.How does NSLA funding work? 2.How do we know if it works? - If it were working, what changes might we expect to see? 3.So, what did we find about possible effectiveness? 4.Given the uncertainty, could we have expected great gains? (How were funds used?) 5.After all this, what would we suggest? 32

33. NSLA: How does it work? In the 2013 Quality Counts report, Arkansas received a B+ on equity funding, ranking it as one of the top states in the nation in distributing equity funding to districts. Arkansas should be commended for its focus on students in poverty, as the formula does channel more resources toward students in poverty, particularly those in very poor districts. 33

34. NSLA: How does it work? Math (GPA Measure), Districts By % FRLLiteracy (GPA Measure), Districts By % FRL We know that districts with 70% or more FRL students see a drop in achievement. NSLA funding seeks to allocate more funding to those districts. 34

35. NSLA Funding: How does it work? The tiered system creates two “cliffs.” “Cliffs” cause districts with very similar demographics to be treated differently in the funding system. For example, a district with 69% FRL receives less funding per FRL pupil than a district with 70% FRL; however, student bodies with 69% and 70% FRL look relatively similar. 35

36. The Big Q – How would we know if NSLA funding worked? We might observe … – Hypothesis 1: Increased scores for FRL students (relative to non-FRL students) … this may be the most important! – Hypothesis 2: Districts just above the “cliffs” performing better relative to those just below the “cliffs.” – Hypothesis 3: Districts with influxes in NSLA funds performing better than in past. 36

37. So, what do we know about NSLA? It is important to note that we do not have the counterfactual to examine how districts would perform without poverty funding. Nevertheless, we do know that: 1.Most agree that additional resources should be provided to schools with higher concentrations of poverty (to help students overcome additional challenges associated poverty). 2.No research indicates exact $$ amount needed to create equal opportunities for poor students. 3.From data presented thus far, no justification for funding “cliffs” (theoretical or empirical). So, how do districts use NSLA funding? 37

38. So, how do districts use NSLA funding? Expenditure Categories Year Coded as Exp. Percent of NSLA Funding in 2011-12 Literacy, Math, and Science Specialists and Coaches200316.51% Other activities approved by the ADE-11.56% High Qualified Classroom Teachers20039.42% Transfer to ALE Categorical Fund-8.63% School Improvement Plan- 8.62% Counselors, Social Workers, Nurses20038.30% Teachers’ Aides20038.17% Curriculum Specialist20034.69% Pre-Kindergarten20033.27% Before and After School Academic Programs20032.76% Supplementing Salaries of Classroom Teachers- 2.77% Tutors20032.35% Transfer to ELL Categorical Fund 2.28% Professional Development in Literacy, Math, and Science20032.02% Summer Programs20031.28% Early Intervention20031.22% Transfer to Special Educations Programs- 0.93% Transfer to Professional Development Categorical Fund- 0.87% District Required Free Meal Program20110.70% Parent Education20030.52% ACT Fees for 11 th Graders and Operating/Supporting a Post-Secondary Preparatory Program20110.10% Scholastic Audit- 0.37% Districted Reduced-Lunch Meal Program20110.05% Remediation activities for college20110.05% Teach For America professional development20110.03% Implementing Arkansas Advanced Initiative for Math and Science20110.01% Hiring Career and College Coaches20110.00% Materials, supplies, and equipment including technology2003- Expenses related to a longer school day2011- Expenses related to a longer school year2011- Shaded box denotes a coded use originally set in 2003. 38

39. How do districts use NSLA funding? The majority of districts distribute funding among 8 or more expenditure codes. Districts seldom focus the money in one or two specific areas; therefore, it seems if many districts use the funding to plug gaps in budgets. It is unclear as to whether all districts are specifically pinpointing the funding towards students in poverty (or schools serving these students). For example, a district may spend a large portion of funding on Highly Qualified teachers or Specialists – these teachers may or may not work specifically with the low-income students. Furthermore, districts do not use all the funding – many have balances at the end of the year. 39

40. How do districts use NSLA funding? Given the uncertainty, could we have expected great gains? – Funds have spent across the board by most districts – No clear evidence that funding has been focused for students in poverty – Money is allocated to district offices and not even to schools with high levels of poverty 40

41. 41 NSLA Funding OEP’s Thoughts

42. What do we recommend? Two main discussions this year: Distribution of funds – “Smooth sliding” scale to replace the current tiered system – Distribute more funding for districts with higher concentrations of FRL students – Weighting the funding to differentiate between poverty levels by factoring in the difference between “free” and “reduced” lunch students – Leftover balances by districts Use of funding: more or less prescriptive? 42

43. Example of a Smooth Distribution “Smooth” sliding scale Weighted to account for differences in “free” and “reduced” Weights are 75% for Reduced-Lunch Students and 100% for Free-Lunch Students. 43 Problem: Several “affluent” districts would lose $$

44. Smoother … but prescriptive? Should the use of NSLA funding be more prescriptive? Long debate over extent of “mandating the spending matrix” Arguments for prescriptive use: – Current lack of focus of funds – Pinpoint only to students in poverty – Use prescriptive manner as a way to figure out what works Arguments against prescriptive use: – Flexibility is necessary: State-wide policies may not fit for all. – What do you prescribe? Research isn’t conclusive on what works best 44

45. Concluding Thoughts Arkansas scores on the Benchmark and EOC have improved in the past 10 years, but… much of the NAEP increases occurred before 2003 (slides 18 and 19) On the NAEP, Arkansas students have only slightly increased scores in 4 th grade and 8 th grade – FRL and non-FRL students have produced gains; but non-FRL students have experienced greater gains It is difficult to determine the effectiveness of NSLA funding over the past 10 years. – The gap between FRL and non-FRL students has not shrunk. Policymakers and districts need to continue to strategically think about how NSLA funding can be pinpointed so that students in poverty can achieve at higher levels. 45

46. 46 A Promising Strategy for School Improvement: Arkansas Teacher Corps

47. Comments? Questions? Thank you for your time and input! 47

48. Hypothesis 1: FRL Students vs Non-FRL Students If NSLA Funding were working, we might expect to see increase in achievement for FRL students relative to non-FRL students. 48

49. Benchmark Achievement Math2005-062011-12 Percentile Point Growth Non-FRL students62 nd 66 th +4 FRL students40 th 0 Literacy2005-062011-12 Percentile Point Growth Non-FRL students63 rd 66 th +3 FRL students39 th 43 rd +4 Math, 2005-06 to 2011-12 Literacy, 2005-06 to 2011-12 In math, the gap between FRL and non- FRL students has widened over time. In literacy, FRL students have slightly closed the gap; but FRL students still perform less well. 49

50. NAEP Growth, 2003 to 2011 50 Math and Reading Score Gains, 2003 to 2011 Over the past decade, Arkansas scores have grown by leaps and bounds, but that statistic is padded by lower baseline scores. The greatest gains come in math and for higher-income students.

51. Hypothesis 1: FRL Students vs Non-FRL Students Achievement gap between FRL and non-FRL students continues to exist. – Benchmark Gap is widening in math performance Gap is slightly shrinking in literacy – NAEP Non-FRL produced higher gains than FRL students over time 51

52. Hypothesis 2: “Cliff” Districts “Cliffs” cause districts with very similar demographics to be treated differently in the funding system. For example, a district with 69% FRL receives less funding per FRL pupil than a district with 70% FRL; however, student bodies with 69% and 70% FRL look relatively similar. The “cliffs” allow us to compare the performance of relatively similar districts (e.g. 69% to 70%) that receive different amounts of funding. Thus, if NSLA were working, we would see greater performance for districts “above the cliffs” 52

53. Hypothesis 2: “Cliff” Districts Benchmark Math GPA, 2007-08 to 2012-13 Benchmark Literacy GPA, 2007-08 to 2012-13 On the math and literacy benchmark exams, the districts just above and below the cliff (thus, districts who are socio-economically “equal”) perform nearly identically. Achievement Comparisons at the 70% “Cliff”* 53

54. On the math and literacy benchmark exams, districts just below the 90% cliff outperformed the districts above the cliff. Hypothesis 2: “Cliff” Districts Benchmark Math GPA, 2007-08 to 2012-13Benchmark Literacy GPA, 2007-08 to 2012-13 Achievement Comparisons at the 90% “Cliff”* 54

55. Hypothesis 3: Increased Funding When a district “moves up a tier” by having a higher % of FRL students, FRL students may perform at higher levels after the district has received more funding. Thus, if NSLA were working, we would see greater performance for districts after the new funds Since 2004-05, some districts have moved into a higher tier of poverty funding. The achievement of these districts was compared and at both the 70% and 90% cliffs, no district showed an increase in achievement as a result of a financial windfall. 55

56. So, what do we know about NSLA? It is important to note that we do not have the counterfactual to examine how districts would perform without poverty funding. Nevertheless, we do know that: 1.Most agree that additional resources should be provided to schools with higher concentrations of poverty (to help students overcome additional challenges associated poverty). 2.No research indicates exact $$ amount needed to create equal opportunities for poor students. 3.From data presented thus far, no justification for funding “cliffs” (theoretical or empirical). So, how do districts use NSLA funding? 56

57. So, how do districts use NSLA funding? Expenditure Categories Year Coded as Exp. Percent of NSLA Funding in 2011-12 Literacy, Math, and Science Specialists and Coaches200316.51% Other activities approved by the ADE-11.56% High Qualified Classroom Teachers20039.42% Transfer to ALE Categorical Fund-8.63% School Improvement Plan- 8.62% Counselors, Social Workers, Nurses20038.30% Teachers’ Aides20038.17% Curriculum Specialist20034.69% Pre-Kindergarten20033.27% Before and After School Academic Programs20032.76% Supplementing Salaries of Classroom Teachers- 2.77% Tutors20032.35% Transfer to ELL Categorical Fund 2.28% Professional Development in Literacy, Math, and Science20032.02% Summer Programs20031.28% Early Intervention20031.22% Transfer to Special Educations Programs- 0.93% Transfer to Professional Development Categorical Fund- 0.87% District Required Free Meal Program20110.70% Parent Education20030.52% ACT Fees for 11 th Graders and Operating/Supporting a Post-Secondary Preparatory Program20110.10% Scholastic Audit- 0.37% Districted Reduced-Lunch Meal Program20110.05% Remediation activities for college20110.05% Teach For America professional development20110.03% Implementing Arkansas Advanced Initiative for Math and Science20110.01% Hiring Career and College Coaches20110.00% Materials, supplies, and equipment including technology2003- Expenses related to a longer school day2011- Expenses related to a longer school year2011- Shaded box denotes a coded use originally set in 2003. 57

58. How do districts use NSLA funding? The majority of districts distribute funding among 8 or more expenditure codes. Districts seldom focus the money in one or two specific areas; therefore, it seems if many districts use the funding to plug gaps in budgets. It is unclear as to whether all districts are specifically pinpointing the funding towards students in poverty (or schools serving these students). For example, a district may spend a large portion of funding on Highly Qualified teachers or Specialists – these teachers may or may not work specifically with the low-income students. Furthermore, districts do not use all the funding – many have balances at the end of the year. 58

59. How do districts use NSLA funding? Given the uncertainty, could we have expected great gains? – Funds have spent across the board by most districts – No clear evidence that funding has been focused for students in poverty – Money is allocated to district offices and not even to schools with high levels of poverty 59