1. Rating and Disseminating Data for Arkansas Schools
June 28, 2012
Gary W. Ritter
James L. Woodworth
OEP Presentation on School Ratings

2. How to Compute and Show AR School Performance Data
What is the status quo? What’s wrong with it?
What are the key improvements needed?
What do we need?
Decide on vehicle for dissemination?
Decide on data and rating schemes?
Which other data?
Which exams? (crt, nrt, other?)
How to incorporate growth?
Feedback and Questions
OEP Presentation on School Ratings

3. Status Quo: Common Measures of School Success
Percent Proficiency
AYP Status
ACT Scores
Graduation Rates
Problems:
Schools that showed growth in student achievement or schools that “out-performed” their “expected” scores would not necessarily be distinguished
OEP Presentation on School Ratings

4. AYP as a Measure of Success?
Schools starting off as low performing schools can make huge gains and continue to be labeled as “failing” while schools making much smaller gains may still be classified as “successful” just because they started from a higher point on the scale.
OEP Presentation on School Ratings

5. Examples of Data Access Vehicles for Constituents and Parents
 NORMES for teachers and parents (media)
ADE for teachers and parents (media)
OEP mostly for parents and media
We will show weaknesses of each and then show the kind of data we are overlooking
OEP Presentation on School Ratings

6. Good info, but perhaps too much; only school by school access
NORMES
Good info, but perhaps too much; only school by school access
OEP Presentation on School Ratings

7. And some silly info that is not very useful … e.g. total financial $
NORMES
OEP Presentation on School Ratings

8. ADE Website
Not visually appealing, all measures are on separate sheets, no obvious destination; but do allow downloads.
OEP Presentation on School Ratings

9. But no overall school or district figures … see tabs for grades
ADE Website
OEP Presentation on School Ratings

10. ADE Website – Individual Downloads
Same issue with school by school downloads from ADE site
OEP Presentation on School Ratings

11. ADE Performance Reports - www
OEP Presentation on School Ratings

12. ADE Performance Reports – snail mail
At about 20 pages, way too much for the average parent – or anyone – to digest.
OEP Presentation on School Ratings

13. What are we missing?
NORMES provides great detail, but no overall score and no downloadable database
How is my school district doing overall?
As compared to what?
Can I create a list of all the schools in NWA?
ADE provides comprehensive state data, but again no overall score
Can I combine data from multiple tests? Can I look at demographic data or last year’s scores alongside of this year’s scores?
OEP Presentation on School Ratings

14. OEP Data Resources Pages
IMPROVEMENTS
Presents subgroup averages
Combines multiple sources (scores + ses)
Presents school / district overall scores
Allows for comparisons of similar schools
OEP Presentation on School Ratings

15. OEP – Compared to Similar Schools
OEP Presentation on School Ratings

16. OEP -- allows datasheet downloads and school lookups
OEP Presentation on School Ratings

17. Examples of What We Overlook
Howard Elementary in Fort Smith looks bad in raw data, but …
Grace Hill looks good on raw data, but when placed alongside of demographics, even better …
OEP Presentation on School Ratings

18. Howard Elementary -25 Math in 2006 -13 Math in 2010
Howard FRL: 95% Enroll: 354
Ark FRL: 59%
Math Growth
Lit Growth
Howard 32 26
Arkansas 20 13
Difference
+12
+13
Successful making continuing gains with low-SES and minority students .
OEP Presentation on School Ratings

19. Grace Hill Elementary Arkansas Outperformed all state subpopulations!+4 +5 +9 +5 -3 -5
GH FRL: 87% Enroll: 461
Ark FRL: 59%
-15
-21
-15
-15 -5
-12
Arkansas
OEP Presentation on School Ratings

20. Key Improvements Needed
A single measure of current status that is based on either:
CRTs, EOCs (weakness = state only)
NRT (weakness = varies every 3 years)
Combination? Other measures?
A good measure of performance that should include either :
Growth over years
Comparisons of status scores to where status scores should be
Sophisticated Value Added Measure (student-level data)
A user-friendly vehicle for dissemination
OEP Presentation on School Ratings

21. Dissemination – Quick Thoughts
Avoid too much information (descriptions and supportive documents can be included on web for those interested)
As a result, must make hard choices – cannot include everything
Use some single measure, however imperfect, for both current status and value-added (more later)
Include both a school by school summary option (for parents) and a spreadsheet download option (for media and researchers)
OEP Presentation on School Ratings

22. Searchable by school and district
Simple OEP Option
Searchable by school and district
Includes financial (per pupil), demographic, and student performance
Allows for summary figures and some backup data
Attached to spreadsheet with all districts; thus useful to researcher and parent alike
Very flexible (just a rough first attempt); content of each block can change
OEP Presentation on School Ratings

23. Tougher Questions – What goes in the boxes?
What is the status quo? What’s wrong with it?
What are the key improvements needed?
What do we need?
Decide on vehicle for dissemination?
Decide on data and rating schemes?
Which other data?
Which exams? (crt, nrt, other?)
How to incorporate growth?
Feedback and Questions
OEP Presentation on School Ratings

24. 3b) Decide on data and rating schemes?
Which other data?
Which exams? (crt, nrt, other?)
How to incorporate growth or value-added?
Remember, we must be parsimonious!
OEP Presentation on School Ratings

25. i. Other Data for Review
Parents and others likely care about money and peers, so our first guess included:
Spending figures (total per pupil, net per pupil, and instructional per pupil)
Student enrollment and demographic characteristics
What else should be included?
Should we included comparisons to recent years or peer districts
Challenges:
Parsimony!
Some data available at school level, some at district
OEP Presentation on School Ratings

26. ii. Which Status Performance Data?
What exams to show (crt, eoc, nrt)
What metrics to use:
Percent of students proficient or advanced or meeting any bar (understandable but not great information) or simple NCE from NRT
GPA Score of 1.00 through 4.00 (better info)
If using NRT or combination, create our own norm-referenced comparison with mean = 80 and standard deviation = 10 (thus, 80 is average, 90 is good, 100 is great, 70 is bad). Understandable to parents?
Should we included comparisons to recent years or peer districts
Challenges:
Parsimony!
Different figures available for different types of schools (NRT = gr 1-9, EOC = HS, ACTAAP = gr 3-8)
Do we do district or school or both?
OEP Presentation on School Ratings

27. ii. Growth or Value Added?
Finally, we need a good measure of performance:
Not the same as current status! Could be something like:
Growth over years
Comparisons of status scores to where status scores should be
Sophisticated Value Added Measure (student-level data)
Strength = easy to understand and will capture schools with changes
Weakness = may overlook schools that are continually good or bad
Comparisons of Status Scores to Where they Should be (e.g. similar schools rankings)
Strength = a couple options that could be easy to understand
Weakness = not as sophisticated as student level value added
Student Level Value Added
Strength = most sophisticated
Weakness = harder to explain and requires student level data
OEP Presentation on School Ratings

28. Raw Growth of Comparison to Similar Schools
The output of something like this would include a simple deviation (+ or - ) either from prior year or from group mean.
OEP Presentation on School Ratings

29. More Complicated Version of “Compared to Expected?
+ 10 pts
Use mathematical model to predict this year’s scores (RED) based on characteristics;
Value added = difference between actual score and expected red line (+ or -)
OEP Presentation on School Ratings

30. Student Level Value Added?
Strength = most sophisticated
Weakness = harder to explain and requires student level data
Here, we look for “deviation’s from expected” for individual students, then sum these up for schools to find schools that consistently foster over-performance in students
This requires complete trust from the readers; the other methods are a bit easier to follow
OEP Presentation on School Ratings

31. Feedback and Questions?
Do we create some overall grade (A, B, C, D)?
Do we aggregate math + literacy?
What status measure?
What growth measure?
How does our “window” look?
Other feedback???
OEP Presentation on School Ratings

32. Searchable by school and district
Simple OEP Option
Searchable by school and district
Includes financial (per pupil), demographic, and student performance
Allows for summary figures and some backup data
Attached to spreadsheet with all districts; thus useful to researcher and parent alike
Very flexible (just a rough first attempt); content of each block can change
OEP Presentation on School Ratings