1. Preliminary Analysis of EOG/EVOS Data – Greene County 2009,2010,2011
Jason Brinkley – ECU BIOS
October 2013

2. Background
Data was provided from Greene County schools in the form of EVOS data for every student in grades 4-8 with a focus on reading and math scores. While there was some variation in what information was pulled, in general the combined data had the following:
Data collection school year: , , Pseudo Student ID: For tracking across three years of data collection Teacher Code: For tracking teachers across three years of data collection Demographics: Gender, Grade, Ethnicity, LEP

3. Student Achievement Data
The current year’s test score
A standardized ‘C Score’
A state assigned ‘predicted score’ and ‘predicted C Score’
A growth variable that measures the difference between the students achieved ‘C score’ and the state ‘Predicted C score’.
For teacher achievement purposes, growth is measured by how well students perform versus predictions.
Test scores are also given ‘level’ which are ordinal categories that represent achievement cut-offs (1,2,3,4) and correspond more to the traditional idea of end of grade testing.
Past Year Scores: Same information as the test scores listed above, but for the previous school year.

4. Demographics

5. Greene County Raw Scores
Math
Year
Score N
1267
1244
1195
Mean
352
354
355
Std Dev 9
Reading
Year
Score N
1259
1212
1184
Mean
348
349
350
Std Dev 10

6. Significant Gains
ANOVA testing reveals (p < ) that there are significant differences between years for both math and reading raw scores.
Further examination shows that each year saw a significant increase in math scores while there is a significant difference between 09/10 and 11 only (no difference between 09 and 10 cohorts).

7. Achievement Level by Year

8. Gains Made
The last three slides illustrate that Greene County has made some improvements over the selected time frame.
We also see that many students are deemed ‘proficient’ (i.e. they reach a level greater than 1), especially in math. Therefore looking at only a variable like ‘level’ as a student achievement outcome may not be as informative.

9. C Scores and Growth
C Scores and Growth Scores are a different type of measure on student impact.
The end goal of using such measures is to provide a more standardized way of measuring student achievement and teacher impact.
However, as we will see, these measures may prove to be difficult to use for research purposes.

11. Ex. Math C Scores by Year 2010-2011 2009-2010 2011-2012
Here we see the distribution of actual C Scores and state predicted C Scores for math tests for all three years of interest. Scores typically range from -2.5 to 2 with Greene County schools averaging slightly less than 0 year. The correlation between predicted and actual scores hovers around 0.80, strong but not perfect.

12. Greene County Performance by C Score
Math
Year
C Score N
1223
1211
1163
Mean
-0.27
-0.03
0.05
Std Dev
0.8
0.9
Reading
Year
C Score N
1259
1212
1184
Mean
-0.44
-0.38
-0.24
Std Dev
0.9

13. Greene County Math Growth by Year
Math Growth Averages:
: -0.3
: 0.19
: 0.17
Percent with Positive Growth:
: 49%
: 58%
: 61%

14. Greene County Reading Growth by Year
Reading Growth Averages:
: -0.1
: -0.03
: 0.10
Percent with Positive Growth:
: 42%
: 43%
: 58%

15. Average Growth in Greene County Across Teachers by Year.
There were 27 teachers in each area in Greene County from the data. Of the math teachers, on 40% had a mean growth value greater than zero. Only 11% of reading teachers showed positive growth.
In there were 32 teachers in each area and 75% had a mean growth greater than zero. 35% of reading teachers showed positive growth.
In there were 32 teachers in each area and 63% of math teachers had a mean growth greater than zero. 63% of reading teachers showed positive growth.

16. Great news, right?
The data shows that Greene County is moving in a positive direction. While we see only mild increases in raw end of grade scores, when those scores are centered toward statewide averages we see that the increases are even greater.
Likewise we see on a one on one basis that many teachers are experiencing higher growth rates, there are more students who perform better than what the state is predicting.

17. The downside of just focusing on growth. Aggregated across all years.

18. Growth also gets harder for better students.

19. Impact on Teachers
Here we see that Growth skews when broken down by teachers with students predicted to get only 3 and 4 levels on reading EOG tests. We see that as Greene County makes continued improvements that those students experience lower growth rates and those teachers have lower mean growth scores. Likely because those students start to hit some sort of glass ceiling in scores.

20. Measuring ECU’s Impact – Greene County Teachers with ECU Interns (Purple = Undergrads, Yellow=TQP)
Teacher Code
Number of Students
Grade
Predicted C Score Mean
Actual C Score Mean
Mean Growth A
111 7
-0.43
-0.36
0.03 99
-0.57
-0.41
0.16 63
-0.56
-0.22
0.33 B 21 4
-0.49
-0.82
-0.40 22
-0.42
-0.86
-0.38 26
-0.54
-0.06 C . 18 5
-0.46
-0.50
-0.03
-0.68
-0.69
-0.01 D
0.00 E
0.01
-0.07
-0.10 24
-0.08
-0.23
-0.15 11
0.99
1.28
0.28 F 40
-0.67
-0.18
-0.55
-0.88
-0.39
-0.71
0.02 G
-0.96 J
112 8
-0.48 K
0.25 P
-0.47
-0.11 17
-0.24
-0.44
-0.20
-0.05 Q 20
-0.02 R
-0.62
-0.09 25
-0.63 27
0.26
Reading EOG Scores and Growth

21. In general we see that the predicted C Score of ECU interns is worse than those classrooms without an intern. Likewise we see a higher rate of students predicted to have a 2 on the EOG Reading test.

22. Quasi-Randomized Study
Another way to look at this data would be to pull off the values that have adequate (randomizable) controls and then do a quasi-randomized case-control analysis.
Pulling off teachers with adequate controls from years and we can use each teachers first ECU intern experience (No TQP) and match with a similar grade and class size for that same year of a teacher with no ECU intern experience:

23. In 40% of matches, classes with ECU interns show higher growth.
Likewise 40% of pairs had classrooms with lower predicted C Score means.
Given that the mean predicted C Score in 90% of these instances are negative, this analysis is likely more appropriate for studying ECU intern impact.

24. Observations
For research purposes, it seems appropriate to study not only growth but either raw or C Scores as well.
The level may also provide useful ways to separate out important analysis.
High growth is much more prevalent in the lower end of the spectrum with students who are predicted to score a 4 less likely to achieve the same amount of growth.
ECU interns (pre-TQP) were seemingly sent to classrooms with lower predicted scores and as such any study that measures their impact has to be mindful to make adequate comparisons with ‘control’ groups.