1. 1 Children First Intensive 2008 Grade 5 Social Studies Analyzing Outcomes for ESO Network 14 March 25, 2009 Social Studies Conference, PS/MS 3 Deena Abu-Lughod, Senior Achievement Facilitator Network Leader: Bob Cohen Brandon Alvarez, Joann Benoit, Deirdre Burke, Freddie Capshaw, Alan Godlewicz, Dr. Pamela McCarthy

2. 2 Learning Intentions  Understand the construction of the Grade 5 Social Studies test  Understand Network 14 performance and improvement trends  How to use benchmark comparisons to reflect on curriculum  Understand how Social Studies outcomes relate to the ELA outcomes

3. 3 Obtaining Information from the State Copies of old state tests are available through NYSED: http://www.nysedregents.org/testing/scostei/socstudies5.html The Core Curriculum and Standards are available through these links. http://www.emsc.nysed.gov/ciai/socst/pub/sscore1.pdf http://www.emsc.nysed.gov/ciai/socst/socstands/socstand.html The last page of the rating guide provides the “specifications grid” that shows the distribution of multiple choice questions by Unit and Standard. The grid shows that questions appear sequentially by grade. In 2008, items 1-11 related to Grade 3 content, items 13-23 related to Grade 4 content, items 27-30 to Grade 4-5 content, and the last four items were cross-topical and skills-based.

4. 4 Screen shot of a Specifications Grid

5. 5 State Social Studies Toolkit: http://www.emsc.nysed.gov/ciai/socst/ssresources.html http://www.emsc.nysed.gov/ciai/socst/ssresources.html

6. 6 Performance Highlights Most schools have improved the percent of students passing the social studies exam by several points each year, notably x3, x20, x24, x86 and x360. x20 has improved the most. What are they doing that has made the difference?

7. 7 Avg. Grade 5 Social Studies Scores, 2006-2008

8. 8 Comparing 2007 and 2008 Results The following two stacked bar graphs show the percent of students in each performance level on the Fall 2007 and Fall 2008 Social Studies exams. Finally, there is a bar chart disaggregating the data by gender. Consistent with the tendency in literacy, girls outscored boys at all schools except x3, x81, and x360.

9. 9

10. 10

11. 11

12. 12 Comparing the Network to all Level 4 students Your annotated copy of the Grade 5 Fall 2008 Social Studies exam indicates the Grade Level, Unit and Standard for each item and the percent of students in Networks 14 and 19 who answered correctly. The percent of students who answered an item correctly is referred to as an “item analysis”. However, there is a better way to use the item response data to evaluate curriculum. That is to compare the percent of students answering correctly to the percent of all level 4 students (or all level 3 students) who answered correctly. This helps you distinguish which kinds of questions separate the 2s from the 3s and the 3s from the 4s. This is called a “benchmark” comparison.

13. 13 Small gap Large gap

14. 14 “Small” gap questions

15. 15 A large gap question

16. 16 Another large gap question

17. 17 Using benchmarks to review curriculum Examine the data comparing the percent of your students who answered each item correctly to the Level 3 or Level 4 benchmarks. Items where the gap was the smallest indicate the types of skill and knowledge that your students were most successful at mastering. Items where the gaps were large provide clues as to the types of skills and knowledge your students were less successful in mastering. For those skills and knowledge, review where in the curriculum they were taught and how they were assessed.

18. 18 Are ELA and Social Studies scores related? The correlation coefficient of the Grade 4 ELA with the Grade 5 social studies is.718. This is very high, and slightly higher than the correlation between these same students’ Grade 4 ELA and Math scores (.694). However, it is not as high as the correlation between the Grade 3 and Grade 4 ELA exams (.777). This means that content-area instruction DOES matter. Note in particular how the correlation between the ELA and Social Studies scores are actually much higher at the middle schools that begin in Grade 5 (.766). That means content area instruction had less of an impact, or independent effect, on student outcomes.

19. 19

20. 20 Is correlation a good thing? Since the social studies exam requires strong literacy skills, it is not surprising to see a high correlation. The fact that the correlation is higher in middle schools with initial 5 th grades suggests a relative weakness in specific social studies skills. A lower correlation is actually a measure of the greater independence of the social studies score, and therefore of the quality of social studies instruction. What do the data tell us about what might be happening in different schools and classes?

21. 21 Feedback and debrief; Evaluation Please complete the Feedback Form now. Did we achieve our intentions?  Better understand the construction of the Grade 5 Social Studies test  Understand what the performance and improvement trends across Network 14 schools have been  How to use benchmark comparisons to revise curriculum  Understand how the results of the social studies exam relate to outcomes on the Grade 4 ELA  How to use old tests to construct reliable measures of mastery in earlier grades