1. 1 Evaluating the NYC Core Knowledge Language Arts Pilot: Summary of Kindergarten and Grade 1 Results HIGHLIGHTS Research and Policy Support Group

2. 2 Summary of Kindergarten Results 2008-2009 School Year Research and Policy Support Group

3. 3 Methodology: A multi-method, longitudinal research design YEAR 1 Literacy Assessments (at 10 CKLA schools & 10 comparison schools, 4 of which are Reading First schools): Pre- and post-test of literacy skills Additional tests of literacy skills at end of each year Periodic assessments throughout the year (DIBELS) Teacher and Administrator Surveys (at 10 CKLA schools): Assesses satisfaction with and impact of CKR Case studies (at 3 CKLA schools): Classroom observations, administrator & teacher interviews Hypothesis: Kindergarteners taught with the Core Knowledge Language Arts (CKLA) Program will gain reading competencies at a faster rate than their peers. Focus of the Evaluation

4. 4 Similar Demographics at CKLA and Comparison Schools CKLA Schools (N = 584) Comparison Schools Note: These and other data were used to select comparison schools (data as of 2007-08 school year). Percent of Students (Number of Students for School Size) (N = 307)* * N = the number of students for whom both fall and spring data were available.

5. 5 Different Literacy Domains: Greater Gains & Higher Spring Scores for CKLA Students than Comparison Students in All Literacy Domains CKLA Schools Compared to… Basic Reading Skills W-J Letter Word Identification Oral Reading Comp- rehension W-J Passage Comprehension Decoding W-J Word Attack Written Spelling W-J Spelling of Sounds Oral Reading Comp- rehension, Vocabulary, Basic Reading, Decoding Terra Nova Reading All Comparison Schools CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Higher Spring Scores CKLA Sig. Higher Spring Scores CKLA Sig. Higher Spring Scores Reading First Comparison Schools CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Higher Spring Scores CKLA Sig. Higher Spring Scores CKLA Sig. Higher Spring Scores W-J Brief Reading

6. 6 5X Greater Literacy Gains for CKLA Students than Students at Demographically Similar Comparison Schools Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKLA Students Significantly Higher p <.001 CKLA Comparison

7. 7 6X Greater Literacy Gains for CKLA Students than Students at Reading First Comparison Schools Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR Students Significantly Higher p <.001 (N = 314)* (N = 103)* * Data presented represent the four comparison schools that are Reading First schools and the four CKR schools who were statistically comparable matches.

8. 8 CKR Schools Comparison Schools CKR & All Comparison Schools CKR & Reading First Schools CKR Schools Reading First Schools Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) At All Achievement Levels, Greater Literacy Gains for CKLA Students than Students at Comparison & Reading First Schools

9. 9 Summary of Findings  By nearly all measures, Core Knowledge Language Arts (CKLA) students made significantly greater gains in early literacy than peer students.  Compared to peers, kindergarteners taught with the CKLA program made more progress in all areas of reading tested: spelling, phonemic awareness, decoding, and comprehension.  These findings hold for two types of peer comparisons: 1) demographically similar schools and 2) demographically similar schools using Reading First (another intensive, focused literacy program).  Surveys and case studies indicate overall high levels of administrator and teacher satisfaction with the CKLA Pilot, while also offering guidance for year 2 implementation and evaluation work.  Administrators would recommend program to others; teachers rate CKLA more favorably than other programs.  Administrators report change in teacher practice: more data- driven instruction & teacher collaboration.  Teachers and administrators feel more positively about the Skills Strand than the Listening and Learning Strand, particularly regarding student engagement.  Teachers may need more support addressing needs of struggling readers with CKLA & managing time to complete lessons.

10. 10 Summary of Grade 1 Results 2009-2010 School Year Research and Policy Support Group

11. 11 Methodology: A multi-method, longitudinal research design YEAR 2 Assessments (at 10 CKLA schools & 10 comparison schools, 3* of which are Reading First schools): Pre- and post-test of literacy skills Additional tests of literacy skills at end of each year Tests of science and social studies skills at end of each year Teacher and Administrator Surveys (at 10 CKLA schools): Assesses satisfaction with and impact of CKLA Case studies (at 5 CKLA schools): Classroom observations, administrator & teacher interviews Hypothesis: First Graders taught with the Core Knowledge Language Arts (CKLA) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers. Focus of the Evaluation * Last year there were 4 Reading First comparison schools. This year there were 3 Reading First comparison schools: one comparison school used Reading First in Year 1 but not in Year 2.

12. 12 Overall, CKLA and comparison schools had similar demographic profiles. However, the differences between groups, especially % of Black/ Hispanic students, was significantly larger than last year. CKLA Students (N = 781) Comparison Students Note: Data from 08-09 was used to select comparison schools (data presented here are from the 2009-10 school year). Percent of Students (Number of Students for School Size) (N = 343)* N = the number of students for whom both fall and spring data were available. A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested. As a result, all analyses control for student demographics.

13. 13 CKLA students displayed greater gains & higher Spring scores in nearly all literacy and content knowledge domains relative to comparison students Woodcock Johnson (Fall and Spring)Terra Nova (Spring Only) TestW-J Brief Reading W-J Word Attack (Decoding) W-J Spelling of Sounds (Written Spelling) Terra Nova Reading (Oral Reading Comprehension, Vocabulary, Basic Reading, Decoding) Terra Nova Social Studies Terra Nova Science W-J Letter Word Identification (Basic Reading Skills) W-J Passage Comprehension (Oral Reading Comprehension) CKLA Schools Compared to… All Comparison Schools CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Spring Scores CKLA Sig. Greater Spring Scores CKLA Sig. Greater Spring Scores Reading First Comparison Schools CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Gains/ Spring Scores CKLA Sig. Greater Spring Scores CKLA Sig. Greater Spring Scores X No Significant Difference Note: All analyses control for student demographic characteristics.

14. 14 Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKLA Students Significantly Higher p <.0001 CKLA students displayed nearly 2x greater literacy gains relative to students at demographically similar comparison schools (N = 758)(N = 342) The magnitude of the group difference was smaller than last year (in Year 1, CKLA students’ gains were 5X greater than their peers).

15. 15 (N = 233)*(N = 100)* * Data presented represent the three comparison schools that are Reading First schools and the three CKR schools who were statistically comparable matches. CKLA Students Significantly Higher p <.0001 Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKLA students displayed 6x greater literacy gains relative to students at Reading First comparison schools The magnitude of the group difference was smaller than last year (in Year 1, CKLA students’ gained 14.3 pts while Reading First schools declined 4.1 pts).

16. School TypeAverage Gain Pilot19.1 Pilot10.2 Pilot13.1 Pilot3.5 Pilot2.6 Pilot3.7 Comparison2.6 Comparison2.1 Comparison2.2 Pilot1.2 Comparison2.4 Comparison1.3 Comparison1.1 Comparison1.1 Comparison1.3 Pilot0.5 Comparison0.3 Pilot-1.5 Comparison-6.0 Pilot-13.2 16 Gains at Each School: All of the schools making statistically significant gains were CKLA pilot schools Significantly Positive Gain Significantly Negative Gain Non-significant Gain

17. 17 Measuring fidelity of implementation to the CKLA curriculum Examining implementation fidelity allows us to better determine: 1) whether achievement gains can be attributable to the CKR program and 2) which components of the CKR program teachers are struggling to implement and those they are implementing successfully We took several different approaches to measuring fidelity in the CKR Pilot schools: Site visits with classroom observations Site visits were conducted at 5 randomly selected Pilot schools 15 classrooms were observed Interviews with administrators and teachers Interviews were conducted with 13 teachers and 7 administrators Additional teacher and administrator survey questions

18. Anecdotes from Classroom Observation The first classroom had a vibrant atmosphere. Approximately 30 students, all of whom were seated on the carpet, appeared completely absorbed, watching their teacher and the images about the U.S. space program that were projected on the screen beside her. The classroom was covered with student work, Core Knowledge posters and chaining charts, and a hanging model of our solar system. The students were eager to share their answers, some of them using challenging vocabulary words that they had learned from earlier lessons like “perseverance.” After a writing/drawing exercise, in the final minutes of the lesson the teacher played a song about the sun that she had found on the internet. Some students pretended to be conductors, which they had learned about in the Mozart unit they had recently completed. This school had a teacher-coordinator who had participated as a kindergarten teacher the year before and was providing specific support for both the Core Knowledge pilot in first grade and the kindergarten classrooms, which had continued to use the curriculum. The school is also using the Core Knowledge curriculum for all grades. 18 Example of a school with high fidelity

19. 19 Administrators’ Views Teachers’ Views Both teachers were very positive about the program’s approach to developing early childhood literacy in their 1 st graders. They liked using the program, its structure, content, and activities, and they also could see the students benefiting from it. Their criticisms of the program were of more “mechanical” elements (e..g, lack of page numbers in the readers, would have liked captions to accompany the pictures and make them more interactive, they felt rushed to get through all the material for any give day) Administrators’ Views Administrators were very supportive of the program They perceived large benefit from the program’s “multi-intelligence” approach and have been excited to watch the students make connections between lessons and to their lives outside the classroom as well. They perceived more parent involvement—children dragging their parents to the library, parents commenting that they are learning from their children. They felt the program worked well largely because their teachers are well supported and have fully embraced the program. Criticisms were that the program needs to include more “higher-order thinking” questions and provide material tailored for special populations (e.g., ELLs). Example of a school with high fidelity

20. Anecdotes from Classroom Observation The structure of the classrooms themselves created a challenge for the three first grade teachers—they only have three walls, and are open onto a hallway on the fourth side. Not only did the teachers have to contend with noise from the other classes, but also with the distractions of passersby using the hallway. In two of the three classes, students were having a difficult time responding appropriately to their teacher’s directions, questions, and assignments. The students in one of these classrooms also had a difficult time listening respectfully to each other during the discussion time at the end of a Listening and Learning lesson. This was the only example we saw of this kind of behavior across all 15 classrooms. One of the classrooms did not have the Core Knowledge materials posted and was missing the chaining and tricky words. Toward the end of the year, the CKR literacy coordinators worked with the teachers to take a different approach to differentiation. They divided the students into three homogenous skill groups and then each teacher worked with one group three days a week for a 45 min. session. 20 Example of a school with lower fidelity

21. 21 Teachers’ Views Did not feel supported by the school’s administrators. This made it that much more difficult to be successful in their implementation of the program because the administrators were out of touch with what the teachers needed in order to be fully successful in their implementation. Sometimes found it difficult to fit in all the content for both the Listening and Learning and the Skills Strand, and would resort to picking out the highlights for a reading or selecting the most feasible activities in order to optimize their time. For all the challenges, the teachers were positive about the program itself—they saw their students engaging with it and connecting it to their lives outside the classroom. Administrator Views The administrator we interviewed was very enthusiastic about the program. Confirmed that the teachers like the program, but needed a little extra support. Researchers and implementation managers also had difficulty communicating with administrator (e.g., email inbox was always so full that any emails we sent were returned). Example of a school with lower fidelity

22. 22 Fidelity of implementation and teacher effectiveness made a difference: Year 2 School Gains on WJIII Brief Reading Assessment Pilot/Comparison School Average Fall to Spring Gain, Year 2 Pilot19.1 Pilot10.2 Pilot*13.1 Pilot*3.5 Pilot2.6 Pilot*3.7 Comparison2.6 Comparison2.1 Comparison2.2 Pilot1.2 Comparison2.4 Comparison1.3 Comparison1.1 Comparison1.1 Comparison1.3 Pilot*0.5 Comparison0.3 Pilot-1.5 Comparison-6.0 Pilot*-13.2 Significantly Positive Gain Significantly Negative Gain Non-significant Gain One of these schools was the low fidelity school described previously. One of these schools had a teacher new to teaching first grade who had been in and out of the classroom, and the relationship between the principal and this teacher was strained. All schools spent less than the required 60 minutes on the Listening and Learning Strand and the Skills Strand. One of these schools also had a very strong focus on writing, which would occasionally cut into their other strand time. Three Site Visit Schools Did Not Significantly Gain or Declined in Performance from Fall to Spring:

23. 23 Executive Summary – Achievement Gains  By nearly all measures, Core Knowledge Language Arts (CKLA) students made significantly greater gains in Year 2 than their peers  Overall Achievement  When compared to peers (both peers at demographically similar schools and peers at Reading First schools), 1 st graders taught with the CKLA program made more progress in reading and social studies.  Spring science scores were significantly greater for students in pilot schools than comparisons schools, but not significantly different from students in Reading First comparison schools.  High vs. Low Performers  High and low performing CK students made larger gains than comparison students  Year 2 vs. Year 1 of the Program.  Both students with only one year of CKLA instruction and those with two years of CKLA instruction made greater gains than their peers at comparison schools.

24. 24  Surveys and case studies indicate overall high levels of administrator and teacher satisfaction with the CKLA Program and high levels of fidelity in program implementation.  While survey respondents had favorable reactions to CKR, they indicated that they felt CKLA rated similarly to other programs they had used.  Surveys and case studies revealed that teachers are struggling with how to differentiate instruction with CKR, in particular for special populations.  Case study schools’ fidelity to the program aligned with their achievement gains: schools with low fidelity demonstrated less gains Executive Summary – Surveys & Case Studies