11 Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 2 Report Research and Policy Support Group CONFIDENTIAL – PLEASE DO NOT DISTRIBUTE

2 Executive Summary – Achievement Gains  By nearly all measures, Core Knowledge Reading (CKR) students made significantly greater gains in Year 2 than their peers  Overall Achievement  When compared to peers (both peers at demographically similar schools and peers at Reading First schools), 1 st graders taught with the CKR program made more progress in reading and social studies.  Spring science scores were significantly greater for students in pilot schools than comparisons schools, but not significantly different from students in Reading First comparison schools.  High vs. Low Performers  High and low performing CK students made larger gains than comparison students  Year 2 vs. Year 1 of the Program  While significant gains were found in almost all domains, the magnitude of these gains was smaller in Year 2 than in Year 1.  Both students with only one year of CKR instruction and those with two years of CKR instruction made greater gains than their peers at comparison schools.

3  Surveys and case studies indicate overall high levels of administrator and teacher satisfaction with the CKR Program and high levels of fidelity in program implementation.  While survey respondents had favorable reactions to CKR, they indicated that they felt CKR rated similarly to other programs they had used.  Surveys and case studies revealed that teachers are struggling with how to differentiate instruction with CKR, in particular for special populations.  Case study schools’ fidelity to the program aligned with their achievement gains: schools with low fidelity demonstrated less gains Executive Summary – Surveys & Case Studies

4 Methodology: A multi-method, longitudinal research design YEAR 2 Assessments (at 10 CKR schools & 10 comparison schools, 3* of which are Reading First schools): Pre- and post-test of literacy skills Additional tests of literacy skills at end of each year Tests of science and social studies skills at end of each year Teacher and Administrator Surveys (at 10 CKR schools): Assesses satisfaction with and impact of CKR Case studies (at 5 CKR schools): Classroom observations, administrator & teacher interviews Hypothesis: First Graders taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers. Focus of the Evaluation * Last year there were 4 Reading First comparison schools. This year there were 3 Reading First comparison schools: one comparison school used Reading First in Year 1 but not in Year 2.

5 Overall, CKR and comparison schools had similar demographic profiles. However, the differences between groups, especially % of Black/ Hispanic students, was significantly larger than last year. CKR Students (N = 781) Comparison Students Note: Data from was used to select comparison schools (data presented here are from the school year). Percent of Students (Number of Students for School Size) (N = 343)* N = the number of students for whom both fall and spring data were available. A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested. As a result, all analyses control for student demographics.

6 Evaluation of Achievement Gains

7 CKR students displayed greater gains & higher Spring scores in nearly all literacy and content knowledge domains relative to comparison students Woodcock Johnson (Fall and Spring)Terra Nova (Spring Only) TestW-J Brief Reading W-J Word Attack (Decoding) W-J Spelling of Sounds (Written Spelling) Terra Nova Reading (Oral Reading Comprehension, Vocabulary, Basic Reading, Decoding) Terra Nova Social Studies Terra Nova Science W-J Letter Word Identification (Basic Reading Skills) W-J Passage Comprehension (Oral Reading Comprehension) CKR Schools Compared to… All Comparison Schools CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Spring Scores CKR Sig. Greater Spring Scores CKR Sig. Greater Spring Scores Reading First Comparison Schools CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Gains/ Spring Scores CKR Sig. Greater Spring Scores CKR Sig. Greater Spring Scores X No Significant Difference Note: All analyses control for student demographic characteristics.

8 Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR Students Significantly Higher p <.0001 CKR students displayed nearly 2x greater literacy gains relative to students at demographically similar comparison schools (N = 758)(N = 342) The magnitude of the group difference was smaller than last year (in Year 1, CKR students’ gains were 5X greater than their peers).

9 (N = 233)*(N = 100)* * Data presented represent the three comparison schools that are Reading First schools and the three CKR schools who were statistically comparable matches. CKR Students Significantly Higher p <.0001 Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR students displayed 6x greater literacy gains relative to students at Reading First comparison schools The magnitude of the group difference was smaller than last year (in Year 1, CKR students’ gained 14.3 pts while Reading First schools declined 4.1 pts).

School TypeAverage Gain Pilot19.1 Pilot10.2 Pilot13.1 Pilot3.5 Pilot2.6 Pilot3.7 Comparison2.6 Comparison2.1 Comparison2.2 Pilot1.2 Comparison2.4 Comparison1.3 Comparison1.1 Comparison1.1 Comparison1.3 Pilot0.5 Comparison0.3 Pilot-1.5 Comparison-6.0 Pilot Gains at Each School: All of the schools making statistically significant gains were CKR pilot schools Significantly Positive Gain Significantly Negative Gain Non-significant Gain

School TypeAverage GainRank in 2010Rank in 2009 Pilot Pilot Pilot Pilot Pilot Pilot3.767 Comparison Comparison2.188 Comparison Pilot Comparison2.411 Comparison Comparison Comparison Comparison Pilot Comparison Pilot Comparison Pilot Gains at Each School in Year 1 and Year 2: Some CKR schools that made significant gains in Year 1 did not make significant gains in Year 2 and vice versa Significantly Positive Gain Significantly Negative Gain Non-significant Gain

12 CKR & All Comparison SchoolsCKR & Reading First Schools CKR Schools Reading First Schools Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test w/ Demographic Controls) Comparison Schools At all achievement levels, CKR students demonstrated greater literacy gains than students at comparison and Reading First schools *Significantly different at p <.05 * * * * * * * While the lowest performing students made greater gains than students at comparison schools, this difference was not significant. All other differences were statistically significant.

CKR students made significant gains regardless of whether it was their first or second year in the program 13 New in Year 2 Years 1 & 2 CKR Comparison CKR students with two years of the program made larger gains than their peers. Similarly, CKR students with one year of the program made larger gains than their peers. CKR Years 1 and 2 (n = 527) Comparison Years 1 and 2 (n = 154) CKR New in Year 2 (n = 237) Comparison New in Year 2 (n = 189)

14 Administrator and Teacher Surveys 14

15 Administrators reported satisfaction with the CKR program No 6 Yes 1 15 Will your 1 st grade classrooms be using the CK Reading program next year? (n = 7) Would you recommend the CK Reading curriculum to other administrators you know? (n = 7) 1 Not Sure 6 Yes Administrators’ overall satisfaction with CK Reading (n = 7*) * Three administrators did not complete the survey. “Although the CK program has a lot of good components, its lack of a writing curriculum and differentiated instruction is a major concern for a program being used for 1st grade.” Very Somewhat Satisfied 3 3 Very Dissatisfied 1

16 Administrators reported that 1 st grade CKR teachers’ practice was about the same as it was the previous year (prior to CKR) 16 Discussing/ sharing ideas on teaching strategies w/ other teachers (n = 7) Collaborating with teachers in other grades (n = 7) Collaborating with other first grade teachers (n = 7) Using assessment data to drive instruction (n = 7) Somewhat less than last year Much more than last year 2 3 About the same as last year 1 Administrators’ Views: “I like CKR. I see improvement from last year…However…it’s the balance that I would like to see CK achieve. More of the sight words that children in early childhood should acquire…I would like to see complete alignment between Pre-K, K, and 1st, and now we’re going to 2.” About the same as last year 4 Much more than last year 2 Somewhat more than last year 1 About the same as last year 3 Much more than last year 2 Somewhat more than last year 1 Much less than last year 1 About the same as last year 3 Much more than last year 3 Somewhat more than last year 1 1 N/A

17 Teachers’ overall satisfaction with CK Reading (n = 29) a Percent of Respondents 50.0% 82.7% Much Better Somewhat Better Teachers’ overall opinion of CK Reading compared with other reading programs (n = 28) b Very Satisfied Somewhat Satisfied Number of teachers selecting the “neutral” response: question a = 1 (3.2%); question b = 6 (19.4%). Teachers’ Views: “I think the skills training really works…the Skills Strand was kind of similar [to last year] because we did Foundations [which also had] lots of cards and decoding.” “I used the Guided reading program. I liked the small groups and being able to move students faster according to their ability with the Guided Reading but I love the phonics and comprehension part of the Skills Strand.” Most teachers were satisfied with the CKR overall but only half reported that CKR was better than other reading program

18 Goals of lessons are clear (n = 29) a I have enough time to complete daily lesson (n = 29) c Students find activities engaging (n = 29) b Somewhat Agree Strongly Agree Goals of lessons are clear (n = 29) d I have enough time to complete daily lesson (n = 29) f Students find activities engaging (n = 29) e Skills Strand Listening and Learning Strand Percent of Respondents Number of teachers selecting the “neutral” response: question a n = 0; question b n = 2 (7.1%); question c n = 2 (6.9%); question d n = 0; question e n = 6 (20.7%); question f n = 4 (13.8%). 96.6% 78.6% 72.4% 93.1% 68.9% 72.4% Teachers thought both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 1, teachers rated the Skills Strand more positively than the L&L Strand and had concerns about time.

Number of teachers selecting the “about the same” response: question a n = 4(13.8%); question b n = 8 (27.6%); question c n = 3 (13.8%); question d n = 9 (31.0%); question e n = 11 (37.9%); question f n = 9 (31.0%). Teachers compare CKR with other programs they have taught: As in Year 1, decoding skills are a strength of CKR while differentiation of instruction is rated less highly. Somewhat Better Much Better Accommodations for different learning needs (n = 29) f Ability to engage students and spark enthusiasm for reading (n = 29) e Compehen -siveness of program (n = 29) d Teaching content/ background knowledge (n = 29) c Teaching decoding skills (n = 29) a Sequence of instruction (n = 29) b Percent of Respondents 68.9% 55.2% 62.0% 51.7% 44.8% 31.0%

20 Site Visits and Interviews 20

21 Measuring fidelity to the CKR curriculum Examining implementation fidelity allows us to better determine: 1) whether achievement gains can be attributable to the CKR program and 2) which components of the CKR program teachers are struggling to implement and those they are implementing successfully We took several different approaches to measuring fidelity in the CKR Pilot schools: Site visits with classroom observations Site visits were conducted at 5 randomly selected Pilot schools 15 classrooms were observed Interviews with administrators and teachers Interviews were conducted with 13 teachers and 7 administrators Additional teacher and administrator survey questions

Examples from Checklist Check to see that CKR materials are being used and CKR visuals are posted Verify that the daily schedule allows for both the Skills and the Listening and Learning Strands to last a full 60 minutes each Whether or not students are familiar with language introduced in the domain Teacher engaging all students in exercises, practice, class discussions, and games Whether or not students are attentive to the teacher during instruction Whether or not students are eager to participate in the daily activities of the program Whether or not students are generally on task and complete assignments in a timely fashion 22 During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form These components were taken from the Core Knowledge Reading Pilot Observation Form These components were added as additional measures of student engagement

23 Students are attentive to their teacher during instruction Students respond appropriately to their teacher’s directions, questions, and assignments Teachers have Skills Strand materials posted and used Word Walls effectively Small group and/or individual practice time Full 60 minutes is allotted for both the Listening and Learning and Skills Strands Number of Classrooms Out of 15 Observed During Visit Not Observed During Visit Most classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons. We observed fewer classrooms engaging in the full 60 minutes of recommended CKR instruction.

24 Overall, teachers and administrators reported that high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction. Fidelity Survey Questions June 2010Teacher Responses Overall, I feel that the CK Curriculum is being implemented consistently in the First Grade classes at my school. 86% Strongly Agreed or Somewhat Agreed Typically, I find I have enough time to complete the daily Skills Strand Lesson 72% Strongly or Somewhat Agreed Typically, I find I have enough time to complete the daily Listening and Learning lesson 72% Strongly or Somewhat Agreed I am able to find time for daily small group instruction46% Strongly or Somewhat Agreed I check with the CK Learning Coordinator before making modifications to the program 56% Strongly or Somewhat Agreed Overall level of satisfaction with the CKR Program83% Very or Somewhat Satisfied

Anecdotes from Classroom Observation The first classroom had a vibrant atmosphere. Approximately 30 students, all of whom were seated on the carpet, appeared completely absorbed, watching their teacher and the images about the U.S. space program that were projected on the screen beside her. The classroom was covered with student work, Core Knowledge posters and chaining charts, and a hanging model of our solar system. The students were eager to share their answers, some of them using challenging vocabulary words that they had learned from earlier lessons like “perseverance.” After a writing/drawing exercise, in the final minutes of the lesson the teacher played a song about the sun that she had found on the internet. Some students pretended to be conductors, which they had learned about in the Mozart unit they had recently completed. This school had a teacher-coordinator who had participated as a kindergarten teacher the year before and was providing specific support for both the Core Knowledge pilot in first grade and the kindergarten classrooms, which had continued to use the curriculum. The school is also using the Core Knowledge curriculum for all grades. 25 Example of a school with high fidelity

26 Administrators’ Views Teachers’ Views Both teachers were very positive about the program’s approach to developing early childhood literacy in their 1 st graders. They liked using the program, its structure, content, and activities, and they also could see the students benefiting from it. Their criticisms of the program were of more “mechanical” elements (e..g, lack of page numbers in the readers, would have liked captions to accompany the pictures and make them more interactive, they felt rushed to get through all the material for any give day) Administrators’ Views Administrators were very supportive of the program They perceived large benefit from the program’s “multi-intelligence” approach and have been excited to watch the students make connections between lessons and to their lives outside the classroom as well. They perceived more parent involvement—children dragging their parents to the library, parents commenting that they are learning from their children. They felt the program worked well largely because their teachers are well supported and have fully embraced the program. Criticisms were that the program needs to include more “higher-order thinking” questions and provide material tailored for special populations (e.g., ELLs). Example of a school with high fidelity

Anecdotes from Classroom Observation The structure of the classrooms themselves created a challenge for the three first grade teachers—they only have three walls, and are open onto a hallway on the fourth side. Not only did the teachers have to contend with noise from the other classes, but also with the distractions of passersby using the hallway. In two of the three classes, students were having a difficult time responding appropriately to their teacher’s directions, questions, and assignments. The students in one of these classrooms also had a difficult time listening respectfully to each other during the discussion time at the end of a Listening and Learning lesson. This was the only example we saw of this kind of behavior across all 15 classrooms. One of the classrooms did not have the Core Knowledge materials posted and was missing the chaining and tricky words. Toward the end of the year, the CKR literacy coordinators worked with the teachers to take a different approach to differentiation. They divided the students into three homogenous skill groups and then each teacher worked with one group three days a week for a 45 min. session. 27 Example of a school with lower fidelity

28 Teachers’ Views Did not feel supported by the school’s administrators. This made it that much more difficult to be successful in their implementation of the program because the administrators were out of touch with what the teachers needed in order to be fully successful in their implementation. Sometimes found it difficult to fit in all the content for both the Listening and Learning and the Skills Strand, and would resort to picking out the highlights for a reading or selecting the most feasible activities in order to optimize their time. For all the challenges, the teachers were positive about the program itself—they saw their students engaging with it and connecting it to their lives outside the classroom. Administrator Views The administrator we interviewed was very enthusiastic about the program. Confirmed that the teachers like the program, but needed a little extra support. Researchers and implementation managers also had difficulty communicating with administrator (e.g., inbox was always so full that any s we sent were returned). Example of a school with lower fidelity

29 There was an alignment between achievement gains and fidelity to the curriculum Year 2 School Gains on WJIII Brief Reading Assessment Pilot/Comparison School Average Fall to Spring Gain, Year 2 Pilot19.1 Pilot10.2 Pilot*13.1 Pilot*3.5 Pilot2.6 Pilot*3.7 Comparison2.6 Comparison2.1 Comparison2.2 Pilot1.2 Comparison2.4 Comparison1.3 Comparison1.1 Comparison1.1 Comparison1.3 Pilot*0.5 Comparison0.3 Pilot-1.5 Comparison-6.0 Pilot*-13.2 Significantly Positive Gain Significantly Negative Gain Non-significant Gain One of these schools was the low fidelity school described previously. One of these schools had a teacher new to teaching first grade who had been in and out of the classroom, and the relationship between the principal and this teacher was strained. All schools spent less than the required 60 minutes on the Listening and Learning Strand and the Skills Strand. One of these schools also had a very strong focus on writing, which would occasionally cut into their other strand time. Three Site Visit Schools Did Not Significantly Gain or Declined in Performance from Fall to Spring:

30 Core Components of Year 3 Evaluation CKR assessments and longitudinal examination of relationship between literacy and content knowledge Survey/ site visits at comparison schools Assess administrators’ and teachers’ thoughts/ plans re: 3 rd grade Assess achievement gains of Kindergarten and 1 st graders - CKR grades that are not receiving as much intensive support (Year 4) Track student achievement through 3 rd grade (NYS tests) Recommendations for Year 3 Implementation Provide targeted support to teachers on: Differentiating instruction for students requiring different accommodations Finding time to complete lessons and have them allot a full 60 minutes to lessons (or adjust appropriately if not allotting full 60) Next Steps: Year 3