1. Deborah Harloff, RCSD Executive Director of Visual and Performing Arts Andrew MacGowan, III Project Administrator and Arts Impact Study Evaluator

2. The research on arts integration is based on the hypotheses that there is a positive impact between integrated arts instruction and increased student achievement.

3. What is arts integration? Arts Integration is the process of combining content of the arts with other classroom content such as English Language Arts, Science, Social Studies, or mathematics. This is done by making clear content connections between the various curricular areas. Visual art Dance Theatre Music

4. The Arts as an intervention Little empirical data Student motivation

5. 1 st Time 2006 2006-2009 Over 200 applicants Ranked 13 th nationally of 25 awardees Perfect score on evaluation design

6. “True Experimental Design” What and Why “Medical model,” “True Experimental,” “Randomized Trial” all mean the same thing Until NCLB, very few (< dozen) randomized trials in education The most rigorous form of evaluation “quasi-experimental” is the next best form of evaluation

7. The RCSD Design 9 randomly selected elementary schools (out of 39) 180 teachers 4,000 students 7 grade levels (K-6) Remaining 29 schools served as the control group Approximately 30,000 students in all, over the course of four years (baseline year + three years of treatment) Statistically “robust”: results can be believed

8. Measures Sixteen major New York and national standardized tests were used, in English Language Arts, Mathematics, Science and Social Studies

9. Components of the grant Ten session teaching artist residency Additional arts experience Supplies to implement the residency Ten hours of PD for teachers

10. Year One: Selection Computer-generated randomization of schools Introduce project to building administrators Interview and hire teaching artists Create planning document Grade level planning sessions

11. 4 Schools Opt Out Based on the following rational: the school based planning team (SBPT) was concerned about the ten hours PD requirement the teachers at one of the schools were concerned that they would be writing the curriculum for the teaching artist another school cited the concern about accepting a new program when the school had a new principal.

12. Years Two & Three Year two one school decided to opt out of the program During year three, some of the faculty at one school refused to meet thereby essentially opting out of the program. Due to scheduling conflicts and organizational skills one teaching artist was not retained. A different teaching artist moved out of state and could not complete the project.

13. Year One: Implementation Grade level planning meetings were scheduled Schedules were arranged to accommodate teaching artist availability as well as classroom continuity. Materials and supplies needed for the project were discussed.

14. Years Two & Three: Implementation Due to inconsistencies in planning meetings, a planning document was developed to create uniform lesson and unit plans Schedule protocol remained the same for year two.

15. Introduction of the 4 Art Forms The art form that was integrated at the school was randomly selected. Two schools were assigned as music two were assigned as dance Three schools were assigned as theatre Three were assigned as visual art

16. Skills and Content Taught Visual Art Music Theatre Dance

17. Arts Impact Study PD Professional development (PD) to reinforce understanding of integration techniques and strategies in music, dance, theater and visual arts. Professional reading 96% approval rating Hands on Models strategies Provides materials

18. PD for Teaching Artists Program objectives, program implementation and classroom residencies. Operational practices were taught to the teaching artists Techniques to reinforce ELA, Math, Science and Social Studies with the teaching artists’ area of expertise. Collegial sharing - discussed best practices and successful strategies for co-teaching and co-planning lessons with classroom teachers.

19. Main Conclusions Using both national and NY tests, positive and significant effects, from 5% to 30% improvements Across most grade levels K – 6 Students with Disabilities showed gains English Language Learners showed gains Gains in both low and high performing schools Gains in ELA and Math Science and Social Studies (not statistically significant)

20. More conclusions The lower the grade level, the greater the effects (e.g., the younger the student, the greater the effects); The more economically disadvantaged the student, the greater the effects. Major, unexpected finding: Arts Impact acts as a protective factor for students who did not have the benefit of attending a higher performing school.

21. Top conclusions (charts to follow) Kindergarten through 2nd grade students in the treatment schools enjoyed the greatest degree of gains, including Students With Disabilities. By Year 3, Kindergarten students realized a 14% differential, 56% of the treatment students reached Hi/Scope Child Observation Record (COR) Proficiency Level 4 (on the COR 5-point scale), versus 41% of the control students. The gains in Year 3 for Kindergarteners With a Disability were even greater, with 25% more gaining on the COR Proficiency Level 4 (on a 5-point scale), 30% for the control students vs. 55% for treatment students. The comparative gains for Kindergarten English Language Learners in Year 2 saw an approximate 4% advantage. By Year 3, 1st grade students on Free or Reduced-Price Lunch (an indication of poverty), realized a 6% comparative gain in Reading, Language Arts and Mathematics (identical in all three) as measured by the CTB Terra Nova (which is configured to New York State Standards.)

22. More conclusions... By Year 3, 1st grade English Language Learners realized a 12% comparative gain in Reading, an 18% comparative gain in Language Arts and a 13% comparative gain in Mathematics. By Year 3, 1st grade Students With Disabilities realized a 15% comparative gain in Reading, a 6% gain in Language Arts and an 8% gain in Mathematics, compared to control school students. For reasons we do not yet understand the intermediate grades (3 – 6) see greater gains for students in 4th and 6th grade than in 3rd and 5th grade. In low performing schools, 4th grade students realized a 13% higher rate of English Language proficiency and a 10% higher rate of Mathematics proficiency (as shown by the New York State Tests), compared to low performing control schools. In low performing schools, 6th grade student gains were noteworthy -- a 13% higher rate of treatment students achieving English Language Arts proficiency (New York State Levels 3 and 4), than control school students, and a 20% higher rate of treatment students achieved Mathematics proficiency compared to students in the control schools.

23. Gains in Kindergarten – all students VII: Rochester Kindergarten Students Improvement in English Language Arts, Math, Science and Behavior 0 10 20 30 40 50 60 70 80 90 Fall, 2006Spring, 2007Fall, 2007Spring, 2008 (p<.01) Fall, 2008Spring, 2009 (p<.001) Percent Scoring at or above 4 on the Total K-COR 5 level Scale Control Schools Program Schools (p<.001) Martin F. Gardiner, Ph.D.

24. Gains for Grade 1 ELL V: Rochester 1 st Grade Students With Disabilities Progress in English Language Arts and Math 0 5 10 15 20 25 30 35 40 45 Reading 2007-2008 (p<.05) Reading 2008-2009 NY Language 2007-2008 NY Language 2008-2009 NY Math 2007-2008 (p <.01) NY Math 2008-2009 Percent Scoring at or above Grade Level Control Schools Program Schools Year 1 Reading (2006- 2007) NY Lang. (2006- 2007) (p <.01) NY Math (2006- 2007) Martin F. Gardiner, Ph.D.

25. Gains for Grade 1 Students with Disabilities V: Rochester 1 st Grade Students With Disabilities Progress in English Language Arts and Math 0 5 10 15 20 25 30 35 40 45 Reading 2007-2008 (p<.05) Reading 2008-2009 NY Language 2007-2008 NY Language 2008-2009 NY Math 2007-2008 (p <.01) NY Math 2008-2009 Percent Scoring at or above Grade Level Control Schools Program Schools Year 1 Reading (2006- 2007) NY Lang. (2006- 2007) (p <.01) NY Math (2006- 2007) Martin F. Gardiner, Ph.D.

26. Quantitative Data 143 teacher interviews were conducted at the end of the third year of the grant project 1. What worked best? the arts activity3021% supported content goals2618% the integration and the artist sessions2115% student performance128% movement activities53% hands-on aspect of the program53% teaching artist was organized53% Particular teaching artist53% flexibility of the teaching artist42.7% class participation32% artist modeling for the students32% PD session2 Culturally based activities1 Planning together1 Picture books1 Illustrating1 Engage students1 Team work activities1

27. 2. What is your evidence of student learning? Response# of responses Percentage Student performance or project produced3122% Students engagement increased2517% Application of knowledge from lesson to classroom discussion 1410% Student recall4028% Use of vocabulary107% Students transfer of knowledge form one content area to another 2 Test scores2 Student writing1 Student confidence increased1

28. 3. What activities that the teaching artist modeled have you tried or would you do? Music activity3625% Movement/ dance activity3424% Theatre activity3424% Visual art activity3122% Changing environment – took kids to another space 2 Cooperative activities43% Make costumes1 Make musical instruments3

29. 4. What support do you need to continue with arts integration? Responses# of Responses Percentage Materials, equipment, resources4531% Professional development4028% Funding75% More time during the day ( time constraints)1813% More Teaching Artist sessions1712% Support for using integrations75%

30. Did it work? In short, Yes. 30,000 students and 16 standardized tests.

31. Summary Overall, we have concluded the earlier the intervention, the greater were the effects sizes: We observe statistically significant positive effects in Kindergarten, first and second grade. We also observed the higher the poverty as determined by the Free and Reduced Price Lunch code the greater the effect size

32. Evaluation Tips If you are contracting with an independent evaluation outfit, get your Non-Disclosure Agreement done ASAP; Work with US DoE on this – and feel free to borrow from Rochester’s model (available upon request) Go with unique IDs, so you can use your Lunch Code data – which can reveal big differences in effects size NEVER, EVER release Lunch Code information with personally-identifiable data

33. What Next? The results of this project – and the independent evaluation of this project – pose significant policy implications for high-poverty urban schools, including those with high populations of English Language Learners and Students with Disabilities.