Evaluation of Shoreline Science Jia Wang & Joan Herman UCLA Graduate School of Education & Information Studies National Center for Research on Evaluation, Standards, and Student Testing (CRESST) The 2005 CRESST Conference Los Angeles, CA September 8, 2005

2 Overview - Purpose - Evaluation Approach - Evaluation Questions - Description of Assessments - Results & Findings - Summary and Conclusion - Example Feedback from Teachers

3 Purpose of the Evaluation Study 1.Evaluate the effects of literacy and science integrated instructional units on student interest, motivation, and learning 2.Evaluate the quality, usability, and utility of Shoreline Science & Terrarium Investigations

4 Evaluation Approach Quantitative Approach: - Descriptive analysis - Hierarchical Linear Modeling (HLM) analysis Qualitative Approach: - Analysis of teacher interviews

5 Evaluation Questions 1.How do the materials “work”? - How engaged and motivated are students? - What are teachers’ reactions to the quality, usability, and utility of the units? 2.What problems and misunderstandings do teachers and students encounter in implementing the materials? 3.What are the effects of using the materials on students’ learning of science and reading of informational science texts?

6 Evaluation Questions (continued) 4.For whom, for what purposes and/or in what contexts do these materials appear most effective? 5.What factors contribute to and/or detract from successful implementation? 6.How can the units be improved? To facilitate implementation? To enhance students’ learning in science and reading of informational texts?

7 Shoreline Science – Description of Assessments Literacy pre- and posttest: - 95 comprehension items 25 narrative items 21 target science items 25 science component items 24 content area comparison items - 34 vocabulary items 12 shoreline science only items 9 terrarium investigations items 13 shoreline science and terrarium investigations items

8 Shoreline Science – Description of Assessments (Science) Science pre- and posttest - 19 multiple-choice (MC) items - 34 short-answer (SA) items

9 Shoreline Science The study consisted of 25 treatment teachers and 10 control teachers. The final science data students - 16 treatment teachers and 8 control teachers The final literacy data students - 12 treatment teachers and 3 control teachers

10 Shoreline Science - Science: Descriptive Results

11 Shoreline Science - Science: Descriptive Results (continued) Students in both treatment and control groups improved in the posttest points improvement for MC items points improvement for SA items Treatment group students had higher gain scores than control group students, about one point higher (one item).

12 Shoreline Science - Science: HLM Results

13 Shoreline Science - Science: HLM Results MC gain scores: - Shoreline science students scored higher than control group students, but the difference was not statistically significant. - Female students improved more than male students SA gain scores: - Shoreline science students scored one point higher than control group students, and the difference was statistically significant. - Female students improved more than male students. Total gain scores: - Shoreline science students scored 1.6 points higher than control group students, and the difference was statistically significant. - Female students improved more than male students.

14 Shoreline Science - Literacy: Descriptive Results

15 Shoreline Science - Literacy: Descriptive Results (continued) Treatment students had higher gain scores than control students. (The difference was about 1.5 higher for the comprehension items, and in the range of for the vocabulary items.)

16 Shoreline Science - Literacy: HLM Results on Comprehension Items

17 Shoreline Science - Literacy: HLM Results on Comprehension Items As expected, -Shoreline Science students outperformed control students on target science gain scores, by 1.5 points. -No differences were found for the two groups of students on narrative and science comparison gain scores. Unexpectedly, - Shoreline Science students outperformed control students on science comparison gain scores, by 1.6 points

18 Shoreline Science - Literacy: HLM Results on Vocabulary Items

19 Shoreline Science - Literacy: HLM Results on Vocabulary Items As expected, -Shoreline Science students outperformed control students on shoreline science only vocabulary gain scores by 1.3 points. Unexpectedly, - Shoreline Science students outperformed control students on shoreline science and terrarium investigations gain scores by 3 points - Shoreline Science students had a higher gain score on terrarium investigation vocabulary items, by 1.2 points

20 Qualitative Evaluation Teacher Phone Interviews: - lasted about one-hour for each teacher - were audio-taped, transcribed, and coded - 7 Shoreline Science teachers - 6 Terrarium Investigations teachers

21 Shoreline Science – Teacher Interview Findings Background Information -4 teachers had bachelor’s degrees, and 3 had master’s degrees. -Teachers’ estimates of their students’ level of proficiency in science and literacy varied greatly from teacher to teacher. It was estimated that the majority of the students were at or below grade level in their science and literacy proficiency.

22 Shoreline Science – Teacher Interview Findings (continued) Implementations -Used the materials at least 3 times a week -All teachers used the teacher’s guide, the readers, the embedded assessments, magazine assessments, and pre- and post-assessments. -All teachers used the home activities, but not every teacher used all of them. -Most teachers used shared or paired reading in the classroom. -Students took active roles throughout the unit and loved it, regardless of their previous achievement level.

23 Shoreline Science – Teacher Interview Findings (continued) Teacher Reactions: -All teachers found the teacher’s guide useful. It was clearly written and planned out, gave lots of ideas, strategies, and background knowledge, and laid out all the experiments. -All teachers found the readers very useful. -The assessments were very useful in measuring student growth, and most teachers thought they were useful for EL students too. -The time estimate was off—too much content in too little time. -All teachers said they would use the Shoreline Science Unit again if they had the choice.

24 Shoreline Science – Teacher Interview Findings (continued) Student Learning: - For second graders, the Unit was most effective for those who were at or above grade level. For third graders, the Unit was effective for all students, especially the high achievers and those at or above intermediate language level. - The Unit was very effective in helping students learn science and literacy, especially science.

25 Shoreline Science – Teacher Interview Findings (continued) Factors for Success: - student engagement - the mix of activities and the sequence of the lessons - use of group activities versus partner activities - creation of a structure for students to internalize concepts - the integration of literacy and science - teacher enthusiasm and willingness to do the preparation work - the building of both background knowledge and concepts - connecting ideas in the books to actual experience

26 Shoreline Science – Teacher Interview Findings (continued) Suggestions for Improvement: - need more ideas for the teachers on how to modify assignments for lower level students and special learners, especially for reading - highlight the must-do’s or prioritize lessons to guide teachers with time constraints and guide teachers who use it as a main or as a supplemental program - have books of a lower reading level. - include more pictures for visual learners, especially for the teacher’s guide - include more anecdotal information and background information in the teacher’s guide

27 Shoreline Science: Summary and Conclusions -Teachers were enthusiastic -Students were motivated to learn -Students of teachers who used Shoreline Science outperformed the control group students -Students’ ethnicity and grade level had no effect on how much they learned -Shoreline Science materials were effective in improving student learning and knowledge, especially for students without much prior knowledge

28 Example Teacher Feedback (1) “The best thing about that program was the readers. They were phenomenal. My kids loved them. They would look at them, even when we were not doing science.”

29 Example Teacher Feedback (2) “They would talk about what they like. In January we had a writing test. This had nothing to do with science, but they had to write and tell about school. Almost all of the students’ favorite thing was science.”

30 Example Teacher Feedback (3) “My students absolutely loved it. …. After the first few lessons, we were in the middle of the second or third hands-on lesson, and some kid was like, ‘Wow, this is the best class ever, we never got to do anything like this before!’ They were always making comments like that. It was very interesting. … They’re pretty much engaged in a lot of things. And I brought that to it, it would be like, ‘Today, guess what we’re doing in science. We’re going to learn about an oil spill!’ And they would cheer and just be so into pretty much everything. So for my students it was very engaging.”

31 Example Teacher Feedback (4) “It was really high interest for the students, really motivated, at least my group. They were very engaged. They were so excited to be doing hands-on lessons and things like that, they love.”