Teaching and Learning Physical Science in Urban Secondary Schools: Assessing the Assessments Joan Whipp, Michael Politano, Michele Korb – Marquette University Mel Sabella, Chicago State University David Hammer, University of Maryland Kristi Gettelman, Wisconsin Conservatory of Lifelong Learning Kelly Kushner, Shorewood Intermediate School Megan Ferger, Roosevelt Middle School

Modeling in Physical Science for Urban Teachers Grades middle and high school science teachers from ten urban Milwaukee schools 2 summer institutes; 8 follow-up half-day modeling workshops; an online community; classroom mentoring Physical Science Modeling Workshop curriculum developed at Arizona State Focus on increasing teacher content knowledge and instructional strategies

Assessing Content Knowledge

Physical Science Concept Inventory (PSCI) Multiple-choice diagnostic test based on: –Lawson Test of Scientific Reasoning (Lawson), –Chemistry Concept Inventory (Mulford and Robinson), –Trends in International Mathematics and Science Study (TIMSS), –Energy Concept Inventory (Swackhamer et al.) served as an assessment of student and teacher content understanding of topics in Physical Science

Gains on PSCI gain calculated using the formula developed by Hake 1. Participants Pre-/post-test during 1 st summer workshop Students in subset of participants ’ classes Baseline established in April 2004 Given as pre-/post-test in Fall 2004/Spring 2005 and in Fall 2005/Spring R. Hake, “ Interactive engagement versus traditional methods: a six-thousand-student survey of mechanics test data for introductory physics courses,: Am. J. Phys., 65 (5) (1997).

Student Performance on PSCI baseline (N=213) (% correct) pre-test (F04) (N=610) (% correct) post-test (S05) (N=395) (% correct) pre-test (F05) (N=298) (% correct) post-test (S06) (N=262) (% correct) 31%30%31%30%36% Student performance was essentially the same for the baseline, the pre-test, and the post-test during the 1 st year of implementation. A mean post-test score of 36% was achieved at the end of the 2 nd year of implementation. Although this is a small gain, it does indicate some improvement over what had been achieved in the past. The 2nd year data focused on the teachers who used the modeling curriculum at a significant level.

Summary Implementation 1 (F04-S05) Looking at the entire set of PSCI data from year 1, there is little evidence for improvement in performance on the diagnostic. There does seem to be clear improvement in student responses to question 13, which deals with the concept of area. Looking at the data for individual sections, one section does seem to show some significant gain – scores went from 37% to 47%. Implementation 2 (F05-S06). Looking at the entire set of data from year 2, there is a greater degree of improvement, but the gain is still small. In one case (section 9), students went from a pre-test score of 33% to a post-test score of 60%, a 39% gain. There does seem to be clear improvement on a greater number of individual questions.

Assessing Attitudes about Science

Student Attitudes Attitudes Survey –consists of twenty-two statements, in which participants are asked whether they agree or disagree on a five point Lickert Scale. –Given as a pre and a post test. Statements from the Attitudes survey were obtained from the: –Maryland Physics Expectations Survey (MPEX, University of Maryland PERG), –the Epistemological Beliefs Assessment for Physical Science (EBAPS, Elby) and –the Epistemological Questionnaire (M. Schommer.)

Sample Statements from the Attitudes Survey I developed a better understanding of science by discussing my ideas with my classmates and my teachers. Learning science makes me change some of my ideas about how the physical world works.

Summary of Results There was little change toward expert-like views after approximately one year of modeling instruction. Researchers have found that attitudes toward science and science learning science are typically very difficult to change in the course of a semester or a year. E. F. Redish, J. M. Saul, and R. N. Steinberg, “ Student expectations in introductory physics, ” Am. J. Phys. 66 (3), (1998).

Middle School Student Responses (revised version of instrument) Results from six sections employing the Modeling Method. –over 64% of the students stated that this course was different than other science courses they have taken (Q2). (Ranged between 64% to 81%.) –Generally students seemed to feel that discussing ideas with their classmates and teachers helped them learn the material (Q11). (Ranged between 50% to 100%.) –Students also seemed to feel that they were ready for their next science course (Q17) and many felt that they were capable of learning science (Q20). (In most cases more than half the class responded favorably to these statements.)

Teacher Self-Assessment Interviews Surveys

Interviews with focus group: Summary Teachers –who participated in the focus group were very committed to the Modeling approach –were comfortable modifying the instructional materials so that it better fit their students –felt that students did have a better understanding - based on classroom observations and alternate forms of assessment. the modeling method created a classroom environment in which: –the classroom was more interactive, –their students began to realize that that they were partly responsible for their learning –the teachers recognized that their students were capable of constructing their own understanding –the teachers were able to better see where their students were in their understanding.

Changes in Teaching Describe how your teaching has changed as a result of the modeling curriculum. The two statements that the teachers agreed with the most: –my knowledge of concepts in physical science has improved. –I now engage my students in more group-work in the classroom.

Changes in Teaching Describe how your teaching has changed as a result of the modeling curriculum. Almost all teachers seem to have adopted the technique of using whiteboards in in their classes to stimulate group interaction. In addition, the use of a standard text book also decreased. 47% of the teachers stated that they used the textbook regularly or frequently before being engaged in modeling and 18% stated regular or frequent use after. In general, these results, show a shift toward the approach emphasized in the modeling curriculum.

Using an Observation Tool to Assess Teaching Reform: The RTOP

Purpose of using the RTOPs To maintain a standard for observation of modeling in the classrooms To provide a common language as reference when evaluating the effectiveness of and implementation of modeling curriculum. To provide a standard for evaluation of dialogue among teachers and mentors (in summer workshops and during online discussions)

Typical Evaluation Items on the RTOP and how they informed the evaluation: Lesson Design and Implementation (1 & 2) Content (6 & 7)

RTOP example questions Procedural Knowledge (11- 14)

Other typical RTOP evaluation points Classroom Culture Communicative Interactions (16 & 17) Student/ Teacher Interactions (22-24)

Results from RTOP and RTOP discussion Teachers –improved most on understanding the modeling approach –struggled with initial implementation due to student resistance Students –use of hands-on manipulatives and experimental design improved –student ability to communicate results did not improve in most cases. The RTOP was only a “ snap shot ” of teacher implementation of modeling, yet fostered discussion and understanding of this method of instruction.

Fascinating results Classroom observations showed that fifth graders ’ response to Modeling were very mature and developed. This occurred in a public school setting. Highest gains in the PSCI were in a classroom where a teacher (according to RTOP data) scored the lowest on modeling skills and implementation of curriculum.

Using Action Research to Assess the Improvement of Teaching and Learning Physical Science

Three Action Research Projects Kushner, K.M. (2006). A comparison of constructivist and traditional methods on students ’ scientific measurement skills. Unpublished master ’ s action research study, Cardinal Stritch University, Milwaukee, Wisconsin. Gettelman, K. (2006). The use of a modeling method to teach physical science concepts at the elementary/intermediate level. Unpublished master ’ s action research study, Marquette University, Milwaukee, Wisconsin. Ferger, M.L. (2007). Traditional vs. modeling instruction of eighth grade chemistry students. Unpublished master ’ s action research study, Cardinal Stritch University, Milwaukee, Wisconsin.

Questions How did student understanding of physical science and related math concepts change? How, if at all, did student attitudes toward science change? In what ways, if any, has student engagement n the scientific process increased – i.e. making hypotheses, collecting and analyzing data, and forming conclusions to understand and explain physical science concepts? What is the impact of a modeling curriculum on student achievement, in comparison to a traditional science curriculum?

Action Research Designs Researcher Kelly Kushner Kristi Gettelman Megan Ferger Subjects & Sample Population 45 – 8 th Grade Students th Grade Students 100 – Eighth Grade Students in 4 Science classes School at which study took place midwestern suburban public intermediate school Midwestern urban public elementary school midwestern urban public middle school

Researcher Kelly Kushner Kristi Gettelman Megan Ferger Content TaughtMeasurementPhysical ScienceChemistry Teaching Techniques Used Modeling vs. Traditional ModelingModeling vs. Traditional Assessment instruments implemented Measurement Pre and Post Test Student Science Attitudes Pre/Post Survey – all participants Properties of Matter Pre/Post- Assessment – all participants Classroom Observations and Reflections Student Reflections in Science Log – all participants Student Interviews – 6 participants A combined fact and critical thinking pre and post-test

Findings Students showed greatest gains when they were taught with a blended approach of traditional and modeling methods Students showed greatest gains in their understanding of principles of matter and the least in their understanding and ability to apply advanced measurement principles Student resistance to modeling was a barrier that needed to be addressed initially; students were too used to being given right answers rather than thinking for themselves. Student attitudes toward science became more positive. Student engagement in scientific processes has increased.

What We Have Learned through Action Research Better understanding of the physical science concepts we are teaching Constructivist principles which underlie the modeling approach to teaching science

Understanding Our Learners Common student misconceptions about physical science and how to address those misconceptions in teaching How changing our practice can impact student learning; Middle school students vary greatly in ability to reason abstractly

Understanding Our Instruction A blended model of traditional and constructivist methods is most effective Became more conscious of our own instructional design and planning and how to use student work samples and classroom discussion to guide and inform planning Changed our focus from teacher to student- driven instruction and assessment. Applied scientific principles to instruction, measured benefits to student learning

Where do we go from here? What will we, as science educators, take forward from this experience?

Ongoing Action Research We will continue to use action research methodology to monitor student learning and inform our teaching Pre/post testing Multiple data sources Comprehensive data analysis

Continuing the Modeling Method Using modeling methods to teach variety of science content, rather than just physical science Using whiteboard presentations in other areas of curriculum to promote student-to-student questioning and understanding

Extending the Modeling Method Sharing findings with colleagues, in order to improve science instruction in other areas of school/district Connecting physical science and math instruction, particularly measurement and geometry strands, to promote deeper student understanding