Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Yaron Doppelt & Christian D. Schunn Learning Research & Development Center University of Pittsburgh Implementing Embedded Assessment in a System Design Approach for Pupil-Centered Learning Paper presented at PATT20 Conference, November 2008, Tel Aviv

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC A teacher reflects on his feelings while participating in the PD A pupil reflects on past experiences in science A teacher: “So we’re getting at it. I’m learning with them. This is my first time so I’m paying attention to this” A pupil:“During sixth and seventh grades I felt that science lessons demand memorizing facts and reading long uninterested stories about phenomena. This year I enjoyed very much the hands-on activities.”

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Focus Questions  How does a design-based approach help researchers collaborate with teachers in order to achieve a wide and sustainable curriculum reform?  How does an embedded assessment assist teachers in identifying misconceptions and adapting their teaching?  What is the impact of professional development on teachers’ practice and pupils’ learning?  To what extent does teachers adopt the new learning module during the implementation in their actual class environment?

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Pedagogical Guidelines Work from pupil’s own needs  Real world examples  Pupils search for examples relating to their home / environment  Search for historical invention  Invent his/her own story  Teach systems perspective rather than parts perspective on electronics Cover core physics concepts  Meet science and technology standards  Implementing hands-on authentic engineering design process Create rich and engaging learning environment  Generative thinking  Teamwork  Reflective learning  Presentation skills

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC The pupils experience an authentic systems design approach to learning scientific and technological concepts Communication Presentation Reflection Self Assessment Analysis & Synthesis  Needs Analysis  Requirements  System/Sub-systems  Decision Matrix Detector Subsystem Indicator Subsystem Power Subsystem Prior Knowledge about alarm systems Construction Predict, Test, Troubleshoot, Improve Science and Technology Education Standards In contrast to typical project based learning in schools Pupils do the work themselves (not parents) No materials needed from home

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Focus Questions How well does a design-based approach help researchers collaborate with teachers in order to achieve a wide and sustainable curriculum reform?

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC I found a gap between the science standards and the existing learning environment National and state science standards  Design thinking  Know and use the technological design process to solve a problem  Systems thinking  Explain the parts of a simple system and their relationship to each other There was a need from the field to assist eight-grade science teachers

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Collaboration on an intervention program to meet the need  Pupil Module  Inserted into Electronics FOSS unit  4-6 weeks to design and construct an alarm system  Learn and come to own authentic design process  Teacher Guide  Professional Development: Series of workshops  Assessment tools for the teachers  On-Site Guidance

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Complex process of recruitment and loss of teachers at each step N=27 Teachers

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC We used a multi-dimensional approach to create a research space

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Proportion of Students in the Experiment and Contrast Groups Coming from High, Medium, and Low Socio-Economic-Status

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Overall Gains in Science Test Scores N=466 N=587 Effect size = 0.89 Design-Based Learning Scripted Inquiry Pupils ’ Gain Score

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Focus Questions How does embedded assessment assist teachers in identifying misconceptions and adapting their teaching?

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Evaluating the Individual Learning Process It is rare that a teacher declares:  The ideal outcome is that all learner achieve 100. (My opinion)  “The normal opinion is that learning achievements in a classroom normally distribute according to the normal distribution is an evidence to a resistance to the success of all learners”.(Mor, 1982)

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Embedded Assessment during the Learning Process  The assessment is designed in such a way that each pupils take a pretest before the first chapter of the learning module.  At the end of each chapter every pupil takes a test that includes: a posttest for the previous chapter and a pretest for the following chapter.  The test’s items were created base upon the experienced gained at the first year’s implementation and based on misconceptions we identified from the literature.  This test includes 17 multi-choice items and 3 open items.

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Adapting the learning process based on preliminary knowledge and identify students’ misconceptions  The teacher receives a deep understanding on students’ misconceptions.  The teacher can easily track the progress of the individual student.  In this process the teacher identifies if the students overcome the misconceptions.  The teacher does not need to invest huge effort to achieve this extra knowledge on the class achievements it is an embedded assessment which becomes a part of the learning process.

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Test Reliability oCronbach Alpah for the instrument was: Alpha = oCronbach Alpah for comprising the split-half test was: Alpha = oAnalyzing each of the instrument’s items – when omitting any of the item the Cronbach Alpha does not increase beyond 0.81

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Wide implementation with all the students from a mid-sized district. N=1324, 19 teachersSD=0.193

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Focus Questions What is the impact of professional development on teachers’ practice and pupils’ learning?

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC What is a Successful PD? Reform Curriculum Professional Development Teacher Practice Pupil Achievement  Distributing the PD throughout the implementation  Engaging teachers in an active learning process  Facilitating a community of teachers professionals

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Score Gains: PD vs. NPD Effect size = 1.12 Effect size = 1.68 PD: N=19 Classes NPD: N=7 Classes Science Questions Design Questions

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Score Gains: PD vs. NPD Existing Curriculum: N=5 NPD: N=5 Teachers PD: N=13 Teachers Effect size = 1.18Effect size = 0.50Effect size = 0.67

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Significant adoption rate  Observations of class activities showed that 27 classes from the 36 participant classes completed the module  In addition, analysis of team presentations showed that in all of these 27 classes, teams created presentations at different stages of the design process  The observations and the portfolios showed also that the teachers developed new learning materials

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC New documentation sheet This is a representative example of documentation from 645 portfolios

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC A Spiral Learning Process Communication Presentation Reflection Self Assessment Analysis & Synthesis  Needs Analysis  Requirements  System/Sub-systems  Decision Matrix Detector Subsystem Indicator Subsystem Power Subsystem Prior Knowledge about alarm systems Construction Predict, Test, Troubleshoot, Improve

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Conclusions Performance  Double the gains in science questions  Low-achievers succeed beyond teacher expectations  Pupils who studied with teachers who participated in PD tripled the gains on the science concept test than the pupils who studied with teachers who did not participate  Teachers covered more investigations

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Conclusions Collaboration  Teachers  Teacher reflected meaningfully on their class experience  Teachers tried and adopted several different team-oriented classroom strategies  Field and academy  Success in achieving the curriculum goals (Zohar & Dori, 2003; Doppelt, Mehalik & Schunn, 2005)  The teachers have created rich and flexible learning environments for their pupils  These learning environments have the potential to develop pupils’ thinking and to increase pupils’ love for science (Dori & Barnea, 1997; Doppelt & Schunn, 2008)

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Expansion Development  Electrical Alarm System - Design, Construction and Reflection  Lift System - Design-based Immersion Module for Force and Motion  Teacher Guides and Research tools Implementation  Middle school (Whole district ~2000 8th grade pupils)  Teacher leaders from last year lead PD  High School (Assumption - 40% of the district)  Researcher lead PD Research (Enhancing science education through design-based learning)  What is the impact of professional development on teacher practice and pupil learning?  What are the characteristics of the learning environment teachers create in their actual classes? Tools  Observation of Class Activities, Video Analysis  Analysis of Pupils’ Portfolios  Findings from: Knowledge Pre/Post test and Learning Environment Questionnaire Expansion  Opportunities for implementing the alarm system learning module in other SCALE districts.

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Other Notable Findings  Reflection as a tool for increasing ownership  Teacher reflection during the workshops enabled them to share ideas and solve problems and to feel ownership of the developed new module  Portfolio and Observation  Teachers started to use new learning materials, which included the infusing of design thinking into their teaching  Pupils documented their class work and reflected on each stage of the design process  Pupils can create, design and document an authentic real- life project instead of solving only well-defined problems

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Team System & Subsystems Diagrams

Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel LRDC Team’s Decision Matrix for 3 Alternatives