1. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 1 LRDC Yaron Doppelt & Christian D. Schunn Learning Research & Development Center University of Pittsburgh yaron@pitt.edu, schunn@pitt.edu Implementing Embedded Assessment in a System Design Approach for Pupil-Centered Learning Paper presented at PATT20 Conference, November 2008, Tel Aviv

2. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 2 LRDC A teacher reflects on his feelings while participating in the PD A pupil reflects on past experiences in science A teacher: “So we’re getting at it. I’m learning with them. This is my first time so I’m paying attention to this” A pupil:“During sixth and seventh grades I felt that science lessons demand memorizing facts and reading long uninterested stories about phenomena. This year I enjoyed very much the hands-on activities.”

3. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 3 LRDC Focus Questions  How does a design-based approach help researchers collaborate with teachers in order to achieve a wide and sustainable curriculum reform?  How does an embedded assessment assist teachers in identifying misconceptions and adapting their teaching?  What is the impact of professional development on teachers’ practice and pupils’ learning?  To what extent does teachers adopt the new learning module during the implementation in their actual class environment?

4. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 4 LRDC Pedagogical Guidelines Work from pupil’s own needs  Real world examples  Pupils search for examples relating to their home / environment  Search for historical invention  Invent his/her own story  Teach systems perspective rather than parts perspective on electronics Cover core physics concepts  Meet science and technology standards  Implementing hands-on authentic engineering design process Create rich and engaging learning environment  Generative thinking  Teamwork  Reflective learning  Presentation skills

5. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 5 LRDC The pupils experience an authentic systems design approach to learning scientific and technological concepts Communication Presentation Reflection Self Assessment Analysis & Synthesis  Needs Analysis  Requirements  System/Sub-systems  Decision Matrix Detector Subsystem Indicator Subsystem Power Subsystem Prior Knowledge about alarm systems Construction Predict, Test, Troubleshoot, Improve Science and Technology Education Standards In contrast to typical project based learning in schools Pupils do the work themselves (not parents) No materials needed from home

6. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 6 LRDC

7. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 7 LRDC Focus Questions How well does a design-based approach help researchers collaborate with teachers in order to achieve a wide and sustainable curriculum reform?

8. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 8 LRDC I found a gap between the science standards and the existing learning environment National and state science standards  Design thinking  Know and use the technological design process to solve a problem  Systems thinking  Explain the parts of a simple system and their relationship to each other There was a need from the field to assist eight-grade science teachers

9. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 9 LRDC Collaboration on an intervention program to meet the need  Pupil Module  Inserted into Electronics FOSS unit  4-6 weeks to design and construct an alarm system  Learn and come to own authentic design process  Teacher Guide  Professional Development: Series of workshops  Assessment tools for the teachers  On-Site Guidance

10. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 10 LRDC Complex process of recruitment and loss of teachers at each step N=27 Teachers

11. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 11 LRDC We used a multi-dimensional approach to create a research space

12. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 12 LRDC Proportion of Students in the Experiment and Contrast Groups Coming from High, Medium, and Low Socio-Economic-Status

13. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 13 LRDC Overall Gains in Science Test Scores N=466 N=587 Effect size = 0.89 Design-Based Learning Scripted Inquiry Pupils ’ Gain Score

14. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 14 LRDC Focus Questions How does embedded assessment assist teachers in identifying misconceptions and adapting their teaching?

15. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 15 LRDC Evaluating the Individual Learning Process It is rare that a teacher declares:  The ideal outcome is that all learner achieve 100. (My opinion)  “The normal opinion is that learning achievements in a classroom normally distribute according to the normal distribution is an evidence to a resistance to the success of all learners”.(Mor, 1982)

16. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 16 LRDC Embedded Assessment during the Learning Process  The assessment is designed in such a way that each pupils take a pretest before the first chapter of the learning module.  At the end of each chapter every pupil takes a test that includes: a posttest for the previous chapter and a pretest for the following chapter.  The test’s items were created base upon the experienced gained at the first year’s implementation and based on misconceptions we identified from the literature.  This test includes 17 multi-choice items and 3 open items.

17. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 17 LRDC Adapting the learning process based on preliminary knowledge and identify students’ misconceptions  The teacher receives a deep understanding on students’ misconceptions.  The teacher can easily track the progress of the individual student.  In this process the teacher identifies if the students overcome the misconceptions.  The teacher does not need to invest huge effort to achieve this extra knowledge on the class achievements it is an embedded assessment which becomes a part of the learning process.

18. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 18 LRDC Test Reliability oCronbach Alpah for the instrument was: Alpha = 0.798 oCronbach Alpah for comprising the split-half test was: Alpha = 0.681. oAnalyzing each of the instrument’s items – when omitting any of the item the Cronbach Alpha does not increase beyond 0.81

19. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 19 LRDC Wide implementation with all the students from a mid-sized district. N=1324, 19 teachersSD=0.193

20. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 20 LRDC Focus Questions What is the impact of professional development on teachers’ practice and pupils’ learning?

21. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 21 LRDC What is a Successful PD? Reform Curriculum Professional Development Teacher Practice Pupil Achievement  Distributing the PD throughout the implementation  Engaging teachers in an active learning process  Facilitating a community of teachers professionals

22. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 22 LRDC Score Gains: PD vs. NPD Effect size = 1.12 Effect size = 1.68 PD: N=19 Classes NPD: N=7 Classes Science Questions Design Questions

23. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 23 LRDC Score Gains: PD vs. NPD Existing Curriculum: N=5 NPD: N=5 Teachers PD: N=13 Teachers Effect size = 1.18Effect size = 0.50Effect size = 0.67

24. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 24 LRDC Significant adoption rate  Observations of class activities showed that 27 classes from the 36 participant classes completed the module  In addition, analysis of team presentations showed that in all of these 27 classes, teams created presentations at different stages of the design process  The observations and the portfolios showed also that the teachers developed new learning materials

25. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 25 LRDC New documentation sheet This is a representative example of documentation from 645 portfolios

26. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 26 LRDC A Spiral Learning Process Communication Presentation Reflection Self Assessment Analysis & Synthesis  Needs Analysis  Requirements  System/Sub-systems  Decision Matrix Detector Subsystem Indicator Subsystem Power Subsystem Prior Knowledge about alarm systems Construction Predict, Test, Troubleshoot, Improve

27. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 27 LRDC Conclusions Performance  Double the gains in science questions  Low-achievers succeed beyond teacher expectations  Pupils who studied with teachers who participated in PD tripled the gains on the science concept test than the pupils who studied with teachers who did not participate  Teachers covered more investigations

28. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 28 LRDC Conclusions Collaboration  Teachers  Teacher reflected meaningfully on their class experience  Teachers tried and adopted several different team-oriented classroom strategies  Field and academy  Success in achieving the curriculum goals (Zohar & Dori, 2003; Doppelt, Mehalik & Schunn, 2005)  The teachers have created rich and flexible learning environments for their pupils  These learning environments have the potential to develop pupils’ thinking and to increase pupils’ love for science (Dori & Barnea, 1997; Doppelt & Schunn, 2008)

29. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 29 LRDC Expansion Development  Electrical Alarm System - Design, Construction and Reflection  Lift System - Design-based Immersion Module for Force and Motion  Teacher Guides and Research tools Implementation  Middle school (Whole district ~2000 8th grade pupils)  Teacher leaders from last year lead PD  High School (Assumption - 40% of the district)  Researcher lead PD Research (Enhancing science education through design-based learning)  What is the impact of professional development on teacher practice and pupil learning?  What are the characteristics of the learning environment teachers create in their actual classes? Tools  Observation of Class Activities, Video Analysis  Analysis of Pupils’ Portfolios  Findings from: Knowledge Pre/Post test and Learning Environment Questionnaire Expansion  Opportunities for implementing the alarm system learning module in other SCALE districts.

30. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 30 LRDC Other Notable Findings  Reflection as a tool for increasing ownership  Teacher reflection during the workshops enabled them to share ideas and solve problems and to feel ownership of the developed new module  Portfolio and Observation  Teachers started to use new learning materials, which included the infusing of design thinking into their teaching  Pupils documented their class work and reflected on each stage of the design process  Pupils can create, design and document an authentic real- life project instead of solving only well-defined problems

31. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 31 LRDC Team System & Subsystems Diagrams

32. Embedded-Assessment-Presentation-PATT20, Yaron Doppelt, TLV, Israel 2008 32 LRDC Team’s Decision Matrix for 3 Alternatives