1. CERA 87 th Annual Conference- Effective Teaching & Learning: Evaluating Instructional Practices Rancho Mirage, CA – December 4, 2008 Noelle C. Griffin, PhD. Implementation of a Formative Assessment-based Middle School Math Intervention: Findings, Considerations and Lessons Learned CRESST/UCLA Victoria Schumacher, PhD. Norwalk/ La Mirada Unified School District

2. NLMUSD Background and Considerations POWERSOURCE© Overview and Implementation POWERSOURCE© Findings and Trends Broader Math Achievement Context Presentation Overview

3. NLMUSD: Background and Considerations

4. Priority 1! 9 out of 10 students reading at grade level. Math: initial focus on middle school Overarching district framework: “The Work” Instructional Context

5. Authentic student engagement Effective questioning Making meaning/text analysis Assessment driven instruction Standards-based instruction across content areas Components of the Work

6. Comparison of Math CST 2008 Results across Schools with Free or Reduced Lunch and English Learners

7. Using CRESST Collaboration to Support “The Work” Initial conversations/planning re: implementation and experimental design (POWERSOURCE and “alternative groups”) Coordination with other district math partners (Dr. Marty Bonsangue, CSUFullerton) Key consideration: How can POWERSOURCE© project compliment existing district initiatives?

8. District Stakeholder Involvement Key role of stakeholders in decision to participate Joint CRESST/NLMUSD presentations/question answer sessions to middle school principals/teachers Involvement of Teachers’ Association (TANLA) Inclusion of CRESST Senior Researcher (Terry Vendlinski) in other district math professional development activities and sessions Teachers invited to participate, vote for “site coordinators” for each school

9. POWERSOURCE© : Overview and Implementation

10. POWERSOURCE© Background Five-year research project Objective: Develop formative assessments and associated instructional support materials to improve student performance in Algebra/pre-Algebra Funder: U.S. Department of Education, Institute of Education Sciences (National Assessment Center Grant) Emphasis on empirical rigor and experimental testing Involves participation of multiple districts over multiple years

11. POWERSOURCE© Foundations: Formative Assessment Formative assessments can be used to guide teaching and instruction When used correctly, it can produce increased learning and achievement. (e.g., Black & William, CRESST) Many teachers don’t have the time, resources, or knowledge to use formative assessments effectively. (e.g., Herman)

12. POWERSOURCE© Foundations: Cognition Expert knowledge is schema-based, organized around complex, interconnected concepts Expert knowledge organization supports problem solving, inferences, transfer Experts provide the basis for material design, scoring, and formative assessment process Application of what we know about experts to support content area expertise

13. POWERSOURCE© Foundations: “Big Ideas” Fundamental principles that underlie Algebra operations, when learned, will enhance the speed and quality of learning Context of expert knowledge/ cognition Identified/vetted through participation of math experts (theory and instruction) POWERSOURCE Focus: Rational Number Equivalence, Arithmetic Properties, Solving Equations, Functions

14. Big Ideas Map of Algebra Knowledge

15. POWERSOURCE© Components Four “units” during the school year (one for each “Big Idea”) A series of brief formative assessment tasks Focused on core concepts in algebra and their interconnection, cognitive demands Using a variety of formats to demonstrate understanding (computation, problem-solving, explanations, word problems, partially worked examples) Focused instructional resources for teachers Teacher Handbook, exemplar lessons, web-based support Professional development activities

16. Explanation Task-Example An open-ended question to see if students can transfer the ideas about the multiplicative identity to a novel situation.

17. Sample Student Responses

18. Sample Student Responses (Cont’d)

19. Sample Teacher Resources (Introduction to Handbook) Information to say to the class Information to write on the board Overheads

20. Reconciling Design/ Rigor with District Needs How to integrate POWERSOURCE© content with on-going initiatives? How to be respectful of teacher time but maintain implementation fidelity? How to make project useful for all participants while maintaining experimental rigor? (I.e., random assignment) How to collect data needed while minimizing district burden?

21. Alternative Professional Development in NLMUSD Developed specifically for implementation in NLMUSD Focus on use of district benchmark data (DataDirector) Teachers randomly assigned to POWERSOURCE© or Alternative Provide a meaningful experience for all teachers (including “controls”) while still maintaining empirical rigor Collaboration with districts to find workable approaches to control group

22. POWERSOURCE© : Findings and Trends

23. POWERSOURCE© 6th Grade Field Testing Random assignment of teachers, 4 districts (6th grade) Preliminary results: POWERSOURCE© students perform significantly better than control students on math learning outcomes. Based on HLM analyses, there were significant differences between PS and control students for all districts and content areas. The average effect size was.88 (with a district by content area range of.44 to 1.75), and the effect size averages by district ranged from.75 to 1.01. No initial broad impact on CST (follow-up being tracked).

24. POWERSOURCE© 2007-08 6th Grade Extended Pilot Test Multiple Districts (7, including NLMUSD) Mixed Design (random assignment by school or teacher) Start with 6th grade in 2007-08, then phase in 7th and 8th grades Analysis in progress (multiple student and teacher outcomes) Preliminary trends NLMUSD

25. Preliminary Findings: Change in Proficient and Above 2006-07 to 2007-08 (Grade 6 Math CSTs)

26. Broader Math Achievement Context

27. NLMUSD Math Achievement POWERSOURCE© part of larger initiative, starting at middle school and eventually moving to other grades Overall trajectory of math gains across grades in the district from 2006-07 to 2007- 08

28. Math 6 Benchmark Comparison 2006 - 07 to 2007 - 08 Percent Proficient and Above = 1% Percent Proficient and Above = 5% Percent Proficient and Above = 6% Percent Proficient and Above = 11% Percent Proficient and Above =17% Percent Proficient and Above = 23% Change +10% Change +12% Change +17%

29. Math 7 Benchmark Comparison 2006 - 07 to 2007 - 08 Percent Proficient and Above = 6% Percent Proficient and Above = 2% Percent Proficient and Above = 3% Percent Proficient and Above = 10% Percent Proficient and Above =6% Percent Proficient and Above = 12% Change +4% Change +9%

30. Math 8 - Algebra I Benchmark Comparison 2006 - 07 to 2007 - 08 Percent Proficient and Above = 2% Percent Proficient and Above <1% Percent Proficient and Above = 15% Percent Proficient and Above =12% Percent Proficient and Above = 5% Change +13% Change +12% Change +5%

31. Grade 6 CST Comparsion 2006 - 07 to 2007 - 08 14% increase

32. Grade 7 CST Comparison 2006 - 07 to 2007 - 08 42% increase

33. Grade 8 CST Comparison 2006 - 07 to 2007 - 08 85% increase

34. griffin@cse.ucla.edu