CERA 87 th Annual Conference- Effective Teaching & Learning: Evaluating Instructional Practices Rancho Mirage, CA – December 4, 2008 Noelle C. Griffin, PhD. Implementation of a Formative Assessment-based Middle School Math Intervention: Findings, Considerations and Lessons Learned CRESST/UCLA Victoria Schumacher, PhD. Norwalk/ La Mirada Unified School District

NLMUSD Background and Considerations POWERSOURCE© Overview and Implementation POWERSOURCE© Findings and Trends Broader Math Achievement Context Presentation Overview

NLMUSD: Background and Considerations

Priority 1! 9 out of 10 students reading at grade level. Math: initial focus on middle school Overarching district framework: “The Work” Instructional Context

Authentic student engagement Effective questioning Making meaning/text analysis Assessment driven instruction Standards-based instruction across content areas Components of the Work

Comparison of Math CST 2008 Results across Schools with Free or Reduced Lunch and English Learners

Using CRESST Collaboration to Support “The Work” Initial conversations/planning re: implementation and experimental design (POWERSOURCE and “alternative groups”) Coordination with other district math partners (Dr. Marty Bonsangue, CSUFullerton) Key consideration: How can POWERSOURCE© project compliment existing district initiatives?

District Stakeholder Involvement Key role of stakeholders in decision to participate Joint CRESST/NLMUSD presentations/question answer sessions to middle school principals/teachers Involvement of Teachers’ Association (TANLA) Inclusion of CRESST Senior Researcher (Terry Vendlinski) in other district math professional development activities and sessions Teachers invited to participate, vote for “site coordinators” for each school

POWERSOURCE© : Overview and Implementation

POWERSOURCE© Background Five-year research project Objective: Develop formative assessments and associated instructional support materials to improve student performance in Algebra/pre-Algebra Funder: U.S. Department of Education, Institute of Education Sciences (National Assessment Center Grant) Emphasis on empirical rigor and experimental testing Involves participation of multiple districts over multiple years

POWERSOURCE© Foundations: Formative Assessment Formative assessments can be used to guide teaching and instruction When used correctly, it can produce increased learning and achievement. (e.g., Black & William, CRESST) Many teachers don’t have the time, resources, or knowledge to use formative assessments effectively. (e.g., Herman)

POWERSOURCE© Foundations: Cognition Expert knowledge is schema-based, organized around complex, interconnected concepts Expert knowledge organization supports problem solving, inferences, transfer Experts provide the basis for material design, scoring, and formative assessment process Application of what we know about experts to support content area expertise

POWERSOURCE© Foundations: “Big Ideas” Fundamental principles that underlie Algebra operations, when learned, will enhance the speed and quality of learning Context of expert knowledge/ cognition Identified/vetted through participation of math experts (theory and instruction) POWERSOURCE Focus: Rational Number Equivalence, Arithmetic Properties, Solving Equations, Functions

Big Ideas Map of Algebra Knowledge

POWERSOURCE© Components Four “units” during the school year (one for each “Big Idea”) A series of brief formative assessment tasks Focused on core concepts in algebra and their interconnection, cognitive demands Using a variety of formats to demonstrate understanding (computation, problem-solving, explanations, word problems, partially worked examples) Focused instructional resources for teachers Teacher Handbook, exemplar lessons, web-based support Professional development activities

Explanation Task-Example An open-ended question to see if students can transfer the ideas about the multiplicative identity to a novel situation.

Sample Student Responses

Sample Student Responses (Cont’d)

Sample Teacher Resources (Introduction to Handbook) Information to say to the class Information to write on the board Overheads

Reconciling Design/ Rigor with District Needs How to integrate POWERSOURCE© content with on-going initiatives? How to be respectful of teacher time but maintain implementation fidelity? How to make project useful for all participants while maintaining experimental rigor? (I.e., random assignment) How to collect data needed while minimizing district burden?

Alternative Professional Development in NLMUSD Developed specifically for implementation in NLMUSD Focus on use of district benchmark data (DataDirector) Teachers randomly assigned to POWERSOURCE© or Alternative Provide a meaningful experience for all teachers (including “controls”) while still maintaining empirical rigor Collaboration with districts to find workable approaches to control group

POWERSOURCE© : Findings and Trends

POWERSOURCE© 6th Grade Field Testing Random assignment of teachers, 4 districts (6th grade) Preliminary results: POWERSOURCE© students perform significantly better than control students on math learning outcomes. Based on HLM analyses, there were significant differences between PS and control students for all districts and content areas. The average effect size was.88 (with a district by content area range of.44 to 1.75), and the effect size averages by district ranged from.75 to No initial broad impact on CST (follow-up being tracked).

POWERSOURCE© th Grade Extended Pilot Test Multiple Districts (7, including NLMUSD) Mixed Design (random assignment by school or teacher) Start with 6th grade in , then phase in 7th and 8th grades Analysis in progress (multiple student and teacher outcomes) Preliminary trends NLMUSD

Preliminary Findings: Change in Proficient and Above to (Grade 6 Math CSTs)

Broader Math Achievement Context

NLMUSD Math Achievement POWERSOURCE© part of larger initiative, starting at middle school and eventually moving to other grades Overall trajectory of math gains across grades in the district from to

Math 6 Benchmark Comparison to Percent Proficient and Above = 1% Percent Proficient and Above = 5% Percent Proficient and Above = 6% Percent Proficient and Above = 11% Percent Proficient and Above =17% Percent Proficient and Above = 23% Change +10% Change +12% Change +17%

Math 7 Benchmark Comparison to Percent Proficient and Above = 6% Percent Proficient and Above = 2% Percent Proficient and Above = 3% Percent Proficient and Above = 10% Percent Proficient and Above =6% Percent Proficient and Above = 12% Change +4% Change +9%

Math 8 - Algebra I Benchmark Comparison to Percent Proficient and Above = 2% Percent Proficient and Above <1% Percent Proficient and Above = 15% Percent Proficient and Above =12% Percent Proficient and Above = 5% Change +13% Change +12% Change +5%

Grade 6 CST Comparsion to % increase

Grade 7 CST Comparison to % increase

Grade 8 CST Comparison to % increase