1. Mathematics Performance Tasks Applying a Program Logic Model to a Professional Development Series California Educational Research Association December 6, 2013 Sally J. Bennett-Schmidt, Ed.D. Assessment Director San Diego County Office of Education

2. Overview of the Session Context of the Professional Learning Experience Summary of the Professional Development Series Application of the Program Logic Model ▫ Vision & Current Situation ▫ Resources & Activities ▫ Outputs ▫ Outcomes: Short-, Intermediate-, Long-Term Learnings and Next Steps

3. Context of the Project San Diego County Office of Education ▫ Support 42 school districts + county-run programs ▫ Curriculum, instruction, and assessment staff working collaboratively to support districts in implementation of the common core state standards San Diego County School Districts ▫ Along a continuum in implementation of CCSS ▫ Substantial needs in mathematics, both content knowledge and instructional approaches

4. The Professional Development Series: Mathematics Performance Tasks & Formative Assessment Purpose: Support district efforts in the use of performance tasks to maximize student achievement Key Activities ▫ Examine assessment in the light of the new Common Core State Standards and work being explored by the Smarter Balanced Assessment Consortium ▫ Experience performance tasks along with scoring protocols ▫ Explore the formative use of performance task results including student re-engagement ▫ Incorporate formative assessment practices in districts’ ongoing work

5. The Professional Development Series Days 1-2: August 2012 ▫ Audience: District teams of 5-7 members (district and site administrators, mathematics teacher leaders) ▫ Facilitated by David Foster from the Silicon Valley Mathematics Initiative ▫ Focus:  Understand the new assessment context under the common core standards  Experience several performance tasks with scoring and analysis protocols  Begin discussion of the formative use of performance tasks and student re-engagement

6. The Professional Development Series Day 3: September – October 2012 ▫ Audience: Expanded district teams with additional classroom teachers ▫ Facilitated by SDCOE Mathematics & Assessment Staff ▫ Focus:  Select an extended response mathematics task to use with students  Anticipate student successes and challenges  Plan for embedding the task in instruction

7. The Professional Development Series Day 4: December 2012 – January 2013 ▫ Audience: Expanded district teams with additional classroom teachers ▫ Facilitated by SDCOE Mathematics & Assessment Staff ▫ Focus:  Score student work from the performance task  Analyze student performance for mathematical understanding and misunderstanding  Plan a re-engagement lesson

8. The Program Logic Model Context for using a Logic Model ▫ Learning and Leadership Services Division began exploring tools to help us better determine the impact of our work on district systems, instructional leadership, teacher practice, and student learning. ▫ Logic model introduced as a tool to help us frame our services within an outcome orientation  For planning  For program evaluation ▫ Logic model completed via a series of guiding questions

9. The Program Logic Model 1.What is our vision and our current situation? 2.Who do we serve? 3.What end result do we desire – how will the “rules of the game” change? 4.What new behaviors will we see? 5.What will change right away? 6.What do we do? 7.What will we count? 8.What will we need to make it happen? 9.How will we know we’re on the right track?

10. The Program Logic Model Situation / Vision: Question 1 Target Audience: Question 2 Assumptions, Environmental / External Factors Resources Question 8: What will we need to make it happen? Outcomes Strategies Activities Question 6: What do we do? Outputs Question 7: What will we count? Short-Term Question 5: What will change right away? Intermediate Question 4: What new behaviors will we see? Long-Term Question 3: How will the rules of the game change? Indicators Question 9: How will we know we’re on track?

11. Vision Mathematics instruction will reflect the common core instructional shifts, and will embed high-quality tasks and formative assessment strategies to gauge students’ conceptual understanding and re-engage students in mathematical thinking.

12. Current Situation Current classroom instructional practices generally reflect training aligned to curricular materials and prior state content standards. Teachers have had limited professional learning and experience with embedding high-quality, open- ended mathematics tasks and formative assessment into instructional practice. Student achievement in mathematics has improved, but continues to show weaker performance in middle and high school grade levels and courses.

13. Target Audience Mathematics teacher leaders Site administrators District curriculum / instruction / assessment administrators

14. Resources SDCOE Mathematics and Assessment staff (managers and support staff) Time for collaborative planning and delivery of professional development Funding to contract for SVMI member resources Funding for costs associated with professional development sessions

15. Activities Four-part mathematics performance task and formative assessment professional development series Contract for SDCOE and small districts in the county to access SVMI resources

16. Output and Indicators Number of professional development sessions ▫ 1 four-part series; 9 total sessions (Day 3 offered four times; Day 4 offered three times) Number of participating teachers and administrators ▫ 135 participants from 14 districts and 4 independent charters Number of teachers that select and administer a task from SVMI MARS or other resource ▫ 88% of participating classroom teachers administered a task Number of teachers that score and analyze student work ▫ 76% brought back student work to score and analyze

17. Short-term Outcomes and Indicators Increased awareness of how mathematics performance tasks and formative assessment can improve instruction and student learning ▫ 94% reported better understanding of assessment in relation to CCSS ▫ 78% reported better understanding of analyzing student work to identify students’ mathematical understanding

18. Short-term Outcomes and Indicators Increased awareness of where to find mathematics performance tasks, rubrics, and scored student papers ▫ 78% reported better understanding of how to find and access identified websites to select performance tasks Increased awareness of how to select and administer a performance task, and score student work using a rubric with training & standardizing papers ▫ 89% reported the PD helped them plan to administer a performance task with their students ▫ 86% reported the PD helped them score and analyze their students’ work

19. Intermediate Outcomes & Indicators Teachers try out performance tasks in classrooms, and score and analyze student work ▫ 88% tried out a performance task ▫ 76% scored and analyzed student work Teachers use student work for professional conversation ▫ 100% of teachers who brought back work engaged in professional conversation on Day 4 ▫ Proposed use of teacher learning logs to document continuing practice Teachers share student work and instructional practices through PLCs ▫ Proposed use of PLC logs and observations at a sample of sites

20. Long-term Outcomes & Indicators Teachers and students engage in rigorous, coherent, and focused mathematics learning experiences Classroom practice reflects high-quality mathematics instruction and assessment PLCs are focused on student learning and instruction Student achievement in mathematics improves ▫ Teacher learning logs and classroom observations ▫ PLC logs and observations ▫ District / Site and Smarter Balanced assessment results

21. Learnings & Next Steps What we learned about supporting teacher use of mathematics tasks and formative assessment strategies ▫ More time was needed on re-engagement portion of mathematics performance task & formative assessment work ▫ Teachers need continued support to embed tasks and formative assessment strategies into ongoing instructional practice ▫ School site structures need to support teacher collaboration around selecting tasks, analyzing student work, and re-engaging students in mathematics learning

22. Learnings & Next Steps What we learned about collecting “evidence of impact” ▫ We were able to collect evidence of short-term outcomes in terms of increased awareness ▫ We were able to collect limited evidence of intermediate outcomes in terms of changing practice ▫ We need a structured plan and resources (human, time) to collect evidence of intermediate and longer- term outcomes

23. Learnings & Next Steps What we learned about the program logic model ▫ The approach helped us better define our service(s) and our desired outcomes ▫ The tool helped us differentiate short, intermediate, and long-term outcomes and identify appropriate evidence to collect to assess impact ▫ The process helped us understand that, to assess long- term outcomes / impact, we need a multi-year project and ongoing data collection methods

24. Contact Information Sally J. Bennett-Schmidt, Assessment Director ▫ San Diego County Office of Education ▫ 858-292-3688 ▫ sbennettschmidt@sdcoe.net sbennettschmidt@sdcoe.net