1. 1 / 27 California Educational Research Association 88 th Annual Conference Formative Assessment: Implications for Student Learning San Francisco, CA November 18, 2009 Jessica Ulloa POWERSOURCE© Formative Assessment Practices Using the POWERSOURCE© Algebra Intervention: A View from the Classroom and Teacher Perceptions CRESST/UCLA

2. 2 / 27 1. POWERSOURCE© Overview and Implementation 2. Teacher Interview & Classroom Observations a. Overview b. Sample c. Data Collection Methods d. Results 3. Conclusion PRESENTATION OVERVIEW

3. 3 / 27 POWERSOURCE© INTERVENTION: OVERVIEW

4. 4 / 27 POWERSOURCE© Background Five-year research project Funder by the U.S. Department of Education, Institute of Education Sciences (National Assessment Center Grant) An algebra formative assessment intervention. SAMPLE: 150 Teachers, approximately 6,000 students 6 th - 8 th grade 8 participating School Districts : CA, AZ Two Randomized control led groups

5. 5 / 27 THE POWERSOURCE© INTERVENTION

6. 6 / 27 POWERSOURCE© INTERVENTION: Professional Development Component GOALS : Develop content knowledge Develop skills in examining student work and explore how student thinking develops Establish teacher-learner community STRUCTURE: Three 90-minute sessions per domain (two-and-a half month intervals) Small groups (5-20 teachers)

7. 7 / 27 POWERSOURCE© : Classroom Observations & Teacher Interviews

8. 8 / 27 TEACHER INTERVIEWS & CLASSROOM OBSERVATIONS RESEARCH QUESTIONS: 1. Are teachers using POWERSOURCE© assessment data in a formative way? 2. How are teachers using POWERSOURCE© formative assessment information to guide their instruction? 3. What instructional practices are teachers implementing in the classroom to present POWERSOURCE© Big Ideas?

9. 9 / 27 POWERSOURCE© Teacher Interviews & Classroom Observations: METHODOLOGY:

10. 10 / 27 Teacher Interviews & Classroom Observations SAMPLE CHARACTERISTICS  Conducted over a two year period for the 07/08 and 08/09 school year:  Participation is voluntary and teachers receive a small stipend  Total Sample Size: n=15 Teachers  Total School Districts: 5 (CA, AZ)  Focus of Observations:  Solving Equations  Review & Application  6 th grade/Treatment (POWERSOURCE ©) Teachers

11. 11 / 27 Interviews & Observations SAMPLE CHARACTERISTICS  Teacher Background:  1-2 years in PowerSource,  5+years of teaching experience (2 first year teachers)  Hold clear and multiple subject credentials  Most possess Bachelor Degrees  2 teachers with Masters degrees  Classroom composition:  ELL, G.A.T.E, R.S.P Students  Average classroom size: 25 students

12. 12 / 27 Teacher Interviews & Classroom Observations Data Collection Methods OBSERVATION MEASURE :  Semi-structured measure  40-50 minute classroom observation  Teachers observed twice (29 separate observations)  Paired Observations OBSERVATION COMPONENTS: 1. Implementation of Big Ideas 2. Teacher Behaviors 3. Lesson Structure 4. Use of POWERSOURCE © materials

13. 13 / 27 Teacher Interviews & Classroom Observations DATA COLLECTION METHODS  INTERVIEW MEASURE  Semi-Structured measure  One 15-30 minute audio recorded interview  INTERVIEW COMPONENTS : 1.Teacher Lesson Reflections 2. Formative Use of Assessments 3. Lesson Structure 4. Professional Development  DATA ANALYSIS:  Qualitative data was coded and analyzed using Atlas Ti.  Interviews coded independently by two coders  Observation frequency data counted and coded across all four observation components.

14. 14 / 27 POWERSOURCE© Teacher Interviews & Classroom Observations: FINDINGS & TRENDS

15. 15 / 27 POWERSOURCE© Teacher Interviews: FINDINGS & TRENDS SAMPLE INTERVIEW QUESTIONS: Category: Use of Formative Assessment: 1.“How did you use the 1 st (or last) Check for Understanding to plan for today’s lesson?” 2.“Based on today’s lesson, what do you think your steps will be instructionally? 3. “What other sources of information, other than the POWERSOURCE © assessments, do you typically use to identify your student strengths and areas of difficulties in math?

16. 16 / 27 Interview Measure Category: Use of Formative Assessment The majority of teachers (n=12) reported using the checks for understanding to identify student areas of difficulty & areas of strength. Teachers (N=15) reported centering subsequent POWERSOURCE© lessons around student’s areas of difficulty—as evidenced by the Checks for Understanding. Teachers (n=12) reported “quickly glancing through” student assessments (Checks for Understanding) to identify gaps in student understanding of POWERSOURCE © math concepts. POWERSOURCE © Teacher Interviews Preliminary Results:

17. 17 / 27 POWERSOURCE © Teacher Interviews Preliminary Results: LIMITATIONS: Use of Formative Assessments Based on interview responses (n=12), teachers’ appear to struggle with articulating how to adjust instruction based on formative assessment information. Teachers did not identify specific actions or strategies used to plan instruction and/or to present the POWERSOURCE © big ideas. Common teacher responses tied to POWERSOURCE © instruction planning and methods sound similar to practices already being used for district modules.

18. 18 / 27 Classroom Observations: Teacher Behaviors: RESULTS YEAR 1 MOST Observed Teacher Behaviors  Uses guided problem solving  Questions for Understanding  Provides procedural feedback  Provides procedural feedback + correct solution. LEAST Observed Teacher Behaviors  References Checks for Understanding, quiz/test/  References homework results  Discussion of misconceptions  Demonstrates incorrect solution methods

19. 19 / 27 Classroom Observations: Teacher Behaviors: RESULTS YEAR 2 MOST Observed Teacher Behaviors : Questions for Understanding Uses guided problem Solving Demonstrates correct solution methods Provides Procedural feedback Provides Procedural feedback plus correct answer LEAST Observed Teacher Behaviors: References Checks for Understanding, quizzes/test or homework results Discussed Misconceptions Demonstrates incorrect solution method

20. 20 / 27 Classroom Observations & Teacher Behaviors Year 1 and Year 2 Comparison

21. 21 / 27 POWERSOURCE © Classroom Observations & Teacher Interviews CONCLUSIONS: Teacher interview responses and observation data, indicate that teachers are using formative assessment to gauge student understanding of math concepts. Observation data shows that teachers consistently and over-time apply similar instructional strategies to present the Big Ideas in math. Teachers seem to struggle with using Formative Assessment as an instrument to guide their instruction. Results suggest that teachers may be at the beginning stages of understanding the use of formative assessment Teachers may still be struggling with transferring and applying content knowledge taught at professional development trainings into the classroom.

22. 22 / 27 Teacher Interviews & Classroom Observations: NEXT STEPS Questions to Further Research: 1. Is nine hours of professional development training sufficient to produce change in teacher’s pedagogical methods and math content knowledge? 2. Is the appeal of the Big Ideas overshadowing the POWERSOURCE © Formative Assessment Piece?

23. 23 / 27 julloa@cse.ucla.edu (310) 206-4273

24. 24 / 27