1 Assessment for learning in science: Issues in learning to interpret student work Center for the Assessment and Evaluation of Student Learning (CAESL) University of California, Berkeley University of California, Los Angeles Lawrence Hall of Science

2 WestEd Diane Carnahan Karen Cerwin Kathy DiRanna Jo Topps U.C. Berkeley Maryl Gearhart Jennifer Pfotenhauer Cheryl Schwab UCLA Shaunna Clark Joan Herman Sam Nagashima Ellen Osmundson Terry Vendlinski Lawrence Hall of Science Lynn Barakos Craig Strang

3 Presentation Program and participants Research framework Design and methods Selected findings Implications 3

4 Situation assessments in materials of variable quality assessments in materials of variable quality teachers lack expertise to revise teachers lack expertise to revise professional practices not well established professional practices not well established Argument science education reform (NRC/NSES) science education reform (NRC/NSES) known benefits of classroom assessment (e.g., Black & Wiliam, 1998; Sloane & Wilson, 2000) known benefits of classroom assessment (e.g., Black & Wiliam, 1998; Sloane & Wilson, 2000) value of reflective practice and long term collaboration (Garet et al, 2001) value of reflective practice and long term collaboration (Garet et al, 2001) 4 Impetus for program

5 Principles integrated with practice long term collaborative reflective practice Core strategy assessment portfolio 5 CAESL Leadership Academy 7/ /04

6 Academy participants 6

7 Interwoven structures district vertical teams w/ administrators district vertical teams w/ administrators cross district grade level teams cross district grade level teams independent classroom implementation independent classroom implementation Series of portfolios repeated opportunities to build expertise repeated opportunities to build expertise 7 Program organization

8 Portfolio: I. Plan Establish learning goals analyze ‘conceptual flow’ of materials analyze ‘conceptual flow’ of materials align with standards align with standards Select assessments choose key assessments to track progress: pre -> junctures -> post choose key assessments to track progress: pre -> junctures -> post identify the concepts assessed identify the concepts assessed anticipate ‘expected student responses’ anticipate ‘expected student responses’ 8

9 Portfolio: II. Implementation Interpret student work refine assessment criteria refine assessment criteria score score chart and identify trends chart and identify trends Use evidence document instructional follow-up and feedback document instructional follow-up and feedback 9

10 Portfolio: III. Evaluate & revise 10 Evaluate using student work alignment with goals and instruction alignment with goals and instruction quality of tasks and criteria quality of tasks and criteria methods of analysis methods of analysis Revise and strengthen

11 Rated for completeness Complete:I, II (some student work), III Complete:I, II (some student work), III Partial: I or III, II (some) Partial: I or III, II (some) Minimal: I or III only Minimal: I or III only None (but participating) None (but participating) Portfolio Completion

12 Portfolio Completion 12

13 Study Focus Growth in understanding and practice Supports and barriers Longitudinal, nested design 18 months = 3 portfolios Cohort: Surveys, focus groups, portfolios Cases: Interviews and observations

14 Framework for classroom assessment expertise Understanding of assessment concepts Facility with assessment practices 14

UNDERSTANDING ASSESSMENT CONCEPTS QUALITY GOALS FOR STUDENT LEARNING AND PROGRESS QUALITY USE QUALITY TOOLS 15

UNDERSTANDING ASSESSMENT CONCEPTS QUALITY GOALS FOR STUDENT LEARNING AND PROGRESS QUALITY USE QUALITY TOOLS SOUND INTERPRETATION 16

ESTABLISH GOALS & DESIGN ASSESSMENT PLAN ASSESS USE INFORMATION TO GUIDE TEACHING & LEARNING REVISE ASSESSMENTS PROVIDE INSTRUCTION INTERPRET STUDENT WORK 17 CLASSROOM ASSESSMENT PRACTICES

ASSESS USE INFORMATION TO GUIDE TEACHING & LEARNING REVISE ASSESSMENTS PROVIDE INSTRUCTION INTERPRET STUDENT WORK Soundness of interpretation: - criteria capture student understanding? - scoring consistent? - interpretation appropriate to purpose? ESTABLISH GOALS & DESIGN ASSESSMENT PLAN 18 USING CONCEPTS TO GUIDE PRACTICE

ASSESS USE INFORMATION TO GUIDE TEACHING & LEARNING REVISE ASSESSMENTS PROVIDE INSTRUCTION INTERPRET STUDENT WORK 19 ESTABLISH GOALS & DESIGN ASSESSMENT PLAN Soundness of interpretation: - criteria capture student understanding? - scoring consistent? - interpretation appropriate to purpose? USING CONCEPTS TO GUIDE PRACTICE

20 Selected findings Portfolios: Patterns of change assessment criteria analysis of whole class patterns alignment Exit survey and focus groups perceived growth supports, barriers, needs

21 Source series of 2 or 3 portfolios (n ≈ 10) Issues & constraints burden of documentation paper & pencil assessments professional choice Patterns in portfolios 21

22 Assessment criteria 22 + from global/holistic to more specific, differentiated, and assessable + from focus on surface features to efforts to capture student understanding + from dichotomous (right/wrong) to attention to qualitative levels of understanding = but … quality variable despite teacher interest (example: reliance on content analysis or notes)

23 Whole class analysis 23 + from few to efforts at systematic analysis using charting or content analysis + from global patterns toward more differentiated analysis (item analysis, item clustering) and efforts to coordinate group & individual patterns + efforts to analyze progress (espec. pre-post) = but … information often unintegrated, inferences unsystematic, comparisons inappropriate

24 Alignmen t 24 + efforts to align interpretations with learning goals, tasks, and criteria + efforts to revise criteria to strengthen alignment + fewer inferences about ancillary skills not assessed = but … problematic alignment of assessments and inferences to track progress

25 Understanding of CAESL 1 (none) 5 (full) Implementation of CAESL 1 (none) 4 (full) 5 (beyond) 25 Exit survey

26 UNDERSTANDING 26

27 IMPLEMENTATION 27

28 Exit focus groups 28

ESTABLISH GOALS & DESIGN ASSESSMENT PLAN ASSESS USE INFORMATION TO GUIDE TEACHING & LEARNING REVISE ASSESSMENTS PROVIDE INSTRUCTION INTERPRET STUDENT WORK 29 PRACTICES STRENGTHENED MOST?

ESTABLISH GOALS & DESIGN ASSESSMENT PLAN ASSESS USE INFORMATION TO GUIDE TEACHING & LEARNING REVISE ASSESSMENTS PROVIDE INSTRUCTION INTERPRET STUDENT WORK 30 PRACTICES STRENGTHENED LEAST?

ESTABLISH GOALS & DESIGN ASSESSMENT PLAN ASSESS USE INFORMATION TO GUIDE TEACHING & LEARNING REVISE ASSESSMENTS PROVIDE INSTRUCTION INTERPRET STUDENT WORK 31 PRACTICES MOST IMPORTANT?

32 Portfolio establishing goals revision and re-application resource for collaboration Resources CAESL framework articles and books grade level teams & facilitators long term program Supports 32

33 Portfolio assessment development only paper and pencil focus on perf. assessments time Resources weak assessments limited frameworks no clear models for progress gaps in teacher knowledge Barriers 33 Context standards testing school & district

34 Portfolio assessment development only paper and pencil focus on perf. assessments time Resources weak assessments limited frameworks no clear models for progress gaps in teacher knowledge Barriers 34 Context standards testing school & district Unnamed challenges inquiry ancillary skills

35 Portfolio …if…. streamline focus on goals, interpretation, and use refinement not development expand assessment types Resources embedded assessments handbook conceptual development grade level collaboration coaching and facilitation Context align with district and state assessments 35 Requests

36 Strengthen materials & resources Expand to unit assessment systems Align assessment content and quality Modify program organization 36 Implications