Assessing instructional and assessment practice: What makes a lesson formative? CRESST conference, September 2004 UCLA Sunset Village, CA Dylan Wiliam ETS

What makes a lesson formative? Moving from “I know it when I see it” Expanding ideas of formative assessment –Feedback –Elicitation and feedback –The student’s role –Systems thinking

Preliminary assumptions The primary purpose of educational research is the improvement of education. The purpose of education is the improvement of student achievement The improvement of student achievement will be achieved primarily through changes in what happens in classrooms The role of the teacher is not to teach but to create situations in which students learn

Functions of assessment For evaluating institutions For describing individuals For supporting learning –Monitoring learning Whether learning is taking place –Diagnosing (informing) learning What is not being learnt –Forming learning What to do about it

Effects of feedback Kluger & DeNisi (1996) Review of 3000 research reports Excluding those: –without adequate controls –with poor design –with fewer than 10 participants –where performance was not measured –without details of effect sizes left 131 reports, 607 effect sizes, involving individuals Average effect size 0.4, but –Effect sizes very variable –40% of effect sizes were negative

Effects of formative assessment Several major reviews of the research –Natriello (1987) –Crooks (1988) –Black & Wiliam (1998) –Nyquist (2003) All find consistent, substantial effects

Kinds of feedback (Nyquist, 2003) Weaker feedback only –Knowledge or results (KoR) Feedback only –KoR + clear goals or knowledge of correct results (KCR) Weak formative assessment –KCR+ explanation (KCR+e) Moderate formative assessment –(KCR+e) + specific actions for gap reduction Strong formative assessment –(KCR+e) + activity

Effects of formative assessment (HE) No.Effect Weaker feedback only Feedback only Weaker formative assessment Moderate formative assessment Strong formative assessment160.51

Formative assessment Classroom assessment is not (necessarily) formative assessment Formative assessment is not (necessarily) classroom assessment

Formative assessment Assessment for learning is any assessment for which the first priority in its design and practice is to serve the purpose of promoting pupils’ learning. It thus differs from assessment designed primarily to serve the purposes of accountability, or of ranking, or of certifying competence. An assessment activity can help learning if it provides information to be used as feedback, by teachers, and by their pupils, in assessing themselves and each other, to modify the teaching and learning activities in which they are engaged. Such assessment becomes ‘formative assessment’ when the evidence is actually used to adapt the teaching work to meet learning needs. Black et al, 2002

Bad items/good items 1 Which is larger: 0.33 or 1/3? A)0.33 is larger than 1/3 B)1/3 is larger than 0.33 C)They are the same D)You need more information to be sure

Bad items/good items 2 What is the general rule for the following pattern: 3, 7, 11, 15, …? A)3 + n B)n + 4 C)3n + 4 D)4n - 1 E)4n + 3

Bad items/good items 3 If f + g = 8, then f + g + h = ? A)9 B)12 C)15 D)16 E)8 + h

Bad items/good items 4 Which of the following statements are true ? A)A square has two 90° angles B)A square is a trapezoid C)A kites are squares D)A trapezoid is a rectangle E)A rectangle is a trapezoid

Bad items/good items 5 Ice-cubes are added to a glass of water. What happens to the level of the water as the ice-cubes dissolve? A) The level of the water drops B) The level of the water stays the same C) The level of the water increases D) You need more information to be sure

Bad items/good items 6 What is the pH of gasoline? A) Less than zero B) Zero C) Greater than zero but less than 7 D)7 E) Greater than 7 F) None of the above

Bad items/good items 7 In what year did World War II begin? A) 1919 B) 1937 C) 1938 D)1939 E) 1941 F) 1942

Good formative items are different… Can have more (or less!) than one correct answer Items need to be generative –of learning –of insights into learning –of insights into how to promote learning Increasing ability may reduce your chance of getting the item correct Distractors must be explicitly connected to incorrect or incomplete conceptions (facets) Item responses must provide clues to effective action Some items support both summative and formative functions well; most do not

How do we make good items? Good items are not universal –In some domains, facets of thinking are reasonably universal (e.g. physics) –In most domains, facets of thinking are dependent on instruction –So there is no ‘universal’ set of facets –And we have to rely on teachers to do it for themselves Good items have to be used –Requires a model of effective, scalable teacher professional development

Why research hasn’t changed teaching The nature of expertise in teaching Aristotle’s main intellectual virtues –Episteme: knowledge of universal truths –Techne: ability to make things –Phronesis: practical wisdom What works is not the right question –Everything works somewhere –Nothing works everywhere –What’s interesting is “under what conditions” does this work? Teaching is mainly a matter of phronesis, not episteme

Countdown Target number: 127 1

Aspects of formative assessment Where the learner is Where they are going How to get there Teacher Evoking information Curriculum philosophy Feedback Peer Peer- assessment Sharing criteria Peer-tutoring Learner Self- assessment Sharing criteria Self-directed learning

A formative lesson Grade 9 Algebra lesson Weather balloon set off, rising at 8fs -1 2s later, a flare is launched, initial velocity 80fs -1 Students complete a table and a graph to answer three questions: –Maximum height of flare –How long to max height –Average velocity of flare

Regulation of learning Teaching as engineering learning environments Key feature: well-regulated –Teaching vs. learning –Regulation of activity vs. regulation of learning –Long feedback cycles vs. variable feedback cycles –Quality control vs. quality assurance in learning

Regulation of learning Proactive (upstream) regulation –Planning regulation into the learning environment –Planning for evoking information Interactive (downstream) regulation –‘Negotiating the swiftly-flowing river’ –‘Moments of contingency’ –Tightness of regulation (goals vs horizons) Retrospective regulation –Structured reflection (e.g. lesson study)