When is assessment learning-oriented? Dylan Wiliam 4th Biennial EARLI/Northumbria Assessment Conference, Potsdam, Germany, August 2008

Overview of presentation Why do we need formative assessment? Clarifying assumptions and definitions about formative assessment A theoretically-based definition of formative assessment How formative assessment relates to other aspects of education

Which of the following categories of skill has disappeared from the work-place most rapidly over the last 40 years? 1.Routine manual 2.Non-routine manual 3.Routine cognitive 4.Complex communication 5.Expert thinking/problem-solving

The changing demand for skill Autor, Levy & Murnane, 2003

There is only one 21st century skill So the model that says learn while you’re at school, while you’re young, the skills that you will apply during your lifetime is no longer tenable. The skills that you can learn when you’re at school will not be applicable. They will be obsolete by the time you get into the workplace and need them, except for one skill. The one really competitive skill is the skill of being able to learn. It is the skill of being able not to give the right answer to questions about what you were taught in school, but to make the right response to situations that are outside the scope of what you were taught in school. We need to produce people who know how to act when they’re faced with situations for which they were not specifically prepared. (Papert, 1998)

A convergence of interests Philosophies of education (Williams, 1966)…  Transmission of culture  Preparation for employement  Self-actualization …all require preparation for future learning (PFL)  Cannot be taught in isolation from other learning  Students still need the basic skills of literacy, numeracy, concepts and facts  Learning power is developed primarily through pedagogy, not curriculum  We have to develop the way teachers teach, not what they teach

Learning power environments Key concept:  Teachers do not create learning  Learners create learning Teaching is the engineering of effective learning environments Key features of effective learning environments:  Create student engagement (pedagogies of engagement)  Well-regulated (pedagogies of contingency)

Why pedagogies of engagement? Intelligence is partly inherited  So what? Intelligence is partly environmental  Environment creates intelligence  Intelligence creates environment Learning environments  High cognitive demand  Inclusive  Obligatory

Motivation: cause or effect? competence challenge Flow apathy boredom relaxation arousal anxiety worry control high low high (Csikszentmihalyi, 1990)

Why pedagogies of contingency? Learners do not learn what they are taught Assessment is the bridge between teaching and learning, and thus the central process of teaching (as opposed to lecturing).  Pedagogies of contingency  Personalisation  Mass customization (rather than mass production or individualisation)  Diversity  A valuable teaching resource (rather than a challenge to be minimized)

What gets learnt? Denvir & Brown (1996)  Understanding of basic number in low-achieving 7-9 year olds  Extensive conceptual hierarchy developed  Students assessed  Teacher plans teaching programme  Students re-assessed

The research evidence Several major reviews of the research…  Natriello (1987)  Crooks (1988)  Kluger & DeNisi (1996)  Black & Wiliam (1998)  Nyquist (2003) … all find consistent, substantial effects

It’s the cost-benefit ratio stupid… InterventionExtra months of learning per year Cost/ classroom/yr Class-size reduction (by 30%)4€25k Increase teacher content knowledge from weak to strong 2? Formative assessment/ Assessment for learning 8€2.5k

Independent dimensions of assessment Scale  Large-scale (nomothetic) versus small-scale (idiographic) Locus  Classroom versus examination hall Authority  Teacher-produced versus expert-produced Scope  Continuous versus one-off Format  Multiple-choice versus constructed response Function  Formative versus summative

No such thing as formative assessment Purposes of assessments  Evaluative  Summative  Formative  Instruments  Purposes  Functions Prospects for integration are bleak Formative assessment involves the creation of, and capitalization upon, moments of contingency in instruction “An assessment functions formatively when evidence about student achievement elicited by the assessment is interpreted and used to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions that would have been made in the absence of that evidence.” Black and Wiliam, 2009 (we hope!)

Some principles A commitment to formative assessment  Does not entail any view of what is to be learned  Does not entail any view of what happens when learning takes place …although clarity on these is essential.

Evolving conceptions of formative assessment “Feedback” metaphor  Components of a feedback system  data on the actual level of some measurable attribute;  data on the reference level of that attribute;  a mechanism for comparing the two levels and generating information about the ‘gap’ between the two levels;  a mechanism by which the information can be used to alter the gap. Feedback system  Importance of eliciting the right data  The role of the learner  The role of the learning milieu (e.g., as activity system)

Unpacking formative assessment Key processes  Establishing where the learners are in their learning  Establishing where they are going  Working out how to get there Participants  Teachers  Peers  Learners

Aspects of formative assessment Where the learner is going Where the learner isHow to get there Teacher Clarify and share learning intentions Engineering effective discussions, tasks and activities that elicit evidence of learning Providing feedback that moves learners forward Peer Understand and share learning intentions Activating students as learning resources for one another Learner Understand learning intentions Activating students as owners of their own learning

Five “key strategies”… Clarifying, understanding, and sharing learning intentions  curriculum philosophy Engineering effective classroom discussions, tasks and activities that elicit evidence of learning  classroom discourse, interactive whole-class teaching Providing feedback that moves learners forward  feedback Activating students as learning resources for one another  collaborative learning, reciprocal teaching, peer-assessment Activating students as owners of their own learning  metacognition, motivation, interest, attribution, self-regulated learning, self- assessment (Wiliam & Thompson, 2007)

…and one big idea Use evidence about achievement to adapt instruction to meet learner needs

Keeping learning on track (KLT) A pilot guides a plane or boat toward its destination by taking constant readings and making careful adjustments in response to wind, currents, weather, etc. A KLT teacher does the same:  Plans a carefully chosen route ahead of time (in essence building the track)  Takes readings along the way  Changes course as conditions dictate

Effects of formative assessment Long-cycle  Span: across units, terms  Length: four weeks to one year  Impact: Student monitoring; curriculum alignment Medium-cycle  Span: within and between teaching units  Length: one to four weeks  Impact: Improved, student-involved, assessment; teacher cognition about learning Short-cycle  Span: within and between lessons  Length:  day-by-day: 24 to 48 hours  minute-by-minute: 5 seconds to 2 hours  Impact: more responsive classroom practice; increased student engagement

System responsiveness and time-frames If evidence is to inform decision-making, the evidence needs to be available before the decision…  Long-cycle: Are our professional development programmes well-aligned with the needs of our teachers?  Cycle-length: two years  Long-cycle: Does our curriculum adequately cover the state standards as operationalized in the annual state test?  Cycle-length: one year  Medium-cycle: Is this student responding adequately to the tier 1 intervention for reading or do they require a tier 2 intervention?  Cycle-length: one month  Short cycle: Does the class understand the generation of equivalent fractions well enough to move on to the addition of fractions?  Cycle-length: five minutes

The formative assessment hi-jack… “Statistical process control” models of learning USA: “Formative tests”  Tests administered at intervals of 6 to 10 weeks  Often not even keyed to instruction England (5 to 16 year olds):“Assessment for learning strategy”  Government policy focused on target-setting and level chasing  Focus on “tracking achievement” England (Higher Education): Portfolio assessment  New focus on formative e-assessment

Ideas whose time has come…or gone… Diagnostic analysis of standardized tests is probably dead  Lack of agreements about models  Models make assumptions not about items, but how students answer them  Dearth of assessment developers who know enough about learners  Poor efficiency More promising developments  Use of Bayesian inference networks to build proficiency models  But  Proficiency models are not necessarily developmental models  Models need large amounts of data to run

Getting the cycle right (and the right cycle) Within this view of formative assessment  feedback is not whole of formative assessment  It’s not even the most important component of formative assessment Medium- and long-cycle formative assessments  Are supported by existing psychometrics  Are easy to manage, but  Generally produce small effects Short-cycle formative assessments  Contradict important psychometric assumptions  Reliability  Monotonicity of ICCs  Are difficult to establish, but  Generally produce large effects

The overlap between age-cohorts is large… The spread of achievement within each cohort is greater than generally assumed

…so individual progress is hard to track On typical standardized tests growth is slow…  Average annual growth of achievement of individuals is around 0.4 sd  So monthly growth of individual achievement is 0.03 sd …and the reliability of the test is limited…  A reliability of 0.90 corresponds to a standard error of measurement of 0.3 sd  In other words, the SEM of a highly reliable test is ten times the monthly growth in achievement. So standardized tests are completely useless for monitoring individual progress in achievement—they are insensitive to instruction.

…and the data is no use when it arrives… Traditional testing deals with individuals, but teachers mostly deal with groups Data-Push vs. Decision-Pull  “Data-push”  Quality control at end of an instructional sequence  Monitoring assessment that dentifies that remediation is needed, but not what  Requires new routines to utilize the information  “a series of unwanted answers to unasked questions” (Popper)  Decision-Pull  Starts with the decisions teacher make daily  Supports teachers “on-the-fly” decisions If a 30-item test provides useful information on an individual, then responses from 30 individuals on a single item might provide useful information on a class

Characteristics of hinge-point questions Relate to important learning outcomes necessary for progression in learning Can be used at any point in a learning sequence  Beginning (range-finding)  Middle (mid-course correction)  End (e.g., “exit pass”) When used in “real-time” teacher must be able to collect and interpret the response of all students in 30 seconds

Low probability of correct guessing In which of these right-angled triangles is a 2 + b 2 = c 2 ? A a c b C b c a E c b a B a b c D b a c F c a b

Build on key (mis-)conceptions…in math What can you say about the means of the following two data sets? Set 1: Set 2: A.The two sets have the same mean. B.The two sets have different means. C.It depends on whether you choose to count the zero.

Wilson & Draney, 2004 …in Science… The ball sitting on the table is not moving. It is not moving because: A. no forces are pushing or pulling on the ball. B. gravity is pulling down, but the table is in the way. C. the table pushes up with the same force that gravity pulls down D. gravity is holding it onto the table. E. there is a force inside the ball keeping it from rolling off the table

… and History. Why are historians concerned with bias when analyzing sources? A.People can never be trusted to tell the truth B.People deliberately leave out important details C.People are only able to provide meaningful information if they experienced an event firsthand D.People interpret the same event in different ways, according to their experience E.People are unaware of the motivations for their actions F.People get confused about sequences of events

Correct Incorrect Requirements for hinge-point questions For an item to support instructional decision-making, the key requirement is that in no case do incorrect and correct cognitive rules map on to the same response (Wylie & Wiliam, 2007)

BCDABCDA Item improvement The discovery of new incorrect cognitive rules that interpret item keys leads to item improvement In which of these figures is one-quarter shaded?

Feedback Kinds of feedback in Higher Education (Nyquist, 2003)  Weaker feedback only  Knowledge of results (KoR)  Feedback only  KoR + clear goals or knowledge of correct results (KCR)  Weak formative assessment  KCR+ explanation (KCR+e)  Moderate formative assessment  (KCR+e) + specific actions for gap reduction  Strong formative assessment  (KCR+e) + activity

Effect of formative assessment (HE) NEffect* Weaker feedback only Feedback only Weaker formative assessment Moderate formative assessment Strong formative assessment *corrected values

Feedback Feedback should  Cause thinking  Provide guidance on how to improve  Focus on what to take forward to the next assignment rather that what is deficient about the last assignment  Be used Techniques  Delayed scores/grades  Learning portfolios  “Five of these answers are wrong. Find them and fix them”  ‘Three-quarters of the way through a unit” test

Sharing learning intentions Effective summative assessment:  Requires raters to share a construct of quality Effective formative assessment  Requires learners to share the same construct of quality as the raters  Requires teachers to have an anatomy of quality Techniques  Explaining learning intentions at start of lesson/unit  Learning intentions  Success criteria  Intentions/criteria in students’ language  Posters of key words to talk about learning  Planning/writing frames  Annotated examples ‘flesh out’ assessment standards (e.g. lab reports)  Opportunities for students to design their own tests

Students owning their learning and as learning resources for one another Techniques  Students assessing their own/peers’ work  with rubrics  with exemplars  “two stars and a wish”  Training students to pose questions/identifying group weaknesses  Self-assessment of understanding  Traffic lights  Red/green discs

The learning milieu Dual processing theory (Boekaerts, 1993)  Self-regulated learning is both metacognitively governed and affectively charged (Boekaerts, 2006 p. 348)  Students form mental representations of the task-in-context and appraise:  current perceptions of the task and the physical, social, and instructional context within which it is embedded;  activated domain-specific knowledge and (meta)cognitive strategies related to the task;  beliefs about motivation (including domain-specific capacity), interest and the effects of effort  Resulting appraisal generates activity along one of two pathways:  ‘well-being’ pathway  ‘growth’ pathway

When is assessment learning oriented? Assessment is learning oriented when it  Is integrated into instructional design so that it becomes invisible  Creates engagement in learning  Helps learners to understand what successful performance looks like  Generates information that can be interpreted in terms of a learning progression  Focuses attention on growth rather than well-being  Provides a focus for supportive conversations between learners