PLASC/NPD User Group Mick Quinlan Change, or the lack of it The Players: suppliers and users NC assessment process A wander through the book… an A-Z of the collection of NC assessment data

Forward Id like to change the current process but: –there are many players in the game each with existing systems and many dependencies –were not good at managing change!

Drivers for Change 1998 Service delivery failure of data collection exercise 1999 Rose report – Weighing the Baby 2004 Beasley report – service delivery failure of KS3 English test results 2005 Statistics Commission report #23 – measuring standards in English Primary schools 2005 Statistics Commission report #26 – school education statistics: user perspectives 2006 NAO 3 rd validation compendium report – quality of data systems underpinning PSA targets –Broadly appropriate but system needs strengthening

Metadata Statistics Commission #23, Feb 2005, §11 –We feel that public presentation of the KS scores in statistical releases should include a clear statement about the uses to which the data may be put, and the limitations of the data in respect of those uses Statistics Commission #26, June 2005, §D –Full and informative statistical commentary should be provided alongside the figures to guide users

Making Good Progress

Changes for 2006 KS2/3 Electronic collection of TA, including pupil-level non-core results for KS3, via S2S – rather than OMR forms –A decision has been taken by the Departments Head of Profession for Statistics not to provide this data until the quality and robustness can be assured (SFR ) KS3 English: Richard III in, Henry V out KS3 ICT onscreen test 2nd pilot (170,000 pupils)

The Players: DfES DfES – Paymaster/customer (TRD) –School Census/NPD and FSP/KS1 data collection –EduBase –School Self-Evaluation/Improvement –Autumn Package (benchmarks, progress, target setting) –PAT/RAISEonline) –School Accountability - Performance Tables –SERAP, FORVUS, FFT –National Statistics –Statistical First Releases –PSA Targets

The Players: NAA A division of QCA (the Regulator) Managers of the process: –Test development (NFER, Edexcel, UCLES, RM) –Pupil registration (DfES, schools) –Test Printing/Distribution (COI/Granby) –Test marking (Edexcel) –Teacher Assessments (schools, LAs) –Data capture (Pearson) –Non-standard processes (in-house)

Mark Scripts Borderlining Marked scripts Level- setting Sample scripts Script scrutiny Cut scores Recom- mend- ation DC results database Borderlined scripts + borderlined marksheets + school report (inc. marks/levels) Request for Review R1: Clerical check R2: Individual Re-marking R3/GR: Group review Script + Revised mark and level Data entry Marker Standardisation Sample scripts Guidance Schools Re-marked script Mark and level Script pool collection 7.3 May-Jun Level-setting data collection 7.5 May/Jun Jun May Jun Jun-Aug Jul/Aug Jul/Sep Jul/Aug Selected scripts Returned scripts Special consideration 7.13 May/Jun Malpractice 7.14 May/Jul Teacher Assessments 7.15 Jul Request for Process review 7.13 Oct Investigate review 7.14 Oct/Nov Upheld review Re-review The Process Pretest 2: Draft Level thresholds

A is for Assessment National Curriculum –key stages: Foundation, KS1, KS2, KS3 –Subjects core: English, maths & science Attainment Targets non-core: Art & Design, D&T, Geography, History, ICT, MFL, Music and PE –Test/Task & Teacher Assessment –Levels

Levels Q: Whats the difference between Level 4 and Level 5? A: 1 mark Making Good Progress (KS2-KS3,page 9) –This shows that 15[%] pupils did not progress by a full level since finishing KS2 and three[%] have actually regressed

A A = pupil Absent from the test Aegrotat Achievement & Attainment Tables Age Standardised Scores (ASS) Additional time Appeals (process reviews)

BCD B = Working Below the level of the test Borderlining Braille Compensatory award Classes/Teaching Groups – collected in 1997 D = pupil Disapplied from the curriculum Date of Birth

E E = Exceptional Performance Extension Papers Early takers Errors

FGH F = Future sitting (subjects early takers not taking) Fine grade Foot and mouth Gender Handwriting

IJKL Independent schools L = pupil Left school (KS2/3) L = below Level 2 in the reading comprehension test (KS1)

MNO M = Missing scripts Maladministration/malpractice Missing pages Mental Maths test Modified large print (MLP) N = No test level (too few marks gained) Names, of pupils Overseas/Offshore schools Optional tests (Y3-5 & Y7/8)

PQR P = Past sitting (subjects early takers took last year) Progress tests (Year 7) P Scales Q = result annulled - malpractice (DfES) Reading Reviews Re-sits

ST Service Childrens Education (SCE) Schools Special Consideration Shakespeare Sub-Levels Spelling test T = Pupil unable to access the tests Tier

UVW U = No TA subject Level calculated (KS1) Unique Pupil Number (UPN) V = mixed tiers Versions W = Working towards Level 1 Writing

XYZ X = Marks pending (KS2/3) X = pupil not entered for reading comprehension test (KS1) Year groups Z = ineligible pupil