Introduction to Value-Added Data Dr Robert Clark

Theory and Stats bits…

. Trend Line/Regression Line Measuring Value-Added – Terminology BASELINE SCORE -ve VA +ve VA Residuals VA Exam grade

Measuring Value-Added – An Example Low AbilityAverage AbilityHigh Ability Baseline Score A* U B C D E F G Result Aldwulf Beowulf Cuthbert +ve (+ 2 grades) -ve (- 2 grades) National Trend ‘Average’ Student The position of the national trend line is of critical importance Subject A Subject B

Some Subjects are More Equal than Others…. A-Level >1 grade A*ABC A B C D E

Some Subjects are More Equal than Others…. BTec National Diploma BCE DDM DDD DMM MMM MMP MPP D PPP Average GCSE Score Grade

Some Subjects are More Equal than Others…. International Baccalaureate CBAA* Average (I)GCSE Score Grade Biology Business and Management Chemistry Design Technology Economics English_A1 Film French_B Geography History Mathematics Music Philosophy Physics Psychology Spanish_B Theatre Arts Visual Arts

F E D C B A A* Test Score GCSE Grades Art & Design Biology Chemistry Economics English French Geography German History Ict Mathematics Media Studies Music Physical Education Physics Religious Studies Science (Double) Spanish Some Subjects are More Equal than Others…. GCSE

Definitions: Residual – difference between the points the student attains and points attained on average by students from the CEM cohort with a similar ability Standardised Residual – the residual adjusted to remove differences between qualification points scales and for statistical purposes Average Standardised Residual – this is the ‘Value Added Score’ for any group of results Subject VA – average of standardised residuals for all students’ results in the particular subject School VA – average of standardised residuals for all students’ results in all subjects for a school / college Confidence Limit – area of statistical uncertainity within which any variation from 0 is deemed ‘acceptable’ and outside of which could be deemed ‘important’

Measuring Ability Key Stage Data or Baseline Test ?

Key Stage Baselines – KS2; GCSE StrengthsWeaknesses Related to curriculumDependent on teaching effectiveness (prior Value- Added) as well as student ability Automatically availableOpen to manipulation In DepthNot all students have KS data Linked to student learning experience Reliability of KS measure Consistency of KS score (i.e. GCSE & Btec First etc) Future of GCSE – EBac certificates ? Future of KS2 ? Impact of Teacher Assessment ? Independent Baseline Test – CEM Adaptive Test

Burning Question : What is my Value-Added Score ? Better Question : Is it Important ?

SPC Chart Year Performance inline with expectation VA Score Performance below expectation Problem with Teaching & Learning ? Performance above expectation Good Practice to Share ?

Subject Summary Standardised Residual Graph

The Scatter Plot Baseline Score Grade Points Equivalent Look for Patterns… General Underachievement / over achievement ? Do any groups of students stand out ? – high ability vs low ability ? – male vs female ?

Other things to look for… Why did these students do so badly ? Why did this student do so well ? How did they do in their other subjects ?

Worked Example – Pre 16

Which Subjects Cause Most Concern ? Danger of Relying on Raw Residuals Without Confidence Limits

Which subjects now cause most concern ?

Business Studies

Religious Studies

Worked Example – Post 16

SPC Chart Year

Subject Summary - 3 Year Average Subject Summary - Current Year

A2-English Literature Statistical Process Control (SPC) Chart Year

Student Level Residuals (SLR) Report Scatter Plot A2 – English Literature General Underachievement ?

Student Level Residuals (SLR) Report Scatter Plot A2 – English Literature Too many U’s ?

Summary of Process Examine Subject Summary Determine ‘interesting’ (i.e. statistically significant) subjects Look at 3 year average as well as single year if available Look at trends in ‘Interesting Subjects’ Examine student data –Scatter graphs Identify students over / under achieving (student list or Paris) Any known issues ? Don’t forget to look at over achieving subjects as well as under achieving Phone / CEM when you need help understanding / interpreting the data / statistics !

Baseline Choice

Do students with the same GCSE score from feeder schools with differing value-added have the same ability ? How can you tell if a student has underachieved at GCSE and thus can you maximise their potential ? Has a student got very good GCSE scores through the school effort rather than their ability alone ? Does school GCSE Value-Added limit the ability to add value at KS5 ? Can you add value at every Key Stage ? How can you check for this ? GCSE or Baseline Test ?

GCSE Value-AddedKS5 ExpectationsKS5 Value-Added ? ?

The Effect of Prior Value Added Beyond Expectation +ve Value-Added In line with Expectation 0 Value-Added Below Expectation -ve Value-Added Average GCSE = 6 Do these 3 students all have the same ability ?

Same School - Spot the Difference ? GCSE as Baseline Test as Baseline

GCSE as Baseline Test as Baseline

A2 Biology GCSE as Baseline Test as Baseline

A2 Biology GCSE as Baseline Test as Baseline Student A Student B

A2 Biology Student A BaselineGradePoints‘Predicted’ Points VA (Residual) GCSEA TestA Student B BaselineGradePoints‘Predicted’ Points VA (Residual) GCSEA* TestA* How well did these students perform ?

National or School Type Specific ?

Comparison to all schools Comparison to Independent Schools Only

Comparison to all schools Comparison to FE Colleges Only

Questions: → How does the unit of comparison used affect the Value-Added data and what implications does this have on your understanding of performance ? → Does this have implications for Self Evaluation ?

Thank You Robert Clark –