Secondary Information Systems Introduction to CEM Secondary Information Systems Dr Robert Clark ALIS Project Manager

The Projects

The Projects Alis Yellis MidYIS A / AS / Btec / IB etc Year 13 GCSE Computer Adaptive Baseline Test (paper test also available) Year 13 Year 12 GCSE Year 11 Yellis Computer Adaptive Baseline Test (paper test also available) Year 10 Combines curriculum tests with developed ability INSIGHT Year 9 MidYIS Computer Adaptive Baseline Test (paper test also available) Year 8 (+ additional) Year 7

Early Testing & Predictions Typical Timeline Testing is available from June at end of previous academic year i.e. Y6 / Y9 / Y11 Early Testing & Predictions The baseline score is measured as early as possible and the information sent to the CEM Centre The institution then receives feedback from the CEM Centre: baseline test results and predictions The public examination results are sent to the CEM Centre when received by the institution The value added data is available from September via the Institution’s website at the CEM Centre

Predictions

How CEM ‘Predictions’ are made… . How CEM ‘Predictions’ are made… Subject X A / B C 0 2 4 6 8 Regression Line (…Trend Line, Line of Best Fit) Outcome = gradient x baseline + intercept Correlation Coefficient (~ 0.7)

Some Subjects are More Equal than Others…. D E >1 grade

Some Subjects are More Equal than Others…. GCSE (MidYIS or Yellis) F E D C B A A* Test Score GCSE Grades Art & Design Biology Chemistry Economics English French Geography German History Ict Mathematics Media Studies Music Physical Education Physics Religious Studies Science (Double) Spanish 1 grade

Baseline Measurement

Problems with Key Stage Baselines Not all students have KS baselines No KS3 Foreign Students Vocational Students Adult Learners KS exams do not always represent ‘Start of Course’ ability Post-16 : Year(s) out or intermediate years Prior Value-Added Can you add value at every Key Stage ? Under achievement leading to under expectation One teacher’s output = another teacher’s input Teaching to the test Does performance represent ability, effort or exam technique ? Aptitude & fluency vs Achievement & knowledge Transfer of Data

Although Key Stage baselines can be a very good indicator of potential attainment, by themselves they are not sufficient. Key Stage baselines are confounded by the effects of prior treatment.

The Computer Adaptive Test Test performed online – results automatically transmitted to CEM. Minimal installation / setup required - if any. Adaptive – difficulty of questions changes in relation to ability of student. Efficient – no time wasted answering questions that are far too easy or difficult. Wider range of ability Less stressful on students – more enjoyable experience than paper test. Less demanding invigilation. Cheaper ! In 2010/ 2011 over 200,000 students across yrs 7-13 sat this test Try it yourself at www.intuproject.org/demos

Baseline Feedback Reports, Graphs & Predictions

IPRs (Individual Pupil Record Sheets) Look for sections that are inconsistent Also available based on MidYIS, Alis, SOSCA & INSIGHT scores

Intake Profiles Also available based on MidYIS, Yellis, SOSCA and INSIGHT scores

Intake Profiles (Historical)

Predictions – MidYIS example Similar spreadsheets available from Yellis, SOSCA, INSIGHT

Predictions - Alis example

Chances Graphs Grade Percent Grade Percent Grade Percent Grade Percent English Language - band D 2 10 23 31 24 9 1 20 30 40 50 U G F E D C B A A* Grade Percent English Language - band C 1 2 6 20 35 29 7 10 30 40 50 U G F E D C B A A* Grade Percent English Language - band B 2 7 24 40 21 5 10 20 30 50 U G F E D C B A A* Grade Percent English Language - band A 1 8 26 35 23 7 10 20 30 40 50 U G F E D C B A A* Grade Percent

Predictions vs Targets

What is a ‘Prediction’ ? Targets ? NOT a forecast of the grade the student will get An indication of the grade (points score) achieved on average by students of similar ability in the previous year Targets ? Minimum Targets – Round CEM prediction down ? Realistic Targets – Use nearest CEM grade ? Challenging Targets – 75th percentile ? Prior Value Added ? Arbitrary grade fraction ?

75th Percentile Predictions If all students attain 75th percentile predictions, School VA will be at top 25% Provided by all Projects in addition to 50th percentile Alis : Excel spreadsheet - ‘Predictions – Spreadsheet (75th Percentile)’ Pre-16 : Adjust button on predictions spreadsheet Prior Value Added Only where prior VA is positive ? 1 year or 3 ? Reasonable to use raw residual figures as this is an approximate measure and raw residuals give grade fractions Alis : Can be calculated using PARIS software Data used to inform, not replace, professional judgement

75th Percentile Prior Value Added

75th Percentile Step 1 Prior Value-Added

Value-Added Feedback Reports & Graphs

Measuring Value-Added – Terminology . Measuring Value-Added – Terminology Exam grade -ve VA +ve VA BASELINE SCORE Raw Residual Trend Line/Regression Line

The position of the national trend line is of critical importance Measuring Value-Added – An Example National Trend ‘Average’ Student A* U B C D E F G Result -ve (- 2 grades) Alf Bob Subject A Subject B +ve (+ 2 grades) Chris Low Ability Average Ability High Ability Baseline Score The position of the national trend line is of critical importance

Standardisation of Residuals (Raw) Residuals can be used to examine an individual’s performance Standardised Residuals are used to compare performance of groups Standardised Residuals are independent of year or qualification type For a class, subject, department or whole institution the Average Standardised Residual is the ‘Value-Added Score’ Standardised Residual = Residual / Standard Deviation (National Sample) When using Standardised Residuals then for an individual subject where N = number of results in the group (for combinations of subjects consult the relevant project) 95% Confidence Limit = 2.0 x Standard Error 99% Confidence Limit = 2.6 x Standard Error 99.7% Confidence Limit = 3.0 x Standard Error

Burning Question : What is my Value-Added Score ? Better Question : Is it Important ?

Statistical Process Control (SPC) Chart Value Added Feedback… Statistical Process Control (SPC) Chart 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Year

Subject Summary Standardised Residual Graph -0.2 -0.1 0.1 -0.6 -0.7 -0.4 -2.2 -0.5 2.0 0.4 -4.0 -3.0 -2.0 -1.0 0.0 1.0 3.0 4.0 Art Biology Chemistry Design and Technology Drama English Language English Literature French Geography German History Home Economics Information Technology Maths Media Studies Music Physics Religious Studies Spanish Average Standardised Residual

The Scatter Plot Grade Points Equivalent Look for Patterns… Baseline Score Grade Points Equivalent Look for Patterns… General Underachievement / over achievement ? Do any groups of students stand out ? – high ability vs low ability ? – male vs female ?

Other things to look for… Why did these students do so badly ? Why did this student do so well ? How did they do in their other subjects ?

PARIS Software

PARIS is ….. PARIS analyses ….. PARIS provides ….. Software to install and use in school An interactive reporting tool Included free with Alis / Yellis / MidYIS PARIS analyses ….. Potential Performance Intermediate Performance Actual Performance PARIS provides ….. Student level reports Subject Level reports Institution Level Reports

Attitudes

There is more to school / college than exams…. Student attitudes Student Welfare & Safety Non-academic activities Support Social and personal development Parental Survey Induction Survey Self Evaluation (Every Child Matters) Attitudinal MidYIS INSIGHT Attitudinal Yellis Full ALIS Try it yourself at www.intuproject.org/demos

Other Issues

Points to mull over… Independence – no agendas Transparency of Analysis Self Evaluation Straightforward and standardised administration Prompt Feedback Full working hours phone / email support Student focus Replacement for KS3 Innovative online adaptive testing available – student experience Longitudinal analysis with appropriate error backgrounds Non-curriculum embedded baselines available Attitudinal surveys available – Every Child Matters…

Dr Robert Clark Alis Project Manager robert.clark@cem.dur.ac.uk 0191 33 44 193