NEW USERS OF ALIS WORKSHOP JUNE 2011 London Conference Geoff Davies

FIVE EXERCISES COVERAGE Baseline data Initial feedback Use of Chances graphs ‘Predictions’ Use of Scatter Graphs Setting ‘target’ grades Interpreting Statistical process charts

Basic ALIS is concerned mainly with value added analysis using baseline data collected early in the sixth form Full ALIS covers this as well as an in depth analysis of attitudes, teaching and learning processes, resources, extra curricular activities, and more Average GCSE score --- repeatedly been found to be the best single indicator of post 16 performance Computer adaptive test---CEM centre’s jewel in the crown across all spheres of education (TDA paper test still available for A level students) TWO BASELINES

INITIAL FEEDBACK

Intake Profiles GCSE (Alis) TDA / CABT (Alis)

Predictions Predictions are available in the following forms: Formal Reports Spreadsheets Paris Software Average performance by similar students in past exams Student IDNameGenderDate of BirthYear GroupGCSE ScoreSubjectLevel GCSE Based Predictions PointsGrade Student Prediction Reports

The ‘prediction’ for each subject is based on the previous year’s ALIS ‘trend- line’ for that subject Example: Lisa Fry has GCSE grades 2A*, 3A, 4B and 1C Total GCSE Points = 66 (i.e. 2x8 + 3x7 + 4x6 + 1x5) AVGCSE = 66 / 10= 6.6 Use ALIS subject formula (21.71 x 6.6) – =99 (B at A level)

Chances Graphs (A2) Geography 2008 Chances graphs Based on students : Correlation coefficient = 0.72, Standard Deviation = Regression Graph The thick region indicates the interquartile range (i.e. the middle 50% of students). (A2) Geography 2008 The thick region indicates the interquartile range (i.e. the middle 50% of students).

Situation You are the subject teacher and are discussing AS target grades with individual students in the Autumn term of Year 12. You are about to talk to Belinda who achieved an average GCSE score of 5.9. This gives a statistical prediction=14.75x = 27.5 UCAS points using the regression formula at AS for this subject (Grade D/E at AS). The computer adaptive test seems to confirm this prediction. Exercise 1 PRACTICAL EXERCISES FOR NEW ALIS USERS

Chances graph showing % of students achieving each AS grade in previous year based on students AS GRADE Belinda (5.9) Individual Chances graph example

Other examples Chemistry different cohorts

‘Most candidates with Belinda’s GCSE background score achieved a D in my subject last year so Belinda’s target grade should be a D’. What are the weaknesses of this statement? a)

What other factors should be taken into consideration apart from chances graph data, when determining a target grade? b)

Exercise 1 The strength of the chances graph is that it shows more than a bald prediction. True, most students starting from an average GCSE score like Belinda did achieve a D grade at AS in examinations for this subject. However the probability of a C grade is also high since her score was not at the bottom of this range. Students are not robots who will always fit with statistics so it is dangerous to make sweeping statements based on one set of results. As well as looking at the prediction you should use the chances graph as a starting point with your professional judgement taking into account factors such as previous performance in the subject, her attitude to work and what she is likely to achieve based on your own experience. You might want to start with the most popular outcome grade D and use your judgement to decide how far up (or down!) to go. She may be a very committed student, so a B grade may be more appropriate though A looks unlikely. If you are using aspirational targets for psychological reasons with students then B may be appropriate

Exercise 2 Situation Jane has completed her 6th form studies and a review has been received for her by the college after A level results. Choose one subject at a time and look carefully at what happened in that subject both from a baseline of average GCSE grades and from a baseline of the computer adaptive test.

Review: Final_Result ReviewReview Average Average Subject Points Grade Points Grade Residual Std. Residual (A1) Health &Social Care30.00D39.64 C (A2) Religious Studies80.00C89.84 B/C (A2) English Literature80.00C87.29 B/C (A2) Drama &Theatre St B91.84 B/C FINAL RESULTS PREDICTIONS RAW STANDARDISED Profile Sheet: Jane (from Average GCSE) Year: 2007 DOB: 01/06/89 (Average GCSE = 6.00 (Band B))

Chances Graphs - Band B from average GCSE

Individual Chances Graphs from average GCSE score

Profile Sheet: Jane from Computer adaptive test Year: 2007 DOB: 01/06/89 (Online adaptive test = 0.11 (Band C)) Review: Final_Result RAW STANDARDISED Review Average Average Subject Points Grade Points Grade Residual St (A1)Health &Social Care30.00D25.97 D/E (A2) Religious Studies80.00C79.01 C (A2) English Literature80.00C74.3 C/D (A2) Drama &Theatre St B85.16 B/C PREDICTIONS

a) Can you suggest possible reasons for the differences between the predictions (average grade) made by the two baselines? b) Did Jane reach her potential in all subjects? c) Jane had been set aspirational targets prior to AS and A level by her teachers as below Health and Social Care C Drama B Religious Studies B English B Where these reasonable target grades? Why should these grades not be used for accountability of her teachers?

Exercise 2 a) This student has been placed in different bands, band B from average GCSE score and band C from the computer adaptive test. This sometimes happens. It may be that the student had an off day when she did the computer adaptive test or it may be that there could have been a lot of ‘spoon feeding’ at GCSE. Jane may do better at coursework! Even though we may not know the cause it can act as a warning when analysing results though the predictions are not wildly out. b) Jane certainly reached her potential in Drama and Theatre studies with positive standardised residuals by both methods. On the basis of the computer adaptive test she broadly reached potential in all subjects. On the basis of average GCSE two A level subjects were broadly down about half a grade and she dropped a grade in the AS c) I hope you agree these were reasonable target grades. Remember we don’t know the student, but use the chances graphs and if these were aspirational grades for the student then accountability of departments staff on that basis is not appropriate, but it certainly is on the basis of a whole classes average standardised residuals particularly over a number of years.

Exercise 3 Scatter Graphs(A2) Subject Report A2 Subject ReportYear: 2008 Review: Final_ResultBaseline: Average GCSE BC

Exercise 3 Hopefully this did not create too much difficulty The arrows are vertical and join the point to the trend line. Not easy to judge from this. It might be worth plotting standardised residual against average GCSE score and look for deviations from a horizontal line (more advanced) Standardised residuals allow one subject to be compared with another because different subjects have varying distributions of grades. These are not dependent on the year either.

Exercise 4 Change in teaching methods used had major effect Sept 2007 probably a factor! Exercise 5 Some schools and colleges use average GCSE points together with chances graphs to tweak appropriate course choices for their students on the basis of probability of success. Perhaps more useful is alerting year 12 tutors to the strengths and weaknesses of students as shown by the computer adaptive test.

Using the chances graphs Of the students with a point-score below 5.0 a total of 11 achieved an ‘A’ grade and a total of 562 achieved a ‘U’ grade! Example: A/S Accounting (2005)

Using the chances graphs Of the students with a point-score above 6.4 a total of 113 achieved an ‘A’ grade and a total of 27 achieved a ‘U’ grade! Example: A/S Accounting (2005)

Comparison between subjects AS Biology (2005) AS Film Studies (2005)

NEW USERS OF ALIS WORKSHOP JUNE 2011 London Conference Geoff Davies