Value Added and Distance Travelled and the Learner Achievement Tracker Yvonne Fullwood New Measures of Success LSC, National Office

Context of the New Measures Fair, valid and ‘fit for purpose’ measures Four partners: LSC, DfES, ALI, OFSTED ‘Basket’ of measures to contextualise performance Application across the sector No additional bureaucratic burden

New Measures of Success Qualification success rate (QSR) Value Added and Distance Travelled measures (VA/DT) Recognising and recording progress and achievement (RARPA) 4. Learner satisfaction Learner destinations Provider responsiveness to employers Value for money (VFM) Staff measures

Purpose of VA and DT for 16-19 learners Improvement and inspection: LSC: Value Added and Distance Travelled data for 16-19 learners Learner Achievement Tracker (LAT) Provider Gateway DfES accountability: VA for 16 – 19 Learners included in the School and College Achievement Tables pilot

Overview (1) Measures the progression in qualifications of 16-19 learners based on their prior attainment ie, how well do learners do post-16 based on their achievement at Key Stage 4? Cover the majority of larger, approved qualifications taken by 16-19 learners, including vocational and ungraded qualifications

Overview (2) VA for 16-19 learners will measure achievement in Level 3 graded qualifications, for example: GCE A and AS level AVCE BTEC National Diploma DT for 16-19 learners will include other approved Qualifications at levels 1,2 and 3 for example: NVQ Foundation and Intermediate GNVQ GCSE BTEC First

Learner Achievement Tracker (LAT) Produces reports for use by teachers, managers, inspectors and LSC Accessible via the Provider Gateway Uses most robust statistical methods available and includes full national sample Builds on proprietary systems currently available Software and data free of charge to all LSC-funded providers and LAs The LSC produced a prototype version of the Learner Achievement Tracker (LAT). The LAT is able to produce reports at different levels of aggregation and can be used by providers, inspectors and the LSC The LAT is available on the Provider Quality Gateway. The measures are calculated using a multi level modelling technique. More information on the statistical methodology will be available on the LSC website for the pilot year. The LAT provides the facility of reporting VA and DT data, but whilst built on similar functionality of proprietary systems, the LAT does not replicate all the functionality or range of services as these systems maybe able to offer. Further development work will continue during the pilot year and following the launch to increase the usefulness of the system and the information it presents. The software will be free of charge and will also include the entire national data set and provider data.

VA and DT data Data used to calculate the VA and DT for 16 – 19 learners is gathered directly from the the awarding bodies Additional data from the ILR is required to calculate DT for 16 – 19 learners Providers will not be involved in collecting data for the measure Providers will be involved in checking the accuracy of the VA data In accordance with the commitment to reduce bureaucracy, the data used to calculate the VA and DT measures will be gained directly from the awarding bodies. This information is sent annually to the University of Bath who match the achievement data for qualifications achieved by learners post-16 to the learners prior attainment at KS4 (or equivalent) which they have collected in the previous year(s). The dataset is credibility checked by a further contractor and the prior attainment for each learner is calculated. The achievement data provided to the University of Bath is in grade format. Each qualification (and grade within a qualification) has been assigned a number of QCA points. The grades for each learners achievement are converted into QCA points, added up and then divided by the number of qualifications achieved to give an average prior attainment in QCA points. The national performance is calculated using MLM and this information as well as the provider level data is fed into the LAT so the provider level performance can be calculated. The process for producing DT information is the same as for VA apart from an additional data source that is needed, given the nature of the qualifications included in the DT measure. The additional data source required is the ILR which is needed to identify those learners who complete a qualification but that do not achieve.

Issues for further exploration Learners with missing or no prior attainment Context variables Customising the LAT Treatment of AS levels in the VA measure Treatment of fails in the VA measure Use of success rate for DT A number of issues have been identified in discussions with providers, partners and stakeholders which are to be investigated further during the pilot year. These include: FAILS – these are not included in the VA measure at present due to an issue with the QCA points scoring system. AT present, the distance between grades is not consistent between pass and fail. In the case of an A level, the distance in QCA points between grades is 30 points (A = 270; B = 240 etc.). However the difference between an E grade (140) and a fail (0) is 140 QCA points. As a result of this, fails can skew the VA results. USE OF SUCCESS RATES FOR DISTANCE TRAVELLED – there are a number different platforms from which the Distance Travelled measure for WBL can be built, depending on whether achievement rate or success rate is used. As part of the work with the Adult Learning Inspectorate and the team leading on the Qualification Success Rate measure, analysis will be undertaken to identify which is the most suitable measure to use. CONTEXT VARIABLES / SOFT INDICATORS – at present, the VA and DT measures do not contain context variables of soft indicators. Feedback from providers particularly, has indicated that this is something they wish to be included in the measure. Further modelling work will be undertaken to ascertain how these can be built into the measures and how the LAT can accommodate them in the calculation.

LAT piloting arrangements January 2006 Updated Quality Improvement Packs available Pilot LAT will be available via the Provider Gateway. This will contain Work-Based Learning Distance Travelled (DT) data and un-amended Value Added (VA) data for schools and colleges March 2006 DT data for colleges and amended VA data for schools and colleges. to April 2006 Short term evaluation. November 2006 The LAT software released for reflection and quality improvement purposes. Ofsted continues to pilot the LAT.

More information http://www.lsc.gov.uk/nms Measuring Success Newsletter The Quality Improvement Pack Implementing New Measures of Success Local LSC

The Learner Achievement Tracker (LAT) Reports

LAT Output reports Type of LAT report What the report shows Summary Charts Overview of performance across different qualifications and across different subjects within a given qualification type compared to the national average National comparison graph Performance in a given qualification/subject compared to the national average across the full range of prior attainment National chances charts Chances of a learner with a given prior attainment achieving a particular outcome

Summary Chart for all Qualifications

A Level Summary Chart

National Comparison Graph: A Level D&T Product Design

National Comparison Graph: A Level Maths A Level Mathematics

National Comparison Graph: A Level Psychology

A Level D&T Product Design Chances Chart (40-46) 34%

A Level Mathematics Chances Chart (40-46)

A Level Psychology Chances Chart (40- 46)

Example of a Distance Travelled National Comparison Graph

Further Information The Quality Improvement Pack contains detailed information about the interpretation and application of each type of LAT reports www.lsc.gov.uk/nms