Hubble Bubble Some Qualifications Bring Trouble June 2012 Debbie Grossman Emma Williams Planning and Statistics

Aim: To understand the way qualification data is collected, processed and used both within the University and by outside organisations.

Qualification Levels GCSE grades G - D GCSE grades A* - C A Levels, BTECs, International Baccalaureate etc. HNC HND, Foundation Degree etc. Honours degree Masters Degree PhD

Qualifications recorded Currently have 231 codes in BANNER 142 Level 3 126 sent to HESA

2010-11 UG New Entrants (Excluding Foundation Degrees) Task Highest Qualification Full time Part time Level 4 and above A Levels BTECs Access Course (accredited) Activity Think about what you know about the student population and in groups estimate the percentage of new entrants that would be in each of these categories. Note: The qualifications listed are not a complete list therefore the total will not equal 100%

2010-11 New Entrants (Excluding Foundation Degrees) Values Highest Qualification Full time Part time Level 4 and above 22.05% 67.74% A Levels 38.39% 5.97% BTECs 8.74% 1.83% Access Course (accredited) 3.94% 0.88% Activity answers Opportunity to discuss the answers and how close the teams answers were to the actual values. Did they come close to peoples perceptions?

Applicant Qualifications submitted as part of the application process L3 and/or above GCSEs & equivalents from 2012-13 Professional qualifications/experience Foundation degrees studied at Greenwich UCAS Electronic Applications System ABL Non UCAS Applicants Entered manually by Admissions Note for Debbie: BOTH go into the Electronic Applications System (EAS) which is an internal system and are then ‘SOATEST-ed’. Some will go in via ABL, but only when the qual comes through UCAS I think i.e. this year’s results. Qualifications held before ABL in August go in manually to SOATEST. Non UCAS applications come in through OLAF (On-Line Application Form) and go into EAS, to then be entered manually by admissions into SOATEST, as stated.

Student OLR Qualifications checked at Stage 2 registration. Review of qualifications held in BANNER Update or comment Qualifications checked at Stage 2 registration. Original Certificates SOATEST (BANNER) updated if required

ELQ Status determined by qualifications held prior to starting the new programme Generated within BANNER Rules updated by PAS

AAB+ and Equivalents Unlimited recruitment for 2012-13 Not based on a tariff score Determined by HEFCE Specific list of qualifications with certain grades Best 3 A Levels, BTECs, Best 5 Scottish Highers, 35 or more International Baccalaureate ABB for 2013-14

Tariff Tariffable Qualifications UCAS determine tariff scores Generally Level 3 Not all level 3 have tariff scores UCAS determine tariff scores Currently based solely on GLH

Tariff Challenge 1 Place the qualifications in order (lowest to highest) according to their tariff score.

Tariff Challenge 1 Position Qualification & Grade Points 1 International Baccalaureate - 31 413 2 BTEC Diploma QCF - D*D* 280 3 CACHE Diploma in Childcare & Education - C 240 4 GCE A Level with additional AS – AB 170 5 AAT NVQ level 3 in Accounting - Pass 160 6 GCE A Level – A* 140 7 Cambridge Pre–U – D3 130 8 Irish leaving Certificate – A1 90 9 Diploma in Fashion Retail – Pass 80 10 Sports Leaders UK - Pass 30

HESA Tariff Most qualifications that have UCAS tariff No qualifications that don’t have UCAS tariff Provided to institutions Provided to league table providers Dependent on league tables request

Internal Tariff Score HESA tariff methodology In session calculation Available for student and applicant Applicant Predicted Actual Student

Tariff Challenge 2 Which of the qualifications listed have tariff points? UCAS HESA

Tariff Challenge 2 Qualification UCAS HESA Access course   British Horse Society Horse knowledge & care BTEC: Subsidiary Diploma QCF Foundation Degree Hong Kong Diploma of Secondary Education iMedia Diploma International Baccalaureate Diploma or Certificate Certificate in Mathematics for Engineers Music Theory Examinations Professional Qualifications at Level 3 Vocational (VCE) A-levels City & Guilds Land Based Services Level 3 Qualification: Extended Diploma Key Skills level 2

Tariff Challenge 2 Answers Qualification UCAS HESA Access course N British Horse Society Horse knowledge & care Y BTEC: Subsidiary Diploma QCF Foundation Degree Hong Kong Diploma of Secondary Education iMedia Diploma International Baccalaureate Diploma or Certificate Certificate in Mathematics for Engineers Music Theory Examinations Professional Qualifications at Level 3 Vocational (VCE) A-levels City & Guilds Land Based Services Level 3 Qualification: Extended Diploma Key Skills level 2

Four Main League Tables Times Complete Guide Guardian Sunday Times There are other league tables as well: Thomson Reuters World University Rankings published by Times Higher Educational Shanghai Jiao Tong University Academic Rankings of World Universities

Measures in League Tables Times Complete Guardian Sunday Times Student satisfaction √ Research quality Student – staff ratio Spend per student on academic services and/or facilities Tariff scores on entry Good honours (1st & 2:1) Completion rates Value added Dropout compared with benchmark Graduate prospects Peer assessments They all use slightly different versions of these measures and the weighting they use and how they are compiled into an overall measure varies. The newspapers vary on how much information they give on this – and this is where some scope for newspapers to add their own interpretative overlay over the HESA data Going to unpack these in detail shortly In brief: All include student satisfaction – from the NSS Research quality (Times & Complete) Student staff ration (not Sunday Times) Other spending measures (Complete & Guardian) All look at tariff on entry Good honours & completion rates (Times & Complete) Guardian & Sunday Times have more sophisticated measures – Guardian value added & Sunday Times dropout compared with expected dropout All look at graduate prospects – from DLHE Sunday Times also uses peer assessment NOTE Times: student satisfaction & research quality 1.5, others 1 Complete Guide: Z-transformation of each measure, then 1.5 weight for Student Satisfaction & Research Assessment, 1.0 for rest Built up from subject level tables: teaching 10%, assessment & feedback 10%, overall satisfaction 5%, value added 15%, student-staff ratio 15%, expenditure 15%, entry score 15%, career prospects 15% Sunday Times: teaching excellence 250 points, student satisfaction +50 to -55 points, peer assessment 100 points, research quality 200 points, tariff 250 points, unemployment 200 points, good degrees 100 points, dropout rate +57 to -74 points

Tariff in League Tables Average tariff First degree, new entrants Level 3 Tariffable qualifications Includes partner college students Excludes those on year 0 Tariff on entry is an important measure in all the league tables. Average UCAS tariff (as counted by HESA) Of first degree new entrants With A-levels or equivalent Includes partner colleges but excludes year 0 Thought to be seen by applicants and their parents as a measure of quality of student population The data comes from the HESA student return – some of you will be aware that we have been making major efforts to improve the quality of the information we provide to HESA about qualifications on entry – and this is one of the reasons.

Tariff on entry 2012 Guides Times Complete Guardian Sunday Times First year, first degree All modes Full-time & sandwich Aged under 21 No age limit Excluding those on foundation years Excluding those with 0 tariff points Based on those with A-levels or equivalent, SQA Higher or Advanced Higher, ONC and OND including BTEC, Baccalaureate ----- Ceiling set at 500 points & %-age of available points calculated – scores then halved to give score out of 250 Data from 2009/10 HESA student return i.e. 2009 entrants 2009 entrants Either Full-time and sandwich only Or Those under 21 only. Excludes those with no tariff points at all And Based on A-levels and close equivalent Sunday Times do a transformation that disregards very high scores.

Tariff on entry 2012 Guide (2009/10 data) Greenwich score Approx equivalent to Greenwich rank Times 215 CDD-CCD 109 of 114 Complete 111 of 116 Guardian 209 115 of 119 Sunday Times 104 (=208) 118 of 122 Based on 2009 entrants we were somewhere like CDD-CCD Target of 50th ranked university is BBB (assuming they don’t improve as well) Two things we can do: Up the tariff we require before we admit students Improve the way we record students’ qualifications on entry so that we really get every tariffable qualification on record Matters more to us than, say, Russell group universities, because most of their entrants would have A-levels only so not so complicated CDD=200 CCD=220 BBB=300 ABB=320

Times and Complete University Guide: Tariff on Entry Changes from 2013 Up to 2012 In 2013 First year, first degree, all modes Aged under 21 Times: aged under 21 Complete: dropped age cut-off Excluding those with 0 tariff points Excluding those on foundation years Based on those with A-levels or equivalent, SQA Higher or Advanced Higher, ONC and OND including BTEC, Baccalaureate Based on wider group than previously: A-levels, Scottish & Welsh Baccalaureate, IB, various qualifications at level 3 and, crucially, ‘Level 3 qualifications of which some or all are subject to UCAS tariff’ Data from 2009/10 HESA student return i.e. 2009 entrants Data from 2010/11 HESA student return, i.e. 2010 entrants Because of problems with the coding of qualifications on entry at some universities (not here), the Times and Complete Guide are changing the way they calculate tariff on entry. Crucially they are proposing to include a group of students who have a mixture of tariffable and non-tariffable qualifications. This is unhelpful to us: if a student has one A-level and then other qualifications that got them the entry but are not tariffable, they’ll look as if they have a very low tariff.

Expected values for 2013 Guides compared with 2012 Times Complete 2012 Greenwich score 215 A Level Grade equivalent CDD-CCD Expected 2013 score including mixed qualifications 254 245 CCC-BCC CCC+ Expected 2013 score excluding mixed qualifications 257 251 Almost BCC We have asked for these students to be excluded from our data –which makes a difference of 6 tariff points in the Complete Guide and 3 tariff points in the Times. Either way, the 2013 Guides (based on 2010 new entrants) should show a substantial improvement. Now in the CCC to BCC range. Remember target was BBB. CDD=200 CCD=220 CCC=240 BCC=260