The economic returns to a degree: how great and how varied are they? (And what might they tell us?) Gianna Boero, Robin Naylor, and Jeremy Smith University of Warwick UEA 7th March 20161

Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation 2.Institutional Arrangements 3.Data and Methodology 4.Results 5.Conclusions and Further Work UEA 7th March 20162

Plan of Talk 1.Context 1:Evidence and Policy Importance of HE Human Capital, R&D, Economic Growth HE Participation and Labour Supply Socio-economic Mobility/Persistence Political Economy: fees and funding Returns to Education Years of Schooling Qualification Levels Grades Performance HE Policy relevance of estimated returns in UK Dearing Report and evidence from Blundell et al. (2000: PTO) Browne Report UEA 7th March 20163

Average Graduate Premium in UK Blundell et al. (2000): NCDS1958 birth cohort 1991 hourly wage data Estimates where HE is Highest Educational Qualification Rich set of observable characteristics Assumes that: Individuals with different HE do not differ on average in unobservables. Results Graduate Earnings Premium (Relative to control group with 2+ A-levels) 17%Men 37%Women UEA 7th March 20164

Variation around Average SubjectHigher for Science, Social Science (Harkness and Machin, inter alia) InstitutionHigher for ‘Elite’ HEIs (Chevalier et al., inter al.) Hence Differential Fees(Greenaway and Haynes) Prior Schooling(Naylor and Smith) CohortWalker and Zhu, 2008 Expansion no effect on average Increased premium in highest quartile (Ability composition effect?) … Degree Class Premia(Not available in NCDS) Expansion UEA 7th March 20165

Interpretation of Graduate Earnings Premium Human Capital Theory Signalling/Screening/Sorting Theories Statistical Discrimination Short-run only? Employer Learning/Statistical Discrimination UEA 7th March 20166

How might we interpret any variation in graduate returns by Grades/Performance? Again: HKT vs Signalling Labour market has various ‘pools’ or non-continuities Employers of some types of occupation recruit only from ‘Graduate pool’ Criteria for successful hiring? Cognitive skills Non-cognitive skills Measured by: Degree Class Transcript/HEAR Subject/HEI Interview Internship/experience Are these abilities: Revealed? (US: Arcidiacono) Signalled?(UK: opposite of US?) UEA 7th March 20167

Jo Johnson, Minister of State for Universities and Science, speaking about BIS’s Green Paper ‘Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice’, stated that, “We want to encourage a (grade point average) system which provides greater information to employers about where attainment really lies. It needs to sit alongside, rather than replace, the honours degree classification… But there is a very big band, the 2.1 band. It disguises very considerable differences in attainment. You can be at the top of the band and then be 50 percentage points below and still be getting a 2.1. And students who worked hard should be able to signal to employers that’s what they’ve achieved.” (Cited on BBC News website 06/11/2015.) UEA 7th March 20168

Degree Class Premia:Evidence from available data BCS70 LFS USR/HESA GCS Background Classification of Honours Degrees: First Upper Second(>=2.1 => ‘Good’) Lower Second(= ‘Lower’) Third Pass (Non-honours) Classification Rules: Based on: Overall average Papers in class Final exams/coursework Viva Anecdotally:‘Achieves’ vs ‘Is’! = HKT vs Signalling! UEA 7th March 20169

Estimated log wage premia (BCS70) UEA 7th March (1)(2)(3)(4)(5) Wages observed in year: 2000 Wages observed at age: 30 Good degree class premium relative to lower degree class (0.007) (0.008) (0.012) (0.014) (0.019) Lower degree class premium relative to 2+ A-levels (0.000) (0.001) (0.000) (0.001) (0.000) Family background NoYes Ability at age 10 No Yes Ability at age 5 No Yes Non-Cognitive ability at ages 5, 10 No Yes Other controls Yes No. of Obs 3046 R2R Notes: p-values in parentheses. Ability controls include: BAS (verbal), BAS (numerical). Background controls include: parental income, parental social class, mother’s interest in education, father’s interest in education, mother’s education, father’s education. Other controls include: region (aged 10), gender, marital status and number of children, ethnicity. Good degree premium over Lower Lower degree premium over A- levels

Estimated log wage premia (LFS): selected birth cohorts in UEA 7th March Wages observed at: Wages observed at age: Good degree class premium (relative to lower degree class) (0.001) Lower degree class (relative to 2+ A-levels) (0.000) Other controls Yes No. of Obs 2930 R2R

Estimated log-earnings premia (USR91, graduate cohort), birth cohort USR-FDS: First Destination Median Occupational earnings Note: p-values in parentheses. Ability controls include: pre-University qualifications. Background controls include: social class of parents, school-type. Other controls include: gender, marital status, University attended and type of degree course. UEA 7th March Earnings observed at:1992 Earnings observed at age:21-23 Good degree class premium relative to lower degree class (0.000) (0.000) Ability and background controlsNoYes Other controlsYes No. of Obs.22,459 R2R

Estimated log-wage premia (GCS1990, graduate): birth cohort Note: p-values in parentheses. Ability controls include pre-university qualifications, background controls include parental education, and other controls include age, gender, ethnicity, and marital status. UEA 7th March (1)(2)(3)(4) Wages observed at: Wages observed at age Good degree class premium relative to lower degree class (0.014) (0.014) (0.014) (0.014) Ability and background controlsNoYesNoYes Other controlsYes No. of Obs R2R

Estimated log-earnings premia (USR; HESA: selected cohorts) and by university type graduates aged UEA 7th March

UEA 7th March

Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation How might we interpret evidence of a premium by class of degree awarded? Why might any premium by degree class change across cohorts? UEA 7th March

Hypothesis 1 Pay Average mark ‘reveals’ ability/productivity to Employer and this is rewarded in the labour market. So when we have data on average mark, we estimate an average mark effect… Average Mark UEA 7th March Contrast with Arcidiacono on US context of high schools vs colleges

Hypothesis 1 Pay Average mark ‘reveals’ ability to Employer. If Econometrician also observes this mark, then estimated coefficient reflects ‘true’ effect of ‘ability’ on pay. Average Mark UEA 7th March

Hypothesis 1 Pay But if Econometrician observes only Degree Class, then there appears to be a Premium by Class: might wrongly interpret this as a discontinuity. Average Mark UEA 7th March

Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability, over and above the average mark, or does not see (or heed) the average mark. Average Mark UEA 7th March

Hypothesis 2 Employer regards Degree Class as a Signal Payof some dimension of ability. But if Econometrician observes only degree class, then we cannot rule out alternative interpretation of (steep) gradient in mark (Hypothesis 1). Average Mark UEA 7th March

A Regression Discontinuity framework offers the prospect of being able to distinguish between the two hypotheses – but requires us to observe both degree classification and underlying marks. Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability. A discontinuity would be Iindicative of signalling or statistical discrimination in the sense of the EL-SD aapproach Average Mark UEA 7th March

A Regression Discontinuity framework offers the prospect of being able to distinguish between the two hypotheses – but requires us to observe both degree classification and underlying marks. Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability. A discontinuity would be Iindicative of signalling or statistical discrimination in the sense of the EL-SD aapproach Average Mark UEA 7th March

Signal-Noise Ratio There might be a difference across subjects in the Signal- Noise Ratio associated with the extent to which the overall average mark indicates ability/productivity. There is some evidence that marks in quantitative subjects have a higher S-NR in terms of cognitive ability. Nature of quantitative material Nature of quantitative tests Labour market value of quantitative skills/abilities UEA 7th March

UEA 7th March Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation How might we interpret evidence of a premium by class of degree awarded? Why might any premium by degree class change across cohorts?

UEA 7th March a f(a) O 30 A 40 L 16 H Ability distribution across broad educational groups; 1995 characterisation.

UEA 7th March a f(a) O 30 A 40 L 16 H Ability distribution across broad educational groups; 1995 characterisation.

UEA 7th March a f(a) O 30 A 40 L 16 H Ability distribution across broad educational groups; 1995 characterisation.

Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation 2.Institutional Arrangements 3.Data and Methodology 4.Results 5.Conclusions and Further Work UEA 7th March

Regression discontinuity and degree class effects Also see:di Pietro (2012); Feng and Graetz (2015) We use individual student data on an anonymous university located somewhere near the centre of England… Anonymised DLHE returns for graduate cohorts of 2011/12, 2012/13, 2013/14. Matched by personal id to extensive individual student records. Data include: Age, gender, nationality, fees status, course, department, previous schooling, family background, degree class, marks per module per year. Labour market outcome, 5-digit SOC, SIC, salary, degree class, location… UEA 7th March

Degree classification Typically, based on performance in modules in years 2 and 3. Overall mark in each module comprises marks in end-of-year examinations and in coursework assessments. All marking is anonymous. Exam boards consider cases anonymously. Each overall module mark is classified as follows: The student’s final degree classification is based on the marks achieved in the individual modules. UEA 7th March

Degree classification Historically, the student’s final degree classification is based on two aspects of the module marks: Overall average mark across modules Distribution of module marks by class of mark However, for cohorts entering from 2008, the harmonisation of classification criteria across all departments means that the overall average mark is the main driver of the final classification: In addition, there are rules for ‘borderline’ cases within 2 percentage points of each critical minimum mark. These rules are based on the class distribution of marks, marks in the final year, and marks in core modules. There is also a system of mitigation. UEA 7th March

Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation 2.Institutional Arrangements 3.Data and Methodology 4.Results 5.Conclusions and Further Work UEA 7th March

3.Data and Methodology To date, we have data only for individuals who have responded to the DLHE in each cohort – we are waiting student records on all students in order to establish the extent to which DLHE respondents might differ in observable characteristics from non-respondents. The DLHE response rate is approximately 63%. Results to be presented today exploit data on graduates who are in full-time employment and have provided Research-Accessible personal salary information. The usable response rate to the salary question is approximately 41%. UEA 7th March

3.Data and Methodology UEA 7th March GraduatesPopulationRespondentsResponse rate Total responses by post/online % of responses by post/online 2011/ % % 2012/ % % 2013/ Total % % % %

UEA 7th March FT-Employed n= 1691 (52%) Population of All Leavers n= DLHE Respondents n= (63%) Telephone n= 9609 (59%) PG n= 3515(52%) UG n= 3265 (48%) Online/Post n= 6751 (41%) Further Study n= 931 (29%) Salary Data n= 1404 (83%) OLFU n= 6751 (19%)

UEA 7th March All 3265 UGs (responding to DLHE online/post) Degree Class Freq. Percent st 1, :1 1, : rd Total 3,

UEA 7th March UG Students. Degree Class breakdown by: All DLHE Respondents, FT-Emp, Salary Info Degree DLHEIn FT Salary ClassRespEmpInfo st 35%34% 35% 2:1 53%54% 53% 2:2 11%11% 11% 3rd 2%2% 2% n

UEA 7th March Degree Class breakdown by Faculty ClassArts Science Soc SciTotal 1 st %41% 26%35% 2: %42% 62%52% 2: %14%11%11% 3 rd % 3% 0%2% Total

UEA 7th March FT-Employed n= 1691 (52%) UG n= 3265 Salary Data n= 1404 (83%) UG HEU+3/4-yr+08-11start n= 2791 FT-Employed n= 1517 (54%) Salary Data n= 1292 (85%) Analysis will be based on a sample which excludes OS students, any students starting later than 11/12 and any students on UG degrees other than 3 or 4 year duration.

UEA 7th March Probability of receiving Treatment (= Good degree) Cut-off at 60 All UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 Based on n=2705

UEA 7th March Probability of receiving Treatment (= Good degree) Cut-off at 60 Science UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 n=1293

UEA 7th March Probability of receiving Treatment (= First) Cut-off at 70 All UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 n=2705

UEA 7th March Probability of receiving Treatment (= First) Cut-off at 70 Science UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 n=1293

UEA 7th March <60>= Compliers and Non-compliers All: 2.1 versus 2.2 n=1707: Compliers=92% All: 1st versus 2.1 n=2422: Compliers=89% Science: 1st versus 2.1 n=1091 : Compliers=90% Science: 2.1 versus 2.2 n=751 : Compliers=94% <70>= <60>= <70>=

UEA 7th March Scatter plot of log(annual pay) versus Overall Average Mark All n=1193(based on 1292, trimmed by payband)

UEA 7th March Density of log(annual pay))

UEA 7th March Scatter plot of log(annual pay) versus Overall Average Mark Science n=646

UEA 7th March Data-driven RD plots See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark All Cut-off 60 n=1193

UEA 7th March Data-driven RD plots: See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark IMSE-optimal quantile-spaced method using polynomial regression. (qspr) All Cut-off 60 n=1193 All (again) Cut-off 60

UEA 7th March Data-driven RD plots See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark Science Cut-off 60 n=646

UEA 7th March Data-driven RD plots See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark IMSE-optimal evenly spaced method using polynomial regression (espr) All Cut-off 70 n=1193

UEA 7th March Data-driven RD plots See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark IMSE-optimal quantile-spaced method using spacings estimators (qs) Science Cut-off 70 n=646

UEA 7th March Continuity (Gender) (RDPLOT) All P(2)

UEA 7th March Continuity (Start_year) (RDPLOT) ALL P(2)

UEA 7th March Continuity (FT-Employment) Cut-off 60 (RDPLOT) ALL P(4)

UEA 7th March Continuity (FS) Cut-off 60 (RDPLOT) ALL P(4)

UEA 7th March Continuity (FT-Employment) Cut-off 70 (RDPLOT) ALL P(4)

UEA 7th March Continuity (FS) Cut-off 70 (RDPLOT) ALL P(4)

3.Data and Methodology Issue of manipulation/precision of control over assignment variable Student manipulation: Those on track for ‘borderline’ after Year 2 work hard to achieve 2.1, those below borderline reduce effort. So we’d see a trough in the density distribution. Note: manipulation of overall average much harder than of a single module Marker manipulation: Eg practice of avoiding ‘9s’ Exam Board manipulation: If Board uses ‘unobservable’ knowledge insight/rules/discretion which correctly assign individuals to treatment/control groups. “This person is a 2.1” UEA 7th March

UEA 7th March Density of Overall Average Mark All n=3210

UEA 7th March Density of Overall Average Mark Science n=1449

Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation 2.Institutional Arrangements 3.Data and Methodology 4.Results 5.Conclusions and Further Work UEA 7th March

UEA 7th March Source | SS df MS Number of obs = F( 8, 722) = Model | Prob > F = Residual | R-squared = Adj R-squared = Total | Root MSE = lannualpay | Coef. Std. Err. t P>|t| [95% Conf. Interval] Upper Second | overall_ave60 | female | | startyr | 9 | | | | Science | Soc Sci | _cons | OLS of log(annual pay) on 2:1 Treatment ALL (2:1 vs 2:2) 48=< Overall Average Mark <70

UEA 7th March Source | SS df MS Number of obs = F( 6, 338) = 6.22 Model | Prob > F = Residual | R-squared = Adj R-squared = Total | Root MSE = lannualpay | Coef. Std. Err. t P>|t| [95% Conf. Interval] Upper Second | overall_ave60 | female | | startyr | 9 | | | | _cons | OLS of log(annual pay) on 2:1 Treatment Science (2:1 vs 2:2) 48=< Overall Average Mark <70

UEA 7th March Number of obs = 863 F( 3, 859) = Prob > F = Total (centered) SS = Centered R2 = Total (uncentered) SS = Uncentered R2 = Residual SS = Root MSE = | Robust lannualpay | Coef. Std. Err. z P>|z| [95% Conf. Interval] upper second | overall_ave60 | female | _cons | IV (2SLS): log(annual pay) on 2:1 Treatment ALL (2:1 vs 2:2) 48=< Overall Average Mark <70 But this includes cases of 3rds and 1sts in the borderline zones. Excluding these cases...

UEA 7th March Number of obs = 731 F( 6, 724) = Prob > F = Total (centered) SS = Centered R2 = Total (uncentered) SS = Uncentered R2 = Residual SS = Root MSE = | Robust lannualpay | Coef. Std. Err. z P>|z| [95% Conf. Interval] Upper Second | overall_ave60 | female | startyr_e | startyr_1 | startyr_3 | _cons | IV (2SLS): log(annual pay) on 2:1 Treatment ALL (2:1 vs 2:2) 48=< Overall Average Mark <70 (Same as previous estimation: but excluding cases of 3rds/1sts in borderlines) So the cases of 1sts in the top borderline was driving an apparent effect of 2:1s (given a discontinuity around 70).

UEA 7th March Number of obs = 345 F( 3, 341) = 9.42 Prob > F = Total (centered) SS = Centered R2 = Total (uncentered) SS = Uncentered R2 = Residual SS = Root MSE = | Robust lannualpay | Coef. Std. Err. z P>|z| [95% Conf. Interval] Upper Second | overall_ave | female | _cons | IV (2SLS): log(annual pay) on 2:1 Treatment Science (2:1 vs 2:2) 48=< Overall Average Mark <70 For First Stage regression see next slide Second Stage regression

UEA 7th March First-stage regression of Upper Second (Science): OLS estimation Number of obs = 345 F( 3, 341) = Prob > F = Total (centered) SS = Centered R2 = Total (uncentered) SS = 270 Uncentered R2 = Residual SS = Root MSE = | Robust good | Coef. Std. Err. t P>|t| [95% Conf. Interval] overall_ave | female | dum60 | _cons | Included instruments: overall_ave female dum Partial R-squared of excluded instruments: Test of excluded instruments: F( 1, 341) = Prob > F =

UEA 7th March Number of obs = 1047 F( 9, 1037) = Prob > F = Total (centered) SS = Centered R2 = Total (uncentered) SS = Uncentered R2 = Residual SS = Root MSE = | Robust lannualpay | Coef. Std. Err. z P>|z| [95% Conf. Interval] first | overall_ave70 | overall_ave70*d| overall_ave702 | female | sci | ssci | startyr_3 | startyr_4 | _cons | IV (2SLS): log(annual pay) on 1st Class Degree Treatment ALL (1st vs 2:1) 58=< Overall Average Mark

UEA 7th March Number of obs = 540 F( 6, 533) = Prob > F = Total (centered) SS = Centered R2 = Total (uncentered) SS = Uncentered R2 = Residual SS = Root MSE = | Robust lannualpay | Coef. Std. Err. z P>|z| [95% Conf. Interval] first | overall_ave70 | overall_ave70*d| overall_ave702 | female | startyr_e | _cons | IV (2SLS): log(annual pay) on 1st Treatment Science (1st vs 2:1) 58=< Overall Average Mark Consistent with RD for 60 < Overall Average Mark < 70

Plan of Talk 1.Context 1:Evidence and Policy Context 2:Theory and Interpretation 2.Institutional Arrangements 3.Data and Methodology 4.Results 5.Conclusions and Further Work UEA 7th March

5.Conclusions and Further Work (i)(Fuzzy) RD to estimate causal effect of degree class on earnings (ii)Evidence of Signalling in early careers Upper Second Class Premium For Science only:27% First Class Premium on average: 16% Strongest for Science: 22% (iii)Other findings Large negative female intercept Premia for Science and Social Science over Arts/Humanities (iv)Further Work/Data 4-digit SOC average occupational earnings Telephone Responses More cohorts 3 ½ year follow-up HMRC link => Non-parametric results UEA 7th March