Widening Participation whilst Narrowing Attainment Gaps between Student Groups: A Realistic Objective for Higher Education? Introduction: How this study came about: Moira’s work on unconscious bias and assessment (presentation in Learning and Teaching conference) An investigation into how the features of programme modules, the formats and timings of assessments and the characteristics of student cohorts interact to influence student attainment Richard McManus Moira Mitchell

The literature… National Studies: Based on HESA data (degree classification) for example: Broecke and Nicholls (2007) 65,000 degree classifications Fielding, Charlton, Kounali and Leckie (2008) 66,000 degree classifications Richardson (2008) 241,300 degree classifications Institutional Studies: Thinner on the ground and often qualitative: Dhanda (2010) (institutional attainment data by school and module, plus qualitative study with 34 students) Richardson (2010,2011) (surveys based on 1,146 and 929 students respectively) The essential role of curriculum design, learning, teaching and assessment practices is recognised (ECU/HEA, 2010; Singh, 2011; Stevenson, 2012) but there is limited evidence about LTA: “We do not know what aspects of teaching and assessment practices in higher education might be responsible for variations in the attainment gap.” Richardson (2015) Richard McManus Moira Mitchell

Our Study… Our study uses CCCU Business School data: Including students who enrolled between 2011 and 2013 incorporating 1,291 students; Considers specific module and assessment results: In total, 179 separate assessments, and over 23,000 separate assessment marks. Simultaneously considers difference across different demographic characteristics; Identifies both the presence of attainment differences and the causes of these in a two-stage approach. Richard McManus Moira Mitchell

Our Study…

Our Study… How could our study enhance what is known? Identifying areas of learning, teaching and assessment practices for adaptation Designing adaptations Evaluating adaptations Richard McManus Moira Mitchell

The sample and attainment gaps… Total BME Cohort 2013 450 32% Cohort 2012 395 37% Cohort 2011 446 34% Significance Ratio Mean Median Min Max Std Dev 0.1 0.05 0.01 Ethnicity -0.10 -0.09 -0.37 0.17 0.09 45% 37% 25% Proportional attainment difference by demographic in a specific assessment

Results… Ethnicity attainment gaps are impacted by: The average mark of the assessment; Attainment gaps fall when the average mark in an assessment is higher; attainment gaps are smaller by three percentage points when the average mark is 10 percentage points higher.

Results… Ethnicity attainment gaps are impacted by: The average mark of the assessment; Whether the assessment is practical based or in class; Practical assignments and in class tests both lead to an attainment gap 2.7 percentage points lower compared to exams.

Results… Ethnicity attainment gaps are impacted by: The average mark of the assessment; Whether the assessment is practical based or in class; If the assessment is numerical; Numerical tests lead to an attainment gap 7.7 percentage points higher.

Results… Ethnicity attainment gaps are impacted by: The average mark of the assessment; Whether the assessment is practical based or in class; If the assessment is numerical; Cohort effects.

Other results… Variables which were not significant (for any demographic): Ratio of BME, international, and disabled students to total population of students in the module; Whether the coursework was a traditional essay style set up or something less traditional like a case study or a specific type advertising proposal (for example) but still written; The term the assessment was performed/submitted (but could be explained by the ‘order’ variable). Variables which were not significant (specifically to BME student): The number of students in the module; Anonymous marking; Coursework; MCQs; Group work; Timing of assessments.

Implications… Richard McManus Moira Mitchell

The Sample: Students Total BME Male International Disabled Cohort 2013   Total BME Male International Disabled Cohort 2013 450 32% 52% 18% 7% Cohort 2012 395 37% 23% 11% Cohort 2011 446 34% 53% 31% 9% Richard McManus Moira Mitchell

The Sample: Assessment and Modules   No. of assessments No. of modules Ave. no. of students per module % of exams % of coursework % of practical assessments Year 1 57 7 288 75% 14% 11% Year 2 89 20 70 31% 49% 20% Year 3 33 16 63 33% 48% 18% Richard McManus Moira Mitchell

Results Stage 1: Prevalence of Attainment Gaps Each attainment gap is presented as the proportional difference of one demographic group against the benchmark, where negative coefficients represent a negative attainment gap. For example, BME students on average score 10% less than the benchmark average. Significance Ratio   Mean Median Min Max StdDev 10% 5% 1% Ethnicity -0.10 -0.09 -0.37 0.17 0.09 45% 37% 25% Gender -0.05 -0.54 0.62 0.12 34% 27% 15% International -0.06 -0.65 0.16 0.11 35% 26% 9% Disability -0.04 -0.02 -0.40 0.43 0.13 17% 8% 2% The benchmark is a white female student from the UK with no disability. Proportional attainment difference by demographic Attainment differences are additive, such that (for example) a male BME student performs on average 15% lower than benchmark.

Results Stage 1: Assessment Method Richard McManus Moira Mitchell

Results Stage 1: Assessment Method Richard McManus Moira Mitchell

Results Stage 2: Determinants of Attainment Gaps   Ethnicity Gender International Disability Students(100) 0.022*** (0.000) -0.028*** (0.005) 0.0278** (0.018) Male Ratio 0.121** (0.032) Average Mark 0.003*** 0.004*** (0.002) Weight 0.001** (0.048) (0.028) Anonymity -0.024* (0.080) 0.048*** (0.009) Coursework 0.059** (0.010) 0.081*** Practical 0.027* (0.078) 0.050* (0.053) 0.072*** (0.001) In Class 0.027** (0.019) 0.036* (0.084) MCQ 0.043** Numerical -0.077*** Group -0.044*** Cohort2012 -0.093*** 0.063*** Cohort2013 -0.028** (0.023) Order -0.047** (0.035) 0.045* (0.070) Year2 -0.071** (0.011) 0.052* (0.094) Year3 (0.006) 0.091** (0.015) Constant -0.261*** -0.322*** -0.231*** (0.003) -0.440*** 𝑹 𝟐 0.325 0.280 0.349 0.321 n 179 Module variables Male and disabled students perform relative better in larger modules, whereas international students do not. International students also perform relatively better in assessments with a higher male-to-female ratio

Results Stage 2: Determinants of Attainment Gaps   Ethnicity Gender International Disability Students(100) 0.022*** (0.000) -0.028*** (0.005) 0.0278** (0.018) Male Ratio 0.121** (0.032) Average Mark 0.003*** 0.004*** (0.002) Weight 0.001** (0.048) (0.028) Anonymity -0.024* (0.080) 0.048*** (0.009) Coursework 0.059** (0.010) 0.081*** Practical 0.027* (0.078) 0.050* (0.053) 0.072*** (0.001) In Class 0.027** (0.019) 0.036* (0.084) MCQ 0.043** Numerical -0.077*** Group -0.044*** Cohort2012 -0.093*** 0.063*** Cohort2013 -0.028** (0.023) Order -0.047** (0.035) 0.045* (0.070) Year2 -0.071** (0.011) 0.052* (0.094) Year3 (0.006) 0.091** (0.015) Constant -0.261*** -0.322*** -0.231*** (0.003) -0.440*** 𝑹 𝟐 0.325 0.280 0.349 0.321 n 179 Assessment variables All attainment gaps are smaller the higher the average mark in the assessment. Male and disabled students perform relatively better in modules with more weighting to the final grade. International students perform better in modules which are anonymously marked, whereas male students do not. International and disabled students perform better in coursework (essay) based assignments. BME, international and disabled students perform relatively better in practical assignments. BME and international students perform relatively better during ‘in class’ tests compared to more formal exams. Disabled student perform relatively better in assessments which contain MCQs. BME students perform relatively worse in assessments with numerical based questions. Male students perform relatively worse in group based work.

Results Stage 2: Determinants of Attainment Gaps   Ethnicity Gender International Disability Students(100) 0.022*** (0.000) -0.028*** (0.005) 0.0278** (0.018) Male Ratio 0.121** (0.032) Average Mark 0.003*** 0.004*** (0.002) Weight 0.001** (0.048) (0.028) Anonymity -0.024* (0.080) 0.048*** (0.009) Coursework 0.059** (0.010) 0.081*** Practical 0.027* (0.078) 0.050* (0.053) 0.072*** (0.001) In Class 0.027** (0.019) 0.036* (0.084) MCQ 0.043** Numerical -0.077*** Group -0.044*** Cohort2012 -0.093*** 0.063*** Cohort2013 -0.028** (0.023) Order -0.047** (0.035) 0.045* (0.070) Year2 -0.071** (0.011) 0.052* (0.094) Year3 (0.006) 0.091** (0.015) Constant -0.261*** -0.322*** -0.231*** (0.003) -0.440*** 𝑹 𝟐 0.325 0.280 0.349 0.321 n 179 Cohort variables: For all demographic groups there are specific cohort (year of intake) effects, suggesting some group dynamics.

Results Stage 2: Determinants of Attainment Gaps   Ethnicity Gender International Disability Students(100) 0.022*** (0.000) -0.028*** (0.005) 0.0278** (0.018) Male Ratio 0.121** (0.032) Average Mark 0.003*** 0.004*** (0.002) Weight 0.001** (0.048) (0.028) Anonymity -0.024* (0.080) 0.048*** (0.009) Coursework 0.059** (0.010) 0.081*** Practical 0.027* (0.078) 0.050* (0.053) 0.072*** (0.001) In Class 0.027** (0.019) 0.036* (0.084) MCQ 0.043** Numerical -0.077*** Group -0.044*** Cohort2012 -0.093*** 0.063*** Cohort2013 -0.028** (0.023) Order -0.047** (0.035) 0.045* (0.070) Year2 -0.071** (0.011) 0.052* (0.094) Year3 (0.006) 0.091** (0.015) Constant -0.261*** -0.322*** -0.231*** (0.003) -0.440*** 𝑹 𝟐 0.325 0.280 0.349 0.321 n 179 Timing effects: Male students perform relatively worse as the year progresses, whereas international students perform relatively better. However, international students perform worse as the academic years progress (perform worse in the 2nd year and worse again the 3rd). The opposite is true for disabled students whose relative performance improves through the academic year.

Our Approach: Advantages Looks at all students who start course, not just those who complete. Analyses final grade and therefore data is continuous rather than discrete (% vs grade boundaries). Considers different demographic groups simultaneously: Ethnicity Gender Disability Nationality Can look for potential causes in attainment differences between: Module characteristics (number of students, demographics of students) Assessment characteristics (type of assessment, type of question, group or individual, anonymously marked, etc.) Cohort effects (defined by the year of the student intake) Timing effects (year of study, time of year) This can then influence decisions on assessment and curriculum. Richard McManus Moira Mitchell

Our Approach: Disadvantages Although a rich dataset, not much scope to look at demographic characteristics interaction: In our methodology attainment gaps are linear and additive Potential for influencing outliers: Only consider modules with more than 20 students; Apply a ‘weighted least squares’ approach. Only consider one school in one university over a three year period of time. Restrained to only consider those things we can measure; what we did not have data for: Attendance; Engagement; Whether students are working; Attitudes; Etc. Richard McManus Moira Mitchell