1. Some case studies from the field
Evaluating Success
Making use of data in HE
Some case studies from the field
Karina Berzins
Continuum, UEL

2. “technological solutionism”
Beware
“technological solutionism”
First they said they needed data about the children to find out what they’re learning. Then they said they needed data about the children to make sure they are learning. Then the children only learnt what could be turned into data. Then the children became data.
Michael Rosen
Feb
The use of learner data at all points in the student lifecycle has exploded over recent years, with no signs of slowing. From the collection of demographic and achievement data at primary and secondary school via the school census and assessments we create data points for learners in all kinds of ways. Then, the use of contextual data for university - school outreach projects and access, to learner analytics once in higher education, and survey based data with graduates, education in the UK establishes some of the most comprehensive longitudinal data sets as standard practice.
This presentation will cover much of the data that is used in the student lifecycle - from school to higher education graduation, providing a critical analysis of the processes behind the data collection, as well as its use, and usefulness.

3. Exercise One List data that we use
The use of learner data at all points in the student lifecycle has exploded over recent years, with no signs of slowing. From the collection of demographic and achievement data at primary and secondary school via the school census and assessments we create data points for learners in all kinds of ways. Then, the use of contextual data for university - school outreach projects and access, to learner analytics once in higher education, and survey based data with graduates, education in the UK establishes some of the most comprehensive longitudinal data sets as standard practice.
This presentation will cover much of the data that is used in the student lifecycle - from school to higher education graduation, providing a critical analysis of the processes behind the data collection, as well as its use, and usefulness.

4. Case Study 1 Pre-entry data, outreach data
Reaching East – UCL and UEL
Reaching London – Greenwich University
Also Two smaller bespoke research projects:
Ravensbourne Reaching Out
Goldsmiths Reaching Out
HEFCE established a National Networks for Collaborative Outreach (NNCO) scheme
explicitly aims to encourage more young people into Higher Education
Nationally coordinated approach to work with schools, colleges and HEIs
Scheme involves around 4,300 secondary schools and 200 universities and colleges
£22 million provided by govt. over two academic years
Each network has appointed a single point of contact (SPoC), who:
helps teachers and advisers find out about the outreach activity which universities and colleges run in their area
provides general advice about progression into HE.

5. Reaching East geography
Eastern arc of low participation – POLAR data

6. Reaching East
providing an in depth understanding of the delivery and impact of outreach work in the East of London area, through the mapping exercise (year one)
Gaps, needs and capacity analysis (year two)
Culminates in a seminar series
by providing an in depth understanding of the delivery and impact of outreach work in the East of London area, through the mapping exercise. As the Linking London network has a college focus, the mapping will highlight FECs and 6th Forms, but will also consider all types of educational institution. Then, we will build on this phase of the research to conduct a gaps and needs analysis so that outreach activities (ultimately) can be better tailored to the individual circumstances of colleges, schools and learners in this part of London. In this way the activities of the Linking London network will be supported, as well as the activities of other institutions who may be members of other networks, who nonetheless operate in this geographic region. This will assist with the HEFCEs aim of cross-collaboration between the regional networks, and will ultimately assist all learners in the East of London area.

7. What is Outreach?
Delivery by HEIs of activities and events with schools
Campus Visits
Summer schools (or other residential events)
Subject taster days
Application support (including folio work)
Mentoring
Student finance talks
Most activities based around “aspiration raising” and/or raising achievement
by providing an in depth understanding of the delivery and impact of outreach work in the East of London area, through the mapping exercise. As the Linking London network has a college focus, the mapping will highlight FECs and 6th Forms, but will also consider all types of educational institution. Then, we will build on this phase of the research to conduct a gaps and needs analysis so that outreach activities (ultimately) can be better tailored to the individual circumstances of colleges, schools and learners in this part of London. In this way the activities of the Linking London network will be supported, as well as the activities of other institutions who may be members of other networks, who nonetheless operate in this geographic region. This will assist with the HEFCEs aim of cross-collaboration between the regional networks, and will ultimately assist all learners in the East of London area.

8. Reaching London
One year project mapping outreach pan-London

9. Research Aims (Year 1)
To map the type and frequency of outreach activity in the East of London arc of low participation
To establish a database of this activity with useful contact details to be shared with our and other NNCOs
To examine the role of colleges in particular, and to look at FECs as a particular case study as they are (potentially) both providers of outreach, as well as receivers of outreach activity
To examine this activity for patterns which may contribute to rates of participation in a local (borough or ward level) area, including the use of datasets such as IMD data
To establish potential connections between rates of participation, outreach, and destination of young learners

10. Research Aims (Year 2)
To conduct a gap analysis to identify “cold spots” where outreach activities are not provided
To conduct a needs analysis to identify where outreach work could potentially be of benefit
To conduct a capacity analysis to help colleges, schools and other providers deliver more tailored, effective, or simply more outreach activity to benefit learners in East London
To develop a toolkit to facilitate the engagement of colleges and schools with appropriate outreach activity and help schools to develop a way to map their own engagement with outreach work
To deliver four seminar workshops to provide a space where stakeholders can come together to collaborate and act as a “matchmaking service” to further facilitate the provision of outreach activity. These seminar workshops will also be of use for teachers, head teachers and other stakeholder staff members’ CPD

11. Data Sources School survey Collecting data from HEIs
All London HEIs involved
From survey and Access Agreements (Nvivo) identified 35 out of London HEIs who work in London/Essex
Informal collaboration of WP and outreach teams made the mapping possible

12. Findings
Strong positive correlation between amount of outreach and participation in HE from a school at both London level and East London level
Schools want to deliver more and better outreach for their learners
From the survey we have found that nearly 70% of schools have at least one dedicated staff member dealing with the organisation of outreach activities.
It is clear from the survey that the majority of activity is taking place with year 12 and year 13 groups. From the 2266 total events reflected in the survey, 1299 of them were for year 12 and 13 students, representing 57% of all activity.
We asked about increases in outreach activity between the 2013/14 and 2014/5 academic years, and nearly 60% of schools surveyed said that there had been an increase in activity, with a further 32% saying the frequency of activity has remained approximately the same.

13. Findings
Across London and into the Eastern regions there was a total of (at least) 5,693 outreach events during the 2014/15 academic year.
Newham schools participated in the most outreach events (325) and Basildon the fewest (15)
The most popular events were Summer Schools, with 1,789 of these events in 2014/15
Further Education Colleges and Sixth form colleges participated in 1,148 outreach event – here the most popular type of events were HE fairs.

16. FEC outreach

18. Mapping Data HESA data Publically available data
Edubase, school census
Achievement
POLAR3
Outreach data from HEIs

20. What Works? Early intervention
Sustained interventions like summer schools
Whole year groups (to avoid self selection of learners who are already going to HE)
HEIs going into schools
Subject based taster sessions

23. Mapping Tool

24. Exercise Two When and where do you use POLAR data?

25. Case Study Two
POLAR3 in a London context

26. Case Study Two POLAR3 in a London context
POLAR has been a very useful tool to establish Higher Education (HE) participation rates across the country. The POLAR classifications (both POLAR, POLAR2 and POLAR3) looks at how likely young people are to participate in higher education, broken down by geographic area in which they live (not the geography of where they study). The first version of the POLAR small area classification was based on young participation rates from the late 1990s. An updated version, POLAR2, was based on the participation rates of young people from 2000 to 2004.
POLAR3 is the latest iteration of this classification system. Based on learners’ home post codes, the data used for the development of the quintiles was HESA enrolment data aggregated over several academic years. It is based on the HE participation rates of those aged 18 between 2005 and 2009, and who entered any HE course in a UK HEI or FEC at age 18 or 19 between the academic years 2005/6 and 2010/11. The unit of geography that the quintiles were developed with are wards, and they are ordered from 1 to 5 with Quintile one being the lowest participation areas (i.e. the 20% of wards with the lowest progression rates for this cohort), and Quintile five being the 20% of wards with the highest participation rates.
Although the HEFCE evaluated the classification in 2014 in terms of the unit of geography used (census wards) and found that the units of geographic measurement was small enough to continue the use of this classification system, this evaluation did not touch on the age of the data used. Of course, patterns in participation were upset by the introduction of fees in 2012 with a dip in enrolments (although this corrected over subsequent years). In addition to this, in a city like London with constant regeneration developments (such as the Olympic regeneration also in 2012) it is increasingly likely that as these areas gentrify and local communities change that the classification system becomes less accurate over time.

27. POLAR3 London
POLAR has been a very useful tool to establish Higher Education (HE) participation rates across the country. The POLAR classifications (both POLAR, POLAR2 and POLAR3) looks at how likely young people are to participate in higher education, broken down by geographic area in which they live (not the geography of where they study). The first version of the POLAR small area classification was based on young participation rates from the late 1990s. An updated version, POLAR2, was based on the participation rates of young people from 2000 to 2004.
POLAR3 is the latest iteration of this classification system. Based on learners’ home post codes, the data used for the development of the quintiles was HESA enrolment data aggregated over several academic years. It is based on the HE participation rates of those aged 18 between 2005 and 2009, and who entered any HE course in a UK HEI or FEC at age 18 or 19 between the academic years 2005/6 and 2010/11. The unit of geography that the quintiles were developed with are wards, and they are ordered from 1 to 5 with Quintile one being the lowest participation areas (i.e. the 20% of wards with the lowest progression rates for this cohort), and Quintile five being the 20% of wards with the highest participation rates.
Although the HEFCE evaluated the classification in 2014 in terms of the unit of geography used (census wards) and found that the units of geographic measurement was small enough to continue the use of this classification system, this evaluation did not touch on the age of the data used. Of course, patterns in participation were upset by the introduction of fees in 2012 with a dip in enrolments (although this corrected over subsequent years). In addition to this, in a city like London with constant regeneration developments (such as the Olympic regeneration also in 2012) it is increasingly likely that as these areas gentrify and local communities change that the classification system becomes less accurate over time.

28. The POLAR3 data for London shows only 17 wards in the lowest category (Quintile 1) as can be seen in the following chart.

29. POLAR3 London
Given that there are a total of 649 wards in London, we thought it would be an interesting project to revisualise the POLAR3 data and divide London wards into quintiles. This way we can more easily see the differentials in participation at a London level which are masked by the quintile development for the whole country.
We used the POLAR3 data and re-developed the quintiles just for London. Through this exercise, many quintiles were shifted down, and the map now looks like this:
This clearly shows areas where there is less participation at ward level by London standards, and may be a much more useful tool for those engaged with delivering outreach activities at London’s many HEIs.

30. Indeed, a heat map of participation rates show the diversity of participation figures across the capital. 
For those institutions with a strong London focus, we suggest not only using POLAR3 alongside a basket of other indicators (given the age of the data), but to consider not only quintile 1 wards, but quintile 2 and 3 also.

31. Exercise FSM data Who uses FSM data? How and what for?
Do you use it with other indicators (NS-SEC, other proxies for socio-economics)?

32. Case Study Three FSM data
Commonly used as a proxy for socio-economic background
Better than NS-SEC (collected via UCAS)
Eligibility verified - robust
Free School Meals (FSM) data is commonly used in education research here in the UK, and is collected annually at the time of the school census for both primary and secondary schools. There are a number of relevant FSM data columns in the publically available census data – both numbers and percentages of those who take up the offer of free school meals, as well as the numbers and percentages of those learners who are eligible. In addition there is a third metric – those who are eligible (for performance tables) which is the most robust as this is an average of three years’ worth of data.
To be eligible for FSM the family must be in receipt of one of a number of state benefits, including jobseekers, disability, housing etc. As such, the FSM eligibility data points have been independently verified and as such are a very robust and useful to use as a proxy to determine the numbers of learners in low income households at a particular school.
Interestingly there is a differential between those learners who are eligible for FSMs and those who take advantage of the offer. This discrepancy varies considerably between Local Authority (LA) areas, and many LAs run programmes and drives to ensure those who are eligible take up the offer, as this also has an impact on the pupil premium funding that the school receives from central government.

33. Using London as an example, based on the latest school census (collected January 2017) we can see that there are differences in the numbers of those who are eligible for FSM, those who are actually taking up free school meals, and the metric of FSM used for the performance tables.
When the latest school census was released, it was noted by various commentators that the numbers of those who were in receipt of school meals were fewer than in the previous years. In fact, across all school types there were 14% of learners eligible for FSM provision, the lowest proportion since 2001 when records began.

34. Graph 3 taken from Department for Education, Statistics: school and characteristics of schools and pupils. Main Text: SFR28/2017, page 1, available from:

35. Graph 3 taken from Department for Education, Statistics: school and characteristics of schools and pupils. Main Text: SFR28/2017, page 1, available from:

36. FSM data Benefits cap (2013) then November 2016
£23,000 (£26,000 in London)
FSM down, but there are more children in poverty
IMD, IDACI
So while FSM data is still a robust proxy for low income households, it is increasingly clear that there are many households who continue to be economically impoverished but no longer are counted by this data point. To understand the recent history of FSM figures, we need to examine the previous coalition government’s policy which began to cap benefits that a household could claim in a year. The first benefits cap was introduced in 2013 and was set at £26,000 (the average income for a family in the UK at the time). In November 2016 there was a further benefits cap introduced by the conservative government which further reduced the income rate at which a household could claim state benefits. This was lowered to £23,000 in London, and £20,000 for the rest of the country. These caps have affected thousands of households with children, the majority of which are in London.
It is clear then that FSM data should no longer be used alone as a proxy for low income households, but instead should be used as one of a bundle of measures to establish the numbers of learners from low income households.
This does present a number of problems for the schools’ data analyst. The other data sets available to establish household income are not based on school level data and these data are not collected annually. The Indices of Deprivation – in particular the Index of Multiple Deprivation (IMD) and the income deprivation affecting children index (IDACI) are geographically based series of data which although robust, are difficult to analyse alongside school based data as the areas of geography do not necessarily correspond to where learners live in relation to the school postcode. Alongside this the latest IMD data is from 2015, so there will be a mismatch of years. While there will be more technical skill involved, and time taken to marry these data sets, it is clear that relying solely on the FSM data as an indicator of poverty at school level is problematic. Certainly any FSM data since 2013, and particularly after 2016 needs to be understood within the wider context of the change in state benefits and therefore no longer adequately reflects the numbers of young people from low income households.

37. Contact Details Karina Berzins Research Fellow, Continuum, UEL