Interactive visualisation of GCSE and A-level results Stephen Rhead 04/07/2019

Ofqual is a non-ministerial department The Office of Qualifications and Examinations Regulation (Ofqual) regulates qualifications, examinations and assessments in England Ofqual is a non-ministerial department We are responsible for making sure that: regulated qualifications reliably indicate the knowledge, skills and understanding students have demonstrated assessments and exams show what a student has achieved people have confidence in the qualifications that we regulate students and teachers have information on the full range of qualifications that we regulate So first of all who are we at Ofqual? We regulate qualifications, examinations and assessments in England including GCSE, AS and A level qualifications and some vocational and technical qualifications.

The regulated qualifications market We regulate 155 awarding organisations who offer around 13,000 qualifications each academic year. Roughly 12.5 million certificates are awarded each academic year. We receive data from all of the awarding organisations we regulate which includes Data on the AOs themselves Data on the qualifications they offer Data on the number of certificates awarded, grades and so on We use this data for regulatory and research purposes and publish research and statistical reports.

Stakeholders Our data and publications are used by a variety of stakeholders including other government departments Teachers and students The awarding organisations we regulate Employers School and subject associations The media

Ofqual Analytics https://analytics.ofqual.gov.uk In the interests of openness and transparency, we wanted to make our data more accessible to our stakeholders and also to provide them with data that we haven’t published before. We wanted to take a fresh, interactive approach to telling data stories and to let users interact with our data and analysis. So in 2017 Ofqual analytics was born. We launched our first three online apps in August 2017 on A level and GCSE results days. https://analytics.ofqual.gov.uk

Variability in results for schools and colleges in 2017: GCSE and A level This is our centre variability app. We refer to all establishments like schools and colleges who offer GCSEs and A levels as centres. Centre variability is the difference in the percentage of students within a centre achieving specific grades in a particular subject between two years. So for example, if in 2017 60% of all the kids in a particular centre taking English language got a grade 4 or above and in 2018 62% of kids in that same centre got a grade 4 or above in English language then the variability would be +2 percentage points. The charts show the distribution of this variability for all the centres included in the analyses for the last two years. You can select: • which subjects you want to see • which grade range you want to look at – here we include just the key grades so 4 and above which equates to C and above and 7 and above which equates to A and above and you can filter the ages of candidates so you can look at everyone taking that subject and achieving those grades or you can just look at students in year 11 so 16 year old students who are the cohort most typically taking GCSEs • You can also filter on centre size so the number of students taking the subject in the centre and whether you want to see all centres or just those who have similar numbers of students each year. Each of our apps has a separate questions and answers tab which explains the app fully and also provides some context to interpret the data. We put all this information in a separate tab because we didn’t want our users to see a wall of text when they load the app.

Grade distributions of reformed (9-1) GCSEs This is our 9 to 1 grade distribution app This app shows the distributions of the number of students achieving each grade for the three reformed GCSE subjects in summer 2017. So you can select a subject from the dropdown and the chart will tell you the distribution of grades awarded for that subject. You can hover over the bars and it will tell you how many students achieved each grade. So for example here you can see … We also included a tab for the most popular grade combinations. One of the key questions people wanted answering for the reformed GCSEs were how many people achieved 3 grade 9s. This pie chart made it very easy to see this.

Ofqual Analytics Development R Shiny applications Shiny Server Computer running R (Server.R) Web Page (UI.R) We built the interactive visualisations using R shiny. This allows you to build interactive web pages with R and no web development skills are required. Which was ideal for us as we had strong R skills and at the time our web development skills were limited. Each shiny app is comprised of two parts: The web page the user sees and a computer that powers the app. The server is responsible for the logic of the app – it’s sets out the instructions that tell the web page what to show when the user interacts with the page. There are multiple ways of publishing these apps online. We decided to build our own server because we did not want the data to leave our organisation and we wanted to retain control of infrastructure and performance. This was by far the biggest challenge before launch

Ofqual Analytics Development R Shiny applications Shiny Server Agile software development We used agile methodology throughout. The power of post-it notes and daily stand-ups helped us to quickly identify and resolve issues. We also invited the leadership team. This helped them feel part of the initiative and gave us sufficient time to react where there were changes to business requirements.

Ofqual Analytics Development R Shiny applications Shiny Server Agile software development Testing User testing Speed testing Load testing We built both a test and production server. We tested the apps fully on the test server before anything went live. This allowed us to go through different rounds of user testing. Collective brainpower noticed things that we didn’t and the product was visibly refined after we rolled it out for testing. We used benchmarking tools to optimise the code to ensure it was as efficient and fast as possible We also used these opportunities to test the scaling of the apps and load testing of the server. ~100 ms

Ofqual Analytics Development R Shiny applications Shiny Server Agile software development Testing User testing Speed testing Load testing Implement Google Analytics We also incorporated google analytics into the apps so we could track the number of users and page views, track how people were getting to our apps and what devices people were using.

Ofqual Analytics Development R Shiny applications Shiny Server Agile software development Testing User testing Speed testing Load testing Implement Google Analytics Published on A level and GCSE results day 2017 We did this all in the space of three months and launched Ofqual analytics at 9.30 am on A level and GCSE results day.

Feedback Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries Ofqual analytics has been a multi-team effort with colleagues from research, data, IT and communications. We’ve had a number of challenges to overcome primarily because we were developing interactive analytics for the first time. After the hard work that has gone into creating Ofqual analytics it has been very rewarding to have some very positive feedback Some highlights include the fact that Ofqual analytics has been accessed over 100,000 times. To put this into context our most popular publications are typically only accessed around 10,000 times.

Feedback has been very positive Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries Feedback has been very positive We’ve had positive reviews from our users where we have been praised for our openness and helping users to understand how their results fit into the bigger picture

Feedback has been very positive Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries Feedback has been very positive And we’ve also been praised for the speed at which the apps were available. All apps were ready and launched at 9.30 am on A-level and GCSE results day.

Feedback has been very positive Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries Feedback has been very positive Ranked highly amongst the UK government tweets around results day* Our tweets always rank highly amongst the government tweets on results day *Source: https://ukgovtweets.measuredvoice.com

Feedback has been very positive Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries Feedback has been very positive Ranked highly amongst the UK government tweets around results day Highlighted by the Office for Statistics Regulation in their innovation review and highlighted by the office for statistics regulation in their innovation review.

Published more apps since launch Apps refined further Moving forward Published more apps since launch 12 apps as of 9th May 2019 Apps refined further Incorporate Ofqual branding Optimised for mobile devices Since 2017 we have launched another 9 apps. As our web development skills have improved we have been able to incorporate the apps with Ofqual branding and optimise the apps for mobile devices. This has been very important as we have found that around 40% of all traffic comes from mobile devices.

Published more apps since launch Apps refined further Moving forward Published more apps since launch 12 apps as of 9th May 2019 Apps refined further Incorporate Ofqual branding Optimised for mobile devices Increased promotion of Ofqual analytics Approached by other departments for advice Traffic increased 266% in month after A-level results day in 2018 compared to same time frame in 2017. As we were more confident in what we were doing in summer 2018 we were able to increase the promotion of Ofqual analytics. This led to a 266% increase in traffic in the month after A-level results day in 2018 compared to the same time frame in 2017. We hoping to see a similar increase in summer 2019.

Published more apps since launch Apps refined further Moving forward Published more apps since launch 12 apps as of 9th May 2019 Apps refined further Incorporate Ofqual branding Optimised for mobile devices Increased promotion of Ofqual analytics Approached by other departments for advice Improving accessibility of apps Lastly we’re working hard to make the apps to meet accessibility standards. We hope to have most of the changes made for the summer 2019.