1. Interactive visualisation of GCSE and A-level results Stephen Rhead 04/07/2019

2. Ofqual is a non-ministerial department
The Office of Qualifications and Examinations Regulation (Ofqual) regulates qualifications, examinations and assessments in England
Ofqual is a non-ministerial department
We are responsible for making sure that:
regulated qualifications reliably indicate the knowledge, skills and understanding students have demonstrated
assessments and exams show what a student has achieved
people have confidence in the qualifications that we regulate
students and teachers have information on the full range of qualifications that we regulate
So first of all who are we at Ofqual?
We regulate qualifications, examinations and assessments in England including GCSE, AS and A level qualifications and some vocational and technical qualifications.

3. The regulated qualifications market
We regulate 155 awarding organisations who offer around 13,000 qualifications each academic year. Roughly 12.5 million certificates are awarded each academic year.
We receive data from all of the awarding organisations we regulate which includes
Data on the AOs themselves
Data on the qualifications they offer
Data on the number of certificates awarded, grades and so on
We use this data for regulatory and research purposes and publish research and statistical reports.

4. Stakeholders
Our data and publications are used by a variety of stakeholders including
other government departments
Teachers and students
The awarding organisations we regulate
Employers
School and subject associations
The media

5. Ofqual Analytics https://analytics.ofqual.gov.uk
In the interests of openness and transparency, we wanted to make our data more accessible to our stakeholders and also to provide them with data that we haven’t published before.
We wanted to take a fresh, interactive approach to telling data stories and to let users interact with our data and analysis.
So in 2017 Ofqual analytics was born.
We launched our first three online apps in August 2017 on A level and GCSE results days.

6. Variability in results for schools and colleges in 2017: GCSE and A level
This is our centre variability app.
We refer to all establishments like schools and colleges who offer GCSEs and A levels as centres.
Centre variability is the difference in the percentage of students within a centre achieving specific grades in a particular subject between two years.
So for example, if in % of all the kids in a particular centre taking English language got a grade 4 or above and in % of kids in that same centre got a grade 4 or above in English language then the variability would be +2 percentage points.
The charts show the distribution of this variability for all the centres included in the analyses for the last two years.
You can select:
• which subjects you want to see
• which grade range you want to look at – here we include just the key grades so 4 and above which equates to C and above and 7 and above which equates to A and above and you can filter the ages of candidates so you can look at everyone taking that subject and achieving those grades or you can just look at students in year 11 so 16 year old students who are the cohort most typically taking GCSEs
• You can also filter on centre size so the number of students taking the subject in the centre and whether you want to see all centres or just those who have similar numbers of students each year.
Each of our apps has a separate questions and answers tab which explains the app fully and also provides some context to interpret the data.
We put all this information in a separate tab because we didn’t want our users to see a wall of text when they load the app.

7. Grade distributions of reformed (9-1) GCSEs
This is our 9 to 1 grade distribution app
This app shows the distributions of the number of students achieving each grade for the three reformed GCSE subjects in summer 2017.
So you can select a subject from the dropdown and the chart will tell you the distribution of grades awarded for that subject.
You can hover over the bars and it will tell you how many students achieved each grade.
So for example here you can see …
We also included a tab for the most popular grade combinations. One of the key questions people wanted answering for the reformed GCSEs were how many people achieved 3 grade 9s. This pie chart made it very easy to see this.

8. Ofqual Analytics Development
R Shiny applications
Shiny Server
Computer running R
(Server.R)
Web Page
(UI.R)
We built the interactive visualisations using R shiny. This allows you to build interactive web pages with R and no web development skills are required.
Which was ideal for us as we had strong R skills and at the time our web development skills were limited.
Each shiny app is comprised of two parts:
The web page the user sees and a computer that powers the app.
The server is responsible for the logic of the app – it’s sets out the instructions that tell the web page what to show when the user interacts with the page.
There are multiple ways of publishing these apps online.
We decided to build our own server because we did not want the data to leave our organisation and we wanted to retain control of infrastructure and performance.
This was by far the biggest challenge before launch

9. Ofqual Analytics Development
R Shiny applications
Shiny Server
Agile software development
We used agile methodology throughout.
The power of post-it notes and daily stand-ups helped us to quickly identify and resolve issues.
We also invited the leadership team. This helped them feel part of the initiative and gave us sufficient time to react where there were changes to business requirements.

10. Ofqual Analytics Development
R Shiny applications
Shiny Server
Agile software development
Testing
User testing
Speed testing
Load testing
We built both a test and production server.
We tested the apps fully on the test server before anything went live.
This allowed us to go through different rounds of user testing. Collective brainpower noticed things that we didn’t and the product was visibly refined after we rolled it out for testing.
We used benchmarking tools to optimise the code to ensure it was as efficient and fast as possible
We also used these opportunities to test the scaling of the apps and load testing of the server.
~100 ms

11. Ofqual Analytics Development
R Shiny applications
Shiny Server
Agile software development
Testing
User testing
Speed testing
Load testing
Implement Google Analytics
We also incorporated google analytics into the apps so we could track the number of users and page views, track how people were getting to our apps and what devices people were using.

12. Ofqual Analytics Development
R Shiny applications
Shiny Server
Agile software development
Testing
User testing
Speed testing
Load testing
Implement Google Analytics
Published on A level and GCSE results day 2017
We did this all in the space of three months and launched Ofqual analytics at 9.30 am on A level and GCSE results day.

13. Feedback
Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries
Ofqual analytics has been a multi-team effort with colleagues from research, data, IT and communications.
We’ve had a number of challenges to overcome primarily because we were developing interactive analytics for the first time.
After the hard work that has gone into creating Ofqual analytics it has been very rewarding to have some very positive feedback
Some highlights include the fact that Ofqual analytics has been accessed over 100,000 times. To put this into context our most popular publications are typically only accessed around 10,000 times.

14. Feedback has been very positive
Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries
Feedback has been very positive
We’ve had positive reviews from our users where we have been praised for our openness and helping users to understand how their results fit into the bigger picture

15. Feedback has been very positive
Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries
Feedback has been very positive
And we’ve also been praised for the speed at which the apps were available. All apps were ready and launched at 9.30 am on A-level and GCSE results day.

16. Feedback has been very positive
Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries
Feedback has been very positive
Ranked highly amongst the UK government tweets around results day*
Our tweets always rank highly amongst the government tweets on results day
*Source:

17. Feedback has been very positive
Since launch Ofqual Analytics has been accessed over 120,000 times in over 100 countries
Feedback has been very positive
Ranked highly amongst the UK government tweets around results day
Highlighted by the Office for Statistics Regulation in their innovation review
and highlighted by the office for statistics regulation in their innovation review.

18. Published more apps since launch Apps refined further
Moving forward
Published more apps since launch
12 apps as of 9th May 2019
Apps refined further
Incorporate Ofqual branding
Optimised for mobile devices
Since 2017 we have launched another 9 apps.
As our web development skills have improved we have been able to incorporate the apps with Ofqual branding and optimise the apps for mobile devices.
This has been very important as we have found that around 40% of all traffic comes from mobile devices.

19. Published more apps since launch Apps refined further
Moving forward
Published more apps since launch
12 apps as of 9th May 2019
Apps refined further
Incorporate Ofqual branding
Optimised for mobile devices
Increased promotion of Ofqual analytics
Approached by other departments for advice
Traffic increased 266% in month after A-level results day in compared to same time frame in
As we were more confident in what we were doing in summer 2018 we were able to increase the promotion of Ofqual analytics.
This led to a 266% increase in traffic in the month after A-level results day in 2018 compared to the same time frame in We hoping to see a similar increase in summer 2019.

20. Published more apps since launch Apps refined further
Moving forward
Published more apps since launch
12 apps as of 9th May 2019
Apps refined further
Incorporate Ofqual branding
Optimised for mobile devices
Increased promotion of Ofqual analytics
Approached by other departments for advice
Improving accessibility of apps
Lastly we’re working hard to make the apps to meet accessibility standards.
We hope to have most of the changes made for the summer 2019.