HESA: HEDIIP Programme Specification of a new Subject Coding Classification Scheme Two years on: experiences of offering assessment improvement services to AOs Andrew Boyle Gillian Whitehouse Zak Horrocks AlphaPlus Consultancy Ltd

Two years ago … We talked about: How and why e-assessment providers should analyse their data. Not because Ofqual says so. But because it is the right thing to do.

Helping Awarding Organisations deliver better qualifications In the interim we have been doing it for clients such as:

Ofqual’s developing approach to validity Ofqual as part of their General Conditions of Recognition require awarding bodies to look at the full qualification cycle for all their qualifications. Risk should be identified within the cycle by categorisation of qualifications according to Ofqual risk characteristics: volume; type of funding; level; size; stakeholders; content; lifecycle stage. AOs should put in place validity strategies and plans that identify and help to ameliorate risk. During an Ofqual validity audit they review evidence to show that an awarding body’s: processes, systems, resources, qualifications and assessment are fit for purpose.

External vs. internal influence So, there is substantial external influence to ‘force’ AOs to analyse their data. But we argue there are substantial reasons why AOs should want to do it regardless of what Ofqual says.

Why should AOs analyse their data? (1) If something’s wrong (with one of your quals), you need to know. Your organisation needs to stand behind the standards of the awards it makes. Analytical techniques (e.g. Cronbach’s alpha reliability) are standard ways of showing assessment quality. If you don’t do this, it’s reasonable enough for stakeholders to ask why not. If you only use content experts and qualitative techniques, you are only getting half the picture. Quant findings are not necessarily definitive, but it’s just another strand of evidence that you have.

Why should AOs analyse their data? (2) Almost all assessment research is mixed methods. Having both quant and qual techniques at hand gives you more information, and information = power. You can think about problems in different ways. That ‘standards problem’ was really an ‘item writing problem’. Doing quant analysis does bring some certainty. Makes it easier to defend one’s organisation against challenge. Can shortcut interminable discussions. Quant analysis has a commercial purpose. It’s about understanding the quality of one’s products.

What have we found out? We have worked mainly with Awarding Organisations at the professional end of the market Success (or failure) in the assessment has profound consequences for candidates. Essential that the holder does possess the skills and knowledge that qualification certifies. AOs’ knowledge and practice vary, from beginning to look at facility values to sophisticated understanding of reliability and the will to improve standard settings.

Challenges to doing statistical analysis It’s hard to get hold of data. Statistical techniques are complex and hard to understand. There can be resistance among AO staff and item writers. Statistical analysis can be unnecessarily complex. More about demonstrating how clever the analyst is, rather than solving a business problem. Yes, it is, but such data is your IPR. You ought to access it. This is a change management problem for AOs. Users need to perceive that statistics help them to do their work. Don’t hire analysts like that. Statistical problems can be complex. But the complexity should relate to the complexity of your design.

A couple of concluding thoughts This work relates to two of the themes in my research and work. E-assessment hasn’t spread as quickly as we might have expected. There are many complex reasons for this, but in part it is because we are a bit of a cottage industry. If we really grasped the nettle with some of this seemingly complex stuff, we would promote e-assessment more. It would be a good thing if we could upskill our profession. English law speaks about being a ‘jurisdiction of choice’. The assessment/awarding industry can also be an internationally renowned centre of integrity and expertise. If AOs routinely analyse their data, then we will be part way along that road.

Thank you for listening. Any questions? 16th March 2016 Andrew Boyle Andrew.boyle@alphaplusconsultancy.co.uk

Complex analysis/complex design May well be effective, sufficient, elegant Simple test design (e.g. one form, comparability not an issue) Simple analysis (e.g. CTT stats, Angoff) Complex requirements/design (e.g. multiple forms each year, comparability between years, etc.) Complex/innovative analysis (e.g. IRT, pre-testing, ACJ, etc.) Risk is of ‘complexity for complexity’s sake’. having very convoluted procedures to mitigate limitations of simple analysis. Risk is either simply giving wrong results OR Although may involve substantial initial costs, may be more effective long term.

English law: jurisdiction of choice … People come here because they want to conduct their business in a country that offers a flexible and dependable legal system. … English commercial law provides predictability of outcome, legal certainty and fairness. … People also come here because they know they will find first class, highly specialised lawyers, arbitrators and mediators and they recognise that a decision from an English court carries a guarantee of judicial excellence and integrity.