From a pyramid to traffic lights Assessing the effectiveness of QAA Douglas Blackstock Director of Administration & Company Secretary

Governance at QAA QAA is a company and a charity Regulated by Companies House and two charity regulators Has a board of 15 – seven independents (one is a student) and nominees from the funding councils and institutions representative bodies from each of the four UK nations Is funded through contracts with funding councils and through institutional subscriptions

Key purposes of QAA Protection Communicate Improvement Understanding More detail in our strategic plan at

About us 140 staff (125 full time equivalent) Up to 1000 auditors and reviewers Over 200 audits and reviews in Budget for is 14.5million C. 200 corporate publications per annum plus review reports

Where I fit in Management of infrastructure services Communications, Finance, Human Resources, Information systems, Office Services Cross QAA work Business development, information management, student engagement and car parking

Assessing our effectiveness 2003 we considered EFQM but couldnt secure commitment restarted 2006 suggesting a simple set of Key Performance Indicators got complicated through 06 and 07 with a proposal for 120+ performance measures and the arrival of a pyramid

Mission & Purpose External profile Working worldwide Offering expertise Safeguarding standards Reducing Regulation Supporting and enhancing quality Management and Governance Executive Board Support People Finance Core Business Reviews and audits Supporting development and enhancement Working with others Knowledge transfer and relationships (outputs) QAA infrastructure and people (inputs) Outcomes Measurement Assessment

But Would this actually tell us if we are meeting our purposes?

three tier system An analysis of Performance & Success Would help us assess Impact and produce a traffic light summary

The traffic lights (based on Committee of University Chairmen guidance) Excellent: this is on track; performing well and is substantially effective. Good; this is broadly on track; performance is satisfactory with one or two areas which could be addressed. Mixed: some significant concerns which could be damaging and result in ineffective performance if not addressed. Problematic: serious concerns threaten this area; this has the potential to become substantially ineffective.

We set about gathering data (in the style of Donald Rumsfeld) What we knew already What we knew someone else knew What we thought someone should know but were not sure they did know What we wanted to know and knew we could find out Does what we know or dont know help us know whether we are achieving our aims?

QAAs performance Programmes of work (quarterly monitoring of operating plan) Key contractual commitments met e.g. reviews and audits completed, reports published Progress against non-contractual activities e.g. papers published, other publications, events held, presentations given, attendance at external events Exceptions and achievements Any shift in risk External evaluation (some annual, some periodic) External auditors Internal auditors External standards – for Information security, human resources management, website accessibility and our Welsh Language Scheme ENQA, Higher Education Regulatory Review Group, quality groups in Scotland and Wales

QAAs performance (2) Staff satisfaction (quarterly and annual) Overall indicators from the annual staff survey Quarterly staff turnover, absence, recruitment analysis Financial performance (quarterly and annual) Monitoring and achievement against budget; budget variances Balance sheet Forecasts from quarter 2 onwards

QAAs success Effectiveness of processes Institutional and reviewer / auditor evaluations of reviews and audits; reviewer / auditor evaluations of their training; participants evaluations of QAA events Renewal of contracts; repeat business New business (from current clients and new clients) Success of engagement strategies – students, employers, bodies representing the professions or with regulatory powers

QAAs success (2) Adding value Demand for our products – hard copy reports, web traffic, external attendance at our events Demand for our services – advice, contribution, participation, expertise, briefing, presentations Hot topics – our take, our engagement Interactions at government level Interactions with non-HEI stakeholders External review of QAA, such as ENQA membership review Institutions taking our reports seriously and putting right any identified problems External partners, the funding councils, student bodies and institution representative bodies

QAAs success (3) The difficult to measure– relationships, mood, reception The mood of the regular meetings with the funding councils and representative bodies (rectors councils?) in each of the countries The mood of the twice-yearly meetings of the Sounding Board group ( made up of all QAA funders) The mood of, and intelligence from, QAA liaison officers interactions with individual institutions The mood of the QAA Annual Reception and the annual Subscribers meeting The amount, mood and tone of press coverage The annual survey of Higher Quality readers

QAAs impact To what extent the performance and success data allowed the Board to make an assessment of the achievement of each of our Key Purposes. The Board would then need to identify whether additional information is needed to help it assess QAAs impact.

What did we find? Key purpose Information Safeguard the student and wider public interest in the maintenance of standards of academic awards and the quality of higher education Communicate information on academic standards and quality to inform student choice and employers understanding, and to underpin public policy-making Enhance the assurance and management of standards and quality in higher education Promote wider understanding of the nature of standards and quality in higher education, including the maintenance of common reference points, drawing on UK, other European and international practice Programmes External evaluation Staff satisfaction Financial management Effectiveness of processes Adding value Reception Overall assessment

What did we learn? The process was very demanding of staff time and we could simplify the process The traffic lights help you focus on risk In some areas we needed to improve information management and sharing Some of the information available did not contribute to an assessment of the achievement of certain key purposes.

and finally Whilst measurement can inform… ultimately, any assessment of the achievement of key purposes is a JUDGEMENT based on an array of evidence, not a measurement.

The Quality Assurance Agency for Higher Education. Registered charity numbers and SC037786