CONSISTENCY OF THE REVIEWS ENQA – IQA GROUP Sandra Marcos Ortega. ACSUCYL (Spain) 2nd ENQA Reviewers’ Seminar Manneheim 10-11 November 2016

INDEX MISSION OUR WORK SOME FINDINGS WHAT´S NEXT DISCUSSION WHAT´S ENQA IQA GROUP MISSION OUR WORK SOME FINDINGS WHAT´S NEXT DISCUSSION 8.12.2018

WHAT´S ENQA- IQA GROUP Mandate for the ENQA –IQA group. April 2015 Permanent working group in charge of the internal quality assurance of the agency external reviews. At the service of the Board Six members Aurelija Valeikienė, SKVC (Lithuania) Núria Comet, AQU Catalunya (Spain) Ole Espen Rakkestad, NOKUT (Norway) Julien Lecoq, HCERES (France) Oana Sarbu, ARACIS (Romania) Sandra Marcos, ACSUCYL (Spain) Chair 8.12.2018

ENQA-IQA GROUP MISSION To analyse and provide knowledge on the outcomes of the ENQA external review process (and particularly the self-evaluations and evaluation reports) 8.12.2018 replace txt View menu > Header and footer

OUR WORK - Definition of the Work Plan Since April 2015. 3 meetings - Definition of the Work Plan - Familiarising with EQAR interpretation - Understanding and interpretation of ESG 2015 - List of possible evidences ... 8.12.2018 replace txt View menu > Header and footer

OUR WORK ENQA understanding and interpretation of ESG 2015 Proposal of ENQA understanding and list of evidences. Analysis of SAR and ERR Two Agencies analysed. First evaluations against new ESG: ACPUA ECCE

OUR WORK ENQA understanding (IQA group’s initial document) Template for IQA-group’s analysis of the review reports – ECCE and ACPUA ENQA understanding (IQA group’s initial document) Agency interpretation in the SARs (match with ENQA´s understanding? Any additional?) Evidence provided in the SARs (indicate the number) How the criteria were interpreted by the panel (match with ENQA´s understanding? Any additional?) How the panel considered the evidences in the external review report? (Everything fully covered? Any additional evidence? Something not considered?) IQA group´s observation (proposed changes in the initial document) ESG Part 3 3.1 Activities, policy and processes for quality assurance Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a regular basis. They should have clear and explicit goals and objectives that are part of their publicly available mission statement. These should translate into the daily work of the agency. Agencies should ensure the involvement of stakeholders in their governance and work. Emphasis on the regularity - agencies should themselves conduct external QA activities on a regular basis. Organisations that only occasionally organise reviews of institutions or programmes do not comply with the standard. QA should be the main/one of the main activities of the agency. Match with ENQA´s interpretation? Yes/No If no, how it was interpreted: Additional comments: Match with ENQA interpretation? Yes/No: 3.1. Activities, policy and processes for quality assurance (Evidences) Mission statement, description of values or principles of work. How the mission is presented by the agency to the evaluated bodies. Strategic Plan The Annual Work Plan stating objectives, activities, units/persons responsible, timeline, budget. Monitoring procedures of annual work plan implementation. Corrective actions taken. Web page its mandate. ... Evidence provided in the SARs (indicate the number): Everything fully covered? Yes/No: If no, specify what wasn´t not considered? Any additional evidence? Yes, No. If yes, please specify:

OUR WORK ENQA understanding and interpretation of ESG 2015 Interviews with secretaries involved in external evaluations using new ESG - GUIDING QUESTIONS 1. Transition to the revised ESG: what are the ESG standards the agencies, in your opinion, find the most difficult to understand/to comply with? 2. Which were the most challenging ESG standards to evaluate, from the panel points of view? Why?

SOME FINDINGS FROM OUR ANALYSIS: Difficult to get consistency: differences in the interpretation between the members of the group when making the analysis Differences in understanding of some standards by Agencies (SAR) Similar interpretation by the experts but not in the final decisions Similar interpretation by the Agencies and experts as the first proposal of the ENQA IQA group 2.1 and 3.4 need more reflection on it

SOME FINDINGS FROM INTERVIEWS: Important the context of the Agency Well appreciate to prepare the review with the secretariat Interesting that someone understand the local language Share responsibilities between different members of the panel 2.1 and 3.4 the most difficult Mission is not always translated into daily work

Expected evidence in the agencies’ SAR FIRST FINDINGS: SOME EXAMPLES 3.4. THEMATIC ANALYSIS Agencies should regularly publish reports that describe and analyse the general findings of their external quality assurance activities. ENQA UNDERSTANDING The reports/thematic analyses published by the agencies should be based on findings from their EQA-activities, NOT on other kinds of information about the agency, the performance of the agency, or on data from the higher education sector in general. The reports/analyses should be clearly distinguishable from the agencies’ yearly activity reports. Expected evidence in the agencies’ SAR Procedure/process description for the production of thematic analyses Examples of analyses done: reports, analyses, magazine articles, journals, books etc. Description/documentation of how the agency’s thematic analyses are typically disseminated FIRST FINDINGS: SOME EXAMPLES

FIRST FINDINGS: SOME EXAMPLES ECCE Compliance: The ECCE President’s annual report to the general council is published on the ECCE website each year. Additionally, the President prepares reports on ECCE’s activities for the general newsletter of the ECU (‘Backspace’). This report is also available on ECCE’s website. Evidences: No evidences highlighted in the SAR Interpretation of the Standard by ECCE: Understood as annual activities report and accountability reports for Council. Panel Conclusion: Substantially compliance

FIRST FINDINGS: SOME EXAMPLES ACPUA Compliance: …producing reports that are useful for HEIs and for the society as a whole, …Evaluation reports. …summary reports on global results at the end of every programme accreditation process. …carrying out studies upon request of the Aragon Government about different key topics, e.g. employability of university graduates, connections of higher education with other education levels, learning outcomes, etc, … …annual report with information about all its activities and results. Report on the status of EQA in Spanish universities, …, summarizing global results of all Spanish QA agencies in different evaluation processes. …design of a new University Information System, defining a set of indicators to reflect the reality with accuracy and reliability and allowing comparative research about the different university systems. … ACPUA’s Board of Directors approved the “Basis for an Excellence Programme” aimed at fostering and recognizing excellence and best practices in the Aragon university system (for both institutions and programmes). Organizes monthly seminars, … as they offer an open forum to reflect, discuss, share ideas and perspectives around quality and improvement in a non-evaluating environment. - Organizes meetings and sessions on different QA topics of interest. It also attends forums and working sessions organized by other institutions …

FIRST FINDINGS: SOME EXAMPLES Interpretation of the Standard by the Agency: - To some extend the interpretation match with the Standard - Still including annual activities report , evaluation reports,… Evidences: A list of evidences of each report and seminar organized and describe in the SAR. Panel Conclusion: Substantially compliance

WHAT´S NEXT TO IMPROVE CONSISTENCY ON REVIEWS To polish the list of evidences and the interpretation Work on a draft of guiding questions and a list of evidences TO IMPROVE CONSISTENCY ON REVIEWS Meeting in January 2017 FUTURE… Work on the analysis of other SARs and ERR to improve consistency

Guiding questions for Agencies Guiding questions for experts WHAT´S NEXT Guiding questions for Agencies Guiding questions for experts List of evidences EXPECTED OUTCOMES

Thank you!

DISCUSSION DO YOU THINK IT WILL BE USEFUL THE DOCUMENTS WE ARE WORKING ON. IN YOUR OPINION, WHAT WILL BE MORE USEFUL : A LIST OF GENERAL EVIDENCES FOR ALL CRITERIA OR A LIST FOR EACH ONE SEPARATELY? DO YOU FIND ANY CRITERIA MORE DIFFICULT TO UNDERSTAND THAN OTHERS?