1. CONSISTENCY OF THE REVIEWS ENQA – IQA GROUP
Sandra Marcos Ortega. ACSUCYL (Spain)
2nd ENQA Reviewers’ Seminar
Manneheim November 2016

2. INDEX MISSION OUR WORK SOME FINDINGS WHAT´S NEXT DISCUSSION
WHAT´S ENQA IQA GROUP
MISSION
OUR WORK
SOME FINDINGS
WHAT´S NEXT
DISCUSSION

3. WHAT´S ENQA- IQA GROUP Mandate for the ENQA –IQA group. April 2015
Permanent working group in charge of the internal quality assurance of the agency external reviews.
At the service of the Board
Six members
Aurelija Valeikienė, SKVC (Lithuania)
Núria Comet, AQU Catalunya (Spain)
Ole Espen Rakkestad, NOKUT (Norway)
Julien Lecoq, HCERES (France)
Oana Sarbu, ARACIS (Romania)
Sandra Marcos, ACSUCYL (Spain) Chair

4. ENQA-IQA GROUP MISSION
To analyse and provide knowledge on the outcomes of the ENQA external review process (and particularly the self-evaluations and evaluation reports)
replace txt View menu > Header and footer

5. OUR WORK - Definition of the Work Plan
Since April 2015.
3 meetings
- Definition of the Work Plan
- Familiarising with EQAR interpretation
- Understanding and interpretation of ESG 2015
- List of possible evidences
...
replace txt View menu > Header and footer

6. OUR WORK ENQA understanding and interpretation of ESG 2015
Proposal of ENQA understanding and list of evidences.
Analysis of SAR and ERR
Two Agencies analysed. First evaluations against new ESG:
ACPUA
ECCE

7. OUR WORK ENQA understanding (IQA group’s initial document)
Template for IQA-group’s analysis of the review reports – ECCE and ACPUA
ENQA understanding (IQA group’s initial document)
Agency interpretation in the SARs (match with ENQA´s understanding? Any additional?)
Evidence provided in the SARs (indicate the number)
How the criteria were interpreted by the panel (match with ENQA´s understanding? Any additional?)
How the panel considered the evidences in the external review report? (Everything fully covered? Any additional evidence? Something not considered?)
IQA group´s observation (proposed changes in the initial document)
ESG Part 3
3.1 Activities, policy and processes for quality assurance
Agencies should undertake external quality assurance activities as defined in Part 2 of the ESG on a regular basis. They should have clear and explicit goals and objectives that are part of their publicly available mission statement. These should translate into the daily work of the agency. Agencies should ensure the involvement of stakeholders in their governance and work.
Emphasis on the regularity - agencies should themselves conduct external QA activities on a regular basis. Organisations that only occasionally organise reviews of institutions or programmes do not comply with the standard. QA should be the main/one of the main activities of the agency.
Match with ENQA´s interpretation? Yes/No
If no, how it was interpreted:
Additional comments:
Match with ENQA interpretation? Yes/No:
3.1. Activities, policy and processes for quality assurance (Evidences)
Mission statement, description of values or principles of work.
How the mission is presented by the agency to the evaluated bodies.
Strategic Plan
The Annual Work Plan stating objectives, activities, units/persons responsible, timeline, budget.
Monitoring procedures of annual work plan implementation. Corrective actions taken.
Web page
its mandate.
...
Evidence provided in the SARs (indicate the number):
Everything fully covered? Yes/No:
If no, specify what wasn´t not considered?
Any additional evidence? Yes, No. If yes, please specify:

8. OUR WORK ENQA understanding and interpretation of ESG 2015
Interviews with secretaries involved in external evaluations using new ESG
- GUIDING QUESTIONS
1. Transition to the revised ESG: what are the ESG standards the agencies, in your opinion, find the most difficult to understand/to comply with?
2. Which were the most challenging ESG standards to evaluate, from the panel points of view? Why?

9. SOME FINDINGS FROM OUR ANALYSIS:
Difficult to get consistency: differences in the interpretation between the members of the group when making the analysis
Differences in understanding of some standards by Agencies (SAR)
Similar interpretation by the experts but not in the final decisions
Similar interpretation by the Agencies and experts as the first proposal of the ENQA IQA group
2.1 and 3.4 need more reflection on it

10. SOME FINDINGS FROM INTERVIEWS: Important the context of the Agency
Well appreciate to prepare the review with the secretariat
Interesting that someone understand the local language
Share responsibilities between different members of the panel
2.1 and 3.4 the most difficult
Mission is not always translated into daily work

11. Expected evidence in the agencies’ SAR FIRST FINDINGS: SOME EXAMPLES
3.4. THEMATIC ANALYSIS
Agencies should regularly publish reports that describe and analyse the general findings of their external quality assurance activities.
ENQA UNDERSTANDING
The reports/thematic analyses published by the agencies should be based on findings from their EQA-activities, NOT on other kinds of information about the agency, the performance of the agency, or on data from the higher education sector in general. The reports/analyses should be clearly distinguishable from the agencies’ yearly activity reports.
Expected evidence in the agencies’ SAR
Procedure/process description for the production of thematic analyses
Examples of analyses done: reports, analyses, magazine articles, journals, books etc.
Description/documentation of how the agency’s thematic analyses are typically disseminated
FIRST FINDINGS: SOME EXAMPLES

12. FIRST FINDINGS: SOME EXAMPLES
ECCE Compliance:
The ECCE President’s annual report to the general council is published on the ECCE website each year. Additionally, the President prepares reports on ECCE’s activities for the general newsletter of the ECU (‘Backspace’).
This report is also available on ECCE’s website.
Evidences:
No evidences highlighted in the SAR
Interpretation of the Standard by ECCE:
Understood as annual activities report and accountability reports for Council.
Panel Conclusion:
Substantially compliance

13. FIRST FINDINGS: SOME EXAMPLES
ACPUA Compliance:
…producing reports that are useful for HEIs and for the society as a whole, …Evaluation reports.
…summary reports on global results at the end of every programme accreditation process.
…carrying out studies upon request of the Aragon Government about different key topics, e.g. employability of university graduates, connections of higher education with other education levels, learning outcomes, etc, …
…annual report with information about all its activities and results.
Report on the status of EQA in Spanish universities, …, summarizing global results of all Spanish QA agencies in different evaluation processes.
…design of a new University Information System, defining a set of indicators to reflect the reality with accuracy and reliability and allowing comparative research about the different university systems.
… ACPUA’s Board of Directors approved the “Basis for an Excellence Programme” aimed at fostering and recognizing excellence and best practices in the Aragon university system (for both institutions and programmes).
Organizes monthly seminars, … as they offer an open forum to reflect, discuss, share ideas and perspectives around quality and improvement in a non-evaluating environment.
- Organizes meetings and sessions on different QA topics of interest. It also attends forums and working sessions organized by other institutions …

14. FIRST FINDINGS: SOME EXAMPLES
Interpretation of the Standard by the Agency:
- To some extend the interpretation match with the Standard
- Still including annual activities report , evaluation reports,…
Evidences:
A list of evidences of each report and seminar organized and describe in the SAR.
Panel Conclusion:
Substantially compliance

15. WHAT´S NEXT TO IMPROVE CONSISTENCY ON REVIEWS
To polish the list of evidences and the interpretation
Work on a draft of guiding questions and a list of evidences
TO IMPROVE CONSISTENCY ON REVIEWS
Meeting in January 2017
FUTURE…
Work on the analysis of other SARs and ERR to improve consistency

16. Guiding questions for Agencies Guiding questions for experts
WHAT´S NEXT
Guiding questions for Agencies
Guiding questions for experts
List of evidences
EXPECTED OUTCOMES

17. Thank you!

18. DISCUSSION
DO YOU THINK IT WILL BE USEFUL THE DOCUMENTS WE ARE WORKING ON.
IN YOUR OPINION, WHAT WILL BE MORE USEFUL : A LIST OF GENERAL EVIDENCES FOR ALL CRITERIA OR A LIST FOR EACH ONE SEPARATELY?
DO YOU FIND ANY CRITERIA MORE DIFFICULT TO UNDERSTAND THAN OTHERS?