1 Combining evaluation and accreditation cultures Caty Duykaerts (AEQES), Teresa Sanchez, Jean-Claude Arditti, Pierre Fleischmann, Bernard Remaud (CTI) A collaboration for the joint evaluation and accreditation of civil and bio engineering programmes in the French Community of Belgium

2 I. Institutional background 1. AEQES 2. CTI II. Origin ans scope of the collaboration III. Phases of the collaboration 1. Preparatory phase 2. Design of the collaboration 3. Execution phase IV. Development and main outomes of the preparatory phase 1. Assessing the feasability of the collaboration 2. Collaboration agreement between AEQES and CTI V. Conclusions and questions for the audience

3 AEQES brief history & legal framework From 1998 to 2002 First evaluations of University programmes conducted by the CRef AEQES 1 : Act of November 2002 All components of higher education Evaluation of study programmes Self-assessment reports, peer reviews and confidential reports AEQES 2 : Act of February 2008 A more independent agency, evaluation of programmes based on a 10-year planning, self-assessment reports, external reviews, publication of results and follow-up procedures Agency review every 5 years (ESG compliance) ENQA full member since september 2011 I. Institutional background II. Origin and scope of the collaboration III. Phases of the collaboration IV. Development and main outomes of the preparatory phase V. Conclusions and questions for the audience

4 AEQES methodology Key features formative QA evaluation process (no accreditation) programme-based quality assurance scope : 1st and 2nd cycle degrees (Bachelor and Master) all similar programmes are evaluated simultaneously, system-wide analysis - Self evaluation reports based on a reference list of indicators No formal effects in terms of institution fundings or authorisation no rankings, no scores I. Institutional background II. Origin and scope of the collaboration III. Phases of the collaboration IV. Development and main outomes of the preparatory phase V. Conclusions and questions for the audience

5 Presentation of CTI I. Institutional background II. Origin and scope of the collaboration III. Phases of the collaboration IV. Development and main outomes of the preparatory phase V. Conclusions and questions for the audience

6 Presentation of CTI I. Institutional background II. Origin and scope of the collaboration III. Phases of the collaboration IV. Development and main outomes of the preparatory phase V. Conclusions and questions for the audience Full member since 2005 Founding Member Mutual Recognition Agreements (OAQ, NVAO) Founding member 2000 (one of the 7 institutions that can deliver the EUR-ACE label) Since 2010 (European Registry of quality agencies)

7 Origin In the FCB: all HEIs must be evaluated / engineering programmes make no exception October 2009: official demand of the four universities concerned for a joint mission AEQES/CTI Twofold objective behind:  Evaluation of the programmes according to AEQES ten-year plan (compliance with AEQES decree requirements)  Accreditation of the programmes according CTI’s criteria (providing access to the « admission » by the French governement and to EUR-ACE label) I. Institutional background II. Origin and scope of the collaboration III. Phases of the collaboration IV. Development and main outomes of the preparatory phase V. Conclusions and questions for the audience

8 Scope Four universities concerned Catholic Universitiy of Louvain (UCL) Free University of Brussels (ULB) University of Liège (Ulg) (including Faculty of Agricultural Sciences of Gembloux – FUSAGx) University of Mons (UMons) Four (4) engineering programmes (BA + MA) in the field of agronomic sciences and biological engineering Fifteen (15 BA + MA) in the field of engineering sciences Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

9 Scope Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

10 Phases of the collaboration  Preparatory phase / December 2010 Two purposes To assess the feasibility of the collaboration To agree on a set of common principles Signature of a formal collaboration agreement/January 2011 Design of the collaboration / from January to December 2011 Execution phase / from January 2012 to September 2013 Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

11 Assessing the feasibility Three main issues 1) Compatibility of two national QA frameworks 2) Compatibility of evaluation / accreditation methods and procedures 3) Applicability of CTI’s accreditation framework to engineering programmes in the FCB Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

12 1) Compatibility of the two national frameworks The two national legal frameworks posed no major barriers to the collaboration However, one difficulty detected: difference between the two periodic calendars Assessing the feasibility Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

13 2) Compatibility of evaluation/ accreditation methods and procedures put in place by the two agencies Methodology: Use of the comparison tools and procedures developed by ECA in TEAM project (as a basis for reflection) Organisation of several work meetings to perform a comparative analysis of standards and procedures Inclusion of observers of the two agencies in a CTI and an AEQES mission (visit) Assessing the feasibility Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

14 2) Compatibility of evaluation/ accreditation methods Conclusions: Global compatibility with regard to principles and conception. However detection of a number of implementation differences regarding: - the dynamics and objectives of the HEI’s self-evaluation phase - the balance between quantitative and qualitative criteria - the role of the panel members during the site-visits - the attitude of the HEI’s with regard to the accreditation / evaluation process Assessing the feasibility Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

15 3) Applicability of CTI’s accredidation criteria to engineering programmes in the FCB. A number of criteria identified as « mandatory ». - A strong and broad basis in fundamental sciences - A guarantee of efficiency and short term adaptation to a professional activity - Business culture and economic, social, human, environmental ethcis awareness - Communication skills and international awareness Assessing the feasibility Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

16 The agreement The main outcome of the preparatory phase has been the establishment of a formal agreement signed by the presidents of the two agencies and covering: - The objectives and the scope of the collaboration - The general organisation of the project - The organisation of the site visits - The composition of the panel - The main outcomes of the mission (productions of several reports) Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

17 Conclusion up to now Benefits of the collaboration from now on: - Agreement on a certain set of basic principles - Setting of a basic work framework Illustrative and interesting exercice for both parties. Opportunity to: - confront procedures and methods - bring to the surface the underlying principles behind two apparently similar procedures This step by step collaboration has built a space of trust and confidence Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience

18 Conclusion up to now Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience Evaluation vs accreditation (1) - Accountability and quality enhancement in a single exercice ? - Accreditation interviews are more « punchy » - Audit report ? Final « product » for the evaluation Work document to the accreditation committee - Oral restitution at the end of the audit ? No problem for the evaluation Ambiguous for the accreditation

19 Conclusion up to now Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience Evaluation vs accreditation (2) Differences in the processes - The common process ends with the publication of the report with the recommendations - For the accreditation, the report is presented and discussed by the (CTI) committee, which may : Put different weights to the recommendations Add new ones (?) Decide accreditation (Y/N, duration) for each program Decide EUR-ACE labels and/or « titre d'ingénieur »

20 Conclusion up to now Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience Evaluation vs accreditation (3) Differences in the processes - The follow-up of recommendations may have not the same objectives

21 Conclusion up to now Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience First audit vs routine audit This joint audit was the first for the belgian engineering faculties. In France, HEI's have about 15 years experience of periodic accreditation Then HEI's are less prepared and suspicious The agency (AEQES) has committed very substantial ressources : prepatory meetings, number of days on site, size of the teams.

22 Conclusion up to now Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience Cultural differences The French-speaking Belgians are soft speaking and they dislike disputes The French like controversy and are more direct Some deans felt uncomfortable during interviews

23 Conclusion up to now Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience CTI is familiar to international accreditation and to distinguish the core of the standards from the elements linked to the local context But Decisions for HEI's a few kilometers apart on both sides of the border must be consistent.

24 Open questions Institutional background Origin and scope of the collaboration Phases of the collaboration Development and main outomes of the preparatory phase Conclusions and questions for the audience What are the fundamental differences between an evaluation and an accreditation approach? To what extent can an agency conciliate the sometimes confronted objectives of the accountability and quality enhancement in a single exercice? When accreditation becomes a routine : what changes in the procedures, should we shift more from program to institution assessment ?

25. Thank you