Indicators&Criteria in External Quality Assessment www.skvc.lt Indicators&Criteria in External Quality Assessment Lithuanian experience 2018 Almantas Šerpatauskas, Acting Director Centre for Quality Assessment in Higher Education (SKVC)

Objectives to contribute to creation of favorable conditions for free movement of persons assessment of foreign qualifications provision of info and consultations oversight of HEI recognition practices to encourage quality of activities of HEI review and accreditation of HEI (institutional review) evaluation and accreditation of HE study programs (ex-ante and ex-post) assessment of (new) establishing HEI

Expertise of SKVC in QA of HE SKVC as QA agency has knowledge and experience in application and interpretation of the ESG, and can assist in capacity building of relevant authorities in developing countries since we: operate as external QA agency in Lithuania for already 23 years; successfully underwent two external international reviews, organised by ENQA (in 2012 and 2016); are listed as trustworthy QA agency on EQAR from 2012, did reviews abroad; work as experts of QA in various capacities (serving on Boards of ENQA and CEENQA; as experts in ENQA reviews of agencies to determine their compliance to ESG; as Bologna experts; as experts in Twinning projects, as members of international and local working groups; as invited speakers to various international events; as consultants to various QA agencies which are in the process of development). 9

SKVC as academic information and recognition centre: Our own quality SKVC as academic information and recognition centre: voluntary self-evaluation and external review by the international panel following SQUARE methodology (end of 2015 – beginning 2016). SKVC as external quality assurance agency: Mandatory self-evaluation and external review against Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) by ENQA: 1st in 2012, 2nd – in 2016.

International relations INQAAHE (International Network for Quality Assurance Agencies in Higher Education) – full member from 1997 ENIC/NARIC (European Network of National Information Centres / Network of National Academic Recognition Information Centres) – full member from 1999 CEENQA (The Network of Central and Eastern European Quality Assurance Agencies in Higher Education) – co-founder and full member from 2000 ENQA (associate member from 2000, full member since 14 Sept 2012) Listed on EQAR as trustworthy QA agency from 2012

SKVC’s commitment to ENQA ENQA GA in Vilnius (October 2013) Former SKVC Director – ENQA Vicepresident Two members of staff – on ENQA Board (in the past) Two members of staff – among ENQA reviewers for QA agency reviews Participation in working groups: on impact of quality assurance staff development stakeholder involvement in quality assurance excellence review of the external review model for quality agencies internal quality assurance and compliance of quality agencies with ESG requirements links between quality assurance and qualifications‘ recognition

External Evaluation of study prorammes 2011-2018 All study programmes must be evaluated at least once every 6 years; Evaluation and accreditation of new study programmes and study programmes in operation; I & II cycle study programmes, non-degree programmes – evaluation by SKVC; III cycle studies – evaluation by Research Council of Lithuania; 7/15/2019

Study programmes in operation All study programmes are subject to evaluation and accreditation Accreditation is based on study programme evaluation report.

Evaluation of study programmes in operation Study programmes of the same (or similar) study field are evaluated at once; SKVC informs HEIs when their programmes going to be evaluated at least a year before submittion of self-evaluation report; 7/15/2019

Evaluation process Follow-up Appeal Consideration of the final report and publication Writing an evaluation report Site visit Preparation for external evaluation Self-evaluation Planning of evaluation

Expert panels International or local Composition of panels: academics, employers and students

Number of study programmes evaluated by … Expert panels Number of study programmes evaluated by …

Study programme evaluation areas Programme aims and learning outcomes Curriculum design Teaching staff Facilities and learning resources Study process and student assessment Programme management 7/15/2019

Evaluation criteria Eg. Programme aims and learning outcomes: Each evaluation area must be evaluated according to set of criteria. Eg. Programme aims and learning outcomes: the programme aims and learning outcomes are well defined, clear and publicly accessible; the programme aims and learning outcomes are based on the academic and/or professional requirements, public needs and the needs of the labor market; the programme aims and learning outcomes are consistent with the type and level of studies and the level of qualifications offered; the name of the programme, its learning outcomes, content and the qualifications offered are compatible with each other.

Evaluation area, criteria & indicators the study programme is provided by the staff meeting legal requirements; the qualifications of the teaching staff are adequate to ensure learning outcomes; the number of the teaching staff is adequate to ensure learning outcomes; teaching staff turnover is able to ensure an adequate provision of the programme; the higher education institution creates conditions for the professional development of the teaching staff necessary for the provision of the programme; the teaching staff of the programme is involved in research (art) directly related to the study programme being reviewed. Must be analysed in SER: the list of the teaching staff complete with information on each member’s academic rank and scientific degree (if any); teaching experience; research interests; subjects taught; practical work experience in the area of the subjects taught; description of each member’s activities; information on the teaching staff’s involvement in applied research, projects and research (artistic) activities directly related to the study programme being evaluated; student/teacher ratio in the provision of the study programme; data on the teaching staff exchange (visiting/outgoing teaching staff ratio).

Evaluation area, criteria & indicators the study programme is provided by the staff meeting legal requirements; the qualifications of the teaching staff are adequate to ensure learning outcomes; the number of the teaching staff is adequate to ensure learning outcomes; teaching staff turnover is able to ensure an adequate provision of the programme; the higher education institution creates conditions for the professional development of the teaching staff necessary for the provision of the programme; the teaching staff of the programme is involved in research (art) directly related to the study programme being reviewed. It is recommended that the analysis of the teaching staff also includes: data to prove compliance of the staff composition with legal requirements; data on the teaching staff turnover; data on the participation of the teaching staff in scientific conferences, wokshops, exchange programmes, long-term visits, etc. methods of professional (educational, scientific, practical) development of the staff; professional development areas and statistics of participation; age profile of the academic staff; workload of the academic staff (in the provision of the programme under evaluation and other programmes; time allocated for research and/or other (professional) activities, etc.)

Study programme accreditation The study programme evaluation is positive or negative. The decision on study programme accreditation is based on external evaluation report. Study programme can be accredited for: 6 years 3 years The programme can be given non-accreditation! 7/15/2019

Evaluation framework to be introduced since 2019 Evaluation and accreditation of study fields instead of individual study programmes. Use of data gathered from national registers, Monitoring agency, Research Council. Importance of scientific level in the field. Simplified vs full procedure

Indicators Threshold and monitoring indicators: Scientific level in the field; Student competitive score in admission; Student drop-out in 3 years period; Percentage of teaching staff working in HEI on full time basis; Percentage of graduates employed in relevant positions 12 months after graduation; Percentage of unemployed graduates 12 months after graduation; Ratio of incoming and outgoing exchange students in 3 years period; Percentage of international students in the study field in 3 years period; Ratio of incoming and outgoing exchange teaching staff in 3 years period; Percentage of study programmes accredited for 6 years; Percentage of study programmes accredited for 3 years twice in a row; Student satisfaction about quality of studies; Number of students in the study field in 3 year period; Percentage of students finishing their studies on time.

Agency’s point of view Quantitative and qualitative indicators. Data must be interpreted in wider context!

Thank you! almantas@skvc.lt www.skvc.lt