WP4- Piloting and Evaluation Activity 4.3: High standards and quality assurance of educational process and teaching Renate Nantschev, MSc Meeting 22 – 24. January, 2019: UMIT; Hall in Tirol; Austria Project web site: www.ph-elim.net

Task: 4.3 Objective To share experience and describing processes in developing and validating quantitative and qualitative evaluation tools. To develop an approach for new evaluation methods for PH-ELIM teaching and learning activities.

Working plan and time table Three Workshops (as demanded by project description) Aims Results Athens: 22nd-23rd Nov. 2018 To analyse all current evaluating approach of each project partner in their universities. All project partner presented their specific evaluation concepts used in their universities. Hall in Tirol: 22nd – 25th Jan. 2019 To find out a practicable solution for an evaluation process that can be useful for Montenegro. To develop questionnaires for all types of courses and lectures.   Debrecen: 7th – 9th May 2019 To finalize the process for implementation

First results We agreed: first, to develop only an evaluation concept for the new master course at UDG we would like to prepare questionnaires to evaluate courses, the study environment and the instructors all documents should be prepared in English – they will be translate later in Montenegrin language

First results We have … different quality assurance requirements examples for evaluation processes examples for questionnaires examples how to manage the results of evaluation examples for evaluation tools What has the presentation shown. In the next slides i will give you a summery to this topics of some instituion they presented in Athens

Evaluation Concept University of Heidelberg Project web site: www.ph-elim.net

Heidelberg: Quality assurance requirement Evaluation Statutes of University of Applied Sciences Heilbronn (cooperation partner of medical informatics programs) §1 Domain §2 Objective of Evaluations §3 Responsibilities §4 Subjects and Deadlines §5 Standardized Data Collection §6 Additional Data Collection by the Faculty §7 Additional Data Collection by Teachers §8 Implemention of Uniform Data Recording §9 Use of Evaluation Results within the University §10 Use of Evaluation Results outside the Univ. §11 Archiving and Anonymization of Data §12 Commencement The quality assurance requirement are regulated in the Statues von Heilbronn

Evaluation Medical Informatics Programmes Heidelberg: Evaluation Process Evaluation Medical Informatics Programmes Evaluation statutes Each course is evaluated every two years Application system: EvaSys TANs provided by teachers for students for each course to access questionaire Evaluation in between lectures to give room for improvement Individual feedback to teacher Evaluation Periode Management of the results

Heidelberg: Questionnaires / Evaluation tool Questionnaires applies to: the course for general evaluation (working atmosphere… ) for evaluation of the University about the workload EvaSys evaluation software of higher education sector

Medical Faculty Heidelberg Heidelberg: Management of the results Medical Faculty Heidelberg 1. Quality Management Team 2. Report for each course 3. General report after each term summary: What was helpful, what to improve comparison to earlier courses or modules detailed results: what was helpful detailed results: what to improve 4. Teachers have to reply to the report

Evaluation Concept National and Kapodistrian Unviersity of Athens Project web site: www.ph-elim.net

Project web site: www.ph-elim.net Athens: Quality assurance requirements 1. Quality Assurance Unit (“MODIP”) Each institution is responsible for ensuring and continuously improving the quality of its educational and research work as well as for the efficient operation and performance of its services, in line with international practices, in particular those of the European Higher Education Area, and the principles and guidelines of HQA. For this purpose, the Quality Assurance Unit is responsible in each HEI. In each Higher Education Institution, an internal quality assurance system is developed and implemented, with the responsibility and initiatives of the relevant “MODIP”, which must cover the whole range of functions and activities of the institution. Each institution is under a modip . Mopid are the requirement by law? John – please can you explane it to you shortly Thee quality assurance unit is responsible for ensuring and improving the quality in higher education and required higher institutions to develop and implement an internal quality assurance system. Modip balance international practices and guidelines of the quality assurance and accreditationa gency Project web site: www.ph-elim.net

Project web site: www.ph-elim.net Athens: Evaluation process “MODIP” - University of Athens The evaluation criteria and indicators used and monitored are reported on the basis of the above institutional framework in the following four axes: the quality of the teaching process; the quality of the research process; the quality of the curricula; the quality of other services (administrative services, student care, ICT infrastructure, transparency in the management of financial resources, etc.). The criteria and indicators used are required to address the following areas 1-4 Project web site: www.ph-elim.net

Project web site: www.ph-elim.net Athens: Questionnaires e.g. Internal Evaluation of MSc Programmes in the University of Athens, Department of Nursing - Evaluation Method and Tools Evaluation Committee of the Department of nursing consisting of professors and the heads of MSc programmes are responsible for constructing the questionnaires. Two Questionnaires are distributed to students regarding the: Evaluation of each course Evaluation of the MSc programme Students are requested to answer anonymously the questionnaires. Project web site: www.ph-elim.net

Athens: Management of the results Actively involving the faculty member in the self-awareness and self-improvement process; using this information as an opportunity to interact with the students and think about alternative teaching methods or material alteration. Regular meetings among the faculty members are held to discuss about individual results and results about the MSc programme. Allowing sufficient time for dialogue and interaction (between the consultant and faculty members) using feedback information and examining and understanding the faculty member’s approach to teaching (e.g. philosophy and paedagogical strategies); and the setting of improvement goals for the faculty member. Faculty members review the institutional materials taking into consideration the results from the surveys, the institutional expectations and University and Ministry of Education guidelines. Analysing the results and giving this ffedback to the instructor can involve him in a self-awareness and self improvement process. The instructor take this opportunity to revise his teaching approach and material used. Adiscusiion amog faculty members follows in order to discuss the results

Evaluation Concept Universtiy of Mediterranian Project web site: www.ph-elim.net

Law of Higher Education in Montenegro- Self-evaluation and Quality Assessment An institution shall conduct the procedure of quality self-evaluation concerning study programmes, teaching and working conditions at the end of each academic year, in compliance with statute of the institution. In the procedure of self-evaluation student surveys should be carried out at all levels, at least once a year with regard to study programmes, teaching, conditions and work of academic staff. Students are obliged to participate in survey. Survey is anonymous. The results of student surveys should be published on the website of institution. Self-evaluation reports of an institution shall be passed by the governing body of the institution. An institution shall submit a self-evaluation report to the Ministry and Agency. Method and criteria of self-evaluation concerning study programmes, teaching equipment, qualifications of academic staff, teaching methods, enrolment of students, exam pass rate, percentage of graduate students and the content of survey are more closely laid down by act of the Agency.

Mediterranean: Evaluation Process University Mediterranean conducts the procedure of quality self-evaluation concerning study programmes, teaching and working conditions twice a year- at the end of winter and summer semester. Students surveys are carried out at all levels of study- undergraduate, post- graduate and PhD level. Students are obliged to participate in survey. Survey is anonymous. The results of student surveys are published on the website of institution.

Mediterranean: Evaluation Process Method and criteria of self-evaluation concerning study programmes, teaching equipment, qualifications of academic staff, teaching methods, enrolment of students, exam pass rate, percentage of graduate students and the content of survey are more closely laid down by Senate of the University. Surveys are carried out on-line. Vice deans and representatives of Students parliament are responsible for the process of carrying out surveys. The results of the survey are analysed by Faculty’s Council and then by Senate of the University. In case of unsatisfactory level of students satisfaction regarding some elements of teaching/learning process, measures for their improvement are defined by members of Faculty’s Council and by Senate of the University

Evaluation Concept Universtiy of Montenegro Project web site: www.ph-elim.net

University of Montenegro (UCG): Evaluation Process Quality control at the University of Montenegro, bodies and documents Centre for study and quality control Preparation of proposals of strategic plans and programs in the field of quality, as well as proposals for improvement of standards, procedures and methods of quality assurance, helps in preparation of documentation for self-evaluation and accreditation, performs all activities related to quality. Rulebook on the organization and operation of the system for quality assurance and improvement of UCG European Standards and Guidelines for Quality Assurance in Higher Education 11th Project Coordination meeting Athens 22-23 November 2018 Project web site: www.ph-elim.net .

Project web site: www.ph-elim.net University of Montenegro (UCG): Evaluation Process Students Questionnaires In April each academic year All students are required Runs and processes by the UMN Information System Center Project web site: www.ph-elim.net

Project web site: www.ph-elim.net University of Montenegro (UCG): Management of results Analizes of survey and actions If the survey for a subjects is rated over 3 (very good / excellent), they are considered to be of high quality teaching. If the survey are rated lower than 3 (bad /very bad), the teacher submits a report to the faculty administration with a proposal that will improve teaching proces. Project web site: www.ph-elim.net

Evaluation Concept Universtiy of Dona Gorica Project web site: www.ph-elim.net

Project web site: www.ph-elim.net University of Dona Gorica (UDG) Quality assurance requirements: Implementation of QA policy and procedures in Montenegro is being promoted by responsible bodies (Ministry of Education, Council for Higher Education, just-established Agency for Quality Assurance in Higher Education) and higher education institutions are encouraged to implement Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG) When it comes to evaluation and assessment of study programs, the process is quite different at different institutions. At UDC developed a framework for quality assurance system The quality assurance requirements are the same as in the other institution of higher education in Montenego Project web site: www.ph-elim.net

Quality assurance framework at UDG I think that is a general framwork for quality assurance: I am not sure if there are the requirement for the evaluation of courses, teacher and so on November 2018, Athens

University of Dona Gorica (UDG) What questionnaires you have? How is the evaluation process organized? How is the management of results organized? Open questions ?

Evaluation Concept UMIT Project web site: www.ph-elim.net

UMIT: Evaluation Process Contents of evaluation concept: Process of the course evaluation Discussion of the evaluation results within study commission Activities to further develop the quality of teaching Feedback to the students of the evaluation results Reporting from the study commission to the senate Content of the concept are described in five chapters Stu

UMIT: Evaluation Process The evaluation will be conducted in all courses in each semester Different questionnaires are used: Lectures; Lectures with exercises; Seminar; Evaluation will be done before the exam All Evaluations are electronic based – Moodle or Zensus

UMIT: Evaluation Process Information to the students are provided by the student management Communication to the instructors about results is provided by the study management Instructors can see the results after the end of evaluation time The results are communicated to the students at the end of each semester

UMIT: Questionnaires Questionnaires about the course: e.g. - The overall organization of the course is good (1-5) - My overall learning gain in this course is high (1-5) 2. Questionnaires about the teacher: e.g. - Teacher can explain complex contents - Teacher gives me opportunities to review my learning progress

Thank you for your attention 

Summary of the first results Report systems/ Management of the results general report after each term Results are available on the website (Debrecen) Informal meetings: Student representatives may have meetings with subject leaders, students give feedback on quality of education (when they are unhappy)(Debrecen) The results of student surveys should be published on the website of institution (Mediterranen) An institution shall submit a self-evaluation report to the Ministry and Agency. (Mediterranen) If the survey are rated lower than 3 (bad /very bad), the teacher submits a report to the faculty administration with a proposal that will improve teaching process (UCG)

Summary of the first results Questionnaires: to the course (for all courses the same) to general aspects of the course (working atmosphere, seminar room, .. to the personal learning process questions addressing the teacher performance, educational materials, exams…

Summary of the first results Evaluation Tools: Online based: EvaSys and Zensus (Heidelberg and Umit) Paper based Via educational offices, departments, student organizations Online (evasys) and paper format (Debrecen) Surveys are carried out on-line (Mediterranen) Runs and processes by the UMN Information System Canter (UCG)

Summary of the first results Responsibility: Quality Management Team Study advisory commission Evaluation Committee of the Department of nursing consisting of professors and the heads of MSc programmes are responsible for constructing the questionnaires. (Athens) The results of the survey are analysed by Faculty’s Council and then by Senate of the University. (Mediterranen)