How are the results of external evaluation used at national level? Ulvi Soomlais 09.03.2012 Tallinn

External evaluation of study results State supervision External evaluation of study results Performance indicators (web-based benchmarking) In my presentation I focus on the following components of the external evaluation and describe how these results are used at national level.

State supervision (1) How are the results published? Year-book Seminar How are the results used? Input into the laws / regulations Input into the teacher training / INSET Individual cases How are the results published? Year-book – every year we, the external evaluation department, publish a year-book which includes lots of information like the results of the state supervision, analyses of internal evaluation, overview about different surveys etc. In terms of the state supervision, we produce only one report that gives main findings and conclusions. This report is available also on the web. Seminar - at the end of each year we organise a seminar where we introduce the results of the state supervision. The audience in this seminar is rather broad: directors, deputy directors, inspectors, school owners, people who are in charge of the teacher training, politicians. How are the results used? Input into the laws / regulations – information received from the state supervision is used for elaboration of or change the laws and regulations. An example: a couple of years ago the theme for the inspection was: effectiveness of measures aimed at supporting academic performance of students and preventing drop-out. At that time, a school was obliged to implement a measure (s) to a child who had negative marks on the school report at the end of the first academic term. During the inspections it was checked whether the schools supported such pupils and what was the effectiveness of these measures. The effectiveness of the repeating of year was also examined. Information received from this inspection was used for the new basic school and upper secondary school act. Input into the teacher training – the impact of the state supervision on teachers´ INSET is two-way – on the one hand the universities offer training in the problematic fields, which have been risen by the state supervision. On the other hand the schools ask for a training in advance. We have heard from the universities that quite often the schools, when they know the theme of the state supervision, they order a training for their teachers. Proactivity - very often the schools , having heard the inspection theme for the next years, act proactively - check the situation in their school in the light of the inspection theme (e.g. check whether their documentations are in line with the regulations). Individual cases – if there is a need for information about one particular school then we also look through the inspection report.

State supervision (2) Challenges Formulation of relevant themes (priorities) Thematic inspection versus whole school (complex) inspection Rigidity versus flexibility The biggest challenge is to formulate a relevant priority (theme) what would serve the needs and wishes of our stakeholders the best. The most important stakeholders are the other departments in the ministry. Our main task is to provide the ministry with the relevant and objective information. We work out the priority in close collaboration with the other departments and every now and then we find that they need information that is impossible to gather during the thematic inspection. Our colleagues in the ministry very often need qualitative information (e.g. What happens in the classroom? What is the quality of teaching or leadership?). According to the regulations we can´t assess whether an action (e.g. Leadership, teaching) is good at schools or not. Why not? According to the law the aim of the state supervision is to check whether the schools´ activities meet the requirements of legal acts. There are no legal acts that say what is a good teaching or good leadership. According to the Administrative Procedure Act (haldusmenetluse seadus) all individual inspection reports are the administrative acts against which an appeal can be lodged. If we evaluated the level of teaching, it would be very difficult to prove in the court why teaching was evaluated satisfactory not good. Just for your information that in Estonia, inspectors don´t visit and observe the lessons during the inspection. This is due to the autonomy of the school- director hires teachers and he/she is also responsible for the quality of teaching. Director can invite an expert to visit teacher's lessons to get feedback on the quality of teaching. Thematic inspection versus complex inspection – before 2006 a whole school inspection was carried out, its report gave a general picture about the school´s performance. At present when we have a thematic inspection, some stakeholders (e.g. School owners) say that they don´t know anymore what happen in their schools, that there is a lack of information. A decision was made 5 years ago that instead of whole school inspection there is a thematic inspection and self-evaluation is compulsory for the schools. Currently, it seems that it would be a vaste of resources (time and human) to keep these two parallel activities (whole school inspection and schools´ self-evaluation) in practice at the same time. However, in extraordinary cases the whole school inspection can be carried out as well. Rigidity versus flexibility As I said before, the inspectors main tasks are to check whether the schools activities are in line with the legal acts and to analyze the problems in implementing the these acts. It seems that in that case an inspection as an action is clear, transparent and easy to carry out. However, we all know that a school life is very rich in different situations, circumstances and problems; there are more colours than black and white. If the inspectors find a violation of legal acts they have to make a prescription in the report. Time after time we have been told that inspectors are too rigid, they follow the regulations too strictly. The explanation for this is that the purpose of legal acts is to enforce the educational policy. The state supervision evaluates whether these regulations function or not, If not, what are the reasons : whether the schools ignore the regulations or is there a time to change a regulation or not to regulate a certain area at all. If the inspectors were very flexible then the ministry wouldn´t receive an adequate information.

External evaluation of study results (1) How are the results published? Statistics and analyses on the Website The results of each school are available on the Internet (web-based benchmarking) How are the results published? Statsitics and analyses of each subject of national test and exams are available on the web of NEQC. There is a general report of the state examinations, which gives general statistics (average, median, min and max results, standard deviation etc). The same data for boys and girls; the schools where the language of instruction is russian, estonian etc. Additionally to this there is a report also for each subject. Regarding the national test, a general statistics and a conclusion of the analyses for each subject are published. Additionally, the NEQC provides each school with a short statistical feedback about its performance. EHIS- public site includes also a web-based benchmarking of the schools. There are average results of the natinal tests, basic school exams and the state examinations for each school. This is a public site, everybody can have a look and compare different schools.

External evaluation of study results (2) How are the results used? Input into the state curricula and syllabi Meetings for schools management and subject teachers organised by the NEQC Regional meetings for school owners and directors A capacity building programme for the schools at risk in the future Input into the state curriculum - information received from the external evaluation of study results is used as an input when we revise our state curricula and syllabi. Meetings for the subject teachers – NEQC organises meetings for the schools´ management and also subject teachers to introduce the analyses of the national test and examinations. The information given in these meetings is not only about the subject matters but also about the quality of marking. (National test and the basic school exam papers are marked by the teachers). Regional meetings for the school owners and directors –We have organised meetings on the regional level where the directors and school owners of such schools whose results in the PISA survey and the national test and exams have been very week. Such meetings have been organised also for the individual schools where a school owner has always been asked to participate. The purpose of such meetings is to introduce and interpret more thoroughly the results of the PISA survey of this particular school. A capacity building programme for the schools at risk – we learned from the other countries that more attention should be paid on the schools at risk, whose performance in the PISA survey and in the national tests and examinations has been low. We have an intention to use the ESF and create a programme focused on these schools to support and raise their capacity.

External evaluation of study results (3) Challenges Improvement in national tests and examination papers Improvement in analyses Implementation of the electronic tests (EIS) Ranking lists Since 2010 we have the new state curriculum for the Basic schools and upper secondary schools and new syllaby. There is also a change in the final examination model at the end of the upper secondary school, which will be introduced in 2014. Our aim is to improve the national tests and basic school exams and state exams that they would be in line with the state curricula and that there would be better link between the state curricula and the external evaluation of study results. Currently, we mainly concentrate on the assessment of the learning outcomes. In the future our focus is on the general and subject specific competences. We want to get more information on the implementation of the state curricula. We would like to raise the quality of analyses - if there is a shift from the the learning outcomes to the general and subject-specific competences in the tests, then there will be more broad view in analyses as well. In the future, coming from the analyses, the NEQC should give concrete recommendations to the Ministry what changes are need in the state curricula and syllabi, recommedations for the teacher training and study materials. EIS - an electronic system has been created where the electronic tests can be carried out. The purpuse is to use the electronic test step by step in the external evaluation of study results in the future. The biggest challenge here is to have a high number of good quality questions in the question bank. There should be questions for the national tests but also for the use of teachers in everyday lessons. Ranking lists – the ministry doesn´t make these lists but the Media does. Until now the Media has compiled these ranking lists according to the results of the final exqaminations of the upper secondary school. This February , when the enrolment process for the 1st graders started in Tallinn, the ranking lists of the schools in Tallinn were published according to the national tests results of the years 3 and 6. It was the first time then the results of the national tests of years 3 and 6 were used. Such lists cause lots of tenses within and among the schools because they don´t give any background information or explanation.

Performance indicators How are the performance indicators used ? Preparation for inspection EHISE Education Information System (www.ehis.ee) provides everyone with an opportunity to review the performance indicators of educational institutions. The goal of performance indicators is to provide educational institutions with an opportunity to monitor their indicators as a trend e.g. over a period of three years and compare them with average indicators of educational institutions of the same type. As performance indicators of educational institutions are publicly available, they also provide information about educational institutions to students, parents, managers of educational institutions and other interest groups. Performance indicators published in the Estonian Education Information System are also used in the preparation and carrying out of inspections. If there is a request for information about a certain school, these indicators have been analysed too.

Thank you!