(representing the project leader and the consortium) Impact of school inspections on teaching and learning. An ongoing project Jan-Eric Gustafsson (representing the project leader and the consortium)

Introduction Project funded by the EU Lifelong Learning Programme. Project leader: Melanie Ehren, University of Twente, the Netherlands Initiated by IPEA, international network chaired by Harvard University European consortium, including the Netherlands, the UK, Austria, Sweden, Ireland, the Czech Republic, (Norway, Switzerland and Harvard University) January 2011-december 2013

Trend towards evidence-based governance regimes Set expectations for the performance of the education system and communicate them clearly. Evaluation measures produce evidence as to whether or not expectations have been met by the practical operation of the system units. Evidence stimulates and orientates system development. Actors on all levels of the system – education politicians, administrators, schools leader, teachers, students etc. – use evaluation information to make more rational choices in developing their contribution to the education system and to improving their performance.

Both strategic behavior such as:.., and improvement such as… Research questions What are effects and negative consequences of school inspections in terms of changes in educational quality of schools and changes in student achievement? What aspects of school inspections (standards and thresholds, sanctions and rewards and, frequency of visits) contribute to these effects and negative consequences? Changes in schools: changes observed and/or initiated by the school leader, for example: …. Both strategic behavior such as:.., and improvement such as… Changes in student achievement: Improved student achievement, declines in student achievement (f.e. in subjects not part of inspection assessment)

Program theories Program theoretic analyses of each of the participating inspectorates: Aims to reconstruct assumptions on the causal mechanisms underlying intended effects of school inspections. Policy scientific approach: Identify the social and behavioral mechanisms that are expected to solve the problem in question. Compile statements indicating the goals of the proposed policy, and the mechanisms that make the program effective. Reformulate the statements into a conditional “if-then” structure. Search for implicit or explicit warrants, which say that B follows from A because of a general principle. Reformulate the warrants in terms of conditional “if-then” propositions and causal links

Program theoretic analysis Seven participating school inspectorates: Netherlands, the UK, Austria (Styria), Sweden, Ireland, the Czech Republic, and Norway. One analysis for each country and a unified conceptual model Document analysis of for example inspection frameworks, legislation and documents describing rationales for inspection methods. Interviews with inspection officials

Intended effects of school inspections All Inspectorates of Education aim for good education in individual schools and/or the education system as a whole. Differences in definitions. Broad objectives, such as contributing to evaluation, to development and to the support of the education system (Ireland, Styria). The right of all students to a good education in a safe environment (Sweden) Students shall develop knowledge and skills which enable them to participate in the society.(Norway) Improve inspected schools (The Netherlands, England, Sweden) Contribute to the improvement of the whole education system (England, Sweden)..

Inspections of schools Cyclical school inspections of every school (Ireland, Czech Republic, Styria, Sweden) Differentiated inspections of weak schools (The Netherlands, England) generally implemented to increase the efficiency of school inspections and target inspection resources to potentially weak schools. Sometimes based on results of self evaluations of schools to target inspection to potential areas for improvement in schools. Combinations of cyclical and differentiated inspections (The Netherlands, England, Sweden) Thematic school inspections (Sweden, Ireland, the Netherlands, Czech Republic) Self evaluations of schools (Ireland, Norway, the Netherlands, England), sometimes including surveys to stake holders (Ireland, the Netherlands, England, Sweden)

Standards, thresholds and feedback Three types of standards: legal aspects, indicating the extent to which schools comply with regulations (Sweden, Norway, the Netherlands) aspects of the context and process quality of education (Sweden, the Netherlands, England, Ireland, Styria) performance or results of schools (England, the Netherlands) Feedback Strong and weak points with respect to standards and thresholds. Recommendations/requirements on how to improve’ Feedback generally provided to the head teacher and/or the entire school staff (the school board in the Netherlands) Generally no feedback to individual teachers

Reporting feedback to stakeholders inspection reports Describe the functioning of individual schools according to the inspection standards Identify areas of improvement. Easily accessible to stakeholders (particularly parents). In some cases arrangements to actively inform stakeholders on the content of these reports Lists of failing schools (the Netherlands). In Norway the results of all the schools in a municipality are presented to the municipal committee for education, and then to the local council. More general reports on the state of education in the country (the Netherlands, England, Sweden)

Consequences Inspectorates generally cannot impose sanctions on schools (the Swedish Schools Inspectorate may withdraw the license and funding of independent schools) Inspectorates can intervene (’ notice to improve’, monitoring of low- performing schools, amendment of identified shortcomings with follow- ups, development of improvement plans, increased support to low- performing schools)

Causal mechanisms to promote intended effects Schools’ alignment of the views/beliefs and expectations of good education and good schools to the standards in the inspection framework. Schools are expected to act on these views and expectations and use the inspection feedback when conducting self evaluations and when taking improvement actions. Stakeholders should use the inspection assessment of the school’s functioning against the standards, to take actions that will motivate the school to adapt their expectations and to improve. Self evaluations by schools are expected to build capacity to improve that will lead to more effective teaching and learning conditions. Improvement actions will lead to more effective school and teaching conditions.

Conceptual model: Intended effects of school inspections

Valid assumptions? Standards on teaching and learning derived from school effectiveness research are generally expected to be the most effective. However, available meta-analyses suggest that teaching/instruction level conditions are more important than school level conditions. Capacity building is taken to be an important objective of the inspection process. However, a detailed description of what capacity- building entails is generally not given. Promoting good education through setting expectations relies on organizational change. But some organizational change may be cosmetic or ceremonial only. Stakeholders are expected to play an important role in how school inspections promote good education. However, parents are often either unable or unwilling to hold the school to account. Such assumptions need to be investigated empirically.

Research design, type I Regression discontinuity design, including Inspectorates of Education that use a threshold to assign different ‘inspection treatments’ to schools (NL, UK)

Research design, type II Interrupted time series with switching replications, including Inspectorates of Education that inspect schools on an ongoing basis (using no specific selection criteria) (Austria, Czech Republic, Ireland, Sweden)

Instruments and data Questionnaire to principals developed on the basis of the conceptual model: Background data concerning the principal and the school Capacity building and improvement processes Inspection characteristics and experiences in the year of the inspection visit Repeated three consecutive years Self auditing assessment blocks in the Netherlands and the UK to measure undesirable test preparation Additional country-specific data collection Rationale for survey to principals (instead of teachers): time limits and response, most school inspections address principals first. Survey: ask for changes and ask principals to provide reasons for why they made these changes. Data collection during three consecutive years (2011, 2012, 2013) in the same schools Effects (improvement of student achievement): national standardized tests, additional test scores (f.e. PIPS in the UK) Unintended consequences (declines in student achievement): national standardized tests, additional test scores and self auditing assessment blocks Reactions of schools (improvement and strategic behavior): survey to principals on changes in teaching (initiated/observed by principal) Aspects of school inspections: databases of Inspectorates of Education and survey

Work conferences and dissemination Symposium Dublin to discuss and evaluate the first results and dissemination of these results (January 2012) Symposium Gothenburg to discuss the final results and dissemination, to evaluate and finish the project and to prepare the final symposium with all stakeholders (December 2013)