Ongoing education policy evaluations Björn Öckert IFAU How to improve school results?, Uppsala, May 24-25, 2016 Responsible for education research at IFAU Education research at IFAU Institutional conditions for policy evaluation in the Swedish context Ongoing policy evaluations

Educational research at IFAU Evaluate effects of educational reforms and policies Policy relevant research on education Evaluate policies and measures Understand the conditions for education policies Applied research Project initiation In-house research ideas (In)formal government assignments Instruction of the institute More focus on school performance 2012 Broad mission From pre-school to university and adult education Restricted to regular education Two areas of research Evaluation of current policies Basic research School market – competition School choice – preferences and information Teacher and student incentives – compensation and grading Empirical research using administrative data and survey data IFAU is a small institute with a broad mission Necessary to prioritize Accumulate evidence about education policies Try to be proactive and start up evaluations of central policies

Sweden’s education system Central government set national goals Municipalities and independent entities run schools Resource allocation and policies National Agency for Education (NAE) Guidelines and support Data collection and descriptive studies Schools Inspectorate Regular school inspections and quality audits To understand the conditions for policy evaluation it is important to know the institutional setting of educational system A goal-steered system with a high degree of local responsibility Education act, school ordinance and national curriculum Municipalities receive grants from central government, but not targeted towards schools Students in independent schools: 15% in compulsory school; 26 % in high school (2015/16). Over 800 independent compulsory schools. Local governance: Public and independent schools design policies to meet national goals. Resources and degree of compensation Hiring and firing of teachers Organization of schools Much variation in resource allocation, organization, teaching practices No information on treatment No control of allocation “Uncontrolled experiments” (treatment without controls) – many different local policies Many different types of policies in different schools Not structured in a way so that we can learn from them No central authority for organizing schools Necessary to contact municipalities and independent school to influence policy design and data collection Central authorities to support and control schools National agency for education Syllabuses for compulsory school Knowledge requirements The Swedish Schools Inspectorate assess if schools comply with the legislations and can sanction schools Quality audits deal with well-defined areas, for example a special matter or problem area within the school.

Recent national educational initiatives Policies to meet falling performance of schools Raising teacher compensation Improving teaching methods Supporting low-performing schools and students National policies with restricted access NAE organize and distribute government grants IFAU involved to make reforms possible to evaluate Implementation Data collection Recently a number of national reforms suited for evaluation Falling student performance in international studies (the PISA 2012 chock) Sweden now performs below both the EU and OECD averages. Increasing performance differences between schools Standardized treatment (to different extent) Make teaching a more attractive profession: Certification of teachers Higher salaries Worry that the learning environment has deteriorated New methods with focus on traditional teaching Increasing differences between schools (and pupils) Support to low-performing schools and pupils Standardized treatment – differences more likely due to heterogeneous effects than different treatment Capacity or resource constraints – possible to find a control group (in theory) IFAU has limited possibilities to influence the implementation of policies IFAU and NAE have different objectives (missions) IFAU has no formal mandate to influence implementation Some success in data collection

Career development reform Career advancement steps for skilled teachers New positions with higher salaries Lead teacher Senior master Teacher-level implementation Municipalities select which teachers to promote Gradual expansion over four years Register data on teacher salaries and job mobility Data on student performance Karriärtjänster (2013/14) Teachers have compressed wages. Starting wages are at the OECD average, but wage growth is among the lowest in OECD. Teachers’ salaries are on average 20% lower than the salaries of other professionals with comparable education levels Attract and retain skilled teachers (boost the status of the teacher profession) Formal requirements for positions Certification Professionally skilled teacher PhD (Senior master) Each municipality can apply for a given number of positions, given by the number of students. Municipalities decide the terms of positions Parmanent/temporary contract Extent and type of work tasks Karriärtjänster Förstleärare (5000 kr / 535 euro / 600 USD)/månad Increase for this group is more than 15 % (average salary for this group is about 30000 kr/ 3600 USD) Lektor (10000 kr / 1070 Euro / 1200 USD) Compare teachers who receive the career positions with teachers who do not (or later) Possible to study the development of salaries and job mobility before and after the reform for treated and untreated teachers. Requires that the selection into the teaching positions are based on time-constant teacher charateristics School FE or Teacher FE depending on question An additional broader teacher compensation reform launched in 2016 (3000 kr)

Boosts for mathematics and reading “Lesson study” with professional coach Discuss problems and methods Apply new approaches in classroom Evaluate and discuss outcomes School-level implementation Trial in about 30 schools with random assignment Staggered national implementation over three years Register data on achievement in grades 3, 6 and 9 Survey data to teachers on training and teaching styles Literacy and numeracy Matematiklyftet (2012/13) and läslyftet (2014/15) Lesson study/learning study Teaching improvement process that originates from Japan Coach is appointed by municpality and trained by NAE In-service training to strengthen the quality of teaching and student performance Municipalities can apply for grants from NAE for teachers participating Municipalities applies for which schools to be part of the training Number of teachers in each municipality given by the number of students (?). The objective of random assignment was to obtain representative sample rather than to facilitate evaluation Control group must be drawn ex post Compliance rate in trial: 70% (mathematics); 40% (reading) Compare schools participating early with those participating later (or never) Difference-in-differences

Policies for newly arrived students Support for municipalities with many new refugees Intervention tailored to meet municipalities’ needs Organization of education Teacher training and methods Municipality-level implementation Municipalities ranked by immigrant share and experience Random selection within municipality pairs 75 municipalities randomly selected over three years Register data on achievement in grades 3, 6 and 9 (?) Massive increase of (school-aged) refugees to Sweden (2016/17) Aim to help refugees improve their school performance Assessment of student’s knowledge – what grade level students should start Introductory class - special classes for immigrants Help on students mother’s tongue Treatment determined jointly by NAE and municipality Grant to municipalities Treatment differ between municipalities NAE interested in an experimental evaluation of the initiative Experience measured as the share of immigrant received earlier 15 municipalities are selected in each of 5 waves Refugees are typically not observed in Swedish registry data (only when they receive permanent residence permit) Caroline will talk about a recent evaluation of a trial aimed at boosting performance in disadvantaged schools