1. Ongoing education policy evaluations
Björn Öckert
IFAU
How to improve school results?, Uppsala, May 24-25, 2016
Responsible for education research at IFAU
Education research at IFAU
Institutional conditions for policy evaluation in the Swedish context
Ongoing policy evaluations

2. Educational research at IFAU
Evaluate effects of educational reforms and policies
Policy relevant research on education
Evaluate policies and measures
Understand the conditions for education policies
Applied research
Project initiation
In-house research ideas
(In)formal government assignments
Instruction of the institute
More focus on school performance 2012
Broad mission
From pre-school to university and adult education
Restricted to regular education
Two areas of research
Evaluation of current policies
Basic research
School market – competition
School choice – preferences and information
Teacher and student incentives – compensation and grading
Empirical research using administrative data and survey data
IFAU is a small institute with a broad mission
Necessary to prioritize
Accumulate evidence about education policies
Try to be proactive and start up evaluations of central policies

3. Sweden’s education system
Central government set national goals
Municipalities and independent entities run schools
Resource allocation and policies
National Agency for Education (NAE)
Guidelines and support
Data collection and descriptive studies
Schools Inspectorate
Regular school inspections and quality audits
To understand the conditions for policy evaluation it is important to know the institutional setting of educational system
A goal-steered system with a high degree of local responsibility
Education act, school ordinance and national curriculum
Municipalities receive grants from central government, but not targeted towards schools
Students in independent schools: 15% in compulsory school; 26 % in high school (2015/16). Over 800 independent compulsory schools.
Local governance: Public and independent schools design policies to meet national goals.
Resources and degree of compensation
Hiring and firing of teachers
Organization of schools
Much variation in resource allocation, organization, teaching practices
No information on treatment
No control of allocation
“Uncontrolled experiments” (treatment without controls) – many different local policies
Many different types of policies in different schools
Not structured in a way so that we can learn from them
No central authority for organizing schools
Necessary to contact municipalities and independent school to influence policy design and data collection
Central authorities to support and control schools
National agency for education
Syllabuses for compulsory school
Knowledge requirements
The Swedish Schools Inspectorate assess if schools comply with the legislations and can sanction schools
Quality audits deal with well-defined areas, for example a special matter or problem area within the school.

4. Recent national educational initiatives
Policies to meet falling performance of schools
Raising teacher compensation
Improving teaching methods
Supporting low-performing schools and students
National policies with restricted access
NAE organize and distribute government grants
IFAU involved to make reforms possible to evaluate
Implementation
Data collection
Recently a number of national reforms suited for evaluation
Falling student performance in international studies (the PISA 2012 chock)
Sweden now performs below both the EU and OECD averages.
Increasing performance differences between schools
Standardized treatment (to different extent)
Make teaching a more attractive profession:
Certification of teachers
Higher salaries
Worry that the learning environment has deteriorated
New methods with focus on traditional teaching
Increasing differences between schools (and pupils)
Support to low-performing schools and pupils
Standardized treatment – differences more likely due to heterogeneous effects than different treatment
Capacity or resource constraints – possible to find a control group (in theory)
IFAU has limited possibilities to influence the implementation of policies
IFAU and NAE have different objectives (missions)
IFAU has no formal mandate to influence implementation
Some success in data collection

5. Career development reform
Career advancement steps for skilled teachers
New positions with higher salaries
Lead teacher
Senior master
Teacher-level implementation
Municipalities select which teachers to promote
Gradual expansion over four years
Register data on teacher salaries and job mobility
Data on student performance
Karriärtjänster (2013/14)
Teachers have compressed wages. Starting wages are at the OECD average, but wage growth is among the lowest in OECD.
Teachers’ salaries are on average 20% lower than the salaries of other professionals with comparable education levels
Attract and retain skilled teachers (boost the status of the teacher profession)
Formal requirements for positions
Certification
Professionally skilled teacher
PhD (Senior master)
Each municipality can apply for a given number of positions, given by the number of students.
Municipalities decide the terms of positions
Parmanent/temporary contract
Extent and type of work tasks
Karriärtjänster
Förstleärare (5000 kr / 535 euro / 600 USD)/månad
Increase for this group is more than 15 % (average salary for this group is about kr/ 3600 USD)
Lektor (10000 kr / 1070 Euro / 1200 USD)
Compare teachers who receive the career positions with teachers who do not (or later)
Possible to study the development of salaries and job mobility before and after the reform for treated and untreated teachers.
Requires that the selection into the teaching positions are based on time-constant teacher charateristics
School FE or Teacher FE depending on question
An additional broader teacher compensation reform launched in 2016 (3000 kr)

6. Boosts for mathematics and reading
“Lesson study” with professional coach
Discuss problems and methods
Apply new approaches in classroom
Evaluate and discuss outcomes
School-level implementation
Trial in about 30 schools with random assignment
Staggered national implementation over three years
Register data on achievement in grades 3, 6 and 9
Survey data to teachers on training and teaching styles
Literacy and numeracy
Matematiklyftet (2012/13) and läslyftet (2014/15)
Lesson study/learning study
Teaching improvement process that originates from Japan
Coach is appointed by municpality and trained by NAE
In-service training to strengthen the quality of teaching and student performance
Municipalities can apply for grants from NAE for teachers participating
Municipalities applies for which schools to be part of the training
Number of teachers in each municipality given by the number of students (?).
The objective of random assignment was to obtain representative sample rather than to facilitate evaluation
Control group must be drawn ex post
Compliance rate in trial: 70% (mathematics); 40% (reading)
Compare schools participating early with those participating later (or never)
Difference-in-differences

7. Policies for newly arrived students
Support for municipalities with many new refugees
Intervention tailored to meet municipalities’ needs
Organization of education
Teacher training and methods
Municipality-level implementation
Municipalities ranked by immigrant share and experience
Random selection within municipality pairs
75 municipalities randomly selected over three years
Register data on achievement in grades 3, 6 and 9 (?)
Massive increase of (school-aged) refugees to Sweden (2016/17)
Aim to help refugees improve their school performance
Assessment of student’s knowledge – what grade level students should start
Introductory class - special classes for immigrants
Help on students mother’s tongue
Treatment determined jointly by NAE and municipality
Grant to municipalities
Treatment differ between municipalities
NAE interested in an experimental evaluation of the initiative
Experience measured as the share of immigrant received earlier
15 municipalities are selected in each of 5 waves
Refugees are typically not observed in Swedish registry data (only when they receive permanent residence permit)
Caroline will talk about a recent evaluation of a trial aimed at boosting performance in disadvantaged schools