Davide Azzolini, FBK-IRVAPP Enrico Rettore, FBK-IRVAPP Antonio Schizzerotto, FBK-IRVAPP MENTEP Kick-Off Meeting Brussels, April 2015

The MENTEP evaluation: a quick overview Day 1

The evaluation question Does the Technology-Enhanced Teaching Self-Assessment Tool (TET-SAT) really improve teachers’ TET self-assessment capabilities and, ultimately, increase their TET competencies? NB: The aim is to evaluate the tool, NOT the teachers, nor the schools!

The basics of the MENTEP evaluation design Counterfactual approach: compare a group of teachers who use the TET-SAT to an equivalent group of teachers who do not use it. The two groups of teachers will be identified in a way that: The two groups will be truly comparable (no «apples vs oranges» comparison!) The “no one forced, no one denied” principle will be satisfied

a) Sampling A random sample of schools will be selected from the relevant population, then a random sample of teachers will be drawn in each school. Reference population: Only schools with adequate ICT equipment levels will be considered; education level to be decided. Target sample size: approximately 1,000 teachers per country over no less than 50 schools (Small countries, ad hoc solutions). Incentives for the participation into the experiment (e.g., a lottery for one teacher per country offered a trip to visit Brussels). Oversampling of schools and teachers will be performed in order to cope with possible refusals. Moreover, administrative information (whenever available) will be used to check whether school/teachers refusing to cooperate are systematically different from the others.

b) Benchmark and follow-up surveys Benchmark survey (within the first two months of school year 2016/2017). Before the intervention all sampled teachers complete an on-line benchmark survey to (a) assess their TET competencies and (b) collect a rich set of information on their educational and professional experience. Follow-up survey (By the end of school year 2016/2017). After the intervention, sampled teachers re-assess their TET competences. This is the outcome variable by which the intervention will be evaluated.

c) Encouragement letters and estimation method The sampled schools are randomly split into two halves: ‘encouraged’ and ‘not encouraged’. A random subgroups of teachers teaching in the ‘encouraged’ schools receive a set of encouragement letters explaining how to use the tool and why they should. All other teachers – both in the ‘encouraged’ and in the ‘not encouraged’ schools - receive no information. Since not all those receiving the letters will make use of the tool while some of the teachers not receiving the letter presumably will do, a rough comparison of the two groups of teachers does not identify the causal effect of the tool. The letter of encouragement is used as an instrumental variable to increase the probability of participation for those who receive it and allow to retrieve the estimate of the impact. We can assess the peer effect of the intervention by comparing teachers in the ‘encouraged’ schools not receiving the set of letters to those in the ‘not encouraged’ schools

Thank you for your attention Contact: