Davide Azzolini, FBK-IRVAPP Enrico Rettore, FBK-IRVAPP Antonio Schizzerotto, FBK-IRVAPP MENTEP Kick-Off Meeting Brussels, April 2015

The MENTEP evaluation: a quick overview Day 1

The evaluation question Does the Technology-Enhanced Teaching Self-Assessment Tool (TET-SAT) really improve teachers’ TET self-assessment capabilities and, ultimately, increase their TET competencies? NB: The aim is to evaluate the tool, NOT the teachers, nor the schools!

The basics of the MENTEP evaluation design Counterfactual approach: compare a group of teachers who use the TET-SAT to an equivalent group of teachers who do not use it. The two groups of teachers will be identified in a way that: The two groups will be truly comparable (no «apples vs oranges» comparison!) The “no one forced, no one denied” principle will be satisfied

a) Sampling A random sample of schools will be selected from the relevant population, then a random sample of teachers will be drawn in each school. Reference population: Only schools with adequate ICT equipment levels will be considered; education level to be decided. Target sample size: approximately 1,000 teachers per country over no less than 50 schools (Small countries, ad hoc solutions). Incentives for the participation into the experiment (e.g., a lottery for one teacher per country offered a trip to visit Brussels). Oversampling of schools and teachers will be performed in order to cope with possible refusals. Moreover, administrative information (whenever available) will be used to check whether school/teachers refusing to cooperate are systematically different from the others.

b) Benchmark and follow-up surveys Benchmark survey (within the first two months of school year 2016/2017). Before the intervention all sampled teachers complete an on-line benchmark survey to (a) assess their TET competencies and (b) collect a rich set of information on their educational and professional experience. Follow-up survey (By the end of school year 2016/2017). After the intervention, sampled teachers re-assess their TET competences. This is the outcome variable by which the intervention will be evaluated.

c) Encouragement letters and estimation method The sampled schools are randomly split into two halves: ‘encouraged’ and ‘not encouraged’. A random subgroups of teachers teaching in the ‘encouraged’ schools receive a set of encouragement letters explaining how to use the tool and why they should. All other teachers – both in the ‘encouraged’ and in the ‘not encouraged’ schools - receive no information. Since not all those receiving the letters will make use of the tool while some of the teachers not receiving the letter presumably will do, a rough comparison of the two groups of teachers does not identify the causal effect of the tool. The letter of encouragement is used as an instrumental variable to increase the probability of participation for those who receive it and allow to retrieve the estimate of the impact. We can assess the peer effect of the intervention by comparing teachers in the ‘encouraged’ schools not receiving the set of letters to those in the ‘not encouraged’ schools

First steps from here to June (…and afterwards) Day 2

1. Organizational model The active involvement of National Authorities is crucial to the success of MENTEP: To contribute to the building of the TET-SAT To get the complete list of schools and teachers To appoint National Coordinators that will be the direct contact with schools

2. TET-SAT surveys The surveys will be administered to all sampled teachers at the beginning and at the end of the school year 2016/2017 The items will be identified through a matrix of "content areas" (e.g. digital communication, digital pedagogy, digital production, e-safety, etc.) and "types of skills": (frequency of use, level of awareness, knowledge, etc.). This would possibly allow us to assess the effects on different content and skills dimensions. The list of the content areas shall be suggested by the NAs and the precise definition of the items by ICT/pedagogy experts. It might be advisable to hide the responses’ ranking to the teachers. This would reduce the risk of “social desirability bias” in teachers answers. This could also make the tool more attractive as the teachers will get at the end a comprehensive assessment of their "skills" without knowing it in advance.

3. Identifying the reference population Two main criteria: 1)Schools that are equipped with sufficient ICT level (to be defined). 2)Focus on the same grades across the 12 countries (according to a very preliminary scrutiny, grades 6 to 8 could be the solution).

4. The lists of schools NAs provide the list of all schools at the elected grades and satisfying the required ICT level (including the contacts, , of the schools). IRVAPP selects the random sample of schools. Within this sample, IRVAPP randomly assigns schools to “encouraged” and “non encouraged” groups IRVAPP/EUN send back the sample lists to the NAs. NAs contact the schools to obtain the teachers’ lists. A key prerequisite: availability of schools and teachers’ s (teachers with no might not be eligible…as they have no ICT skills, presumably) The role of National Coordinators will be crucial for establishing contacts and coordinating the MENTEP schools. NB: Next meeting (June) will be dedicated to all these «sampling» details!

Thank you for your attention Contact: