1. Teacher Professional Development and its Effects on Students: Evidence from the M@t.abel Program in Italy Gianluca Argentin Aline Pennisi Daniele Vidoni Giovanni Abbiati Andrea Caputo

2. The problem Weakness of Italian students in international assessments on mathematics and science (i.e. IEA, TIMSS and OECD PISA). North South gap achievement.

4. North/South gap increasing over grades (difference from national average in the percentage of correct answers in grades 2,5,6 and 8 – source: SNV)

5. The opportunity Thanks to the EU funding, there has been a boost in initiatives to help schools and teachers to improve student achievements. (PON Istruzione 2007-2013 I-3-FSE-2009-2)

6. Target regions of PON EU funding.

7. One of the solutions A professional development program, called M@t.abel, was offered to tenured math teachersM@t.abel in lower/upper secondary schools. Enrolment on voluntary basis at school/teacher levels.

8. The questions Does M@t.abel work? Does it increase students’ math achievment? Does it change teachers’ way of teaching?

9. We are tryng to get an answer using a rigorous method, a Randomized Control Trial. To our knwoledge, this is an absolute first in the Italian school system

10. How did we make randomization acceptable to Italian school authorities? Delaying by one year the PD of the control group, instead of excluding it completely.

11. The M@t.abel programM@t.abel in the Italian context

12. The M@t.abel program:M@t.abel math applied to daily life problems is based on a mixture of formal lectures and on-line mentoring; offers a huge repository of scripts for math lessons; it lasts one entire school year; it requires to be implemented in classes (at least 4 units); it promotes teacher community.

13. M@t.abelM@t.abel seems promising, according to the literature on professional development [Garet et al 2001; Desimone et al 2002] : content focused; extended in duration; active learning processes; implemented directly in classes; based on peer collaboration.

14. M@t.abelM@t.abel seems promising also looking at the Italian teachers: the oldest worldwide [OECD 2007] ; the majority did not have any specific training in teaching; the majority of math teachers did not graduate in math/physics.

15. Moreover, during their career, the Italian teachers: rarely attend PD; are not assessed at all; do not have any salary differentiation based on merit; even do not have feedbacks about their job [Talis 2008].

16. The Randomized Control Trial

17. The RCT – 1st year 1.Teachers applied for the PD through their school 2. The school must send at least 2 teachers 3. Schools were randomly assigned to treatment in the current year or to delayed treatment next year

18. The estimate of the effect of the PD is the difference between T & C Effect = T - C For the first year, we have a classical experiment

19. Randomization September 2009 Sent to the treatment Control group Schools12550 Teachers473193

20. We checked the equivalence on an unusually wide set of characteristics at schools/teachers/students levels. The equivalence is guaranteed. We found only minor differences, anyway controlled in our estimates.

21. The outcomes Students: math test scores on the standardized National Assessment attitude toward math Teachers: self reported teaching behaviours attitudes toward math teaching

22. Data collection November 2009/January 2010 CATI survey pre-intervention on teachers’ attitudes May 2010 Standardized math tests + questionnaires on students (background & attitude vs math) + data from teachers’ logs December 2010 CATI survey post-intervention on teachers’ attitudes and evaluation of the experience.

23. Intervention November 2009 – May 2010 TEACHERSSent to the treatment (473) Control group (193) Lost – no data available 7927 Compliers156166 Non compliers2380 STUDENTS7.6923.372

24. Only 39% of teachers are compliers …. Drop out Attended only lecturesAttended course Attended course + classroom activities with students

25. Effects estimations ITT Comparison of sent to treatment and control group ATT Comparison of actually treated and control group (Instrumental Variable regression)

26. The short term effects 1st year for students beginning of 2nd year for teachers

27. Effects on students math achievment No effect. Math achievment scaled to an average of 500 and standard deviation of 100 for the 7th grade.

28. Effects on students attitudes Some slight effects: more interested in maths (1 item out of 4); feeling more time costraints; less frequently attributing academic failure to chance or to bad luck; more anxious during the assessment; more frequently skipping at least one item during the assessment. Development of a perfectionist attitude?

29. Estimated effects on teachers Increase: more exchange with colleagues more frequent lessons based on group activities production of didactic material Decrease: use of the school textbook mnemonic approach to math learning perceived self efficacy in making students work in groups

30. Further data collection on teachers May-June 2012 Good reponse rate: 90% Teachers who answered all the surveys: 85% Survey on Attitudes Instructional practices Use of Matabel 2 years after of the experiment

31. Further data collection on students May 2011-May 2012 National Students Assessment Measure of math performance All previous classes (6/7th grade, now 7/8th on 2011, 6th grades now 8th grades on 2012), but not all students (dropping out, failures)

32. Further step Merging data at student individual level, obtaining a panel and assessing the PON M@t.abelPON M@t.abel effect on the increase in math achievement across school years Long term effects estimation on teachers

33. Concluding... it is possible to run a RCT evaluation also in the Italian school system; first year results do not show any effect of M@t.abel on the main outcome... M@t.abel... but we found promising effects on teachers and students attitudes second and third year estimation will be crucial to evaluate the effectiveness of the program

34. Thank you!

35. Further steps

36. …in Year 2 many (56%) of the former controls are treated Computing the difference T Y1&2 - T Y2 we will get the effect of one additional year of exposure to the treatment.

37. If we want the effect of 2 years of exposure to the new method We have to compute the difference T Y1&2 - C Y1 The non-testable assumption is that there is no cohort effect.

38. Additional analysis 1. Effects on the score distribution (quantile regression) 2. Subgroup analyses (effects’ heterogeneity)

39. Eterogeneous effects by teachers age But too much uncertainty...

40. Probability of being full complier Compliance is associated mostly with individual factors, such as: o Age (50-55 year old, - 16 perc. points; over-55, - 22 perc. points) o Previous training experience (+11 perc. points) o Personal motivation to enroll (-25 perc. points if forced to enroll) (Binary logistic regression models)

41. 2.Data collection (3rd year) 3.Data collection on a brand new cohort of teachers

42. Effects on students’ attitudes