1. The literacy divide: territorial differences in the Italian education system Claudio QUINTANO, Rosalia CASTELLANO, Sergio LONGOBARDI University of Naples “Parthenope” claudio.quintano@uniparthenope.it lia.castellano@uniparthenope.it sergio.longobardi@uniparthenope.it Statistical Methods for the analysis of large data-sets University G. d'Annunzio - Chieti-Pescara September 23, 2009 – September 25, 2009

2. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 2 Overview Investigating the determinants of student achievement Highlighting the influence of the TEST-TAKING MOTIVATION on territorial differences Italian data from the last PISA (Programme for International Student Assessment) survey edition (2006) A MULTILEVEL REGRESSION MODEL is applied This approach is suggested by the hierarchical structure of the PISA data where students (level-one units) are nested in schools (level-two units) DATA GOAL METHOD

3. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 3 PISA 2006 400.000 students (21.773 in Italy) 14.300 schools (806 in Italy) 57 Countries The survey has involved The OECD’s PISA “Programme for International Student Assessment” survey is an internationally standardised assessment administered to 15 years old students

4. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 4 The survey assesses the students’ competencies in three areas Reading literacy Scientific literacy Mathematical literacy The OECD collects data on STUDENT QUESTIONNAIRE SCHOOL QUESTIONNAIRE SCHOOL CHARACTERISTICS FAMILY ENVIRONMENT OF STUDENT PISA 2006 COGNITIVE TEST

5. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 5 International ranking -Reading literacy- ITALY: 469 points OECD average: 492 points Italy ranked 33th in reading among 57 countries

6. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 6 ITALY:462 points OECD average: 498 points Italy ranked 38th in mathematics among 57 countries International ranking -Mathematical literacy-

7. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 7 Italy ranked 31th in Science among 57 countries ITALY:475 points OECD average: 500 points International ranking -Scientific literacy-

8. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 8 The literacy divide READING MATHEMATICS SCIENCE

9. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 9 Determinants of learning outcomes Many studies (Marks, 2006; Korupp et al., 2002) emphasize the role of socio-economic background for determining learning outcomes and explaining the territorial differences. This work aims to asses how much the DIFFERENCES IN THE TEST-TAKING MOTIVATION “boost” the effect of socio economic background on the Italian literacy divide

10. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 10 Low stake test The PISA test is considered as a low stake test since the students perceive an absence of personal consequences associated with their test performance. Without an adequate effort, test performance is likely to suffer, resulting in the examinee’s test score underestimating his or her actual level of proficiency (Wise and De Mars, 2005; Wolf & Smith, 1995; Wolf, Smith, & Birnbaum, 1995)

11. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 11 Index of student effort 1.the “Test non-response rate” computed on the basis of the number of missing or invalid answers in the PISA cognitive test An index of student effort is computed on the basis of three variables : 3. the Students’self-report effort in the PISA questionnaire measured on a 10- point scale 2. the “Questionnaire non-response rate” computed on the basis of the number of missing or invalid answers in the PISA student questionnaire (family environment data)

12. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 12 Effort and performance Variation of the index of student effort in correspondence of the average performance at macro region level. Italy=100 The correlation coefficient between this index and the science performance is equal to 0,553 at national level

13. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 13 A two-level random intercept regression model is adopted Multilevel approach Variables at student level Variables at school level Error components ε ij ~ IID-N(0, σ 2 ) U 0j ~ IID-N(0, τ 2 ) cov(U 0j, ε ij )= 0 Outcome of i th student of j th school (Science plausible values)

14. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 14 Student performance (Science Test score) Scholastic context -School mean of ESCS -Parents’pressure -Private vs public Study programme Technical, Professional, Vocational, Lower secondary vs Classical studies Macro area North East, North West, South, South and Islands vs Centre Scholastic resources -Computers with web -Quality of educational resources -Teacher shoratge Gender Girls vs boys Home educational resources Self confidence in ICT Hours per week spent on homework Immigration background 1. or 2. generation vs native LEVEL 2 School j LEVEL 1: Student i within School j Causal structure of multilevel model

15. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 15 Estimation strategy A block entry approach is adopted (Choen and Choen, 1983) which consists to the gradual addition of the first and second level covariates The process starts with the simplest model, denoted the empty model, and then progressively adds complexity introducing school and student variables In the last model the index of student effort is considered with 8 school variables and 5 students variables

16. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 16 Block Entry Approach Student variables Scholastic context Study programme Scholastic resources School location Student effort Empty model

17. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 17 Pre-R 2 Proportion reduction in variance or “variance explained” (PRE-R 2 ) Variance component EMPTY MODEL Full model without effort index Full model with effort index Variance between schools 5,347.981,025.56589.76 Variance within schools 4,674.314,470.304,474.87 Variance expalined at school level - 80,82%88,97%

18. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 18 Effect of the student variables

19. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 19 Effect of the school variables

20. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 20 The index of student effort allows to explain a larger amount of variance, indeed after controlling for students motivation the accounted total variance total variance among schools increases from 80% to 89% socio-economic context The analysis confirms that the socio-economic context plays an important role on the student achievement The North-South divide has been overestimated by the PISA test since the score differences are also influenced by the lower effort and engagement of the Southern students Main findings

21. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 21 Territorial dummies With effort index:-25.46% With effort index:+0.82% With effort index:-48.47% With effort index:-47.50%

22. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 22 Collaboration indexes

23. The literacy divide: territorial differences in the Italian education system C.QUINTANO, R.CASTELLANO, S.LONGOBARDI 23 Transformation steps Rescaling procedure in order to obtain a common scale (0- 1) for each indicator (growing when the non- response rate declines) COLLABORATION INDEX 1) 2)