EARLI 2009, Amsterdam Roel van Steensel (SCO-Kohnstamm Instituut – UvA) Project 4: Reading and writing development Assessing the reading and writing proficiency of at-risk adolescents in multilingual contexts Roel van Steensel, Ron Oostdam, Amos van Gelderen and Mirjam Trapman Study into Adolescent Literacy of Students At-risk SALSA

2 Roel van Steensel, EARLI 2009 Project SALSA Large groups of students in secondary education lag behind in literacy skills OECD (2001) Relatively little is known about the factors that influence at-risk students’ literacy development Curtis (2002) Hence: Project SALSA

3 Roel van Steensel, EARLI 2009 Project SALSA Project 4 Reading and writing development Environment Student Project 1 Literacy context in school De Milliano Project 2 Literacy context outside school Van Kruistum Project 3 Literacy-related skills and attitudes Trapman

4 Roel van Steensel, EARLI 2009 Project SALSA Central research question Which factors contribute to/impede the reading and writing development of at-risk adolescents in multilingual contexts? Sample –Lower tracks of prevocational secondary education (vmbo): basic and middle-management program –Grades 7, 8 and 9 (age range: appr ) –30% lowest scoring on CITO End of Primary School Test –Year 1 (2007/08): N=63 –32 monolinguals, 31 bilinguals (based on student questionnaire) International context Parallel projects in Canada en Switzerland

5 Roel van Steensel, EARLI 2009 The test Newly developed literacy test: –Reading: text comprehension –Writing: text production This presentation: (construct) validity of the reading test

6 Roel van Steensel, EARLI 2009 The test 9 reading tasks Text selection on the basis of: –Text type: narrative, expository, instructive, argumentative –Medium: school books, newspapers/magazines, internet, ‘official documents’ (house rules) –Subject: context-neutral, relevant for the target group

7 Roel van Steensel, EARLI 2009 The test NarrativeExpositoryInstructiveArgumentative School booksNAR1EXP1 Newspapers/ magazines NAR2EXP2 EXP3 ARG1 ARG2 Official documents INS1 InternetINS2

8 Roel van Steensel, EARLI 2009 The test Item construction on the basis of hypothesized subskills: –Retrieving: retrieving relevant details from the text –Interpreting: making inferences from shorter passages in the text (e.g. cause-effect relationships) –Reflecting: reflecting on larger passages/the text as a whole Davis (1968); Rosenshine (1980); Goldman & Durán (1988); Alderson (2000); OECD (2003); Song (2008); Cerdán et al. (2009)

9 Roel van Steensel, EARLI 2009 The test Text comprehension Local level/microstructure Retrieving Interpreting Global level/macrostructure Reflecting

10 Roel van Steensel, EARLI 2009 Validity study Messick (1989) - aspects of (construct) validity: –Construct (ir-)relevance: does the test only reflect aspects of the construct to be measured? Or are the test results influenced by variables irrelevant to the construct? –Internal structure: can (theoretical) expectations about relations between items be confirmed? –External structure: can (theoretical) expectations about relations with other measures be confirmed?

11 Roel van Steensel, EARLI 2009 Validity study Test administered to 200 students in the lower tracks of prevocational secondary education (school year ) Additional tests administered to a subsample of 63 students

12 Roel van Steensel, EARLI 2009 Construct (ir-)relevance Assumption: difficulty of the tasks is determined by difficulty of the texts Comparison between: –Text difficulty measured by Flesch/Douma readability index –Task difficulty indicated by mean task score as % of the maximum task score

13 Roel van Steensel, EARLI 2009 Construct (ir-)relevance DoumaRank Mean/max*100% NAR NAR EXP INS EXP ARG ARG EXP INS

14 Roel van Steensel, EARLI 2009 Construct (ir-)relevance INS1 en INS2 are both discontinuous texts: –INS1 is a set of house rules discontinuity makes the task easier –INS2 is a website discontinuity makes the task more difficult

15 Roel van Steensel, EARLI 2009 Internal structure Text comprehension Local level/microstructure Retrieving Interpreting Global level/macrostructure Reflecting

16 Roel van Steensel, EARLI 2009 Internal structure Do the data show a: –1-factor model? –3-factor model? –2-factor model? Confirmatory factor analysis using Structural Equation Modeling (EQS 6.1) Individual test items were combined into 6 parcels (RTR1, RTR2, INT1, INT2, RFL1, RFL2)

17 Roel van Steensel, EARLI 2009 Internal structure 1-factor model Text comprehension RTR1 RTR2 INT1 INT2 RFL1 RFL2 Excellent fit! χ 2 =6.27, df=9, p=.713, N=175, SRMR=.026, RMSEA=.000, 90% confidence interval=

18 Roel van Steensel, EARLI 2009 Internal structure 3-factor model Text comprehension RTR1 RTR2 INT1 INT2 RFL1 RFL2 No improvement! Difference score 1-factor/3- factor model: χ 2 =3.36, df=3, p>.05. retrieving interpreting reflecting

19 Roel van Steensel, EARLI 2009 Internal structure 2-factor model Text comprehension RTR1 RTR2 INT1 INT2 RFL1 RFL2 No improvement! Difference score 1-factor/2- factormodel: χ 2 =2.41, df=1, p>.05. local level skills global level skills

20 Roel van Steensel, EARLI 2009 External structure Assumption: text comprehension is facilitated by: –Well-developed decoding skills –A rich vocabulary –Well-developed morpho-syntactic skills –A good working memory –(Metacognitive) knowledge of reading and writing And: text comprehension is associated with IQ

21 Roel van Steensel, EARLI 2009 External structure VariableCorrelation with SALSA reading score (N=63) Decoding skills.22 Receptive vocabulary.59** Morpho-syntactic knowledge.66** Working memory.42** Metacognitive knowledge.41** Nonverbal IQ.36** ** p<.01

22 Roel van Steensel, EARLI 2009 Conclusions: support for validity? Construct (ir-)relevance: – Task difficulty largely corresponds with text difficulty –Exceptions (INS1, INS2) are explainable Internal structure: –  No support for the presence of subskills – Data do show that the test measures one construct External structure: – Substantial correlations with other skills –Exception (decoding skills) is explainable

23 Roel van Steensel, EARLI 2009 Thank you for your attention! Contact? –Or visit our website: