24 October 2018 Lilli Pretorius ADAPTING EGRA TO ASSESS EARLY LITERACY IN THREE PROVINCES LESSONS FROM THE ZENLIT PROJECT 2015-2017 24 October 2018 Lilli Pretorius

OUTLINE Some issues around reading assessment Theoretical reading framework underlying the Zenlit intervention EGRA – background, components Adapting EGRA for reading assessment in the Zenlit context – Zulu, Xhosa and English Some results Suggestions for way forward

DOES THE LOGIC MAP GET US TO OUR DESTINATION? High quality PD programme Increased teacher knowledge Improved classroom practices Improved learner performance Even if PD packages have high-quality features, there is no guarantee that they will work. Contention around sequence of Steps 2 and 3; if Step 3 comes first, then Step 2 may be “a dividend earned later” (Walpole & McKenna 2015) In California, significant gains in teacher knowledge, but classroom practice did not solidify or extend the knowledge (Goldschmidt & Phelps 2010). (Step 2 happened) In a large scale Grade 2 study in the USA, teacher knowledge improved, teachers taught more explicitly, but no difference were found in learner outcomes between interventions and control (Garet et al. 2008). (Step 2 and 3 happened, but not Step 4)

SOME ISSUES AROUND ASSESSMENT who gets tested what gets tested when testing occurs how tests are conducted why the test is conducted (Sorenson 2015) “To be useful, tests have to be reasonably accurate. But to be accurate, tests have to be fairly complex, and complex assessment procedures contradict the fundamental notion that assessment should be simple and easy to deliver.” (Singleton 1997 ‘Screening early literacy’) Tension between monitoring and evaluating a project and collecting data for research purposes

COMPONENTS OF READING LANGUAGE PROFICIENCY EXECUTIVE CONTROL Working memory Cognitive self-control Cognitive flexibility LANGUAGE PROFICIENCY Listening comprehension Vocabulary Grammar (morphology/ syntax) COMPREHENSION READER RESPONSE DECODING

STRUCTURES AND PATHWAYS SOCIOECONOMIC AND CULTURAL CONTEXT HOME SCHOOL COMMUNITY EXECUTIVE CONTROL Working memory Cognitive self-control Cognitive flexibility LANGUAGE PROFICIENCY Listening C Vocabulary Grammar (morphology/ syntax) Cognitive-linguistic COMPREHENSION READER RESPONSE DECODING NEUROLOGICAL STRUCTURES AND PATHWAYS

EGRA use worldwide It arose from the need for the creation of simple, effective, and low-cost measures of learning outcomes, specifically in reading. Development of EGRA began in October 2006. RTI International was contracted to develop an instrument to assess early grade reading, with input from a panel of reading experts. The objective was to help countries begin a process of systematically measuring how well children in the early grades of primary school are acquiring reading skills, in order to inform ministries and donors regarding system needs for improving instruction. It is not intended as a diagnostic tool to screen individual learners. It is also not intended to be a high-stakes accountability measure for determining grade passing.

COMPONENTS OF READING ASSESSED BY EGRA (Gove & Cvelich 2011:20)

THE EGRA JOURNEY in ZENLIT 1. EGRA from EC 2. Pilot EGRA 2015 & CEA report 3. Tweaked & EGRA 2016 (& a little in 2017) 4. What can we learn from it?

2015 2016 2017 Use to 2016 results to report back to schools 2015 2016 2017 Obtained test from EC D of E Assessed 5 learners per grade per school - workbooks Trialled testing time Budget challenges and constraints ERA evaluators assessed the learners Tests evaluated by Prof Sarah Howie at CEA, UP Adapted the test to reflect orthographic features of Xhosa and Zulu Increased sample size to 15 learners per grade per school , random selection Trained the coaches to administer the tests About 15-18 learners per morning (3 fieldworkers per school per morning) Use to 2016 results to report back to schools The dictation test was dropped – too easy Tweaked the test a little bit – ironed out problem items Included 4 control schools in each province

ORAL READING COMPREHENSION EGRA COMPONENTS Included digraphs and trigraphs LETTER SOUNDS Adapted a PA test for the Nguni languages (syllable awareness, phoneme awareness, phoneme deletion – 13 items) PHONEMIC AWARENESS Included longer words – 3-, 4- and 5-syllable words WORD READING Used grade appropriate text for each grade (longer and more difficult with each grade) ORAL READING FLUENCY 5-8 questions, depending on grade at least one inferential question ORAL READING COMPREHENSION ASSESSMENT AND EVALUATION

DEMOGRAPHICS 2017 Grade 1 Grade 2 Grade 3 Mean Age 6.7 (5 – 9) 7.8 (6 – 10) 8.8 (6 - 11) Boys Girls 49% 51% 50% 55% 45% KZN EC WC Control Intervention 180 360 270 301

RURAL/URBAN CONTROLS AND INTERVENTIONS

CORRELATIONS between EGRA components NGUNI (Zulu/Xhosa) n = 988 correlations** significant at the 0.01 level Phonemic awareness Word Reading ORF Comprehension Letter sounds .65** .75** .67** .58** Phonemic Awareness .62** .55** .51** .89** .69** .80**

CORRELATIONS between EGRA components English n = 481 correlations CORRELATIONS between EGRA components English n = 481 correlations** significant at the 0.01 level Phonemic awareness Word Reading ORF Comprehension Letter sounds .63** .57** .55** .51** Phonemic Awareness .68** .92** .76** .80**

MULTIPLE REGRESSION: What predicts WORD READING in NGUNI languages(Zulu/Xhosa) n = 988 Adjusted R square = 0.59 Beta Standard error 𝜷 p Letter sounds .37 .017 .60 .000 Phonemic Awareness .84 .100 .22

MULTIPLE REGRESSION: What predicts WORD READING in English n = 481 Adjusted R square = 0.50 Beta Standard error 𝜷 p Letter sounds .34 .061 .23 .000 Phonemic Awareness .35 .275 .54

MULTIPLE REGRESSION: What predicts READING COMPREHENSION in Nguni languages(Zulu/Xhosa) n = 988 Adjusted R square = 0.66 Beta Standard error 𝜷 p Letter sounds .006 .002 .08 .004 Phonemic Awareness .046 .01 .10 .000 Word reading -.028 -.22 ORF .089 .89

MULTIPLE REGRESSION: What predicts READING COMPREHENSION in English n = 481 Adjusted R square = 0.74 Beta Standard error 𝜷 p Letter sounds .006 .004 .04 .16 Phonemic Awareness .062 .023 .09 Word reading -.028 -.29 .000 ORF .075 1.03

Falling at the first hurdle… If children do not learn to read in Grades 1-3, they cannot read to learn in Grade 4 onwards Components of reading Spaull & Pretorius (2018) Falling at the first hurdle: Early grade reading in South Africa. Springer.

SOME TAKEAWAYS It’s important to adapt the EGRA tool to the orthographic features of the target languages The adapted EGRA results discriminated between learners’ reading abilities in control and intervention schools The KZN and EC results provide preliminary data for differences in reading development between rural and urban schools The results suggest that in transparent agglutinating languages, letter sound knowledge is important for word reading The results indicate that in transparent agglutinating languages, ORF is a strong predictor of reading comprehension. This has been confirmed by other studies in SA (Spaull, Pretorius & Mohohlwane 2018) Given the relatively small sample sizes, these results could serve as a starting part for further research into benchmarks for letter-sound knowledge and ORF

Building impetus for comprehension Components of reading

DECODING AND COMPREHENSION BENCHMARKS Independent level 98% decoding accuracy 95% level of comprehension These are highly skilled readers who can effectively learn from texts appropriate for that specific maturational level. Instructional level 95% decoding accuracy 75% comprehension These are readers who do not have major reading problems but who benefit from reading instruction at their maturational level. Borderline level 90-94% accuracy in decoding 55-74% accuracy in comprehension These readers need to be given additional reading exposure and practice. Frustration level less than 90% decoding accuracy and about 50% or less comprehension These are readers who have major reading problems and who are reading well below their maturational level. They need intensive reading programmes to increase their reading level.

SHARING ZENLIT 2016 ASSESSMENT RESULTS Teachers and HODS sat at their school tables How well are your learners reading in Zulu compared to the Zenlit mean results in KZN? How can we use these results to help us pass the baton and set appropriate high standards (benchmarks)? HODS to set up a reading and assessment plan for the year and report back (after tea) Prov X School Y Grade 1 Grade 2 Grade 3   Prov School Phonemic awareness % 31.7 28.2 50.7 40.5 59.5 37.4 Letter sounds score Min – max 18.9 0- 70 16.3 0-31 36.1 0-96 23.5 0-46 35.7 0-93 21.9 0-55 Word reading score 12.7 10.8 25.9 20.8 55.2 42.2 ORF score 11.3 0-36 7.8 0-13 30.8 0-58 22.9 0-54 0-79 21.4 Oral Reading comprehension % 12.6 8 51.3 13.3

GROUP GUIDED READING – identifying cracks, modelling good reading, providing practice opportunities Components of reading

PAIRED READING – provides practice opportunities Components of reading