Advisory Group Meeting, 6 th November 2009 Student Performance in National Examinations (SPINE): the dynamics of language in school achievement (ESRC/DfID RES )

Research Team Changes Bristol: –Dr Neil Ingram (Biology) –Dr Federica Olivero (Maths) Zanzibar: –Dr Haji Mwevura (overseas: post-doctoral attachment); Shumbana Said (overseas: doctoral studies) –Mohamed Abeid Mbarouk (Maths & Science) 2

Study 2 : System wide examination study Study 4: Classroom study: focus on Teacher Assessment Issues Study 5: Examination achievement: focus on the learners Update on data collection for Process study (5.2) and Product study (5.3) Dissemination: Conferences, Publications RESEARCH UPDATE AND FINDINGS

4 STUDY 2: ENG and MATH Total: , school=29.18%, pupil=70.82% ENGLISH explains ( )/90.4=28.41% of the maths total variance Total=90.4, School=30.75%, Pupil=69.25%

5 STUDY 2: ENG and BIO 5 Total=65.646, school=18.17%, pupil=81.83% ENGLISH alone explains ( )/ =42.90% of the total variance in BIOLOGY Total= , school=18.79%, pupil=81.21%

6 STUDY 2: English and CHEM Total= , school=23.06%, pupil=76.94% ENGLISH explains ( )/ =42.65% of the total CHEM variance Total= , school=22.05%, pupil=77.95%

7 STUDY 2: Summary of multilevel models ENGLISH is a significant and substantial predictor of student performance in MATH, BIO & CHEM. School-level variances – all models (using ENGLISH) as the single explanatory variable demonstrated a substantial proportion of the variance attributable to school factors

Study 4: Focus on Teacher Assessment In Form II, Teacher Assessment (TA) was implemented based on the following assessment procedures: Class work Oral questions Homework Weekly tests Terminal tests 8

Study 4: Focus on Teacher Assessment Understanding of what assessment is to be done Awareness of guidelines Percentage of marks that go to the final examination Sharing what teachers know and practice about assessment guidelines among teachers within and across schools Level of monitoring of Teacher Assessment by head teachers.

Study 4: Focus on Teacher Assessment However: significant variability among teachers: the types of specific activities that constitute teacher assessment the number of activities from which they take marks for each month decisions on how to handle student absenteeism how to get 10% of marks for each month; some picked the one that students did better on, some picked randomly, and some put the average of all marks Study 2

Study 5.1: What are the four things that man could resemble whales? 45 students took this item: 35.6% = no answer 26.7% = wrong answer 28.9% = partially correct answer 8.8% = correct answer 11

Study 5.1: Interview: D1 who didnt answer Q3 explains D1:because I did not understand by this this … resemble (lines ) Int:If I tell you that resemble means to look like … can you do the question now? D1:Yes Int:OK so whats the answer? D1:Man … is warm blooded … and whales also … whales have lungs and man also have lungs … ( ) 12

13 Study 5 - Biology: responses to original item Give a brief explanation on what will happen on the following: A locust not suffocating when its head is immersed in water while the remaining part of the body is outside. No answer = 67.4% Wrong answer= 21.7% Partially correct answer = 6.5% Correct answer = 4.4%

14 Biology: on locusts

15 Biology: on locusts

16 Biology: on locusts Questions: a) In which picture do you think the locust will/may die? b) Why do you think it will/may die?

17 Original item modified: Greater contextualisation Simplification of instruction 2 structured parts: A & B Visual clues to support information retrieval Rephrasing of the item Altering item layout

18 Results Original item –Only 32.6% of students wrote an answer –Just under 11% gave a partially correct or correct answer Modified item –100% responded to this item –42% gave a partially correct answer to Part A –53% gave a partially correct answer to Part B

19 Changes in student response OR Sc Written responses on Modified Item MOD Score Comment (Neil Ingram: biologist) H30In picture A the locust may/will die I think it will/may dies because the locust get its breathing by using its body More than 0 Understands that locusts breathe using the body C21I think it will/may die because the boy is dipping the locust in the water to all the bodies with its trachea that used to respiration as a respiratory surface of a locust more than 0 Getting very close to a very complex answer I think a very able pupil indeed. The right answer but with great difficulty in expressing this in English

Study 5: Language Factors in Exam Performance Considerable challenges faced by students in understanding English lexis/vocabulary Piloted an off the shelf vocabulary test (receptive) Developed a measure of vocabulary knowledge based on corpora (an innovation in the design) developed from all past examination papers & current textbooks (Maths, Chemistry, and Biology) Piloted (Sept09), revised administered (Nov09) To our knowledge this is the first time a special purpose receptive vocabulary measure has been developed for English L2 school-age learners in sub-Saharan Africa. 20

Study 5.2: Focus on Process – CH & MT Aims: to capture in-depth insights on how students: –process and produce their answers to examination questions –use both English and Kiswahili in so doing –use English and Kiswahili in class Stimulus –video clips from Study 4 –exemplar examination items 21

Study 5.3: Focus on Product (achievement) Aim: to probe further the complex relationships between the use and effects of learning and assessment through English. developed exemplar examination items for Maths, Chemistry and Biology 3 different forms of the examination for c Form II learners: – English only –Kiswahili only –bilingual: Kiswahili and English. 22

Study 5.3:Focus on Product (achievement) Receptive vocabulary measure Student questionnaire (capture variables e.g. English language exposure, SES data) Head Teacher questionnaire (capture variables e.g. teacher qualifications matched with subjects they teach) 23

Progress of SPINE anticipated all data collection activities will be completed on schedule as well as the bulk of the analyses – end July 2010 refinement of analyses, writing & publication, & ongoing dissemination of research findings will be ongoing post July

Next Steps Data analysis ongoing & preparation of project outputs Dissemination in Zanzibar & regional engagement: practitioners, academics, NGO, policy makers SPINE International Symposium, Zanzibar 2011 SPINE mark 2: what do we - SUZA/MoEVT - want to do next? 25