IELTS and the Academic Reading Construct Tony Green Cyril Weir Centre for Research in English Language Learning and Assessment The researchers would like to acknowledge the support of the British Council in funding this study

EALTA Athens May 2008CRELLA University of Bedfordshire2 Test validation from the user perspective CRELLA programme of research to explore how far IELTS academic reading test reflects the reading practices of university students. analysis of undergraduate texts vs IELTS academic reading texts analysis of student vs IELTS academic reading tasks student reading processes vs IELTS academic reading test taking processes

May 2008CRELLA University of Bedfordshire3 Comparisons between IELTS and undergraduate reading Weir et al. (2007) compared IELTS academic reading to student experiences based on survey of 1,000 UoB students IELTS was said to under-represent: expeditious reading skills (requires avg. reading speed of c. 60 wpm) integration of information beyond the sentence level information at level of the whole text information accessed across texts Current study intended to extend self-report data to larger sample of test takers in variety of contexts.

May 2008CRELLA University of Bedfordshire4 Instruments IELTS academic reading has 3 parts 1 Test Part has an input text of c.800 (min 586 – max 1036) words and 13 or 14 associated questions. Used 2 IELTS academic reading tests from C.U.P. Cambridge Practice Tests for IELTS: Volume 2 (released material that has passed through Cambridge ESOL test development procedures). These… Only employed currently approved Q types (see Required both explicit and implicit information sources Were judged to encourage both expeditious and careful reading types Contained texts well within typical IELTS ranges for readability, vocabulary range and syntactic complexity IELTS academic reading test

May 2008CRELLA University of Bedfordshire5 Instruments Groups of students were administered one Test Part (20 minutes) Test Part = 1 text + up to 4 Sections of different Q types = 13/ 14 Qs Followed by a retrospection form eliciting… Background information (age, gender, L1, nationality, previous IELTS, uni. subject) Text preview – did test takers read the text before looking at the questions? Strategies for responding – how did test takers go about looking for the answers? Information base for the response – where did the the test takers find the information they needed to answer the questions? Retrospection form

May 2008CRELLA University of Bedfordshire6 Participants 352 participants participants per Test Part 16 languages 79% L1 Chinese, 4% Arabic, 4% Thai 59% female Median age 22 Divided into 3 broad score levels, loosely interpreted (based on equivalences suggested at as representing… 0-5 points c. IELTS 5.5 or below 6-8 points c. IELTS points c. IELTS 6.5 or above Background and score levels

May 2008CRELLA University of Bedfordshire7 Text Preview PR1read the text or part of it slowly and carefully PR2read the text or part of it quickly and selectively to get a general idea of what it was about PR3did not read the text

May 2008CRELLA University of Bedfordshire8 Text Preview Over ½ of all report quickly and selectively previewing text Highest scoring test takers less likely to preview the text Lowest scoring most likely to preview slowly, carefully 1: slowly, carefully, 2: quickly, selectively, 3: no preview

May 2008CRELLA University of Bedfordshire9 Response strategies ST1match words that appeared in the question with exactly the same words in the text ST2quickly match words that appeared in the question with similar or related words in the text ST3look for parts of the text that the writer indicates to be important ST4read key parts of the text such as the introduction and conclusion ST5work out the meaning of a difficult word in the question ST6work out the meaning of a difficult word in the text ST7use my knowledge of vocabulary ST8use my knowledge of grammar ST9read the text or part of it slowly and carefully ST10read relevant parts of the text again ST11use my knowledge of how texts like this are organised ST12connect information from the text with knowledge I already have

May 2008CRELLA University of Bedfordshire10 Response strategies 83% use ST2: quickly match words that appeared in the question with similar or related words in the text 77% use ST10: read relevant parts of the text again 76% useST3: look for parts of the text that the writer indicates to be important 8% useST8: use my knowledge of grammar Most and least popular strategies

May 2008CRELLA University of Bedfordshire11 Response strategies ANOVA reveals differences in strategy use by level for: Used more often by higher scoring test takers ST2quickly match words that appeared in the question with similar or related words in the text ST10read relevant parts of the text again Used more often by lower scoring learners ST5work out the meaning of a difficult word in the question Differences by level

May 2008CRELLA University of Bedfordshire12 Response strategies Example ST3look for parts of the text that the writer indicates to be important ST4read key parts of the text such as the introduction and conclusion Both associated with higher scores on the following item set: Choose the most suitable heading for paragraphs A-G from the list of headings below. Patterns by item type (Test Section) viThey can't get in without these viiHow does it work? viiiFighting corruption ixSystems to avoid xAccepting the inevitable iCommon objections iiWho's planning what iiiThis type sells best in the shops ivThe figures say it all vEarly trials

May 2008CRELLA University of Bedfordshire13 Location of necessary information L1within a single sentence L2by putting information together across sentences L3by understanding how information in the whole text fits together L4without reading the text L5could not answer the question

May 2008CRELLA University of Bedfordshire14 Location of necessary information Test Part & SectionWithin sentenceAcross sentencesWhole text E1.1 + E1.2 + E1.3 + E2.1 + E2.2 + E2.3 + E2.4 + E3.1 + E3.2 + Test E

May 2008CRELLA University of Bedfordshire15 Location of necessary information Test Part & SectionWithin sentenceAcross sentencesWhole text F1.2 + F2.1 + F2.2 + F3.1 + F3.2 + Test F

May 2008CRELLA University of Bedfordshire16 Conclusions Response strategies cannot be assumed from item type or predicted with sufficient accuracy via expert judgement Protocol forms potentially of great value in routine piloting Can highlight issues with particular items as part of the item QA process – e.g. ‘guessability’ Can help to confirm that required range of reading skills are addressed in every test form IELTS test takers do locate necessary information across sentences, but whole text level not always required use more expeditious reading strategies than predicted from Weir et al 2007, but few items require these