CRELLA Centre for Research in English Language Learning and Assessment Однажды…

CRELLA Centre for Research in English Language Learning and Assessment Однажды… ….. в 1974-oм году …

CRELLA Centre for Research in English Language Learning and Assessment Assessing reading – how and why? An introduction to assessing reading for language teachers Dr Lynda Taylor

CRELLA Designing tasks for testing L2 reading

We need to think about…. 1.What types of mental (or cognitive) processing are typically involved in reading? 2.What types of text are typically involved in reading activity? 3.How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests?

Starting from the right place…. 1.the nature of the cognitive processes in reading 2. the nature of the reading texts 3. the nature of the reading question formats

Starting from the WRONG place? 1.the nature of the reading question formats 2. the nature of the reading texts 3. the nature of the cognitive processes

Good reading task design 1.What types of mental (or cognitive) processing are typically involved in reading? 2.What types of text are typically involved in reading activity? 3.How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests?

CRELLA What actually happens when we read? A brief look at cognitive processes in reading

Visual input Clearly, the starting point for any reading activity is the set of marks on a handwritten or printed page, or on a computer screen – i.e. a combination of letters, symbols, pictures, etc.

Word recognition Word recognition involves matching the form of a word in a written text with a mental representation of the orthographic forms of the language. Field (2004:234) refers to this as “the perceptual process of identifying the letters and words in a text”.

Lexical (vocabulary) access Accessing the lexical entry containing stored information about a word’s form and its meaning from the lexicon. The form includes orthographic and phonological mental representations of an item and possibly information on its morphology. The lemma (the meaning-related part of the lexical entry) includes information on word class and the syntactic structures in which the item can appear and on the range of possible senses for the word.

Syntactic parsing Once the meaning of words is accessed, the reader has to group words into phrases, and into larger units at the clause and sentence level to understand the text message.

Establishing propositional meaning at the clause or sentence level An abstract representation of a single unit of meaning: a mental record of the core meaning of the sentence without any of the interpretative and associative factors which the reader might bring to bear upon it.

Inferencing (1) Inferencing is necessary so the reader can go beyond explicitly stated ideas as the links between ideas in a passage are often left implicit. Inferencing in this sense is a creative process whereby the brain adds information which is not stated in a text in order to impose coherence.

Inferencing (2) A text cannot include all the information that is necessary in order to make sense of it. Texts usually leave out knowledge that readers can be trusted to add for themselves. If there was no such thing as inferencing, writing a text which includes every piece of information would be extremely cumbersome and time consuming.

Constructing a mental model Once the reader has processed the incoming sentence and elaborated it where necessary and possible through inferencing, the new information needs to be integrated into a mental representation of the text so far. This process entails an ability to identify main ideas, to relate them to previous ideas, distinguish between major and minor propositions and to impose a hierarchical structure on the information in the text.

Creating a discourse-level structure At a final stage of processing, a discourse-level structure is created for the text as a whole. The skilled reader is able to recognise the hierarchical structure of the whole text and determines which items of information are central to the meaning of the text. The skilled reader determines how the different parts of the text fit together and which parts of the text are important to the writer or to reader purpose.

Establishing a mental representation across texts In the real world, the reader sometimes has to combine and collate macro-propositional information from more than one text. The need to combine rhetorical and contextual information across texts would seem to place the greatest demands on processing.

Cognitive processing at A2 to C2

Goal setter The goal setter in the left hand column is critical in that the decisions taken on the purpose for the reading activity will determine the relative importance of some of the processes in the central core of the model.

Types of reading Reading is either careful or expeditious and comprehension takes place at the local and global level.

Local comprehension Local comprehension refers to the understanding of propositions at the level of microstructure i.e., the sentence and the clause. Basic comprehension questions are used to assess lexical, syntactic, and semantic abilities and the ability to understand important information presented in sentence-level propositions. The information used in the question and the information required for the answer are usually in the same sentence.

Global comprehension Global comprehension refers to the understanding of propositions beyond the level of microstructure, that is, any macro- propositions including main ideas, the links between those macro-propositions and the way in which the micro-propositions elaborate upon them.

Careful reading Careful reading is intended to extract complete meanings from presented material. This can take place at a local or a global level, i.e. within or beyond the sentence right up to the level of the complete text or texts. The approach to reading is based on slow, careful, linear, incremental reading for comprehension.

Expeditious reading Expeditious reading involves quick, selective and efficient reading to access desired information in a text. Expeditious reading would appear likely to include skimming, search reading, and scanning.

Skimming Skimming is generally defined as reading to obtain the gist, general impression and/or superordinate main idea of the whole text.

Search reading Search reading involves locating information on predetermined topics. The reader only wants information necessary to answer set questions or to provide data for example in completing written assignments. It differs from skimming in that the search for information is guided by predetermined topics so the reader does not necessarily have to establish a macro- propositional structure for the whole of the text.

Scanning Scanning involves reading selectively, to achieve very specific reading goals, e.g. looking for specific words/phrases, figures/percentages, names, dates of particular events or specific items in an index at the local word level.

Types of reading tested at levels A2 to C2

Let’s look at a reading test task Look at Task 1 which comes from a reading test. Which of the core cognitive reading processes we have discussed does this task seem to elicit? Which processes does this task apparently NOT elicit?

Let’s look at a reading test task Word recognition Lexical access Syntactic parsing Establishing propositional meaning at clause/sentence level Inferencing Building a mental model Creating a text-level representation Creating an intertextual representation

Let’s look at a reading test task Word recognitionYes Lexical accessYes Syntactic parsingYes Establishing propositional meaning at clause/sentence levelYes Inferencing? Building a mental model Creating a text-level representation Creating an intertextual representation

When we design a reading test task… We need to be explicit about: the types of reading that the task requires the various cognitive processes we believe the reading task is eliciting how well matched the cognitive processes are to the level of our students E.g. Is there a shift from tasks that focus on decoding to tasks that focus on meaning building from main ideas, to a text level representation to intertextual representation

Cognitive validity in the testing of Reading The extent to which the tasks we employ in a reading test elicit the cognitive processing involved in target reading contexts beyond the test itself.

CRELLA What is the nature of the texts we read? A brief look at what makes a text difficult to read

Contextual features of texts and tasks in reading tests This relates to the appropriateness of both : the linguistic and content demands of the text to be processed (i.e. read and comprehended) and the features of the task setting that impact on task completion (e.g. responding to comprehension questions or writing a summary)

Controlling contextual features of reading texts and tasks – 3 questions What are the key contextual features (within-text) that we need to think about when selecting texts for reading test tasks? What degree of complexity should we aim for in these features across each of the proficiency levels we want to test? What are the methods that can help us to understand the difficulty level of a reading text?

Contextual features in reading Some illustrative features relating to: lexical complexity (decoding) structural complexity (syntactic parsing) cohesion (the construction of meaning)

Some key lexical parameters L1Syllables per word L2Type token ratio L3Word frequency L4Lexical density L5Proportion of academic words

L1 : Average syllables per word the mean number of syllables per content word multisyllabic words take longer to read and process than monosyllabic words [Rayner & Pollatsek, 1989] “In general, the more syllables per word and the more words per sentence, the higher the associated grade level of the text” [White, S. (2011) Understanding Adult Functional Literacy: Connecting Text Features, Task Demands, and Respondent Skills. Taylor & Francis]

L2 : Type token ratio the number of unique words divided by the number of tokens of the words Each unique word in a text is a word type. Each instance of a particular word is a token. When the type: token ratio is 1, each word occurs only once in the text; comprehension should be comparatively difficult because many unique words need to be encoded and integrated with the discourse context. A low type: token ratio indicates that words are repeated many times in the text, which should generally increase the ease and speed of text processing. [Templin, M (1957) Certain Language Skills in Children: Their development and interrelationships. Institute of Child Welfare Monograph Series No. 26. Minneapolis, MN: University of Minnesota Press]

L3 : Word frequency the relative frequency of occurrence of words Frequency effects have been shown to facilitate decoding: – frequent words are processed more quickly and understood better than infrequent ones (Haberlandt & Graesser, 1985; Just & Carpenter, 1980). – rapid or automatic decoding = strong predictor of L2 reading performance (Koda, 2005) – texts which assist such decoding (e.g., by containing a greater proportion of high frequency words) are easier to process…. The more frequent a word, the more likely it is to be processed with a fair degree of automaticity, thus increasing reading speed (even among lower level learners) and freeing working memory for higher level meaning building. (Crossley, Greenfield and McNamara, 2008)

L4 : Lexical density depends on distinguishing between different word types, i.e. lexical (content) and function words – lexical: verbs, nouns, adjectives, adverbs – function: auxiliaries, determiners, pronouns, prepositions, conjunctions Accessing the meaning of lexical items requires accessing the mental lexicon, function words can be dealt with by pattern matching. Reading focuses mainly on lexical items and readers tend to skip function words.

L5 : Proportion of academic words the incidence of academic words in a text proved to be a good predictor of level in a study of FCE, CAE and CPE reading texts (Weir et al, 2012) Mean SD FCE (B2)1.61%1.26% CAE (C1)1.63%1.41% CPE (C2)5.82%2.84%

Some key syntactic parameters S1Sentence length S2Readability formulae S3Higher level constituents

Syntactic complexity linear processing of text in careful reading, with the reader decoding word by word assembly of decoded items into larger scale syntactic structure (Just & Carpenter, 1987; Rayner & Pollatsek, 1994) cognitive demands imposed vary considerably according to how complex the structure is (Perfetti, Landi & Oakhill, 2005) (Crossley,Greenfield and McNamara, 2008)

S1 : Sentence length in Cambridge ESOL reading papers Main Suite LevelAverage number of words per sentence Range KET (A2) PET (B1) FCE (B2) CAE (C1) CPE (C2)

S2 : Readability formulae are long-established and widespread in use rely heavily on word length and sentence length ignore many language and discourse components that are theoretically expected to influence reading and comprehension difficulty

…nevertheless… texts with longer words and lengthier sentences are more difficult to read – longer words tend to be less frequent in the language and infrequent words take more time to access and interpret during reading – longer sentences place more demands on working memory – real-time processing means holding information in your head until you can parse sentences syntactically – the longer the sentence, the more difficult this may be

S2 : Difficulty/readability estimates in Cambridge ESOL reading papers Main Suite level Flesch reading ease score Flesch-Kincaid grade level Flesch-Kincaid range KET (A2) – 7.4 PET (B1) – 10.1 FCE (B2) – 12.3 CAE (C1) CPE (C2) – 16.1

S3 : Higher level constituents the number of main verbs in a sentence is broadly indicative of the number of clauses - thus of complex syntactic composition the more complex the syntactic composition, the greater the load on cognitive processing the more clauses you have to process in a sentence, the more propositions you have to hold in working memory and link together

Cohesion (and coherence) Cohesion is an objective property of the explicit language and text. There are explicit features, words, phrases, or sentences that guide the reader in interpreting the substantive ideas in the text, in connecting ideas with other ideas, and in connecting ideas to higher level global units (e.g., topics and themes). These cohesive devices cue the reader on how to form a coherent representation. The coherence relations are constructed in the mind of the reader and depend on the skills and knowledge that the reader brings to the situation…coherence is a psychological construct, whereas cohesion is a textual construct. [Graesser et al 2004: 193]

Cohesion two forms of textual cohesion can be estimated : referential cohesion (the extent to which words in the text co-refer) conceptual cohesion (the degree of similarity between concepts in different parts of a text)

Let’s look at a reading text Read the text and, as you do so, think about aspects of the text’s complexity related to : – the difficulty of the lexis/vocabulary – the difficulty of the syntax – the cohesion of the text Don’t worry about answering the questions over the page. We shall look at those later.

Analysing the text’s complexity

CRELLA What is the nature of the reading task? A brief look at question formats in reading tests

Response method Selected response Multiple choice (MCQ) True/false Right/wrong doesn’t say Multiple matching Gapped text Constructed response Short answer questions Information transfer Random deletion cloze Selective deletion gap- filling Reading into writing (e.g. summary)

Position of the test questions Before the reading text? or After the reading text?

Let’s look at some test questions Question 1 Question 2 Question 3 Question 4

Context validity in Reading Context validity relates to the appropriateness of both the linguistic and content demands of the text to be processed, and the features of the task setting that impact on task completion.

When designing reading test tasks we need to take account of…. 1.What types of mental (or cognitive) processing are typically involved in reading? 2.What types of text are typically involved in reading activity? 3.How can different test tasks (question formats, e.g. MCQ, short answer) help us to elicit the relevant cognitive processes using the relevant text-types in our reading tests?

Спасибо за внимание!