SOCIO-COGNITIVE APPROACHES TO TESTING AND ASSESSMENT by Marina Gvozdeva, Natalya Milyavskaya, Tatiana Sadovskaya, Violetta Yurkevich Based on material by Cyril Weir PROSET - TEMPUS

Proficiency levels “How do examination boards make distinctions between the tests they offer at different levels on the proficiency continuum?” (Prof. Cyril Weir) What does it mean to say that one language learner is more proficient than another? What makes some language tasks easier to perform than others? What makes a test suitable for higher or lower ability learners? PROSET - TEMPUS

Common European Framework of Reference (CEFR) The CEFR describes language ability on a scale of levels from A1 (Breakthrough) for beginners up to C2 (Mastery) for those who have reached a high level of proficiency in a language.

From http://www.cambridgeenglish.org/exams-and-qualifications/cefr This is how one international examination board (Cambridge ESOL) relates its tests to the CEFR. From http://www.cambridgeenglish.org/exams-and-qualifications/cefr

A SOCIO-COGNITIVE FRAMEWORK

Construct Validity TEST TASK CONTEXT VALIDITY COGNITIVE PERFORMANCE SCORING VALIDITY This is a validity framework suggested by Cyril Weir as a useful way of investigating assessments. We will look at each of the parts of the framework in turn. We will use this framework throughout the course to look at how assessments are designed and to consider how effective they might be at giving us the information we need about test takers. Centre for Research in English Language Learning and Assessment

Cognitive Validity The extent to which the tasks we employ elicit the cognitive processing involved in task solving. If an assessment has cognitive validity, then the mental processes that language learners use when they carry out assessment tasks will be very similar to the mental processes they use when they deal with real life tasks involving language.

Receptive language processing Input Word Recognition Lexical access and syntactic parsing Propositional meaning Inferencing Mental representation Productive language processing Output Phonological/ orthographic encoding Grammatical and lexical encoding Planning and organisation Mental representation Here are two (simplified) models of language processing.

Context Validity Context validity relates to the appropriateness of both the linguistic and content demands of the text to be processed, and the features of the task setting that impact on task completion. If we want test takers to use the same mental processes that they use in the real world when they take an assessment, the tasks should also be very much like real world tasks. The kinds of language used should be realistic and the conditions (such as the time available for reading) should be similar. If we want them to be able to read newspapers and learn the facts of a story, newspaper texts with factual questions should appear in our tests. If we want learners to read novels and understand the feelings of the characters, extracts from novels with questions about emotions should appear in tests.

COGNITIVE VALIDITY: the example of reading

Cognitive demand at different proficiency levels In many ways the CEFR specifications are limited in their characterisation of task-solving ability at the different levels and we need to be more explicit for testing purposes about: the types of tasks demanded at each of the stages; how well calibrated the cognitive processing demands made upon candidates are in the design of the tasks; the cognitive load imposed by relative task complexity at each stage. According to the CEFR, as learners progress through the different levels, they can use their language abilities to carry out a wider range of tasks. At the A1 level, for example, they can just talk aboiut things they know well – themselves, their family, the classroom in concrete terms. At B2 level they can use their more sophisticated language abilities to give opinions and back them up with justifications. This is rather vague in the CEFR, but we can see how these ideas are put into practice at the different levels of the Cambridge tests.

Word Recognition Word recognition is concerned with matching the form of a word in a written text with a mental representation of the orthographic forms of the language. A basic step in understanding any text is word recognition. Being able to connect the sounds you hear or shapes you see to a ‘word‘. At the lowest levels, learners find this very demanding and it takes a lot of mental processing. Reading/ listening and recognising individual word shapes and sounds can be challenging for beginners.

Lexical access Accessing the lexical entry containing stored information about a word’s form and its meaning from the lexicon. The form includes orthographic and phonological mental representations of an item and possibly information on its morphology. The lemma includes information on word class and the syntactic structures in which the item can appear and on the range of possible senses for the word. This involves matching word forms to basic meanings. Beginners have only very limited vocabularies and know few of the possible senses of the words they recognise. Multi-word expressions – idioms and phrasal verbs – can cause difficulty.

Syntactic parsing Once the meaning of words is accessed, the reader has to group words into phrases, and into larger units at the clause and sentence level to understand the text message. Awareness of syntax is needed to build comprehension of phrases and clauses.

Establishing propositional meaning at the clause or sentence level An abstract representation of a single unit of meaning: a mental record of the core meaning of the sentence without any of the interpretative and associative factors which the reader might bring to bear upon it. A basic understanding of what a clause or sentence means in terms of the words and grammar. This is not always enough to understand what the speaker/ writer intends to communicate. If a teacher says ‘You are late,‘ the expected response from the student may not be ‘Yes, I know‘.

Inferencing Inferencing is necessary so the reader can go beyond explicitly stated ideas as the links between ideas in a passage are often left implicit. Inferencing in this sense is a creative process whereby the brain adds information which is not stated in a text in order to impose coherence. If there were no such thing as inferencing, writing a text which includes every piece of information would be extremely cumbersome and time consuming. Picking up the intended meaning involves awareness of context and conventions. If the teacher points out that you are late, an apology is expected.

Establishing a mental representation across texts In the real world, the reader sometimes has to combine and collate macro-propositional information from more than one text. The need to combine rhetorical and contextual information across texts would seem to place the greatest demands on processing. Particularly when reading or listening to learn about a topic, the reader/ listener has to understand connections between different texts. How does writer/ speaker A‘s information or point of view connect to the information provided by other speakers/ writers? This is challenging to do, even for advanced (C1 or C2) level learners.

Cognitive processing at A2 to C2 This is how Khalifa and Weir (2009) see the language processing demands of Cambridge reading tests that target the different levels of the CEFR. As learners progress up through the levels, the tasks they are given make increasing demands on language processing. At C2 level they are asked to make one summary of three different texts.

CONTEXT VALIDITY

Contextual Parameters in Reading Task Setting: text length. Linguistic Demands: discourse mode lexical resources structural resources functional resources nature of information. These are some of the contextual characteristics that affect how easy it is to understand a text. Longer texts are more difficult to understand. Argument texts may be harder to follow than simple narratives. Words with longer sentences and long, unusual words are harder to understand. Abstract ideas are harder to grasp than concrete descriptions.

The cognitive demands imposed by relative text complexity at each stage This slide (and the two that follow) shows how these contextual features change as the texts and tasks become more challenging in Cambridge tests.

Text length