Standardisation of judgements: reading and writing Ülle Türk University of Tartu/Estonian Defence Forces EUROPOS SĄJUNGA

Reading

30-31 March Training with standardised samples for reading Phase 1: Illustration – a first assessment of the level of one text and its corresponding tasks and items –At what CEFR level can a test taker already answer the following item correctly? Phase 2: Controlled practice – different texts with their corresponding tasks and items are assessed under the guidance of the co-ordinator Phase 3: Individual assessment of the remaining items in chunks of 4-6 items The session continues until a spread of results of one and a half levels is achieved. EUROPOS SĄJUNGA

30-31 March Benchmarking local samples of reading Individual rating followed by (pair) discussion leading to group consensus Discussion of spread across individual ratings and iteration until suitable agreement (spread equal to one and a half levels) is reached Ratings of the CEFR standardised samples should be analysed statistically in order to a)Confirm the relationship to the levels and b)Calculate intra-rater reliability (consistency) and inter- rater reliability EUROPOS SĄJUNGA

30-31 March Task 4: Items 1–4 Text Characteristics: –Text source: magazine –Authenticity: authentic –Discourse type: mainly argumentative –Domain: public –Topic: other/politics –Nature of content: fairly extensive abstract content –Text length 350 words –Vocabulary: extended vocabulary –Grammar: wide range of complex structures –Text likely to be comprehensible by learner at CEF level: C1 EUROPOS SĄJUNGA

30-31 March Items 1 & 2 Item 1 –Item type: Selected response/multiple choice –Operations: Infer implicit main idea/gist –Item level estimated: A2/B1 –Key: B –Per cent correct: 90.5% –Discrimination (item/total correlation):.31 Item 2 –Item type: Selected response/multiple choice –Operations: Make explicit inferences about the writer’s mood –Item level estimated: C2 –Key: B –Per cent correct: 36.1 % –Discrimination (item/total correlation):.45 EUROPOS SĄJUNGA

30-31 March Items 3 & 4 Item 3 –Item type: Selected response/multiple choice –Operations: Make inferences about implicit main idea/gist –Item level estimated: C2 –Key: B –Per cent correct: 43.3 % –Discrimination (item/total correlation):.20 Item 4 –Item type: Selected response/multiple choice –Operations: Make inferences about implicit detail –Item level estimated: B1 –Key: C –Per cent correct: 83.6 % –Discrimination (item/total correlation):.43 EUROPOS SĄJUNGA

30-31 March Task 5: Items 5–8 Text Characteristics: –Text source: magazine –Authenticity: authentic –Discourse type: mainly expository –Domain: public/personal –Topic: other/contemporary life –Nature of content: mainly abstract content –Text length 418 words –Vocabulary: extended vocabulary –Grammar: mainly complex structures –Text likely to be comprehensible by learner at CEF level: C1 EUROPOS SĄJUNGA

30-31 March Items 5 & 6 Item 5 –Item type: Selected response/multiple choice –Operations: Infer implicit detail –Item level estimated: B2 –Key: B –Per cent correct: 65,9 % –Discrimination (item/total correlation):.52 Item 6 –Item type: Selected response/multiple choice –Operations: Evaluate implicit opinion –Item level estimated: C2 –Key: C –Per cent correct: 49,2 % –Discrimination (item/total correlation):.32 EUROPOS SĄJUNGA

30-31 March Items 7 & 8 Item 7 –Item type: Selected response/multiple choice –Operations: Make inferences about implicit main idea/gist –Item level estimated: C1 –Key: C –Per cent correct: 52,1 % –Discrimination (item/total correlation):.36 Item 8 –Item type: Selected response/multiple choice –Operations: Evaluate implicit attitude/conclusion –Item level estimated: B2 –Key: A –Per cent correct: 74,4% –Discrimination (item/total correlation):.24 EUROPOS SĄJUNGA

Writing

30-31 March Training with standardised samples for writing Phase 1: illustration with about three standardised performances – 60 minutes Phase 2: controlled practice with 3–5 standardised performances – 60 minutes Phase 3: individual assessment of 3–5 standardised performances – 60 minutes The session ends when the vast majority of participants agree on the level, with the spread not exceeding one and a half levels. EUROPOS SĄJUNGA

30-31 March Benchmarking local samples of writing Individual rating and group discussion of high, middle and low performances – 60 minutes Individual rating of about five more performances – 60 minutes EUROPOS SĄJUNGA

30-31 March Sample task 1 Your English Teacher has asked you to write a story. Your story must have the following title: The most important day of my life Write your story on your answer sheet. EUROPOS SĄJUNGA

30-31 March Example answer 1 The most important day of my life. During a lifetime there is so many days you could call, the most important day of your life”. It could be the day you chose wich school you are going to, or what you want to work with the rest of your life. Another important day is when you get married, or you chose where to live. But most of all it must be a very important day when you give birth to a child. I think that changes everything you have been doing until then. Than you have to realice that somebody are more important than yourself. EUROPOS SĄJUNGA

30-31 March Sample task 2 This is an extract from a letter which you receive from the World Opinion Organisation. We are carrying out an international survey on families around the world and the importance of family relationships. Please write us a report for our survey outlining the typical family situation in your own country and suggesting how you think the situation might change in the future. Write your report. Write approximately 250 words. EUROPOS SĄJUNGA

30-31 March Sample task 3 You have read the extract below as part of a newspaper article on the loss of national and cultural identity. Readers were asked to send in their opinions. You decide to write a letter responding to the points raised and expressing your own views. ‘We are losing our national and cultural identity. Because of recent advances in technology and the easy availability and speed of air travel, different countries are communicating more often and are therefore becoming more and more alike. The same shopping malls and fast food outlets can be found almost everywhere. So can the same types of office blocks, motorways, TV programmes and even lifestyles. How can we maintain the traditions that make each nation unique?’ Write your letter. Do not write any postal addresses. EUROPOS SĄJUNGA

30-31 March Sample task 3 You recently entered a competition and have just received this letter from the organiser. Read the letter, on which you have made some notes. Then, using all the information in your notes, write a suitable reply. Write a letter of between 120 and 180 words in an appropriate style on the opposite page. Do not write any postal addresses. EUROPOS SĄJUNGA

30-31 March EUROPOS SĄJUNGA

30-31 March Sources Manual for relating language examinations to the Common European Framework of Reference for Languages – a preliminary pilot version: ocuments_intro/Manual.html ocuments_intro/Manual.html The CEFR Grid for Writing, v. 2.: The Final Grid for analysing reading and listening items: EUROPOS SĄJUNGA