The New SHAPE Listening & Reading Tests OR HOW DID THEY DO THAT?! BILC Seminar – Stockholm, Sweden October 2013 Michael Adubato SHAPE Language Testing Centre Belgium

GERMANY 5 – 3 SWEDEN Friends Arena, Stockholm 16 October 2013

Designing tests for SHAPE – a true learning process University of Westminster

Aim To explain the challenges we faced designing the Listening and Reading tests and how we overcame them.

The Format – Listening and Reading SLP 2: 20 x passages 20 x MCQ SLP 3: ? x passages 20 x MCQ SLP 4: ? x passages 20 x MCQ/ SA SLP 2 – 3 Listening: 60 mins Reading: 90 mins SLP 3 – 4 Listening: 75 mins Reading: 120 mins

Listening: Speed per level Rubric Orientation message Time to complete MCQ Reading Reading speed per level Types of reading: skim/ scam/ read for detail Factors affecting Passage Length Required test length

Validity Contexts EnglishGenres

Paper question Task FocusCategoryPaper ATopicWord Length EnglishAnswerAuthor 1Answer who, what, when questions PersSerial 5: Invitation to partyScottish Dancing109 BritishaKW 2Answer how and why questions WorkSerial 31: Travel orderBrussels Airport127 USAbRW 3Answer how and why questions GenNewpaper articleSleep patterns119British aRW 4Answer how and why questions WorkEquipment request formsNew system for booking equipment 124 USAaMR 5Identify explicity stated main ideas WorkSerial 4: Routine office exchange Revised HR policy requiring checking for legality 112 CanadiancRW 6Distinguish fact from opinion WorkSerial 4: Routine office exchange Brown and Laing Account106Australian bCH 7Answer who, what, when questions and how,why questions WorkSerial 40: Request for translator/interpreter services. Include details of requirement Interpreter required for Ukrainians166 USAcRW 8Interpret complex sentences WorkSerial 11: How to use the airconditioning Fire safety210 BritishcJAG Sample from Examination Schematic

Creating the texts………

Which factors affect the complexity of a text? Task focus Criteria eg STANAG or CEFR Grammar Lexis Sentence length Density of text/passage Speed (listening) Interference (listening) Level of inference

Answer who, what, when questions Answer how and why questions Distinguish fact from rumour or hearsay Recognise author's intentions Recognise attitudes and emotions of the writer Outline the development of an argument Task Foci

Writing and Setting Process Undertake TNA Allocate test items to writers Train writers Scrutinise in-office Writer edits Internal Setting Board In-house edits Pilot papers Produce paper Produce Schematic Allocate items to writers Scrutinise in-office Writer edits Order tasks Initial draft Internal Setting Pilot and adjust Select writers Select final writers

Recording

Why do people offer to sit a pilot examination? For a reward Joy To please others They have no reason to do so

Piloting Exam Paper in the UK

Piloting the paper overseas SLP 2/3/4??

What did SPSS do for us? (Remember, items increase in difficulty throughout each level of the test.) Enabled us to see how many people answered a question with a, b, c or d.

Pre-test questionnaires –Gender –Nationality –Language level (variation required) –Educational level

NOT SHOWN! IF YOU’RE INTERESTED IN THIS, I CAN PROVIDE YOU WITH A COPY… Demonstration of Paper C Spreadsheet

Questions?