Challenges in Developing and Delivering a Valid Test Michael King and Mabel Li NAFLE, July 2013

Contents 1. Integrated Language Education System 2.Basic Concepts in Language Assessment 3. Considerations in Language Assessment – IELTS Approach 4.Q&A

1. Integrated Language Education System Learners Curriculum Delivery Curriculum Delivery Curriculum Language Assessment

2. Language Assessment: Basic Concepts

What is assessment? Noun The evaluation or estimation of the nature, quality, or ability of someone or something: "the assessment of educational needs". Synonyms appraisal - rating - valuation - estimation - evaluation

Why do we need to do assessment? Evaluate learning Measure achievement Inform future teaching Rank students Benchmarking Motivate students Evaluate teaching Selection Find gaps in learning

What kinds of assessment are there? Assignment Conversation Presentation Self assessment Observation Test Essay Portfolio Peer assessment Questionnaire Checklist Quiz Group task

3. Considerations in Language Assessment

What are the qualities of a good test?

IELTS Approach to Test Qualities Validity Reliability Impact Practicality R P I V

Validity: test content  Test content matched to needs of candidates  Task-based approach to testing  Testing of integrated skills  Variety of Listening and Reading item types used  Face to face Speaking Test and direct marking  Accepting all standard varieties of native-speaker English

Validity: continual research based development  1980 ELTS introduced (English Language Testing Service)  Original 4 skills test  1989 tri-partite international partnership created, 1st delivery of IELTS  1995 major systems and content revision  2001 revised Speaking Test  2005 revised Writing assessment criteria  2007 introduction of half band for Speaking and Writing  Constants up to date: 9-band scale, skill profile & overall score, f2f interview

Reliability of test materials QP Production Process Commissioning Pre-editing and Editing Pretest Construction Pretesting Pretest Review Banking of Materials Standards Fixing Construction Live Test Construction & Grading Live Test Release

Reliability of results  Scoring system Scoring system  Marker/Examiner selection and training  Double marking and EOR  Research and validation analysis  Monitoring: sample and targeted  Results benchmarked with international standards Results benchmarked with international standards

How do you create effective test tasks and rating scales for Writing and Speaking? Theoretical relevance The test’s criteria and tasks should be defensible in terms of valid theories of oral performance Discriminating power The test criteria and tasks should enable relevant distinctions to be made between candidates Assessability The features selected for assessor attention should be salient enough for a single assessor to notice them

IELTS W/S Band Descriptors WritingSpeaking Task Achievement/Response Fluency and Coherence Coherence and CohesionLexical Resource Grammatical Range and Accuracy Pronunciation

Results benchmarked with int’l standards

Impact  Long-standing concern for the relationship between IELTS and its contexts of use  Test development partly shaped by this concern  Early 1990s growing interest in test impact/washback  Systematic investigation of IELTS impact began in 1995

Consultation processes with stakeholders to ensure - IELTS is fit for purpose - cut scores are set appropriately Use of GT module for immigration and employment Standards setting events General Medical Council, UK (June 04) National Council for State Boards of Nursing, USA (Oct 04) Canadian Immigration and Citizenship (May 05)

Main outcomes of IELTS Impact Studies IELTS is a reasonable direct communicative performance test of the four macro skills IELTS is challenging but fair IELTS uses authentic and appropriate content IELTS is motivating for learners and teachers IELTS engenders a positive effect on classroom activities and learning processes

Practicality 可行性  48 test administrations per year  Global network of 900 test venues in over 130 countries/regions  Emphasis on standards, consistency and security global standards test centre selection, training, auditing secure stock management on-line results verification  Provision of information through a variety of channels  On-line registration  Results available 13 days after the test  Post test services

4. Q&A

Acknowledgement References: Presentation for teachers event, Nick Charge