Www.britishcouncil.org1 Challenges in Developing and Delivering a Valid Test Michael King and Mabel Li NAFLE, July 2013.

Slides:



Advertisements
Similar presentations
Assessment types and activities
Advertisements

Performance Assessment
Qualifications Update: English Qualifications Update: English.
Qualifications Update: Sciences Qualifications Update: Sciences.
English language requirements in immigration Focus on IELTS
Session 4: ASSESSING SPEAKING
Training teachers to use the European Language Portfolio Former les enseignants à lutilisation du Porfolio européen des langues.
TESTING SPEAKING AND LISTENING
Assessment Assessment should be an integral part of a unit of work and should support student learning. Assessment is the process of identifying, gathering.
1 © 2006 Curriculum K-12 Directorate, NSW Department of Education and Training English K-6 Syllabus Using the syllabus for consistency of assessment.
1 SESSION 3 FORMAL ASSESSMENT TASKS CAT and IT ASSESSMENT TOOLS.
Testing What You Teach: Eliminating the “Will this be on the final
HONG KONG EXAMINATIONS AND ASSESSMENT AUTHORITY PROPOSED HKDSE ENGLISH LANGUAGE ASSESSMENT FRAMEWORK.
Constructing the Foundations of Capacity Building An Activity Theory Analysis of the English in Action Baseline Studies Jan Rae and Adrian Kirkwood.
1 Functions of Assessment Why do we assess students? Discuss in your group and suggest the three most important reasons.
English language requirements in immigration Focus on IELTS
Internal verification and external standards moderation.
Pearson Test of English (PTE)
Consistency of Assessment
An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.
An overview of Assessment. Aim of the presentation Define and conceptualise assessment Consider the purposes of assessment Describe the key elements of.
Developed by Marian Hargreaves for NEAS 2013
Listening Task Purpose of the test:
How to use the scores achieved by university graduates under CEPAS as reference in hiring the right employees 1.What is CEPAS? 2.CEPAS Statistics.
UK NARIC Annual Conference 2008 English Language Requirements and Immigration The role of IELTS Carly Moulding, IELTS Stakeholder Relations Manager, British.
EXAMS International English Language Testing System.
Mathematics Assessment Standards The Mathematics Standard-Assessment should reflect the mathematics that all students need to know and be able to do. The.
Weaving Pathways: Interculturalism and Language
General Update. New Quality Assurance arrangements Will cover new National 1 to National 5 from 2013/14 New arrangements will promote shared understanding.
Qualifications Update: ESOL Qualifications Update: ESOL.
6 th semester Course Instructor: Kia Karavas.  What is educational evaluation? Why, what and how can we evaluate? How do we evaluate student learning?
New qualifications, assessment & quality assurance to support the Senior Phase of Curriculum for Excellence.
© Myra Young Assessment All rights reserved. Provided for the use of participants in AM circles in North Lanarkshire Council.
Quality in language assessment – guidelines and standards Waldek Martyniuk ECML Graz, Austria.
Reading and Writing National Standards workshop phase 2 Term
Lynn Thompson Center for Applied Linguistics Startalk Network for Program Excellence Chicago, Illinois October 16-18, 2009 Formative and Summative Assessment.
ASSESSING LANGUAGE SKILLS
UKNARIC conference Understanding IELTS scores explanation and practical exercise.
The second part of Second Language Assessment 김자연 정샘 위지영.
What do IELTS candidates have to do?  Candidates must do all four test modules:  Listening  Reading  Writing  Speaking.
Literacy of Assessment Karen Yager Knox Grammar School & University of NSW
Understanding Meaning and Importance of Competency Based Assessment
CCSSO Criteria for High-Quality Assessments Technical Issues and Practical Application of Assessment Quality Criteria.
Nick Saville Bridging the gap between theory and practice EALTA Krakow May 2006 Investigating the impact of language assessment systems within a state.
International Conference on Enhancement and Innovation in Higher Education Crowne Plaza Hotel, Glasgow 9-11 June 2015 Welcome.
Stages 1 and 2 Wednesday, August 4th, Stage 1: Step 5 National and State Standards.
Workshops to support the implementation of the new languages syllabuses in Years 7-10.
FLAGSHIP STRATEGY 1 STUDENT LEARNING. Student Learning: A New Approach Victorian Essential Learning Standards Curriculum Planning Guidelines Principles.
Qualifications Update: Sociology Qualifications Update: Sociology.
Trinity College London provides respected international qualifications across a range of disciplines in the performing and creative arts, and in English.
Assessment. Workshop Outline Testing and assessment Why assess? Types of tests Types of assessment Some assessment task types Backwash Qualities of a.
Sue Hartga, Quality Reviewer OCN Eastern Region Simplifying Standardisation.
 Teaching: Chapter 14. Assessments provide feedback about students’ learning as it is occurring and evaluates students’ learning after instruction has.
‘Education has for its purpose not the imparting of particular knowledge but the strengthening of mental faculties’ (Kant c1770)
FSM NSTT Teaching Competency Test Evaluation. The NSTT Teaching Competency differs from the three other NSTT tests. It is accompanied by a Preparation.
What is IELTS ?  The International English Language Testing System (IELTS) is widely recognized as a test of language ability of candidates who need to.
Language Assessment.
What is the purpose of the IELTS test?
Smarter Balanced Assessment Results
Training in Classroom Assessment Related to the CEFR
ASSESSMENT OF STUDENT LEARNING
CHAPTER 10 ASSESSMENT AND LANGUAGE LEARNING
Assessment of student achievement of outcomes from the 7-10 Science syllabus Use this as a screen saver.
Test format Total test time:
Evaluation and Testing
Week 9 English, Literacies and Policy Contexts A.
Jennifer Rodriguez TB610-60
Presentation transcript:

Challenges in Developing and Delivering a Valid Test Michael King and Mabel Li NAFLE, July 2013

Contents 1. Integrated Language Education System 2.Basic Concepts in Language Assessment 3. Considerations in Language Assessment – IELTS Approach 4.Q&A

1. Integrated Language Education System Learners Curriculum Delivery Curriculum Delivery Curriculum Language Assessment

2. Language Assessment: Basic Concepts

What is assessment? Noun The evaluation or estimation of the nature, quality, or ability of someone or something: "the assessment of educational needs". Synonyms appraisal - rating - valuation - estimation - evaluation

Why do we need to do assessment? Evaluate learning Measure achievement Inform future teaching Rank students Benchmarking Motivate students Evaluate teaching Selection Find gaps in learning

What kinds of assessment are there? Assignment Conversation Presentation Self assessment Observation Test Essay Portfolio Peer assessment Questionnaire Checklist Quiz Group task

3. Considerations in Language Assessment

What are the qualities of a good test?

IELTS Approach to Test Qualities Validity Reliability Impact Practicality R P I V

Validity: test content  Test content matched to needs of candidates  Task-based approach to testing  Testing of integrated skills  Variety of Listening and Reading item types used  Face to face Speaking Test and direct marking  Accepting all standard varieties of native-speaker English

Validity: continual research based development  1980 ELTS introduced (English Language Testing Service)  Original 4 skills test  1989 tri-partite international partnership created, 1st delivery of IELTS  1995 major systems and content revision  2001 revised Speaking Test  2005 revised Writing assessment criteria  2007 introduction of half band for Speaking and Writing  Constants up to date: 9-band scale, skill profile & overall score, f2f interview

Reliability of test materials QP Production Process Commissioning Pre-editing and Editing Pretest Construction Pretesting Pretest Review Banking of Materials Standards Fixing Construction Live Test Construction & Grading Live Test Release

Reliability of results  Scoring system Scoring system  Marker/Examiner selection and training  Double marking and EOR  Research and validation analysis  Monitoring: sample and targeted  Results benchmarked with international standards Results benchmarked with international standards

How do you create effective test tasks and rating scales for Writing and Speaking? Theoretical relevance The test’s criteria and tasks should be defensible in terms of valid theories of oral performance Discriminating power The test criteria and tasks should enable relevant distinctions to be made between candidates Assessability The features selected for assessor attention should be salient enough for a single assessor to notice them

IELTS W/S Band Descriptors WritingSpeaking Task Achievement/Response Fluency and Coherence Coherence and CohesionLexical Resource Grammatical Range and Accuracy Pronunciation

Results benchmarked with int’l standards

Impact  Long-standing concern for the relationship between IELTS and its contexts of use  Test development partly shaped by this concern  Early 1990s growing interest in test impact/washback  Systematic investigation of IELTS impact began in 1995

Consultation processes with stakeholders to ensure - IELTS is fit for purpose - cut scores are set appropriately Use of GT module for immigration and employment Standards setting events General Medical Council, UK (June 04) National Council for State Boards of Nursing, USA (Oct 04) Canadian Immigration and Citizenship (May 05)

Main outcomes of IELTS Impact Studies IELTS is a reasonable direct communicative performance test of the four macro skills IELTS is challenging but fair IELTS uses authentic and appropriate content IELTS is motivating for learners and teachers IELTS engenders a positive effect on classroom activities and learning processes

Practicality 可行性  48 test administrations per year  Global network of 900 test venues in over 130 countries/regions  Emphasis on standards, consistency and security global standards test centre selection, training, auditing secure stock management on-line results verification  Provision of information through a variety of channels  On-line registration  Results available 13 days after the test  Post test services

4. Q&A

Acknowledgement References: Presentation for teachers event, Nick Charge