THE CONTEXT OF THE PROJECT – Language Testing in Russia Carolyn Westbrook Southampton Solent University PROSET: A step forward in testing and assessment for Russian teachers of English Library of Foreign Literature, Moscow 17 September 2013

Outline 1.Construct validity - what are we assessing? 2.Test validation - quality control in language assessment 3.The context of the ProSET project 4.Key project milestones a.Language testing and assessment training – Luton, UK b.Development of modules c.Piloting d.Monitoring visits

Construct validity – what are we assessing? German listening test - what are we testing here? Listen to two people talking about their pets and match the words from the text to the definitions: 1. Katzea. dog 2. Hundb. mouse 3. Kaninchenc. cat 4. Mausd. rabbit

Construct validity – what are we assessing? English reading test - what are we testing here? Read the text and answer the questions below in full sentences using your own words. 1. According to the report, what other role have public relations firms taken on?

Construct validity – what are we assessing? The moral of the story … Think very carefully about the purpose of your test: What do you want to find out? Which skills do you want to test? How are you going to test them? Are the tasks going to provide the information you need?

Test validation – quality control in language assessment Validation aims to achieve “explanation and understanding. Therefore, the profession is coming around to the view that all validation is construct validation”. (Cronbach, 1984: 126, in Messick, 1989: 19).

Test validation – quality control in language assessment Threats to construct validity: 1.Construct underrepresentation: the test “fails to include important dimensions or facets of the construct” 2.Construct irrelevance:the test is irrelevantly difficult or easy. (Messick, 1989: 34)

Test validation – quality control in language assessment Test design: test specifications How will it be scored? Who? How many? Why? What? How long? How will it be answered? Test taker characteristics Test purpose / abilities / characteristics Test tasks / types of items Number of items and tasks Length of test Response type Scoring criteria

Context of the project High quality language teaching Background in language assessment Move to EGE Need for understanding of theoretical principles of testing

Project milestones TASKFMAMJJASONDJFMAMJJASOND LTA training - Luton ECTS training - Dresden Development of modules 1st round monitoring Revision of modules 2nd round monitoring Final revision of modules Handover conference Publishing of materials

Development of Russian LTA modules Module 1 Unit 1Assessment literacy & the socio-cognitive approach to assessment Unit 2The assessment cycle, test design and specifications Unit 3Classroom assessment Unit 4Assessing reading Unit 5Assessing listening Unit 6Assessing speaking Unit 7Assessing writing Unit 8Assessing language knowledge

Development of Russian modules Module 2 Unit 1Test qualities and usefulness Unit 2Validity Unit 3Reporting test scores Unit 4Reliability and item analysis Unit 5Washback and consequences Unit 6Standard setting

Quality assurance workflow RF partners RF coordinator EU LTA/ECTS team EU co- ordinator

Monitoring visits 1st round2nd round INSTITUTIONEU MONITORING PARTNER(S) Chelyabinsk State Pedagogical UniversityN/A - visit to SamaraLucie Betáková Far Eastern Federal UniversityCarolyn Westbrook / Gerry Cullen Carolyn Westbrook Immanuel Kant State University of RussiaGerry Cullen Karelian State Pedagogical AcademyCarolyn WestbrookN/A Mordovian State UniversityTony Green / Lucie BetákováGerry Cullen National Association of Teachers of English N/A Nizhny Novgorod Linguistics UniversityLucie Betáková Samara State UniversityTony GreenCarolyn Westbrook Southern Federal UniversityCarolyn Westbrook / Gerry Cullen Carolyn Westbrook Surgut State University of Khanty- Mansyisk Autonomous Okrug-Ugra Carolyn Westbrook / Gerry Cullen Carolyn Westbrook Tatar State University of Humanities and Education Kazan Federal University Lucie Betáková Ulyanovsk State UniversityTony Green Lucie Betáková

The monitoring process (round 1) Observation of module one delivery by RF partner teams Completion of monitoring reports Aims Content Methodology Student-teacher interaction Room and seating Overall ‘Master lessons’ and methodological school presentations by EU partners Conference presentations by EU partners (FEELTA, Vladivostok) Inspection of PDCs

Post-monitoring (1 st round) Amendments to report template Revision of modules based on RF context and 1 st round monitoring Revision of delivery based on 1 st round monitoring reports Content generally very well understood Some learner-centred / interactive delivery of classes More learner-centred approach to delivery needed in some cases Continue to set up PDCs

The monitoring process (round 2) Observation of module two delivery by RF partner teams – looking for improvements to teaching methodology as applicable Completion of amended monitoring reports ‘Master lessons’ and methodological school presentations by EU partners Conference presentations by EU partners (NATE Surgut) Inspection of PDCs

The future for language testing in Russia Language testing and assessment and ECTS modules published Regional PDCs to train both pre- and in-service teachers for long-term sustainability Improved knowledge and understanding of theoretical language testing

Any questions or comments?