1 THE INTO EUROPE SERIES IN IN-SERVICE TEACHER TRAINING Presentation by Györgyi Együd ‘Into Europe - European Standards in Language Assessment’ Conference Budapest, February 9-10, 2006

2 WASHBACK  What is the washback effect? What studies have been conducted?  Are there preparatory materials?  How are teachers trained (encouraged) to prepare their students for the exam? (questions by Prof. Alderson)

3 COURSES AVAILABLE: Modern English Examinations Teacher Training (60 hours)  Module 1: Introduction to skill-based language testing & assessment  Module 2: Assessing Reading  Module 3: Assessing Listening  Module 4: Assessing Use of English  Module 5: Assessing Writing  Module 6: Interlocutor Training  Module 7: Assessing Speaking

4 Assessing Receptive Skills (30 hours) Interlocutor Training + Assessing Speaking Skills A2/B1 (30 hours) Interlocutor Training + Assessing Speaking Skills B2 (30 hours)

5 Assessing Writing Skills A2/B1 (30 hours) Assessing Writing Skills B2 (30 hours)

6 The materials used on the courses (tasks, rating scales, interlocutor's frame, etc.) have all been piloted in real Hungarian schools with real Hungarian students.

7 All of the training courses have gone through the accreditation process of the Hungarian in-service teacher training framework. They run live.

8 Basic training principles as presented in ’In-service teacher development: some basic principles’ in ELTJ 49/3 by Hayes, D (1995) ‘All teacher development activities should be classroom-centred.’ (Hayes, 1995: 256)

9 Basic training principles ‘Teachers should be involved in the preparation of courses.’ (Hayes, 1995: 257)

10 Basic training principles ‘Trainers should themselves be teachers.’ (Hayes, 1995: 257)

11 Basic training principles Training methodology should be largely task-based and inductive.’ (Hayes, 1995: 257)

12 Basic training principles ‘Training / development sessions should value participants’ existing knowledge.’ (Hayes, 1995: 258)

13 Basic training principles Teacher development sessions should offer opportunities for participants to share knowledge and ideas.’ (Hayes, 1995: 260)

14 The general training approach of the courses is based on the cyclical process of experiential learning (Kolb: 1984).

15 Stage 1: Concrete experience Stage 2: Reflective observation Stage 3: Abstract conceptualisation Stage 4: Active experimentation

16 Course design Each course cycle contains the following recurring elements : a demonstration of s uitable alternative assessment approaches & techniques

17 familiarising participants with all the possible task- and text types of the given skill presenting piloted, calibrated, benchmarked examples from the Into Europe series with the results from the pilots

18 opportunities to actually do the sample tasks presented opportunities to reflect on the teaching implications

19 reference to the teachers’ own classroom experiences, examining the possible problem points, sharing experiences, offering guidance, solutions

20 Participant feedback from an awareness raising course ‘Teaching and testing are becoming friends, walking in the classroom hand in hand, like learners and teachers, with washback sponge.’ ‘I can't wait to use today's ideas at home, with my students.’

21 ‘ ‘Now I'll be able to judge critically the listening exercises in the course book that I use.’ ‘Thank you for making me aware of even those "small things" (the tricky ones) in designing tests. I will definitely be more careful when writing test papers for my students.’

22 The desired impact of the training programmes teachers will use better teaching and testing techniques: positive washback teachers will be better-informed about European good practice in language testing & assessment

23 teachers’ awareness will develop in terms of what skills can be tested and how they can be tested, and also how students can perform at these tests as a result of the above, language learners’ language skills and test performance will improve

24 COURSE PARTICIPANTS ARE FAMILIARISED WITH GOOD PRACTICE THROUGH  standardised examination procedures demonstrated by a Model Examination developed for training purposes  examiner training focusing on both objectively & subjectively assessed skills  the use of benchmarked performances in the training courses for speaking and writing skills

25 In the Hungarian context the model consisted of five main phases: 1.selecting sample performances from the pilot examinations 2.selecting judges 3.home marking by judges 4.live benchmarking exercise 5.editing and standardising justifications A model of benchmarking procedures was designed and piloted by the project team under the guidance of Prof. Charles Alderson (the advisor of the Project).

26 The main purpose of the benchmarking exercise: to reach agreement on grades using the analytic rating scales developed by the Project Team. Relating the performances to the Common European Framework could be a supplementary exercise.  For this purpose the 9-point scales (Overall Spoken / Written Interaction) of the CEF were used.

27 The Use of Benchmarked Performances in the Training of Assessors The benchmarks and justifications produced by the judges in the benchmarking sessions are used for supporting the pre-course tasks and the face-to-face assessor training course.  Benchmarked performance samples illustrate candidate performance at different levels of the scales.

28 In the face-to-face training phase, the benchmarks and justifications are revealed to course participants in different ways at different stages of the training.

29 The training procedures developed by the Project have the following aims: to provide future examiners with sufficient information about the model language examination they are going to be trained for (outline, task types, mode) to familiarise future oral examiners with standard interlocutor behaviour to familiarise participants with the main principles and procedures of assessing speaking and writing performances According to the CEF (Guideline 4), future examiners should undertake appropriate training.

30 Further aims: to introduce the idea and practice of using analytic rating scales for assessing speaking and writing performances to enable participants to develop the necessary interlocuting and assessing skills to ensure valid and reliable assessment of live performances through standardisation  to equip trainees with transferable skills (there is a special need for this in Hungary)

31 The Outline of the Training Model Stage 1: Stage 1: pre-course distance learning self-study of an Introductory Training Pack  accomplishing the pre-course tasks (analysing and marking writing scripts or sample video performances)

32 The Introductory Training Pack contains An overview of the Speaking / Writing Model Examination Guidelines for interlocutor behaviour Guidelines for assessor behaviour Pre-course tasks Self-assessment questions Appendices: Benchmarks & Justifications for the Sample Tests, Examples of Candidate Language, CEF Scales, Glossary of Useful Terms

33 The Outline of the Training Model Stage 2: Live training courses (a series of workshop sessions)

34 Example 1: Speaking – Interlocutor Training discussing the experiences of the distance phase discussing the experiences of the distance phase analysing video samples of both standard and non- standard interlocutor behaviour analysing video samples of both standard and non- standard interlocutor behaviour standardisation of the administration procedure through simulated examination situations (role plays) standardisation of the administration procedure through simulated examination situations (role plays) demonstrating alternative approaches (paired vs individual tests) demonstrating alternative approaches (paired vs individual tests)

35 Role-play Cards for Part 1 (The Interview) Candidate are a shy, not very talkative candidate who tends to wait for guiding questions. You often reply with one or two short sentences only. You are a shy, not very talkative candidate who tends to wait for guiding questions. You often reply with one or two short sentences only. Interlocutor You are the interlocutor who asks the questions of the first part of the speaking test. You have to elicit as much speech from the candidate as possible. Please remember to ask the questions listed in the Interlocutor Frame.

36 discussing the experiences of the distance phase discussing the experiences of the distance phase introduction to assessing oral performances: modes and techniques of assessment introduction to assessing oral performances: modes and techniques of assessment familiarisation with the analytic rating scale familiarisation with the analytic rating scale standardisation of the assessment procedure standardisation of the assessment procedure comparing performances at different levels comparing performances at different levels Example 2: Speaking – Assessor Training

37 Stage 3: a distance phase application of the acquired skills: Practical application of the acquired skills: mock tests conducted in the participant’s own environment. The Outline of the Training Model

38 Conclusions formal training It is not impossible to become a trained examiner without formal training. distance and face-to-face elements Training should involve distance and face-to-face elements as well to ensure that future examiners go through each and every phase of the difficult and complex standardisation process. One training course is not enough One training course is not enough. Only further practicemonitoring examiner behaviour Only further practice and monitoring examiner behaviour can ensure that candidates’ language abilitiy is assessed in a standard manner, and the assessments are valid and reliable.

39 WASHBACK Are there preparatory materials? How are teachers trained (encouraged) to prepare their students for the exam? (questions by Prof. Alderson)

40 ‘Teaching and testing are becoming friends, walking in the classroom hand in hand, like learners and teachers, with washback sponge.’