1. 1 Examination Impact: Monitoring how teachers cope with change Dianne Wall Tania Horák Lancaster University UK

2. 2 Background Next Generation TOEFL (NGT) to be launched September 2005 (in certain sites only) Two impact studies commissioned by Educational Testing Service (producer of TOEFL)

3. 3 Our impact study Where? Central and Eastern Europe Who? TOEFL preparation course providers

4. 4 Phase 1Phase 2Phase 3 BaselineTransitionImpact Jan 03 to Oct 04Oct 04 to Mar 06After introduction of NGT -Description of current TOEFL preparation classes -Class observations, interviews with teachers, DoS and students. -10 sites in 6 countries -Tracking study - ’ Interviews ’ via Internet -Description of NGT preparation classes -Class observations, interviews with teachers, DoS and students. - same sites

5. 5 Theoretical underpinnings 1) Henrichsen (1989) – Hybrid Model of the Diffusion / Implementation Process 2) Messick (1996) - ‘ evidential link ’

6. 6 Henrichsen ’ s model AntecedentsProcessConsequences

7. 7 Process The Receivers The Innovation Characteristics of Communication The Resource System The User System Inter-elemental factors

8. 8 How our study fits the model: AntecedentsProcessConsequences BaselineTransitionImpact

9. 9 Messick ’ s ‘ evidential link ’ AntecedentsProcessConsequences ‘ evidential link ’

10. 10 Phase 2 Who? 6 teacher participants from Phase 1 Where? 5 countries: Bulgaria, Croatia, Lithuania, Poland, Slovakia. When? data collection period: January - May 05 How? Computer Mediated Communication What? investigation of pps ’ awareness and understanding of and reactions to NGT, to help us understand how they are preparing for the arrival of NGT

11. 11 What? Interviews based on monthly tracking questions Completion of tasks designed to familiarise participants with NGT Interviews discussing the task responses

12. 12 Data collection activities Interview 1 Task sent to pps Pps return task Researchers discuss responses Interview 2

13. 13 The Tasks January- Features of TOEFL preparation classes February - Awareness of current and future TOEFL March - Focus on the new writing tasks April - Focus on the new speaking tasks May - Prediction of actual content and methodology of pps ’ NGT preparation classes

14. 14 Findings Understanding of NGT Materials Teacher Support

15. 15 Understanding of NGT Teachers seem clear about the format of NGT but less clear about the actual constructs.

16. 16 Understanding of NGT ‘ I don't think the matter of accent was really addressed, though maybe those who grade the tests have an implicit understanding of how accent plays a role and will score accordingly. In preparing students for tests, it would help to have the issue of accents and how they affect scoring addressed directly in the rubric. ’

17. 17 Understanding of NGT Conclusion Awareness-raising requires more than providing examples of the test.

18. 18 Materials Teachers rely heavily on published materials.

19. 19 Materials ‘I can’t even dare think of something like that but I suppose I’ll have to improvise. The other possibility is to reschedule the [NGT] classes to the moment when there will be materials.’

20. 20 Materials Conclusions Impact will probably be affected by the textbook writers Impact of the test may well be hard to distinguish from impact of the textbook

21. 21 Teacher Support The test provider has supplied various forms of teacher support but they are not necessarily accessible to the teachers who need them.

22. 22 Teacher Support ‘ I just heard it mention by [the Director] … She also said it is very expensive …. I cannot afford, the center cannot afford. ’

23. 23 Teacher Support Conclusion Careful consideration must be paid to the logistics of teacher support if it is to be effective and reach those who need it, when they need it.

24. 24 Professionalism Test Provider - by commissioning impact studies TOEFL teachers – by participating in our study

25. 25 For further information About the project: d.wall@lancaster.ac.uk t.horak@lancaster.ac.uk About NGT: http://toeflpractice.ets.org

26. 26

27. 27 Tracking questions Have you learnt anything about New TOEFL which you didn ’ t know last month? If so, what? Is the new exam being discussed in your institution (e.g. by other teachers/ management)? If so, what are the main issues being discussed? Have students asked about the new TOEFL? If so, what have they asked about? Are you worried/ concerned about anything concerning the new TOEFL? If so, what? Is there anything else of interest concerning new TOEFL which has happened this month?