1. Making a Difference: Exploring the Impact of the Tempus- ProSET Project Ludmila Kozhevnikova Samara National Research University Foreign Languages and Professional Communication Department

2. Aim to explore the impact and sustainability of the TEMPUS - ProSET project

3. Overview Background and project description Survey results Conclusion

4. Background and project description Radical transformation of the Russian educational system since 1980s High-stakes exams: Russian State Exam (EGE, Unified Exam, since 2001) and middle school achievement test (OGE- 2009) Federal Exam for university students (FEPO)

5. Main problems Extended group of item writers had no formal training in testing ( the first group – BC support: training at Lancaster University; second group – support from the English Language Office of the US Embassy in Moscow: occasional seminars and training from the well-known scholars – J. Purpura, JD Brown, A. Palmer, C. Coombe, N.Hubley) Poor training of assessors and interlocutors –the Speaking part was removed in 2008

6. Score distribution. English (FIPI, 2008,p.372) % of students

7. German (ibid, p. 373) % of students

8. French(ibid)

9. EGE 2009. Reading scores [fipi.ru]

10. Federal i-Exam for universities http://fepo.i-exam.ru/ Developed in Yoshkar-Ola 2009 – 2010 Statistics: No of universities – 1,344 No of participants – 1,252 389

11. English (http://fepo.i-exam.ru/) Accessed 1 July, 2008http://fepo.i-exam.ru/ Total: 26 MC items Time: 90 min No specs, description on the website. Vocabulary(4 items) Grammar (11 items) Knowledge of the rules of etiquette (??)(3 items) Culture and traditions(2 items) Writing letters, memos, applications(2 items) Reading (1 text, 4 items)

12. GE. Vocabulary ://fepo.i-exam.ru/ [Accessed 1 July 2008] Let’s stop at this restaurant for a quick _____. 1) walk 2) menu 3) meal 4) smile

13. Grammar. Prepositions(ibid) Dear Tom, Mary and me would like to ________________ you______your birthday Congratulate ____ with Congratulate ___ in Congratulate ____to Congratulate _____from

14. Rules of Business Etiquette (ibid) Choose the most appropriate response Woman: I’m Laura Miles from London office. How do you do? Man: Hi, Laura. Pleased to meet you, Ms. Miles. Hello, everybody! What do you want?

15. Socio- business domain(?) Colleague: Can you lend me some money? Peter: Sorry, but I am short of money at the moment. Yes, do, please Nobody likes to lend money You see, money is not everything.

16. ‘Writing’ http://fepo.i-exam.ru/ [Accessed 1 July 2016]

17. Testing Reading [ibid]

18. http://eacea.ec.europa.eu/tempus Promoting Sustainable Excellence in Testing and Assessment of English

19. http://eacea.ec.europa.eu/tempus Partners EU Partners  University of Bedfordshire  Southampton Solent University  Technische Universitat Dresden  University of South Bohemia RF Partners  Ministry of Education and Science  Mordovia N.P. Ogarev State University  Karelian State Pedagogical University  Linguistics University of Nizhny Novgorod  Southern Federal University  Tatar State University of Humanities and Education  Chelyabinsk State Pedagogical University  Immanuel Kant State University of Russia  Far Eastern Federal University  Samara State University  Surgut State University  Ulyanovsk State University  National Association of Teachers of English  Cambridge University Press

20. http://eacea.ec.europa.eu/tempus

21. Slide by prof. A.Green(CRELLA)

22. http://eacea.ec.europa.eu/tempus Transfer expertise in assessment literacy training

23. http://eacea.ec.europa.eu/tempus

24. Training in Dresden

25. http://eacea.ec.europa.eu/tempus Materials development

26. http://eacea.ec.europa.eu/tempus EU partners’ monitoring and assessment literacy seminars for local teaching communities

27. http://eacea.ec.europa.eu/tempus EU partners’ monitoring and assessment literacy seminars for local teaching communities

28. http://eacea.ec.europa.eu/tempus EU partners’ monitoring and assessment literacy seminars

30. Making a difference: Ex-Post evaluation Online questionnaire Telephone and skype interviews Respondents: 11 universities NATE CUP

31. Making a difference: Ex-Post evaluation questionnaire 15 Yes/(Partially)/No questions 20 sub-questions (Yes/No, MC and open- ended questions) Impact Effectiveness Sustainability Dissemination Spin-off effects

33. Impact 1. New methods of testing and assessment

34. Impact New methods of testing and assessment have been introduced, such language portfolio, writing portfolio, reading logs, collection portfolio, project work, drama, self- and peer- assessment Final examinations were completely redesigned in four universities to meet the CEFR requirements and the principles of communicative approach to testing

35. Impact “The project itself and practical results were of great support and help to the University during the very important procedure of accreditation in 2016. We worked out a lot of different tests to test the students' knowledge not only in languages but also in other subjects according to the general rules of testing and practical recommendations of the handbook issued as a result of the PROSET”.

36. Impact

38. Effectiveness To what extent the programme's objectives can be said to have been achieved Project objectives to develop and introduce new curricula for English language teacher education in the area of assessing learning outcomes to improve the awareness of testing and assessment issues among teachers to improve academic cooperation between universities and build partnerships between RF and EU institutions to develop sustainable inter-university co-operation on assessment literacy training to establish regional centres for professional development in quality assurance and language assessment to promote best practice in the development of instructional modules for higher education institutions in line with the EU Tuning initiative

40. Effectiveness Objective to develop and introduce new curricula for English language teacher education in the area of assessing learning outcomes Outcome 2000 pre-service teachers have received T&A training since 2013

41. Effectiveness Objective to improve the awareness of testing and assessment issues among teachers Outcome 4300 in-service teachers have received assessment literacy training

42. Effectiveness Objective to improve academic cooperation between universities and build partnerships between RF and EU institutions Outcome ‘The outreach of the project was not limited only by the T&A issues. All the partners managed to stay in touch and continue with the applications to other educational projects. They also facilitated the development of T&A into new level of dissemination’ Five joint projects; workshops and seminars with Carolyn Westbrook (Samara, Chelyabinsk, Moscow ); partnership relations between Surgut State University and Southampton Solent University; joint research, etc

43. Effectiveness Outcome Ulyanovsk State University: partnership relations with universities in Germany, Czech Republic. Nizhni Novgorod started cooperation with several EU universities which they… ‘expect to bring further improvements in curriculum design and program development’ Kaliningrad… ‘established new partnerships for new project applications and for new projects within NATE’

44. Effectiveness Objective to develop sustainable inter-university co-operation on assessment literacy training Outcome Conferences in Kazan(2014, 2015, 2016), Samara(2013, 2014, 2015, 2016), Saransk (2014), Nizhni Novgorod(2013, 2014), Petrozavodsk(2016), Vladivostok(2016), Ulianosvk(2014), Surgut(2015, 2016), Chelyabink (2014, 2016), Rostov/Don(2014). Joint presentations in Smolensk, Nizhni Novgorod, Saransk, Samara, Limassol, Cyprus (coming)

45. Sustainability ‘A project can be considered as sustainable if relevant activities are pursued and outputs are maintained or developed after the end of the EU funding (i.e. duration of new courses, up- dating of new tools)’ ‘Sustainability of international cooperation projects in the field of higher education and vocational training. Handbook on Sustainability’, p.3 http://ec.europa.eu/dgs/education_culture

46. Sustainability (slide by prof.A.Green)

48. Sustainability PDC: Teacher training seminars Webinars Outreach training programs (supported by local Ministries of Education) PDCs in Samara and Kazan – Academic Writing Centers. New rubrics to assess writing skills are being developed with the focus on ‘assessment for learning’ strategies INTEX Centre (Center for International Examinations, Surgut, northern part of West Siberia)

49. Russian Science Index ‘…was presented to the teaching staff of the Pedagogical Faculty of the University of Constantine the Philosopher in Nitra (Slovakia) and was found very useful for their similar course in this field’(Natalia Burenina)

51. Dissemination ‘Sustainability can be described by the maintenance, further development and/or dissemination of outputs and outcomes’ (Handbook on Sustainability, p.7)

52. Dissemination Circa 98 conferences(international, national and regional) 79 publications

53. Positive spin-off effects Joint degree program with Southampton Solent University 11 PhD researches (Samara, Moscow) 40 MA dissertations (Kazan, Samara, Moscow) Participation in Phase 2 and Phase 3 of the Validation of Extended Set of CEFR Illustrative Descriptors (Mediation)(Samara) Russian –English Dictionary of Testing and Assessment terms( Vladivostok) Membership in EALTA(Samara, Kazan, Nizhni Novgorod, Surgut), ALTE (Kazan)

54. Positive spin-off effects English Language Olympiads as alternative forms of assessment: For secondary school children (Moscow, 2014, Surgut,2015) For university students (Inter-regional, organized by Samara University, 2015, in cooperation with Cambridge Assessment and Cambridge University Press) – 940 students from Almetyevsk, Astrakhan, Kazan, Moscow, Orenburg, Rostov, Samara, Syzran, Togliatti, Ulyanovsk)

55. Conclusion The Tempus-ProSet project has proved to be effective and sustainable to have a positive impact on the development of the National Qualifications framework in Russia to develop sustainable inter-university co-operation Participants of the project have become agents of change: they have contextual and collaborative expertise they empower in-service and pre-service teachers with the assessment literacy skills thereby pursuing continuous improvement of the assessment- based accountability systems in the Russian education they initiate ‘bottom –up’ cascading effect changes that are likely to lead to systemic change in the Russian educational system

56. Thank you Ludmila Kozhevnikova laksvet@mail.ru