Making a Difference: Exploring the Impact of the Tempus- ProSET Project Ludmila Kozhevnikova Samara National Research University Foreign Languages and Professional Communication Department

Aim to explore the impact and sustainability of the TEMPUS - ProSET project

Overview Background and project description Survey results Conclusion

Background and project description Radical transformation of the Russian educational system since 1980s High-stakes exams: Russian State Exam (EGE, Unified Exam, since 2001) and middle school achievement test (OGE- 2009) Federal Exam for university students (FEPO)

Main problems Extended group of item writers had no formal training in testing ( the first group – BC support: training at Lancaster University; second group – support from the English Language Office of the US Embassy in Moscow: occasional seminars and training from the well-known scholars – J. Purpura, JD Brown, A. Palmer, C. Coombe, N.Hubley) Poor training of assessors and interlocutors –the Speaking part was removed in 2008

Score distribution. English (FIPI, 2008,p.372) % of students

German (ibid, p. 373) % of students

French(ibid)

EGE Reading scores [fipi.ru]

Federal i-Exam for universities Developed in Yoshkar-Ola 2009 – 2010 Statistics: No of universities – 1,344 No of participants – 1,

English ( Accessed 1 July, 2008http://fepo.i-exam.ru/ Total: 26 MC items Time: 90 min No specs, description on the website. Vocabulary(4 items) Grammar (11 items) Knowledge of the rules of etiquette (??)(3 items) Culture and traditions(2 items) Writing letters, memos, applications(2 items) Reading (1 text, 4 items)

GE. Vocabulary ://fepo.i-exam.ru/ [Accessed 1 July 2008] Let’s stop at this restaurant for a quick _____. 1) walk 2) menu 3) meal 4) smile

Grammar. Prepositions(ibid) Dear Tom, Mary and me would like to ________________ you______your birthday Congratulate ____ with Congratulate ___ in Congratulate ____to Congratulate _____from

Rules of Business Etiquette (ibid) Choose the most appropriate response Woman: I’m Laura Miles from London office. How do you do? Man: Hi, Laura. Pleased to meet you, Ms. Miles. Hello, everybody! What do you want?

Socio- business domain(?) Colleague: Can you lend me some money? Peter: Sorry, but I am short of money at the moment. Yes, do, please Nobody likes to lend money You see, money is not everything.

‘Writing’ [Accessed 1 July 2016]

Testing Reading [ibid]

Promoting Sustainable Excellence in Testing and Assessment of English

Partners EU Partners  University of Bedfordshire  Southampton Solent University  Technische Universitat Dresden  University of South Bohemia RF Partners  Ministry of Education and Science  Mordovia N.P. Ogarev State University  Karelian State Pedagogical University  Linguistics University of Nizhny Novgorod  Southern Federal University  Tatar State University of Humanities and Education  Chelyabinsk State Pedagogical University  Immanuel Kant State University of Russia  Far Eastern Federal University  Samara State University  Surgut State University  Ulyanovsk State University  National Association of Teachers of English  Cambridge University Press

Slide by prof. A.Green(CRELLA)

Transfer expertise in assessment literacy training

Training in Dresden

Materials development

EU partners’ monitoring and assessment literacy seminars for local teaching communities

EU partners’ monitoring and assessment literacy seminars for local teaching communities

EU partners’ monitoring and assessment literacy seminars

Making a difference: Ex-Post evaluation Online questionnaire Telephone and skype interviews Respondents: 11 universities NATE CUP

Making a difference: Ex-Post evaluation questionnaire 15 Yes/(Partially)/No questions 20 sub-questions (Yes/No, MC and open- ended questions) Impact Effectiveness Sustainability Dissemination Spin-off effects

Impact 1. New methods of testing and assessment

Impact New methods of testing and assessment have been introduced, such language portfolio, writing portfolio, reading logs, collection portfolio, project work, drama, self- and peer- assessment Final examinations were completely redesigned in four universities to meet the CEFR requirements and the principles of communicative approach to testing

Impact “The project itself and practical results were of great support and help to the University during the very important procedure of accreditation in We worked out a lot of different tests to test the students' knowledge not only in languages but also in other subjects according to the general rules of testing and practical recommendations of the handbook issued as a result of the PROSET”.

Impact

Effectiveness To what extent the programme's objectives can be said to have been achieved Project objectives to develop and introduce new curricula for English language teacher education in the area of assessing learning outcomes to improve the awareness of testing and assessment issues among teachers to improve academic cooperation between universities and build partnerships between RF and EU institutions to develop sustainable inter-university co-operation on assessment literacy training to establish regional centres for professional development in quality assurance and language assessment to promote best practice in the development of instructional modules for higher education institutions in line with the EU Tuning initiative

Effectiveness Objective to develop and introduce new curricula for English language teacher education in the area of assessing learning outcomes Outcome 2000 pre-service teachers have received T&A training since 2013

Effectiveness Objective to improve the awareness of testing and assessment issues among teachers Outcome 4300 in-service teachers have received assessment literacy training

Effectiveness Objective to improve academic cooperation between universities and build partnerships between RF and EU institutions Outcome ‘The outreach of the project was not limited only by the T&A issues. All the partners managed to stay in touch and continue with the applications to other educational projects. They also facilitated the development of T&A into new level of dissemination’ Five joint projects; workshops and seminars with Carolyn Westbrook (Samara, Chelyabinsk, Moscow ); partnership relations between Surgut State University and Southampton Solent University; joint research, etc

Effectiveness Outcome Ulyanovsk State University: partnership relations with universities in Germany, Czech Republic. Nizhni Novgorod started cooperation with several EU universities which they… ‘expect to bring further improvements in curriculum design and program development’ Kaliningrad… ‘established new partnerships for new project applications and for new projects within NATE’

Effectiveness Objective to develop sustainable inter-university co-operation on assessment literacy training Outcome Conferences in Kazan(2014, 2015, 2016), Samara(2013, 2014, 2015, 2016), Saransk (2014), Nizhni Novgorod(2013, 2014), Petrozavodsk(2016), Vladivostok(2016), Ulianosvk(2014), Surgut(2015, 2016), Chelyabink (2014, 2016), Rostov/Don(2014). Joint presentations in Smolensk, Nizhni Novgorod, Saransk, Samara, Limassol, Cyprus (coming)

Sustainability ‘A project can be considered as sustainable if relevant activities are pursued and outputs are maintained or developed after the end of the EU funding (i.e. duration of new courses, up- dating of new tools)’ ‘Sustainability of international cooperation projects in the field of higher education and vocational training. Handbook on Sustainability’, p.3

Sustainability (slide by prof.A.Green)

Sustainability PDC: Teacher training seminars Webinars Outreach training programs (supported by local Ministries of Education) PDCs in Samara and Kazan – Academic Writing Centers. New rubrics to assess writing skills are being developed with the focus on ‘assessment for learning’ strategies INTEX Centre (Center for International Examinations, Surgut, northern part of West Siberia)

Russian Science Index ‘…was presented to the teaching staff of the Pedagogical Faculty of the University of Constantine the Philosopher in Nitra (Slovakia) and was found very useful for their similar course in this field’(Natalia Burenina)

Dissemination ‘Sustainability can be described by the maintenance, further development and/or dissemination of outputs and outcomes’ (Handbook on Sustainability, p.7)

Dissemination Circa 98 conferences(international, national and regional) 79 publications

Positive spin-off effects Joint degree program with Southampton Solent University 11 PhD researches (Samara, Moscow) 40 MA dissertations (Kazan, Samara, Moscow) Participation in Phase 2 and Phase 3 of the Validation of Extended Set of CEFR Illustrative Descriptors (Mediation)(Samara) Russian –English Dictionary of Testing and Assessment terms( Vladivostok) Membership in EALTA(Samara, Kazan, Nizhni Novgorod, Surgut), ALTE (Kazan)

Positive spin-off effects English Language Olympiads as alternative forms of assessment: For secondary school children (Moscow, 2014, Surgut,2015) For university students (Inter-regional, organized by Samara University, 2015, in cooperation with Cambridge Assessment and Cambridge University Press) – 940 students from Almetyevsk, Astrakhan, Kazan, Moscow, Orenburg, Rostov, Samara, Syzran, Togliatti, Ulyanovsk)

Conclusion The Tempus-ProSet project has proved to be effective and sustainable to have a positive impact on the development of the National Qualifications framework in Russia to develop sustainable inter-university co-operation Participants of the project have become agents of change: they have contextual and collaborative expertise they empower in-service and pre-service teachers with the assessment literacy skills thereby pursuing continuous improvement of the assessment- based accountability systems in the Russian education they initiate ‘bottom –up’ cascading effect changes that are likely to lead to systemic change in the Russian educational system

Thank you Ludmila Kozhevnikova