1. Usefulness of participants’ reports in studying and analyzing outcomes and learning effects in Erasmus+.
Siru Korkala
29 Nov 2018

2. Example SCIENTIFIC MONITORING „APPLIED METHODS OF IMPACT ASSESSMENT“
Showing and Identifying Impact of Erasmus+ on EU and National Level
TCA
Estonia, Finland, Hungary, Iceland, Netherlands, Norway, Sweden, Slovenia and Austria
31/07/2019
Finnish National Agency for Education

3. The data and aim of the model
Erasmus+ KA1 Mobility in VET, Participant’s reports from 9 countries
Learners and staff
Aim: to develop an impact model for ERASMUS +, illustrating the effects of the program for learners, teachers and trainers and educational institutions
31/07/2019
Finnish National Agency for Education

4. Impact assessment model MIA
Methodological Approach
Selection of Questions
Scaling and transfer of answers to scales
Calculation the sub-indicators for issues selected
Calculating the overall indicator
The questions are selected to the themes
Correlation analysis: to assess the correlation patterne between all indicators
Confirmatory factor analysis: to test for uni-dimensionality and meassure the indicators ability to explain the variance of the theretical concept
Cronbach's alpha: a single paramater meassuring to what extent a group of indicators are meassuring «the same thing» - assuming unidimensionality
Six major themes:
Competence
Innovation
Employability
Professional development
System improvement
EU-citizenship
31/07/2019
Finnish National Agency for Education

5. Country comparison: all topics 2014 + 2015 (1 strongly disagree – 5 strongly agree)
31/07/2019
Finnish National Agency for Education

6. Challenges
The empirical data are given – vi can not design a new measurement instrument but will have to rely on existing data
The values reflect the participants' assessment of the effects of mobilities on the selected topics - no "objective" measures or empirically verifiable actual effects!
Benefits
Nevertheless reliable measure of individually perceived or expected effects of mobilities, due to the high response rates
Longitudinal comparison of indicators (and sub-indicators)
Comparison between participating countries
In principle also analysable in a greater disaggregation: level of individual questions, socio-economic characteristics (such as gender, nationality, age, type of vocational training, level of education), characteristics of the sending or receiving institutions (country, region, type of institution), implementation characteristics ( Mobility type, duration of mobility)
31/07/2019
Finnish National Agency for Education

7. Participant reports as a research data
General remarks
Some questions are asked too soon after the mobility period
Questions about the effects
Easier for staff to answer right after the mobility period
Questions about administration etc.
Open questions gives more information than statements
31/07/2019
Finnish National Agency for Education

8. Advantages
Comprehensive data on user experience from the biggest education cooperation program in the world.
Official data from E+/Commission
National monitoring
Comparative data between countries and between different actions and levels of education
Comparative data that have not been explored – interesting to find its potential
One important data source that should be used together with other relevant sources (administrative data, surveys, interviews, observation, expert evaluation…)
Important information for educational institutions (succeeding)
Institutions can promote themselves
31/07/2019
Finnish National Agency for Education

9. Disadvantages
Extensive (too long) questionnaire, not adapted to different target groups
Some questions and concepts too abstract for young people
Little control on data collection process/non existing guidelines
Danger of biased self reporting due to economic binding
Yearly changes to the questionnaire
Not possible to add any national questions
Scale-variables are as text, must be recoded
31/07/2019
Finnish National Agency for Education

10. Improvement ideas
Commission could produce more country specific comparison
Would enhance the service that NA provides to educational institutions
More clear questions / one issue per question
31/07/2019
Finnish National Agency for Education

11. Thank you