Rolling Review of Education Statistics Working Group Meeting “Education and training statistics” Luxembourg, 23 May 2011 Antonio Consoli, Eurostat Unit B1 23-May-2011

Background information Definition of rolling reviews Objectives of a rolling review Identify strengths and weaknesses of the process Help to improve the quality of the process Report on achievements of the process Value added from Rolling Reviews Assessment of the process from a different angle Independent expert assessment External assessment of data quality Tools for rolling reviews 23-May-2011

Rolling Review of Education Statistics Fit within the Eurostat quality assurance framework Focused mainly on CVTS and UOE Included Self-assessments from quality review of CVTS and UOE Global user satisfaction survey via the web-site (74 answers) Partner satisfaction surveys to all MS, Candidates and EFTA (27 answers for CVTS, 23 for UOE) Final result: improvement actions and recommendations 23-May-2011

User satisfaction survey (1) Respondents NSA (28), government (22), university or research (17), private (3), business (2), other (2) Users were asked about: Users themselves Purposes they use data for Relevance of data for their work Other data sources they use except Eurostat Quality of data Quality of the service provided by Eurostat 23-May-2011

User satisfaction survey (2) Users’ opinion is generally positive about data quality Strengths: Education statistics are relevant for users, for all data collections UOE, AES and CVTS. Overall quality is rated high, especially for quality dimensions important for users (accuracy, clarity and accessibility of data). Users are very satisfied with Eurostat’s support services. Weaknesses: User needs more data across European countries and at a more detailed level. All UOE data published by OECD should be made available by Eurostat. Many users consider that data on IVT have limited comparability over countries and are the least important for them. In addition, the timeliness of CVTS results needs to be improved. Users are less satisfied with the coherence and the spatial and temporal comparability of data, due to the differences in the national methodologies and the frequent changes in the survey concepts. 23-May-2011

Partner satisfaction surveys Partners were asked about: Planning and coordination structures in the ESS Functioning of the ESS Direction of change Eurostat’s role in the field of data production Problems with the data production and suggestions Assessment of data quality Resources used to produce the data and for CVTS response burden on enterprises 23-May-2011

Partner satisfaction survey (CVTS) Strengths: Very positive perception of Eurostat’s role in the preparation of the CVTS 4 and satisfaction with TF meetings. It is anticipated that CVTS 4 will be much better than CVTS 3. The relevance of all or part of the information collected is high. Even if the Commission Regulation did not require them to carry out this survey, more than half of the reporting countries would continue it. Data accuracy of the two main CVTS indicators (Percentage of training and non-training enterprises and CVTS strategies and difficulties) and coherence and completeness of CVTS 3 statistics disseminated in Eurostat’s web site are satisfactorily appreciated. Weaknesses: High burden imposed to the enterprises and the length of the questionnaire. Some CVTS concepts are difficult to understand by respondents, in particular the IVT concept should be removed from future collections. High cost, both in terms of human and financial resources. It’s the reason why some countries would stop it without the regulation. Need to make CVTS micro-data available for analytical purposes and lack of sufficient documentation that accompanies the published CVTS 3 figures. Timeliness of CVTS 3, as well as the accuracy of some indicators are rated below adequate. Part of the problem can be in the shortage of human resources inside Eurostat. 23-May-2011

Partner satisfaction survey (UOE) Strengths: Partners are very satisfied with planning and coordination structures in the ESS, the functioning of the ESS in this field and the role of Eurostat in the UOE data production. Partners do not perceive that the problems they face with the UOE data production are serious or major problems. In particular the joint administration of the UOE data production by UIS/OECD/Eurostat is not perceived as a source for problems. Partners have been following closely the development in various related topics of the UOE and they are mostly satisfied with recent developments on topics that are most important to them (i.e. ISCED revision and the Commission Regulation). Partners anticipate improvements, especially in the most ‘problematic’ quality dimensions like timeliness and comparability across countries, with the implementation of the new Commission Regulation. Weaknesses: Need for communication of best practises. Difficulty to find all the requested information and at the detail it is requested. As a consequence, these data need to be estimated to a large extent. The topic ‘Modernization of the UOE expenditure data’, of high importance to more than half of the partners, should be given more attention by the three organizations. Timeliness and comparability across countries are the quality dimensions with which partners are least satisfied (heavy burden to validate and prepare data for so many countries and the lack of resources inside Eurostat, methodology not harmonized enough among countries). 23-May-2011

Checklist The Eurostat Statistical Processes Assessment Checklist is a self-assessment tool, addressed to the production unit and covering the quality of the production process and of the statistical output. It includes information on: Users Data Providers Validation (by the national data providers) Validation (at Eurostat level) Statistical confidentiality Documentation (at Eurostat level) Data dissemination Follow-up of the statistical production process IT conditions Management, Planning and Legislation Staff, work situation and competence Conclusions can be cross checked with opinion of partners and users 23-May-2011

Checklist (CVTS) Strengths: Weaknesses: Users’ interest, relevance of the statistics. Based on regulations. Completeness of data and of information on methodology and quality. Efficiency of data validation at Eurostat. Current IT application used at Eurostat is reliable enough, integrated and shared. CVTS production team is well organized. Weaknesses: Insufficient available resources at Eurostat for further development and improvement of the production process. Eurostat does not have the necessary resources to tackle the unmet needs of the users, especially concerning timeliness that is affected by delays at various stages. Lack of availability of anonymized micro-data. Heavy burden imposed to the data providers which also results in high unit non-response (not corrected). More resources would be needed for validation of both primary and secondary variables. Need to further enhance the completeness of the publications of quality reports. The IT system used for the CVTS 3 is considered reliable but lacking flexibility and stability. 23-May-2011

Checklist (UOE) Strengths: Weaknesses: Establishment of a regulation coinciding with revision of ISCED Good contact with users Limited burden on respondents Low impact of validation and revisions Good process documentation Some balance between the available human resources and the work required Weaknesses: Need for more explanations on the data disseminated and more information on quality available to users Problems with timeliness and missing values in data delivery (addressed in the regulation) Validation at Eurostat needs to be made automatic Timeliness, Availability of metadata and Completeness need to be improved Current IT application at Eurostat not satisfying 23-May-2011

Recommendations (CVTS, 1) Data production: Reduce the length of the questionnaire with the ultimate aim to reduce response burden (L) Further work on the harmonization of methods and definitions (also with other surveys) (L) Improve the flexibility and stability of the IT applications used for data validation (S) Improve and update regularly the internal documentation about CVTS production (L) Consider a revision of the CVTS regulation (L) Improving data and metadata dissemination: Disseminate the anonymised micro-data based on agreed criteria (S) Improve timeliness of the data release (M) Improve the content of ESMS metadata (M) 23-May-2011

Recommendations (CVTS, 2) Improving data quality: Improve timeliness and coherence (M-L) Improve comparability between countries (L) Other matters: Assess resource needs for CVTS data production inside Eurostat (S) Assess the benefit from the use of alternative sources for CVT on the frequency of data release and the burden on statistical authorities (L) 23-May-2011

Recommendations (UOE, 1) Data production: Increase the sharing of knowledge between partners (national and best practices) (S) Continue with the revision of the production process in line with the ISCED revision (L) Reduce changes in the design of the data collections from year to year (L) Assist data providers with the collection of the requested information (M) Provide more information about UOE estimation practices (including examples) (M) Boost the interest of both partners and users on the Bologna process indicators (M) Allocate more effort in the developments on the ‘Modernization of the UOE expenditure data’ topic (but no application on the 2010 ESSnet call for proposal on lifelong learning accounts) (L) Consider reviewing the UOE tables used for data transmission (L) Improving data and metadata dissemination: Improve timeliness of the data release (M) Improve the content of ESMS metadata (M) Improve the functionality of Eurostat’s website (look at OECD) (S) 23-May-2011

Recommendations (UOE, 2) Improving data quality: Improve timeliness and clarity (M-M) Improve coherence with National Accounts data (L) Other matters: Communicate to the Member States the scope of the data collection (M) Communicate to the Member States the deadlines of the data collection (M) Assess resource needs for UOE data production inside Eurostat (M) Consider migration to another IT tool than Oracle/OLAP (M) 23-May-2011