1. ERA Roadmap – Monitoring and Indicators
Luxembourg 6-8 July

2. Roadmap Monitoring - mandate
“INVITES ERAC to propose by the end of 2015 a set of core indicators and, where appropriate, qualitative methods allowing to monitor the implementation of the ERA Roadmap. STRESSES that the monitoring of the ERA Roadmap should be put in the context of the monitoring of the ERA progress and CALLS ON the Commission to consider possible integration of the monitoring of the ERA Roadmap into the ERA Progress Report 2016 and the following ERA Progress Report exercises, in close cooperation with the Member States, while avoiding creation of unnecessary administrative burden.”

3. Roadmap Monitoring- approach
Work closely with Commission, national experts, representatives ERA Groups and Stakeholder Groups
One short high level indicator per implementation priority ( so eight in all) to provide political focus. This should be linked to the priority area, not necessarily to the individual actions.
Should draw on information which is already available for all MS/Acs (Eurostat, databases, …)
In developing the list we need to be pragmatic and realistic; the ideal indicator may well not exist at the moment.

4. Part of a wider monitoring process
Recognize that a single indicator may not fully capture complex developments
Therefore ERA Groups are also encouraged to undertake further work to develop additional/more accurate indicators.
Reminder that this is just part of a wider ERA monitoring process linked to Progress Reports (debate with Com necessary) with much more qualitative and quantitative information

5. Outcomes so far
Fair degree of consensus on most indicators; in other cases still working on alternatives.
Some wider political messages as well (e.g. on data availability/collection cycles for Priorities 4, continued data collection in 5b)
Building on the outcomes there, we do further work with relevant ERA Groups and other experts to resolve the outstanding issues : nominator/denominator, data quality issues, plausibility of results,…

6. Summary Very advanced P1 (effect. Res. Syst) P2a (jointly…)
P3 (open lab market)
P4 (gender)
P5a (kt)
P5b (OA)
P6 (int coop)
More work needed P2b

7. Next steps Stakeholder meeting : 14 september
The Working Group will meet again on 28 September to agree a final set of indicators and wider political messages
This will go to the 13 November ERAC Plenary for endorsement.
Afterwards : further refinement work by JRC/Eurostat needed

8. Messages and debate Are you satisfied so far ?
How does this relate to wider ERA monitoring process : comments by delegates and COM requested
How can ERA related groups feed in this wider process ?
To ERAC : can we debate this in November ?

9. PRIORITY 1 (Effective national research systems)
Research Excellence Indicator, a composite indicator prepared annually by the Joint Research Centre (JRC) of the European Commission
4 components: Highly cited publications; PCT patents; ERC grants; Quality of universities & research organizations
Pro’s : Captures key aspects of research excellence, no country size bias, data available and tested by JRC
Con’s : composite indicator (4 numerators + 4 denominators), time lags for some components, number of top institutions often 0 in small countries
Recommendation on possible improvement to be tested by JRC/Eurostat/COM

10. PRIORITY 2a (Jointly Addressing Grand Challenges)
Share of national GBARD allocated to Europe-wide, bilateral or multilateral transnational public R&D programmes (other denominator ?)
GBARD directed towards : Europe-wide transnational public R & D programmes and bilateral or multilateral public R & D programmes established between MS or with EFTA and candidate countries
Pro’s : Reflects cooperation and budgetary efforts; input indicator showing the results of policy decisions
Con : May understate the "true" figure; no information on alignment to EU grand challenges; potential country size bias (countries with low GBARD will come out high)

11. PRIORITY 2b - Make optimal use of public investments in Research Infrastructures
Availability of national roadmaps detailing investments in research infrastructure
Pro’s : gives information on the investments at national level into research infrastructure
Con’s : we need more to measure yearly progress
(ideal indicator would be  Share of GBARD allocated to RI investment)
Help needed : can Commission (with the help of ESFRI) provide the data ? or can ESFRI suggest a better alternative ?

12. PRIORITY 3 (Open Labour Market for Researchers)
Open recruitment: Researcher posts advertised through the EURAXESS Jobs portal per thousand researchers in the public sector per year
Pro’s : directly related to priority action; will encourage institutions to make use of Euraxess; directly measures a country's (or at least its institutions) commitment;
Con’s : no absolute reference point (reflects changes in relation to all R&I posts available in country); some MS have national job portals preferred by their institutions;
Question for MS/AC : what portals do national systems use if they do not use Euraxess

13. ERA PRIORITY 4 (Gender Equality and Gender Mainstreaming in Research)
Proportion of women A grade in Higher Education Sector (HES)
Pro’s : corresponds well with the roadmap top priority; data are based on a known methodology;
Con’s : covers HES-based researchers only; data are provided only every three years; what is the optimum here? 50%? How to interpret results ?
Help from Commission needed : explore the possibility of producing the data every two years and the extension to other public sector research organisations

14. PRIORITY 5 a (Scientific knowledge transfer)
Innovative firms cooperating with (1) universities and HEI’s and/or (2) public research institutes
Pro’s : addresses knowledge transfer; readily available indicator; it is being part of the European legislation (innovation survey)
Con’s : some data issues to be resolved

15. PRIORITY 5b (Promoting Open Access to scientific publications)
Proportion of Open Access papers per country, The indicator shows the proportion of Open Access papers published in peer reviewed Journals at the European and World levels over the period based on data from the Scopus database.
Pro : A relative simple indicator to measure the proportion of Open Access papers per Country
Con : statistical problems and shortcomings have to be investigated as well as alternative data sources
Help from Commission needed : a systematic update depends on a contract by the Commission

16. PRIORITY 6 (International cooperation)
International scientific co-publications with non-EU countries per ‘000 researchers (of public sector, FTE)
Pro’s : proxy for assessing international activity and cooperation between countries;
Con’s : does not show the positive effects of national cooperation strategies per se, it just shows how researchers collaborate (which can be the result of many factors); country size bias and linguistic/location bias possible.
Help from Commission needed : depends on a contract with institution that can produce bibliometric indicators (cfr 5b)