Evaluation culture in SEE (Public procurement in SEE innovation evaluations: A comparative and needs assessment study; Lena Tsipouri – Nikos Sidiropoulos University of Athens, Centre of Financial Studies December 12, 2013, Zagreb

2 Outline  Does a comparative study and benchmarking make sense (objective/subjective indicators)?  Methodological remarks (literature; research)  Experiences from the EVAL INNO countries  A Synthesis  Lessons from intelligent benchmarking

3 Does a comparative study and benchmarking make sense The background of the countries studied differs considerably (Austria-Greece-Hungary-Bulgaria / Montenegro-Serbia) The legal framework is hardly applicable to the size of the procurement studied Objective indicators (measurable such as number of evaluations; budgets; frequency, but also qualitative regarding quality and response) are limited Subjective indicators can be obtained but are subject to criticism (the solution is stakeholder approval)

4 Methodological remarks (literature ) Common rules for public procurement, as a tool of the Single Market (contracts over ; Open procedures, Restricted procedures, Negotiated procedures and Competitive dialogue). Usually RTDI Evaluations fall in the budget range of Euros There is no real academic literature; There is a broad number of tender documents and terms of reference available, mostly (but not exclusively) originating in procurement by the European Commission; Information of the later stages of the procurement, namely monitoring of the evaluations and their acceptance, is practically absent;

5 Methodological remarks (research) Borrow notions for benchmarking from the pverall literature on public procuremnt Quality provision of the service Knowledge of the international standards and good practices Knowledge of the local needs and capabilities Knowledge of international capabilities and interest to respond to national tenders (price and reputational issues) Good and timely decisions

6 Public procurement of RTDI evaluations: our approach Decompose the process: Identifying the requirements and user readiness Market intelligence Tendering process (Terms of reference: background, data availability, questions and methods) Assessing tenders and awarding contracts Managing contract delivery Response to recommendations

7 The basic dimensions The institutional set up (formal and informal rules) Key organisations involved Tendering process

8 The institutional set up (formal rules) The budget thresholds for general provisions for public tendering The existence (or not) of special provisions for RTDI evaluations (e.g. specific thresholds; individual selection procedures etc.) Explicit legislation (or not) regarding the legal obligation of awarding authorities to evaluate their programmes or organisations. The existence (or not) of evaluation standards

9 The institutional set up (informal rules) Relevant parameters for launching tenders (strategy issues): Frequency Type of evaluations

10 Key organisations involved Awarding Authorities (how many, how good, how can they improve) Evaluators (local, national, international; issues of independence, expertise and reliability for evaluators called for direct or restricted tenders; how good, how can the market evolve) Other stakeholders (exercise pressure for RTDI evaluations)

11 Tendering process Terms of Reference (how good they are/could be) Smooth process (no legal or other complications) Time to contract (benchmarks) Monitoring (hands on or off?) Content (how ambitious are the Terms of Reference?) Adoption of recommendations (of the specific evaluation and more in general)

12 Type of parameters

13 Type of parameters

14 Experiences from the EVALL-INNO countries  Measuring/subjective rating per parameter decomposed  Comments per parameter  Comments per country

15 An example of experiences from the EVALL-INNO

16 Experiences from the EVALL-INNO countries The European Commission defined PRAG – Practical Guide to Contract Procedures for EU External Actions

17 Experiences from the EVALL-INNO countries Smooth process Time to contract MonitoringContent Adoption of recommendations Comments per country AustriaYes***Good/variableVariable60% Implementation is smooth but can be further improved BulgariaYes***Limited/variableStandard40% Need to improve monitoring, content of the ToR and relevance of recommendations GreeceYes*LimitedStandard20%“ HungaryYes**Limited/variableStandard40%“ MontenegroYes**LimitedStandard30%“ SerbiaYes**LimitedStandard30%“

18 A topic for benchmarking

19 What next: Do countries want to incorporate the benchmarking lessons (what should be their priorities; what is their distance to top; who are the top performers; from whom to learn)? Should/could the EU play a role in taking the (obvious but documented) results a step further? If yes how? Are other stakeholders interested in the results (to be read horizontally or vertically)

Thank you for your attention!