THE OECD APPROACH TO ASSESSING ORGANISATIONAL EFFECTIVENESS Frank van Tongeren Head of Division, Policies in Trade and Agriculture (OECD) ICAE pre-conference workshop: “Measuring the value of policy oriented research” Milan 8 August 2015 Session SE 01-06, 08/08/2015, room 311

End-user survey In-depth evaluation of committees Monitoring effectiveness of communication Outline 2

Organisational effectiveness and impact are important dimensions of the OECD’s approach to delivering Value for Money for our Member Governments. Online end-user Survey is part of assessment and reporting –The Survey obtains feedback on the Quality, Impact and Usage of OECD knowledge-based outputs Survey is completed by policy makers in Member capitals, and each Member country determines their list of invitees. –As some of the OECD’s products are intended for Non-member end-users, a separate Survey will be undertaken in 2015 for selected Non-members of the OECD. Assessing Organisational Effectiveness at the OECD 3

4 Survey Scope End-user Survey (conducted in 2015) … All relevant ‘Output Results’ (i.e. groupings/ packages of knowledge-based products) were included in the Survey Knowledge-based Products included in the Survey … … End-users responded to the Survey, across all 34 Member countries and the European Commission Output Results rated by at least 25 Members … … Output Results rated by at least 17 Members

End-users enter the online Survey, select OECD products that they have used, and: –provide a rating for each of Quality and Impact on a five-point scale (1=Very Low … 5=Very High); – Indicate Usage from a hierarchy of statements. Overall results indicate that OECD products are consistently of at least High Quality and most have at least a Medium to High Impact. Usage provides useful insights into the influence of the OECD in policy reform over time. Quality, Impact and Usage 5

Not all of the OECD’s products are going to be used by all of the Members, all of the time. –Policy reform cycles will vary across sectors and across Member countries. The OECD Secretariat is responsible for delivering outputs on time, on budget, and to a high level of Quality. Member Governments are responsible for the use and impact of those products. Important to Note 6

OECD Outputs – Overall Usage by Member Countries 7

Over the and biennia, Member countries used Agricultural Policy Monitoring and Evaluation products in the following ways: Example: Usage of Agricultural Policy Monitoring and Evaluation products Usage RatingNumber of Members Overall As a basis for policy change235 As a reference (data or analysis) for policy discussion Has potential for authoritative use in policy development 379 8

Members Using OECD Agriculture Products as a Basis for Policy Change 9

Members Using and OECD Agriculture Products as a Basis for Policy Change 10

Members Using and OECD Agriculture Products as a Basis for Policy Change or Used as a Reference in Policy Discussions 11

Members Using and OECD Agriculture Products as a Basis for Policy Change, Used as a Reference in Policy Discussions, or indicating Potential for Future Use 12

End-user survey In-depth evaluation of committees Monitoring effectiveness of communication Outline 13

In-depth Evaluation (IDE) - overarching objective The objective of IDE is to provide a mechanism through which Council can assess whether OECD Committees are… – conducting processes… – delivering outputs… – achieving impacts… Frequency – about every 5 years 14 … in line with Members’ policy expectations and with the OECD’s comparative advantage 14

Evaluation Methodology Relevance Effectiveness Efficiency Assessment (Quantitative and Qualitative analysis per evaluation criterion) End user Survey: Quality; Impact; Relevance Survey priority members Delegate attendance in committee meetings Interviews with policy makers in capitals Output & budget reporting Committee documents Interview with Stakeholders Recommendations (following the good practices framework) 15

 Committee performances are ranked on the following scale: very low, low, medium, high or very high.  Lessons are also drawn with the aim of supporting improvements in performances across OECD committee structures.  Results of Initiatives launched; Reviews and interviews with delegates on the implementation Conclusions and Good Practices 16

Monitoring the implementation of recommendations 17 Council approves Recommendations Committee develops action plan EVC reviews action plan Committee implements action plan EVC reviews progress made in implementing action plan Council reviews progress made in implementing action plan IDE + 6 months IDE + 12 months Council dialogues with Committee Chair IDE + 18 months 17

Committee for Agriculture (i.e. the Level I body) and the following sub-bodies: –Working Party on Agricultural Policies and Markets (APM) - created in 1987; –Joint Working Party on Agriculture and Trade - created 1962; –Joint Working Party on Agriculture and the Environment - created –Group on Commodity Markets which reports to the APM. - created in 2009 December 2014 – January 2016 Many sources: including end-user survey, policy makers in capitals, delegates and chairs of bodies, outside users and partner IOs etc. IDE of our work on agriculture 18

End-user survey In-depth evaluation of committees Monitoring effectiveness of communication Outline 19

The goal of measuring communication statistics is to evaluate the effectiveness of our actions. Why communicate if no one is listening? We are able to quantify the effectiveness of our communication using tools across all channels including , web, social media and press Measuring communication outcomes 20

Analysing the click-through rates towards content from s sent to mailing lists, comparing them to known industry averages Measuring the impact of campaigns by analysing web traffic to specific pages and downloads from OECD iLibrary Evaluating the reach on social media of our messages and how and with whom they are shared Collecting press coverage following launches and events and reviewing the reach of the outlets Examples of quantitative measurements 21

Example: Agricultural Policy Monitoring & Evaluation – Campaign 22

For more information Visit our website: Contact us: Follow us on 23