Independent IOD and CDIP Project Evaluations - An external perspective WIPO 2016 Evaluation Seminar 29 January 2016 Glenn O’Neil Evaluation consultant oneil@owlre.com www.owlre.com

Use Methodology Implementation Findings Evaluation This is a simplified diagram of the evaluation process Working in a cyclical way; each step leading into the next Ultimately the whole experience should feed into improved methodology, etc. I will use this to make some reflections on evaluation in WIPO

Methodology Use Implementation Findings Procedures in place Evaluation Implementation Findings Procedures in place Variety of methods Pre-post designs Largely inclusive Reaching stakeholders Isolating WIPO’s contribution Often direct & intended use Input into program design/direction Here I’ve now highlighted some points that I’ve noticed about evaluation in WIPO – per step Methodology: Procedures in place that are largely respected A variety of methods can be used; often programs/projects collect interesting data (n.b. need for monitoring..) 90% of evaluations are post only – with WIPO programs/projects, if prepared well, some pre-data can exist, allowing for a stronger design Evaluations at WIPO are inclusive – stakeholders and staff do participate Implementation Reaching all relevant stakeholders is often difficult; also given that WIPO is often 1 or 2 removed from beneficiaries Difficulty in analysing data to isolate WIPO’s contribution – often given that WIPO activities (and IP in itself) are part of a larger “package” that relies on many different inputs to “succeed” Findings Feedback is manageable in that there are not 100s of people commenting on drafts (in my experience..) Evaluation reports are received in a relatively structured way – CDIP for example Use Often seen that reports are used directly and what they were intended for Program/projects do use findings as an input into program/project design Manageable feedback Structured reception

Evaluation policies & institution Field Program design & management Organisational setting Funding People Use Methodology Evaluation Implementation Findings However.. Evaluation doesn’t occur in isolation – here I’ve introduced what I identify as the main influences on evaluation These factors can be both enabling and hindering – depending upon the situation Context

Evaluation policies & institution Field Program design & management Organisational setting Funding People Use Methodology Evaluation Implementation Findings What I highlight in bold are the three influences that I see as the most important (in WIPO and elsewhere) Note – they are all internal factors and therefore “possible” to control (to a certain extent…) - Evaluation policies and institutions – are key to enabling evaluation in an organisation; WIPO has built its institution over time and its policy is flexible (and not “dogmatic”, e.g. it doesn’t enforce a given method/approach) - Organisational settings – the notion of culture is in important and elements such as RBM and CDIP have encouraged M&E in WIPO - Program design/mgt – the acceptance of M&E by programs is key; incorporating monitoring elements, indicators, baselines, measurable objectives – this we see starting to happen in WIPO Context

Evaluation policies & institution Field Program design & management Organisational setting Funding People Use Methodology Implementation Findings Isolating WIPO’s contribution In this final slide, I highlight areas where I think improvements could be seen: five points: Link between evaluation policy and program - programs need to ensure that they adopt the minimum (and more!) required for M&E, monitoring, etc. and staff have time and resources to be involved Funding needs to be budgeted for by programs for evaluation 3. Isolating WIPO’s contribution – deserves more thought – does the evaluation field (academic, prof. associations, evaluation community) have methods to tackle this better? Such as contribution analysis, comparative studies (e.g. country with and country without WIPO support), etc. People – participation of WIPO staff in evaluation design/TOR and inputting at all steps is important – must be kept manageable – but link clearly shown between participation and use 5. Use – use of findings can be seen directly in WIPO – however, these “squiggles” show other ways that use occurs; It can be unintended and direct It can be unintended and indirect It can be intended and indirect It can be intended, indirect and not occur Not an improvement per se - just to be aware that evaluation reports can be used in unintended and indirect ways…. Context

Thank you! @glenn_oneil Contacts: oneil@owlre.com glennoneil www.owlre.com www.intelligentmeasurement.net