1. Independent IOD and CDIP Project Evaluations - An external perspective WIPO 2016 Evaluation Seminar 29 January Glenn O’Neil Evaluation consultant

2. Use Methodology Implementation Findings Evaluation
This is a simplified diagram of the evaluation process
Working in a cyclical way; each step leading into the next Ultimately the whole experience should feed into improved methodology, etc.
I will use this to make some reflections on evaluation in WIPO

3. Methodology Use Implementation Findings Procedures in place
Evaluation
Implementation
Findings
Procedures in place
Variety of methods
Pre-post designs
Largely inclusive
Reaching stakeholders
Isolating WIPO’s contribution
Often direct & intended use
Input into program design/direction
Here I’ve now highlighted some points that I’ve noticed about evaluation in WIPO – per step Methodology:
Procedures in place that are largely respected
A variety of methods can be used; often programs/projects collect interesting data (n.b. need for monitoring..)
90% of evaluations are post only – with WIPO programs/projects, if prepared well, some pre-data can exist, allowing for a stronger design
Evaluations at WIPO are inclusive – stakeholders and staff do participate
Implementation
Reaching all relevant stakeholders is often difficult; also given that WIPO is often 1 or 2 removed from beneficiaries
Difficulty in analysing data to isolate WIPO’s contribution – often given that WIPO activities (and IP in itself) are part of a larger “package” that relies on many different inputs to “succeed”
Findings
Feedback is manageable in that there are not 100s of people commenting on drafts (in my experience..)
Evaluation reports are received in a relatively structured way – CDIP for example
Use
Often seen that reports are used directly and what they were intended for
Program/projects do use findings as an input into program/project design
Manageable feedback
Structured reception

4. Evaluation policies & institution
Field
Program design & management
Organisational setting
Funding
People
Use
Methodology
Evaluation
Implementation
Findings
However.. Evaluation doesn’t occur in isolation – here I’ve introduced what I identify as the main influences on evaluation
These factors can be both enabling and hindering – depending upon the situation
Context

5. Evaluation policies & institution
Field
Program design & management
Organisational setting
Funding
People
Use
Methodology
Evaluation
Implementation
Findings
What I highlight in bold are the three influences that I see as the most important (in WIPO and elsewhere)
Note – they are all internal factors and therefore “possible” to control (to a certain extent…)
- Evaluation policies and institutions – are key to enabling evaluation in an organisation; WIPO has built its institution over time and its policy is flexible (and not “dogmatic”, e.g. it doesn’t enforce a given method/approach) - Organisational settings – the notion of culture is in important and elements such as RBM and CDIP have encouraged M&E in WIPO - Program design/mgt – the acceptance of M&E by programs is key; incorporating monitoring elements, indicators, baselines, measurable objectives – this we see starting to happen in WIPO
Context

6. Evaluation policies & institution
Field
Program design & management
Organisational setting
Funding
People
Use
Methodology
Implementation
Findings
Isolating WIPO’s contribution
In this final slide, I highlight areas where I think improvements could be seen: five points:
Link between evaluation policy and program - programs need to ensure that they adopt the minimum (and more!) required for M&E, monitoring, etc. and staff have time and resources to be involved
Funding needs to be budgeted for by programs for evaluation
3. Isolating WIPO’s contribution – deserves more thought – does the evaluation field (academic, prof. associations, evaluation community) have methods to tackle this better? Such as contribution analysis, comparative studies (e.g. country with and country without WIPO support), etc.
People – participation of WIPO staff in evaluation design/TOR and inputting at all steps is important – must be kept manageable – but link clearly shown between participation and use
5. Use – use of findings can be seen directly in WIPO – however, these “squiggles” show other ways that use occurs;
It can be unintended and direct
It can be unintended and indirect
It can be intended and indirect
It can be intended, indirect and not occur
Not an improvement per se - just to be aware that evaluation reports can be used in unintended and indirect ways….
Context

7. Thank you! @glenn_oneil Contacts: oneil@owlre.com glennoneil