Learning from Existing Evaluation Practices on the Impacts and Effects of Intellectual Property on Development Geneva 6th/7th October 2011 Evaluation Section Internal Audit and Oversight Division (IAOD)World Intellectual Property Organization (WIPO)

Using Facilitated Self-Evaluations for Intellectual Property Projects Daniel P. Keller, Director Swiss Consulting Co. Ltd. Hanoi – Vietnam Management & Development Consultants

Professional experience in technical cooperation Evaluation experience in the field of IP focuses mainly on national projects that use a comprehensive approach of strengthening different aspects of IPRs as a tool to promote social and economic development. One project evaluated used a regional approach: Strengthening a regional IPR system in, combined with support at national level to Least Developed Countries (LDCs) within ASEAN. Designed 5 IPR projects (for different national contexts, including 2 pilots to respond to Art. 67 TRIPS agreement (TA to LDCs). Beyond this, over 30 other evaluations, mainly in the area of trade related technical assistance, many of which also include IPR aspects within a broader approach to trade capacity building, SME development, etc. Extensive own experience in managing projects.

Introduction My presentation will introduce the methodology applied as well as the key lessons learned from the facilitated self-evaluation of the Swiss-Vietnamese Intellectual Property Project (SVIP). SVIP: 2007 – 2010 (phase II), executed by the Swiss Federal Institute of Intellectual Property, budget of 1 million CHF, funded by Swiss State Secretariat for Economic Affairs, SECO. Purpose: share personal experience as a facilitator of this exercise. Why to use facilitated self-evaluations? What conditions need to be in place? Explain the self-evaluation process and steps used Highlight key lessons learned identified by the self-evaluation report Briefly discuss future trends in evaluation Draw conclusions and provide some recommendations.

Why to use self-evaluations instead of external evaluations? Self-evaluations are a capacity building process. Partners learn to assess whether the project did the right things and did things right rather than merely reporting on implementation of activities. Stronger ownership of evaluation results. This increases likelihood that partners apply recommendations and lessons learned in practice. Particularly important for countries that shift from the role of an aid recipient to providing themselves technical assistance. While external evaluations might provide a more independent, fresh view on the project, self-evaluations benefit from detailed and specific insight on the project and its context. Self-evaluations are a cost effective alternative to external evaluations: Funding for external in-depth evaluations of small projects is often not available, so they are not evaluated at all.

What conditions need to be in place for successfully conducting a self-evaluation? Only in a relatively advanced development context: Partners should have a certain experience in implementing technical cooperation projects that enables them to make meaningful contributions to the self-evaluation. Prior capacity building in monitoring & evaluation essential. The methodology is only suitable if project design & monitoring tools have been properly applied (logical framework, result-based financial and operational reporting). Reconstituting an intervention logic ex post (method typically used in evaluations of projects that did not properly apply planning tools) would not be feasible. Good partnership among partners/stakeholders. Self- evaluations require mutual trust. They are not suitable for problematic projects (e.g. with a conflict situation).

Self-evaluation process and steps used Workshop with partners on monitoring & evaluation. Establishing factual basis: For each of the performance indicators in the logframe, partners were required to assess expected against planned outputs and outcomes (using OVIs) in the form of a table. Partners were provided with a simple evaluation framework for rating relevance, effectiveness, efficiency and sustainability for each of the expected outputs/outcomes (high/medium/low) on a scale plus a brief explanation for their assessment. Draft findings, conclusions and recommendations (report) were discussed at a stakeholder workshop with all direct/indirect beneficiaries. Purpose to independently validate evaluation results. Report was discussed by the Steering Committee, amended based on feed-back and then formally approved.

Key lessons learned on IPR projects derived from the self-evaluation of the SVIP (1) Developing well-functioning IPR systems in developing countries calls for a comprehensive, coordinated support, addressing all subject matters in parallel and combining the strengthening of demand/supply side of IPRs in addition to the legal and regulatory framework. The three-pronged approach used under the SVIP (strengthening legal framework, IP administration and institutions that promote IPRs among key users) was successful. Sustainability of capacity building requires institutionalizing training functions rather than only train-the-trainers. Follow- up trainings organized by training divisions of local counterparts contributed to strengthening staff training within institutions in a sustainable way. Projects should decisively shift away to provide direct training at the level of IPR users.

Key lessons learned on IPR projects derived from the self-evaluation of the SVIP (2) Decentralizing day-to-day management of the SVIP to the field level, while strengthening financial and operational monitoring was the right management structure (seen as a key success factor for effectiveness and efficiency of implementation). Joint-implementation of technically complex projects allows the project partners to contribute what they are best at. This is a good way to ensure ownership, capacity building and sustainability without compromising on aid effectiveness. Selecting the right key management staff for projects: Important are leadership skills, result-orientation, ability to move the project forward, network, and knowledge of the country. Added benefits, but not crucial, were technical in-depth knowledge about IPR and seniority.

Key lessons learned on IPR projects derived from the self-evaluation of the SVIP (3) Project management and technical cooperation capacities of implementing agencies crucial: The SVIP also showed the importance that implementation agencies strengthen their specific capacities/know-how in technical cooperation and project management capabilities in the context of a developing country. IPR specific know-how is important, but not sufficient. Physical presence of the implementing agency in Vietnam was a key success factor (partners compared the SVIP with other project managed from outside the country).

Future trends in evaluation (1) Trend to strengthen the monitoring & evaluation framework for projects, starting at the preparation stage. Without a proper framework and a result-based operational and financial reporting system, evaluations suffer from considerable limitations! Pay attention to this during the appraisal stage for new projects. Still weak in many IPR-projects, but increasingly looked at by donors and beneficiary governments. Trend towards a more participatory approach to monitoring & evaluation: Increased stakeholder involvement (self-evaluation combined or not with external evaluation) is one way to respond to this. This is in line with international efforts to strengthen the role of partner countries in project implementation (monitoring & evaluation are a part of this).

Future trends in evaluation (2) Trend towards a more regular, external or internal result- oriented monitoring (ROM) of projects: Evaluations come often at a stage where it is too late to address fundamental problems. ROM is more suitable as a basis for timely strategic decision making, because the picture provided is more up-to-date. ROM could replace mid-term evaluations and feed into final evaluations. EU applies external ROM and Switzerland starts piloting a modified version of this approach for more complex projects. Shift towards assessing impact- rather than only output/outcome level 2 – 3 years the after project ends. Evaluations often come too early for assessing impact and partially also sustainability. This requires impact indicators in the logical framework and the collection of relating baseline data. External impact assessment of SVIP by SECO is planned.

Conclusions and recommendations Self-evaluations respond well to the Paris Declaration on Aid Effectiveness, by increasing role (ownership) of partner countries, enhance mutual accountability, contributing to managing towards results and reducing transaction cost (aid effectiveness). We advocate for the use of facilitated self-evaluations as a tool of organizational learning in a more advanced development context, where project partners have a certain degree of experience and a successful track-record in technical cooperation. Conducting facilitated self-evaluations might be an alternative for smaller WIPO-Projects, for which the cost of an external evaluation would not be commensurate to the overall project budget. They could be used instead of a final evaluation for projects that subsequently undergo an impact assessment.

Thank you for your interest! Questions and comments Discussion Daniel P. Keller, President Swiss Consulting Co. Ltd. Hanoi – Vietnam Management & Development Consultants