Contribution Analysis A different approach to measurement of impact

The challenge Justify your programme’s impact

Typical metrics Number of students Length of courses Cost comparisons Benchmark against similar institutions … But do these show impact?

Options Call for an evaluation –Usually time consuming and one off –Often struggles to get to level 4 of the Kirkpatrick Evaluation

Possible approach Set up regular measuring to: Paint a picture about the attribution / contribution to corporate goal routinely provide an indication on efficiency

Contribution Analysis “ Addressing Attribution Through Contribution Analysis: Using Performance Measures Sensibly ” John Mayne Office of the Auditor General of Canada June 1999

WMO and Contribution Analysis 2007 UN commissioned a consultant to recommend a process to examine impact of fellowships 2008 UN agreed to use the “Contribution Analysis” approach 2009 WMO commissioned a study on the impact of the WMO Fellowship Programme

Contribution Analysis Acknowledge the problem Present the logic of the program. Identify and document behavioural changes. Use discriminating indicators. Track performance over time. Discuss, and test alternative explanations. Gather additional relevant evidence. Gather multiple lines of evidence. When required, defer to the need for an evaluation.

Financial Resources (WMO Regular Budget, VCP, Trust Fund) WMO Secretariat Staff (Education and Fellowship Division, BO, Finance, travel…) NMHS Human Resource development needs 1Pre-screening of incoming requests 2Consideration by FELCOM 3Approval by SG 4Fellowship administration prior to, during and after the fellowship (payments, contact with host training institutions, PRs and fellows, travel arrangements,…) Efficiency and effectiveness of existing services is improved New services are being delivered by NMHSs New technologies/ methods/practices/ procedures are being introduced in NMHSs Fellows’ career within the services is enhanced Fellows use their acquired knowledge upon return to their services Successful completion of studies Fellows are satisfied with their studies Fellows have acquired new skills/knowledge in relevant fields Programme Inputs Programme Activities Programme Outputs Programme Outcomes 1 Programme Outcomes 2 Programme Impact (at national level) PRs are satisfied with the trained fellows (meeting NMHSs’ HR development needs) Assumptions: Fellows continue their employment within the NMHS (retention); PRs facilitate fellows’ reintegration into the services and make use of their new knowledge and skills Assumptions: Fellowship administration procedures and practices are efficient; selection criteria are clear and are applied consistently; only qualified candidates are selected; only those requests are approved which satisfy a genuine HR development need within the NMHSs Assumptions: Fellows continue their employment within the NMHS (retention); PRs facilitate fellows’ reintegration into the services and make use of their new knowledge and skills Assumptions: Requests for fellowships are made by PRs based on the HR development needs of their services; suitable candidates are suggested for fellowships; sufficient WMO secretariat staff resources are available to deal with fellowship administration; sufficient funds are available to offer WMO fellowships.

Summary WMO using contribution analysis for assessing the impact of the Fellowship Programme The logic map has assisted us to identify the key steps in the fellowship approach and the things we need to monitor and watch We need to devise and test the discriminating indicators Look for an update in 12 months