1. DISCUSSION PAPER PREPARED FOR THE WHO’S DEPARTMENT OF HUMAN RESOURCES FOR HEALTH ON BEHALF OF THE UN TASK FORCE ON IMPACT ASSESSMENT OF FELLOWSHIPS BY ARIE ROTEM WITH THE SUPPORT OF MICHAEL A. ZINOVIEFF Impact Assessment of United Nations System Fellowship Programmes: An Evaluation Framework

2. Fellowships in the United Nations System “ A Fellowship in the United Nations System is a specially tailored or selected training activity that provides a monetary grant to a qualified individual for the purpose of fulfilling special learning objectives; such training,......., should be in response to nationally approved human resources policies and plans, and should aim at impact and relevance for all stakeholders involved.”

3. Observations concerning fellowship programs Variability in design and implementation Large investment of efforts and resources Insufficient evidence to support accountability Need to ascertain contribution beyond benefit to individuals Need to coordinate approaches Potential for strengthening fellowship system

4. Evaluation assumptions Evaluation aims to support decision making An integral part of all phases of planning and implementation Main purpose to improve rather than to prove Engagement of stakeholders essential for success

5. Key Challenges Conceptual logic: Why do we expect fellowship program to contribute? Attribution: What else may have caused observed outcomes? Fidelity: Did we follow the necessary steps? Modalities of fellowships: What kind of impact could we reasonably expect? Time frame: How long is it likely to take before impact could be observed?

6. Mapping the pathway A reliable “map” of the fellowship pathway, required to:  Monitor progress towards the desired destination  Manage road blocks and unintended consequences  Replicate successes and prevent mistakes

7. Theory based evaluation “ Theory or program based approaches map out the channels through which the activities, inputs, and outputs are expected to result in the expected outcomes. It also allows for the identification of unintended effects”. Inder Jit Ruprah, Office of Evaluation and Oversight, IADB.

8. Logic Pathway and Benefit Chain INPUTACTIVITYOUTPUTSUSAGEIMPACT Improved Outcomes For Clients Improved Capacity Improved Services Policies Plans Agreements Fellows Funds Selection Placement Training Mentoring M&E Satisfaction Gained knowledge and skills Behaviour change Use of gained skills Contribution to performance Bridging gaps

9. Contribution analysis An analysis of activities and interim outcomes that could reasonably be expected to contribute to a positive impact for the beneficiary institution. Confirming the logical link among these outcomes and excluding other known influences (Conceptual logic )

10. Reducing uncertainty Identifying outside factors not related to the fellowship that may have an effect ( Attribution) Ascertaining that the fellowship program has been implemented properly within a logical conceptual framework ( Fidelity)

11. Verification of fidelity Clear objectives for fellowship addressing agreed priorities Selection of appropriate fellows and training programs Appropriate support during training Opportunity to apply knowledge and skill Utilisation and support after return home Opportunity to contribute and to “echo” learning

12. Assess the existing evidence on results Set indicators and obtain evidence to verify achievement of the desired results at each level ( Monitoring key milestones) Identify weak links between elements of the results chain and obtain further evidence to enable adjustments.

13. World Bank Indicators Relevance: consistent with organizational objectives and HRD plans Efficiency: value for money and adequate administrative and financial procedures Effectiveness: major objectives were achieved. Sustainability: continuation of benefit, once fellowships investment completed. Impact: improved performance of organization and benefit to community

14. Extended Kirkpatrick’s framework Planning, design and implementation- evidence of the appropriateness, relevance, efficiency and sustainability of the program. Reaction – evidence of satisfaction (what the trainees/fellows thought and felt about the training); Learning – evidence of learning (the resulting increase in knowledge or capability as reflected in end of course assessment);

15. Extended Kirkpatrick measures of training Behaviour – evidence of behaviour change or capability improvement as reflected in job performance; Results – evidence of contribution to the institution resulting from the fellows’ performance. Mega Impact- evidence of the long term benefit of the improved performance of the institution and the services it provides to specific communities or target groups.

16. Evidence of Contribution Self perception of enhanced capacity\contribution Others’ perceptions of enhanced capacity\contribution (360 degrees) Continuing to study and develop professionally Recognition by professional bodies of gained qualification (eligibility for higher studies and duties) Change in behaviour\attitude\performance

17. Evidence of Contribution Change of work practice\procedures \ways of doing things associated with learning Initiation of a new program or aspects of a program Attainment of specific institutional objectives related to added capacity Evidence of bridging a performance gap in the institution\program related to added capacity Evidence of contribution to institution key success factors

18. Evidence of Contribution Utilization of knowledge\skills by home institution Passing on knowledge\skills to others (dissemination) Assignment to higher or different duties that reflect use of learning Improved career path \progression Demonstration of leadership Improved prospect of retention due to higher motivation\morale

19. Use of all plausible evidence Use any plausible evidence regardless of the design, method or source used to obtain it. Use both qualitative and quantitative evidence (triangulation of findings) Use information obtained through current monitoring and evaluation approaches and techniques. (Less costly and less bias)

20. Assemble the performance story Document a Performance Story based on the evidence available showing the attainment of intermediate outcomes (critical milestones) and the logical link among them.

21. Performance story The performance story describes the journey from the inception of a fellowship program to the attainment of its immediate and long term goals. The important events and experiences along the way are identified as mile stones that could be monitored in order to ascertain that we are moving in the right direction and ultimately that we have reached our destination.

22. Performance story The performance story should enable ‘reasonable’ observers to determine whether it is plausible that particular interventions led to certain results.

23. “Plausible Association" “Plausible Association“ implies that reasonable people with adequate information concerning what has occurred or is occurring in the programme, agree that the programme contributed or is likely to contribute to the desired outcomes.

24. Seek out additional evidence. Where an alternative explanation cannot be discounted, or the program cannot be shown to be a more likely contributor, the program logic should be reviewed and/or additional data gathered and evaluated. Where this can’t be done, further evaluation is required or it should be recognised that the program is not the key contributor to the outcomes.

25. Work in progress The approach is well suited to development programs where data is likely to reflect ‘progress toward results’, rather than a definitive statement of final outcomes.

26. A credible performance story includes; The program context (including the results chain), Planned and actual accomplishments, Lessons learnt, Approach for assuring the quality of information The main alternative explanations for the outcomes occurring including reasons why they had limited or no influence. Mayne (2003)

27. Engagement of stakeholders Requires collaboration of stakeholders Provides better contextual understanding and engenders ownership Partnership and collective responsibility for success

28. Next Steps Construct benefit chain for selected programs Review existing evidence in relation to mile stones Identify weak links Seek additional evidence Construct performance story Confirm conceptual logic and fidelity of programs

29. Next steps Assess “plausibility” of contribution to impact Derive lessons and strengthen fellowship programs Disseminate findings and establish benchmarks and guidelines Engage stakeholders in all steps  ( Particularly, fellowship authorities, training institutions, beneficiary institutions, and fellows)

30. Recommendation Recommend to the 17 th Meeting of Senior Fellowship Officers to allocate existing resources and/or seek additional support for the conduct of pilot studies in selected countries and/or sectors, to test the appropriateness of the contribution analysis approach in the context of impact evaluation of UN system fellowship programmes.