Module 2 Making Program Evaluation Useful

2 Overview n Consider some real evaluations n Analyze what affects use n Report Back n What the theory says n Things to think about

3 Case Study 1- Evaluation of Management Education in E. Europe Background n Western donors developed many MBA programs in Central and Eastern Europe in the 1990s n Professors from the West taught students and faculty were trained in the West n Students were required to pay tuition fees n After 3-7 years, the western professors left and the MBA program was left to survive on its own

4 Key Evaluation Findings n The MBA programs could not be sustained after post-assistance because... Profs in Eastern Europe were poorly paid The State was reluctant to recognize the MBA degree They did not integrate the management approaches already used in E. Europe Students could not afford expensive fees

5 Evaluation Use n The major recommendations were... More investment in program development over a longer time period Establishment of student loan funds n The donors did not make these changes because they felt... The cost in money and time was too high It was too difficult to manage loans It was too late to make a difference

6 Program Evaluation Case Study 2-- Search & Rescue Background n A national coast guard has the responsibility of saving lives of people in distress at sea n The coast guard has 75 vessels -- 6 are large cutters that cost 70% of all the vessels n On average, 543 marine incidents with1,650 lives at risk per year and 148 people die

7 Key Evaluation Findings n Estimated 110 lives saved in distress incidents that would have been lost without assistance n Over 50% of $68 million operating expense is used to operate six large vessels n Average cost per life saved = $430 K $2.8 million for cutters $141 K for small vessels

8 Evaluation Use n The Government eliminated 4 of 6 cutters, saving $21 million per year n Sale of 4 cutters enabled purchase of 25 more small boats that were stationed in small ports previously without protection n Overall result was better search and rescue at lower cost

9 Use and the Purposes of Evaluation n Program Accountability Summative Decision-Making Conformity to standards n Use ensured through Resource decisions Effects on reputation n n Program Development Formative Learning Change to improve practice n n Use ensured through Empowerment Persuasion

10 Types of Use n Accountability - to an outside body (government, donor, public, beneficiaries, etc) n Decision making - program managers and organization leaders n Decision making - funders and clients n Learning - for generalization to other programs n Individual Learning - informally changing behavior of related personnel

11 Ensuring use through the Accountability Approach n Accountability standards need to be understood n Credibility ensured through standard conventions n Methods of checking for compliance must be objective and verifiable n Evaluators are experts in the subject n Consequences of non-compliance are less resources for the program

12 Ensuring Use through the Program Development Approach n Program/organizational conditions support efforts to improve the program n Credibility ensured through stakeholder support for conclusions/recommendations n Methods can vary by situation n Evaluators are experts in helping programs through evaluation n Consequences of not using evaluation results are lack of rewards for leaders

13 Factors that support Evaluation Use n Openness of decision-makers n A ‘champion’ to follow through n Involvement of stakeholders at all levels n An organizational will to improve n Negative consequences -- resources

14 Reasons Organizations Do Not Use Evaluation Results n Complacency -- it is easier to do nothing than to try to be better n Concerns about the image/reputation of the organization or program (in particular if major ‘visible’ changes are suggested). n The evaluation is poorly done - including poor process and/or content n The form of reporting may create difficulty for the organization

15 Things to Think About n Some evaluators feel that a focus on use leads to compromise in the objectivity of evaluations - - they become political compromises; not scientific information n It is said that “What gets measured gets done.” Measure it and the program will respond. n It is also said that “What gets measured gets corrupted”