Evaluation Proposal Defense Observations and Suggestions Yibeltal Kiflie August 2009

The proposal What it is? –An evaluation proposal is your plan to conduct an evaluation. –Like any other plan, evaluation proposals should: Convince readers on why we need to invest on it Who will benefit from the evaluation (users) Set clear objectives Describe how the objectives will be achieved How the plan will be operational How it will be monitored and evaluated

Observations General observations on presentations –Time management Time management is beyond the value of time An audience/examiner upset because of time management usually gives priority on your weaknesses Avoid hyperlinks and over use of animations –Too many slides Presentations should highlight most important parts of your document Focus on methodology rather than background –Overloaded slides and unreadable texts in boxes –Inability to handle power point presentations

General observations on presentations cont… Eye contact –Don’t read slides –Putting only reminders on slides rather than full sentences help avoid reading from slides –Rehearse your presentation –Don’t present what you are not comfortable with

Content of proposals Background/Introduction –Expectation Why do we need the program – the problem targeted by the program What is the program – description of the program including related policies, strategies, objectives, activities, resources and the context in which the program is functioning Why do we need the evaluation –Clearly show where the information gap is –How this information gap is affecting implementation and or achievement of outcomes

Background/Introduction cont… Observations –Lack of focus in describing programs Area Program Program Component –Need for program Vs Need for the evaluation Problem statements focusing more on description of why we need the program rather than why we need the evaluation –Use of outdated data while recent data is already available

Stakeholders Analysis Expected: –Description of steps undertaken to involve stakeholders and the purpose of doing it –Description of stakeholders roles in the program and evaluation making clear intended users and potential uses of evaluation findings –Description of how stakeholders will get involved throughout the evaluation process

Stakeholders Analysis cont… Observations –Unnecessary classification of stakeholders as high, medium and low level of importance without adequate justification of why? –Failure to describe specific intended uses of evaluation findings for each intended user –Failure to mention the plan on how stakeholders will be involved throughout the evaluation process “Use evaluation findings for improvement” is like an excuse for not identifying intended use by intended users.

Literature review Expected: –Either as a separate chapter or integrated in different chapters – wherever relevant –Review should cover: Review of literature on methods Review of literature on findings from other settings on the issues related to what your evaluation will address Look for supporting findings to justify that your evaluation will contribute for identification of problems and thus solving them

Literature cont… Observations: –Copying from text books –Plagiarism??? –Failure to mention the set up while presenting findings from other similar studies

Eval Questions and Objectives Expected: –Clearly mention what the evaluation will answer by the time it gets completed –directly related to the information gap discussed in the statement of the problem –Objectives need to be specific and comprehensive enough –Avoid the use of overlapping objectives

Methods Recommended Contents –Focus of evaluation –Dimensions of the evaluation –Approach to the evaluation –Study Area and Period –Study Design –Target Population –Study Population –Study units and sampling units –Sample size and sampling procedures –Eligibility criteria –Data collection tools and procedures –Data collectors –Instruments for data collection –Pretesting –Data collection –Data quality control (Could be omitted if addressed in the meta evaluation section) –Data Management and Analysis –Basis for Judgment –Ethical Issues (could be omitted if addressed in the meta evaluation section) –Limitations of the Study –Report Writing and Dissemination Plan –Meta-evaluation

Methods cont… Observed problems –Incoherent flow of topics –Inadequate to answer evaluation questions Target population level objectives but method focusing on organizational factors Question on entire program but method addressing only a component of a program –Sample size Vs Case selection –Case selection criteria against what the evaluation question requires –Multistage sampling without design effect –Exclusion of segments of target population without adequate reason

Methods cont… Observed problems –Instruments for data collection Not mentioning source of adapted tools Observation tools for set up assessment and repeated provider – patient interaction should be separated –Pretesting Not mentioning where and how pretesting will be conducted Planning pretesting while no need

Methods cont… Observed problems –Analysis plan Mentioning software without any plan to use specific statistical analysis –Limitation Mentioning challenges as limitations Considering something which is out of the scope of the study as limitation –Meta evaluation Duplicating what is already mentioned Copying definitions from books