1 of 20 Evaluating an Information Project From Questions to Results © FAO 2005 IMARK Investing in Information for Development Evaluating an Information Project From Questions to Results

2 of 20 Evaluating an Information Project From Questions to Results Learning Objectives At the end of this lesson you should be able to: identify the specific impacts that you want to measure; be aware of the assumptions involved in measuring change; define the most appropriate indicators for your selected impacts; be aware of several methods for collecting data.

3 of 20 Evaluating an Information Project From Questions to Results Introduction In order to examine the issues involved in the planning of an evaluation, we are following a scenario: Dr Kumar, the Director of publications at SMAU University, must evaluate a newsletter. Dr Kumar and his colleagues have held the first Evaluation Management Committee meeting, to reach an agreement on the focus of the evaluation. The EMC conducted a preliminary analysis of the RICE NEWS project, examining Needs, Beneficiaries, Activities, Outcomes. Also, a SWOT analysis identified potential risks and benefits of the evaluation. Dr Kumar, Jon and Lara have built a Logic Model, that has provided an intellectual basis for the evaluation. Each of the "if-then" links identified has helped define what to measure.

4 of 20 Evaluating an Information Project From Questions to Results Introduction Now, here are two important sets of questions: If any given impact contains the word "more" - for example, "more awareness", "more knowledge", or "more skill" - how can you best measure such a change? Are all potential impacts of your project equally important? Do you want to try to measure all of them?

5 of 20 Evaluating an Information Project From Questions to Results Setting priorities Lets start with the first question. Which impacts are most important to your stakeholders? A priority setting process will help you avoid two common mistakes: 2. Collecting too much data 1. Taking the easy way out

6 of 20 Evaluating an Information Project From Questions to Results Setting priorities Here is a useful method to avoid these mistakes: 1.Call another EMC meeting. 2.Use the card technique. 3.Ask each member of the EMC to write down specific questions that (s)he thinks the evaluation should focus on (one question per card).

7 of 20 Evaluating an Information Project From Questions to Results Setting priorities 3.Gather up the cards, pin or tape them to a large board and lead a discussion in which members try to group them. 4.Raise four supplementary issues (for each of the questions on the cards): Who is the information for? How will this intended user actually use it? How will you collect it? What resources will you need to collect it?

8 of 20 Evaluating an Information Project From Questions to Results Measuring change If any given impact contains the word "more" – for example, "more knowledge"– how can you best measure such a change? Ideally, we need both before and after data if the evaluation is going to measure change. But what if your information project lacks this "before" data? Now lets take the second set of questions:

9 of 20 Evaluating an Information Project From Questions to Results Measuring change The best approach is to look at any question that contains the word "more: How do we measure the change that that word implies? Do we really have questions (indicators) to measure such a change? Are our questions (indicators) valid, reliable and convincing?

10 of 20 Evaluating an Information Project From Questions to Results Measuring change The biggest challenge in deciding on questions (data) is to balance what you would like to collect with what you can collect. Here are four "rules" regarding question selection. Make sure that: you've done a systematic analysis of all your questions; you know what types of data they're likely to generate; you're aware of any potential limitations in these data; and you've already thought about how they should be interpreted.

11 of 20 Evaluating an Information Project From Questions to Results Collecting data Let's go back to MSAU. The EMC has used the "card technique" to come to agreement on questions to be asked and data to be collected. Now the issue is HOW to collect these data. Where are these data? How can we best collect them? What resources do we need?

12 of 20 Evaluating an Information Project From Questions to Results Collecting data WHERE ARE THE DATA? 1. In project records 2. In peoples attitudes and behaviours Content determines location, location determines method, and method determines resources.

13 of 20 Evaluating an Information Project From Questions to Results Collecting data Lets start with project records. Some of the data that could be found in project records are: data on funds spent; number of person-days gone into the project; number of copies produced; number of requests for hard copies; and number of hits on the web site.

14 of 20 Evaluating an Information Project From Questions to Results Collecting data Suppose your EMC has developed a list of questions to find out about farmer knowledge. How are you going to ask these questions?..and the second category: in peoples attitudes and behaviours. The most common ways are questionnaires and face-to-face interviews. In both cases, it is possible to use both open-ended questions or/and fixed-response questions.

15 of 20 Evaluating an Information Project From Questions to Results Collecting data Interviews or questionnaires? Here are some issues to consider: But do we have the organizational capacity to analyze these data once we have collected them? 1. We will probably get a higher response rate from interviews. But how high a response rate do we really need? 2. Interviews will be more expensive and time-consuming. Will the benefits that we gain from face-to-face contacts outweigh the costs? 3. During interviews, well be able to ask follow-up questions.

16 of 20 Evaluating an Information Project From Questions to Results Collecting data Whichever technique you use, youll have to decide about respondents. Whom should you interview? To whom should you send questionnaires? What proportion of stakeholders should you interview or send questionnaires to?

17 of 20 Evaluating an Information Project From Questions to Results Analyzing results After we collect our data, the next step will be to analyze them. I identified three last stages in our evaluation: From data collection to organization From organization to analysis and conclusions From analysis to a report and publication

18 of 20 Evaluating an Information Project From Questions to Results Analyzing results Each of these stages represents a process, that: must be planned; requires resources; must have clearly defined responsibilities and procedures; and produces outputs.

19 of 20 Evaluating an Information Project From Questions to Results Analyzing results If I had to do it all over, I would think about evaluation right at the beginning of the project, and define stakeholders, impacts and evaluation questions right from the start. I think that an evaluation is really a project in itself. It has to be planned, managed, and it could even be evaluated.

20 of 20 Evaluating an Information Project From Questions to Results Summary When planning an evaluation, important issues are: setting priorities and measuring change. It is important to: a) conduct a systematic analysis of all your questions; b) know what types of data they're likely to generate; c) be aware of any potential limitations in these data; and d) think about how they should be interpreted. The next issue will be how to collect data. The questions are: Where are these data? How are we going to collect them? What resources will we need? And the final issue will be how to analyze the data. The analysis is conducted in three stages: a) from data collection to organization; b) from organization to analysis and conclusions; and c) from data analysis to a report and publication.