Ensuring rigour in qualitative research CPWF Training Workshop, November 2010

1. Validity: does your information reflect reality? In quantitative research, statistical tests are used to test whether findings can be considered valid, i.e. whether they are likely to reflect reality. In qualitative research there is no formal test of confidence. Your honest conviction of having found something close to the truth, reached with an open mind and based on multiple sources of information and sources which are expected to be reliable, is the test. Critically assess the reliability of different sources of information: How expert / experienced is this informant? Is s/he in a good position to know about the subject? Is this statement fact or opinion? What kind of document is this? Who wrote it, and for what purpose? But be prepared to challenge your own assumptions. Look at what the data is telling you, not what you expected it to tell. You should be continually reflecting during the research. Triangulate using different sources and methods to cross-check your findings.

Triangulate If you’re consulting/interviewing only one informant, it is hard to tell how reliable the information is. It is therefore recommended to triangulate with multiple sources of information. Ask multiple informants: if three or more informants agree on something, you can be fairly confident in the picture which is emerging. Use your own observations : e.g. If you are told about an intervention that has taken place, try to observe it in the field. Consult documentary evidence: e.g. If you are told that three different sector offices are involved in planning decisions for rainwater management, confirm whether this is an official arrangement by reading the guidelines used by staff of those offices. (N.B. policies and practices set out on paper may differ from those used in practice, which is an important finding and does not mean one or the other is ‘wrong’.) You can also triangulate among different methods: do the results of interviews, a survey and participatory techniques agree? Triangulation: When more than two methods or sources are used in a study in order to increase confidence in the findings.

Minimise bias Bias: A preference or an inclination, especially one that inhibits impartial judgment Systematic error leading to false results What types of bias might affect our research? How can we avoid them? There are 5 major categories of bias: Interviewer bias Biased questions Biased answers Biased samples Biased reporting

Common types of bias Types of bias can include: Spatial bias - The tendency to favour locations that are easily accessible. This may affect the choice of the village for the assessment (how much mud do you have to go through to get to the village). Male bias – In most research, the views of men are usually heard more than those of women. Politeness - People in most cultures have a tendency to tell interviewers what they think the interviewer wants to hear. If researchers are conscious of this and express genuine interest in learning the truth, this effect can usually be counteracted. Expectations - Where the research is perceived to be associated with possible benefits for the individuals or villages being studied, answers are likely to reflect the villagers’ strategy for capturing those benefits. Researchers need to carefully plan how they will present themselves to the village in order to minimise or offset this effect. Perception bias – All researchers bring their own opinions, expectations and assumptions to qualitative research. It is important to reflect on these, and acknowledge them transparently.

2. Reliability: Is your method replicable? For findings to be considered reliable, the research must be replicable by others. During the research and the reporting: Provide a clear description of the sampling system, methodology adopted, data collection and analysis. Justify and explain why these are appropriate. Maintain a transparent paper trail (field diary, interview notes, sampling lists, documents collected) Include original data (quotations) in the analysis and presentation of findings, to justify conclusions.