Ensuring the quality of Qualitative Data Presented by Lorie Broomhall, Ph.D. Senior HIV/AIDS Advisor Nigeria Monitoring and Evaluation Management Services The Mitchell Group Abuja, Nigeria

Mini Pop Quiz Please indicate T if the statement is true and F if the statement is false: 1.Quantitative methods are more accurate than qualitative methods. 2.Qualitative methods are not as credible as quantitative methods because they cannot be replicated. 3.Quantitative methods are more generalizable than qualitative methods. 4.Qualitative evaluations are more costly and time consuming that quantitative evaluations – less bang for the buck. 5.Quantitative evaluations are less biased than qualitative evaluations 6.Qualitative methods are useful for explaining findings derived from quantitative methods – telling the story – but not the other way around. 7.Qualitative data analysis is not systematic 8.Focus group discussions are the most useful and reliable qualitative evaluation method.

Workshop Objectives Definition of high quality qualitative data Uses of qualitative data in M&E Quality standards Qualitative method designs Analysis and reporting

High Quality Data  Ensure that program and budget decisions are as well informed as practically possible  Support efficient use of resources  Are credible  Meet reasonable standards accuracy, objectivity, and consistency

“In principle, the same quality standards for quantitative data apply to qualitative data” USAID Functional Series 200 – Programming Policy ADS 203 – Assessing and Learning

High quality can be achieved using any qualitative method (as long as it is used appropriately):  Interviews  Mapping/Walk-throughs  Directed observations  Participant observation  Focus Group discussions  Elicitation techniques  Pile sorts  Ranking  Free lists

High quality data have Validity (internal and external) Reliability Timeliness Precision Integrity

Validity “Data are valid to the extent that they clearly, directly, and adequately represent the result that was intended to be measured.”* * Quotations in this presentation from USAID ADS 203, and Tips #12

Validity increases with  Clear goals and objectives  Sound method design and data collection plan - triangulation  Adequate training and supervision  Systematic data analysis  Method triangulation

Badly designed data collection instruments Imprecise definitions Inappropriate sampling techniques Inadequately trained data collectors Transcription error Validity is compromised by

A Sound Sampling Plan is Representative: Are all people or groups affected by program represented in the sample? Are they represented in proportion to their size? Inclusive: Are all the different views and perspectives represented in the sample? Appropriate: Are the right people being sampled? Consistent: Is the sampling strategy applied to all sites over the length of the evaluation? Documented and justified

Examples of sampling strategies for qualitative evaluations: Convenience/purposive sampling Stratified sampling (significant variation) Random sampling (people, time, location, etc.) Snowball sampling (social networks)

Reliability “ Data should reflect stable and consistent data collection processes and analysis methods over time.”

Reliability in assured with:  Consistent data collection processes  Field supervision and observation  Checks for bais and ‘drift’  Data replicability

Precision is improved with:  Direct, understandable data collection instruments  Concise and operationalized definitions  Clear, descriptive, objective and decipherable documentation (e.g. field notes, summaries)  Scrutinize!  Frequent checks of data bases and inter-coder reliability Precision “Data should be sufficiently accurate (precise) to present a fair picture of performance...”

Timeliness “Data should be current and available frequently enough to permit management decisions to be made” Integrity Data should be free from manipulation for professional, political or personal purposes.

Timeliness can be achieved through  Building an appropriate data collection strategy – minimum methods for the maximum return.  Well planned data management/processing  Adequate training and supervision  Effective communication  Efficient analysis  Contingency plans for unforeseen events (strikes)

Maintaining Integrity  Secure data storage: names on folders, electronic files, registers and logs  Supervision – scrutinizers  Awareness of potential data manipulation

Thank you