Data Collection Methodologies Module 7

2 Overview n Definition of data collection n Data sources n Characteristics of data collection strategy n Questionnaires, focus groups, interviews: pros and cons

3 Data Collection n Observable, measurable units of analysis n Relevant to the purpose of the evaluation n Specific to the unit of analysis n Appropriate procedures and tools n Collected, recorded, analyzed, to draw conclusions

4 Data Sources n People: program managers,administrators, beneficiaries, donors, program staff, government officials, etc. n Documents: program strategy, annual plans, progress reports, financial reports, n Infrastructure observation: building and grounds, laboratories, program sites, n Observation of group dynamics: nature of meetings, Board meetings, etc.

5 Tools for Data Collection n Questionnaires n Interviews n Focus groups n Observation n Performance tests

6 Experimentation n To ensure an optimal validity of data collected, am ideal model would the experimentation approach. n Given resources constraints in many evaluations experimentation is not always feasible nor accepted n More sophisticated techniques exist to ensure validity and reliability of data

7 Data Collection Strategy n Consider the following: Size, geographical dispersion, literacy level of target population Resources available for data collection (time, people, money, technology) Sensitivity of issues to evaluate Reliability and validity sought Needs of the evaluation Developing an adequate strategy is key!

8 Questionnaires and Surveys n Printed or electronic list of questions n Distributed to a predetermined group n Completed and returned to evaluator

9 When to use a questionnaire n Target population is literate, and large (over 200) or dispersed n Need for categorical data n Need for quantitative data and statistical analysis n You have access to people who can process and analyze this type of data n Need to examine responses by sub-groups

10 6 Steps To Effective Questionnaires 1. Develop the purpose 2. Draft the items 3. Sequence the items 4. Design the questionnaire 5. Pilot the questionnaire 6. Develop a strategy for data collection and analysis

11 1. Develop the Purpose n Specific to the information you need n Add value n Relate to the evaluation n Refined to relevant questions only

12 2. Draft the Items Formulate into the following item types: n Fill in the blank Question requiring one word answer n Multiple Choice Question with 4-8 defined answers n Comment-On Open ended question, written response

13 2. Draft the Items (cont`d) n List Requesting list of 3 to 5 short responses Can be in order of importance n Likert Scales Statement to be ranked for agreement n Rank List of items to be ranked by importance

14 3. Sequence the Items n Group into themes, or by question type n Rewrite questions, eliminate redundancies n Where necessary, use filter questions guide respondents to different branches n Sequence with logical progression

15 4. Design the Questionnaire n Group similar themes together n Organize into titled sections n Format into booklet n Use space effectively n Decide on the need for pre-coding n Include a title and introductory explanation

16 5. Pilot the Questionnaire n Test questionnaire with small group n Clarify wording n Identify ambiguity n Revise n Retest if necessary

17 6. Develop the Strategy n Select the sample n Determine distribution plan n Develop a cover letter n Prepare distribution package, and send n Monitor responses, follow-up n Enter data n Analyze data

18 Effective Questionnaires n Gather valid information n Provide logical and organized data n Facilitate process of data entry and analysis n Simple, timely and cost efficient

19 Interviews n Dynamic process of gathering information n Personal interaction between two individuals n Follow a guide or protocol n Require recording of responses

20 When to use interviews n Need to have the views of key informants n Target population is small n Need for depth of information rather than breadth n You have reason to believe that people will not respond to a questionnaire n Informants are willing and available to meet

21 7 Steps to an Effective Interview 1. Define interview purpose 2. Draft interview questions 3. Sequence the questions 4. Consider process needs 5. Prepare introduction and closing 6. Prepare to record responses 7. Pilot test questionnaire

22 1. Define the Interview Purpose n Define objectives for the interview n Link objectives to the evaluation n Identify the data needed n State the purpose for each part of interview n Focus on content rather than process

23 2. Draft Interview Questions n Carefully drafted and worded n Minimize ambiguity n Combine open-ended with close-ended

24 3. Sequence the Questions n Vary question types Open-ended (What do you think about…) Close-ended (Did the program begin in 1998?) n Organize questions into themes or sections

25 4. Consider Process Needs n To manage the interview process, use: transitions to move to each theme reminders to paraphrase or summarize lists of standard probes

26 5. Prepare Introduction and Closing n Introduction to set tone and rapport state purpose of interview explain who the interviewer is confirm confidentiality and use of data n Closing to signify the end of interview reinforce usefulness of interview thank respondent confirm arrangements for follow-up

27 6. Prepare to Record Responses n For key informant interviews: tape record (with permission) take brief notes on notebook detail notes immediately after interview n For normative interviews: check boxes space for efficient data recording

28 7. Pilot-test Interview n Validate the interview: protocol content flow arrangement to record responses n If more than one interviewer, validate: ability to use protocol

29 Sample Protocol - Program Evaluation Interview n Introduction: “We have been asked by.. To conduct a.. We would like to discuss the...” n Theme 1 - Program relevance n Theme 2 - Program success n Theme 3 - Program cost effectiveness n Conclusion: “In conclusion, what are the main strengths and weaknesses of this program?”

30 Effective Interviews Are: n Planned sufficiently clear objectives n Focused on the process control over context and process two-way communication awareness of non-verbal communication n Conducive to establishing trust n Dependant on a cooperative interviewee

31 Effective Interviewing Skills Include: n Active Listening relaxed, eye contact, verbal feedback n Openness and Empathy accepting what is said without judgement n Paraphrasing/Summarizing rewording to clarify response n Controlling the Process setting context, shaping responses

32 Focus Groups n Group discussions of a predetermined issue n Members share common characteristics n Moderated by facilitator n Responses are recorded

33 When to Conduct Focus Groups n You need rich description to understand client needs n Group synergy is necessary to uncover underlying feelings n You have access to a skilled facilitator n Informants are willing to speak in groups

34 5 Steps to Effective Focus Groups 1. Define the purpose 2. Select an appropriate sample 3. Determine date and location 4. Develop Focus Group Guide 5. Plan to record responses

35 1. Define the Purpose n Based on the objectives of the evaluation n Identify information needed and why n Develop the purpose of the focus group

36 2. Select an Appropriate Sample n Individuals who are: informed about the topic of discussion in the best position to share information represent all target populations n Pre-screen participants to ensure they meet the required characteristics n Select participants

37 3. Determine Date and Location n Make clear arrangements regarding purpose starting time how long it will last neutral convenient location participants feeling at ease

38 4. Develop Focus Group Guide n Construct effective questions content related to evaluation purpose open-ended to facilitate discussion limit to five or six n Sequence questions ensure natural flow facilitate transition from one to the next lead logically through variety of topics

39 5. Plan to Record Responses n Taping gives full record of discussion inform and attain consent in advance n Detailed notes are indispensable taken by assistant to moderator organize data in predefined categories document relevant observations avoid cues about value of responses underline or highlight reference points

40 Effective Moderation Includes n Active listening n Facilitation of discussion n Awareness of group dynamics n Suspension of personal bias n Alertness to time schedule n Probing for elaboration n Affirmation of responses n Respect for different perspectives

41 Observation: when to use n When the setting is important for the evaluation n When the interaction amongst people is a key factor of the evaluation n Before developing a questionnaire, in order to get a feeling for some of the issues n To understand cross-cultural issues

42 Steps in Observation n Define the population or setting to observe n Select representative places to visit/observe n Develop an observation schedule with the dimensions to observe n If there are many observers, train observers for consistency

43 During Observation n Prepare for an entry phase so the observer does not impact the setting too much n Record data according to the observation dimensions selected: interaction, environment, light, smell, infrastructure, etc. n Record qualitative (how good, how bad) as well as quantitative (how many) data n Exit from the setting and record data immediately

44 Performance Tests: When to use n When there is a need to define aptitudes or abilities n When the study requires to select good performers (as opposed to people to train for good performance) n When there is a need to diagnose performance problems

45 Categories of performance tests n Aptitude tests n Attitude scales n Psychological tests n Performance tests

46 Choosing a performance test n First criteria is to match the test to the purpose n Second is to match the resources (time, money, technology), context and culture to the purpose n If the purpose is to screen out the best individuals, aptitude tests make sense n If the purpose is to assess progress, pre- and post-activity tests make sense

47 Scoring performance tests n Criterion referenced (Can a person perform according to a given standard?) n Normative (Who is the best in a group?)

48 Summary of Methodologies