1. 1 Session 20: Data Collection in the Field Ina F. Wallace RTI International IES/NCER Summer Research Training Institute, June 2007

2. 2 Data Collection in the Field Topics Covered Methods of Data Collection Use of State-of-the-Art Data Collection Procedures Ensuring Data Quality Conducting Small-Scale Validation Studies Ethical Issues in Data Collection

3. 3 Methods of Data Collection Interviewer Administered –In Person Surveys Assessments –Telephone Self-Administered –Paper and Pencil Mail-in In-person –Computer Assisted Self Interview –Web procedures

4. 4 Data Collection Mode Characteristics High Data Collector Involvement Low Data Collector Involvement PaperComputerPaperComputer Direct Contact with Respondent Face-to-Face Paper and Pencil Interviewing/ Assessment (PAPI) Face-to-Face Computer Assisted Personal Interview/ Assessment (CAPI) (Audio) Computer Assisted Self- Interview (CASI, A-CASI) Indirect Contact with Respondent Telephone PAPI Telephone Computer Assisted Telephone Interview (CATI) Mail, Fax, E- mail questionnare Touch-tone Data Entry (TDE), Web, Electronic Mail Survey (EMS) No Contact with Respondent Direct Observation Computer Assisted Data Entry (CADE) Administrative Records Electronic Data Interchange (EDI) Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality

5. 5 Face-to-Face Interviewing Advantages Maximum degree of communication and interaction Can utilize PAPI or CAI Allows for long, complex interviews Variety of tactics can be used to gain cooperation Response rates usually high Disadvantages High cost due to travel Can generate social desirability for sensitive questions Interviewers can affect respondents Can generate interviewer variance Presence of others can affect responses Falsification of interview

6. 6 Telephone Interviewing Advantages Cost lower than face to face More easy to provide training, supervision, and feedback Social desirability bias less than with face to face Interviewer variance less Disadvantages Less flexible No ability to use visual aids Must be shorter Questions must be less complicated Less capacity for motivating the respondent No coverage of non- telephone units Response rates lower

7. 7 Mail Surveys Advantages Lowest cost Low social desirability Can utilize visual aids Allows respondent to take time and look up information (if necessary) No interviewer variance Disadvantages Little control over who completes the survey Item non-response high Response rates lower Require a long field period to obtain an acceptable response rate Respondents must have good literacy skills Limit to closed-format questions

8. 8 Computerized Procedures Computer Assisted Personal Interview (CAPI) Computer Assisted Telephone Interview (CATI) Computer Assisted Self Interview (CASI) Audio Assisted Self Interview (ACASI) Telephone Audio Assisted Self Interview (T-ACASI) Computer Assisted Data Entry (CADE) Web Surveys

9. 9 Computer Assisted Procedures Advantages Complex skip patterns and branching are easier to follow Previously acquired information can be programmed Ranges, inconsistencies, and illegitimate values can be checked Help screens can be included Data are available for analysis more quickly Disadvantages Time consuming to develop and program computer Must be adequately tested Cant change things easily once fielded Risk of computer crashing Costly

10. 10 Direct Assessment Used to obtain psychological and educational information about an individual Individually administered Research driven NOT for clinical or diagnostic purposes Follows a standardized protocol

11. 11 Observational Techniques Involve a researcher making observations within a natural setting Direct Observation - individuals know they are being observed –Continuous Monitoring recording as much behavior as possible –Time Allocation randomly visiting the sample to observe at different times Unobtrusive Observation – individuals do not know they are being observed

12. 12 Observational Techniques Advantages No or limited participation needed May be good for generating hypotheses Sensitive social issues can be obtained more accurately High in validity because of the depth of information collected Disadvantages Variance associated with observers is similar to that found with interviewers Can be very costly as they are often labor intensive Time consuming Reliability can be low Generalizability can be low

13. 13 Summary: Issues in Choosing a Strategy Type of Population Nature of Research Issue Question Form Question Content Response Rates Costs Length of Data Collection

14. 14 Methods of Administration Standardized Interviewing Approach –A protocol in which interviewers interact with the respondent in a manner which is consistent across all interviewers Conversational Interviewing –A protocol in which interviewers modify and adapt questions to the respondents situation Hybrid Interviewing –Combination of the two approaches

15. 15 Factors that Can Explain Interviewer Variance Interviewer Characteristics Appearance Motives Beliefs/attitudes Perceptions Expectations Skills/Knowledge Respondent Characteristics Knowledge Interest/motivation Confidence Strength of convictions Expectations Interviewer Effects Questionnaire Definition clarity Terminology/jargon Question form Instructions Question wording Question topic Survey Conditions and Setting Mode of interview Standardization Interviewer training Interviewer supervision Monitoring/observation From Biemer & Lyberg (2003). Introduction to Survey Quality

16. 16 Factors that Can Explain Interviewer Variance Interviewer Characteristics Appearance Motives Beliefs/attitudes Perceptions Expectations Skills/Knowledge Respondent Characteristics Knowledge Interest/motivation Confidence Strength of convictions Expectations Interviewer Effects Questionnaire Definition clarity Terminology/jargon Question form Instructions Question wording Question topic Survey Conditions and Setting Mode of interview Standardization Interviewer training Interviewer supervision Monitoring/observation From Biemer & Lyberg (2003). Introduction to Survey Quality

17. 17 Role of the Interviewer/Assessor Gaining Cooperation –Converting reluctant participants –Recognizing and dealing with refusals Training and Motivating Respondents –Interviewers implicitly and explicitly teach respondents how to behave Obtaining Quality Data Ensuring Confidentiality

18. 18 Assessing Children Assessor Characteristics –Flexibility –Vigilance –Self-awareness –Eye contact Establishing Rapport –Allaying apprehensions –Helping children feel at ease –Maintaining limits on permissible behavior Arranging the Physical Environment

19. 19 Ensuring Quality Data Recruiting interviewers/assessors –Characteristics of all data collectors –Characteristics of interviewers –Characteristics of assessors Training on standardized procedures Certification as data collectors Monitoring performance

20. 20 Training Content of Training Presenting the study Contacting participants Administering the questionnaire/ assessment Probing Recording responses Confidentiality Interpersonal relations Administrative procedures Procedures for Training Written materials Lecture and presentation Computer-based tutorials Planned exercises Role Playing Observation Certification

21. 21 Interviewer Training Agenda Introduction to the Study Role of the Interviewer Confidentiality Procedures Review of Questionnaire and Protocols Standardized Interviewing Techniques Round Robin Interview Gaining Cooperation Presentation of Scenarios Round Robin Demonstrations Paired Mocks Certification

22. 22 Assessor Training Agenda Day 1: Large group Introduction to Project Role as an Assessor Best practices with children Small groups Pre-training quiz Introduction to all materials Learn each assessment –Observation –Round Robins Day 2: Small groups Individual practice with assessments Paired practice with assessments Large group Building rapport and gaining cooperation Administrative responsibilities Data security Informed Consent Day 3: Small groups Review of assessments Paired practice Post-training quiz Large group Concluding remarks Certification

23. 23 Monitoring Performance What to Monitor Detection and prevention of falsified information Compliance with rules and guidelines about administering the questionnaire or assessment Performance on non- interview/assessment tasks How to Monitor Telephone verification with a sample of respondents Direct observation – in- person or by telephone Audio-recording – CARI Review of materials Performance and production measures

24. 24 Human Subjects Issues Institutional Review Boards Informed Consent –Who is carrying out research under whose sponsorship –Description of purposes and procedures –Duration of involvement –Risks and Benefits –Confidentiality –Voluntary nature of research including stopping early and skipping questions –Compensation –Contact information Confidentiality Issues

25. 25 Human Subjects Issues Institutional Review Boards Informed Consent –Who is carrying out research under whose sponsorship –Description of purposes and procedures –Duration of involvement –Risks and Benefits –Confidentiality –Voluntary nature of research including stopping early and skipping questions –Compensation –Contact information Confidentiality Issues

26. 26 Data Collection Evaluation Stage of Data Collection Evaluation MethodPurpose DesignExpert review of instrumentIdentify problems with instrument PretestBehavior coding Cognitive interviewing Evaluate one or more stages of the response process Pretest/Post-Data Collection Respondent focus groups Interviewer debriefings Evaluate instrument and data collection procedures Pretest/Data Collection Supervisor observation Telephone monitoring CARI Evaluate performance of data collector Post-Data Collection Experimentation Internal Consistency External validation Re-interview/retest Compare alternative methods of data collection Evaluate components of MSE Validate estimates Adapted From Biemer & Lyberg (2003). Introduction to Survey Quality

27. 27 Post-Data Collection Evaluations Experiments Observational Studies Internal Consistency Studies External Validation Studies Administrative Record Check Studies Re-interview/Re-Test Studies

28. 28 Post-Data Collection Evaluations Experiments Observational Studies Internal Consistency Studies External Validation Studies Administrative Record Check Studies Re-interview/Re-Test Studies