1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University.

Slides:



Advertisements
Similar presentations
Outstanding Interviews.
Advertisements

Description of the Project The Consumer Operated Service Program (COSP) will study the impact of participating in a consumer-operated service by examining.
Cognitive Testing: Process and Results Roger Levine, Ph.D. American Institutes for Research (AIR) CAHPS AIR Team.
Coverage error Survey Research and Design Spring 2006 Class #3.
Measurement Reliability and Validity
Evaluating tests and examinations What questions to ask to make sure your assessment is the best that can be produced within your context. Dianne Wall.
Assessment: Reliability, Validity, and Absence of bias
Chapter 41 Training for Organizations Research Skills.
Journal Club Alcohol and Health: Current Evidence November–December 2006.
Instrumentation: Performance & Surveys : Research I Spring 2010 Dr. Leonard.
1 Adapting Research Methods for Diverse Ethnic Groups Eliseo J. Perez-Stable, M.D. Anita Stewart, Ph.D. Anna Nápoles-Springer, Ph.D. University of California,
Developing a Questionnaire. Goals Discuss asking the right questions in the right way as part of an epidemiologic study. Review the steps for creating.
Improving Patient Outcomes Through Effective Teaching The Teach Back Method.
Developing a Questionnaire Chapter 4. Using Questionnaires in Survey Research Construction is key to valid and reliable research Well written and manageable.
Notes for Social Sciences Constructing Survey Questions.
1 MTN-003 Training General Interviewing Techniques Some specific tips for administering the Screening interviewer-administered CRFs SSP Section 14.
Mood Disorder PHQ-9© PHQ-9-OV© SECTION D MOOD June 3, PM.
Collaborating for Quality: A Cross-Discipline Approach to Questionnaire Content Evaluation in Business Surveys Diane K. Willimack Peter Gibson U.S. Census.
Descriptive and Causal Research Designs
‘Hints for Designing Effective Questionnaires ’
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.
1 Class 8 Pretesting Measures and Creating a Questionnaire November 12, 2009 Anita L. Stewart Institute for Health & Aging University of California, San.
1 Using Qualitative Methods to Develop Culturally Adequate Surveys Anna Nápoles-Springer, Ph.D. University of California San Francisco Center for Aging.
Copyright 2010, The World Bank Group. All Rights Reserved. Testing and Documentation Part I.
Cognitive Interviewing for Question Evaluation Kristen Miller, Ph.D. National Center for Health Statistics
Construction and Evaluation of Multi-item Scales Ron D. Hays, Ph.D. RCMAR/EXPORT September 15, 2008, 3-4pm
SPECA Regional Wrokshop on Disability Statistics, Dec 13-15, 2006 Issues Related to Disability Measurement: Cognitive testing and mode Jennifer Madans.
EFFECTIVE RECRUITMENT AND SELECTION
Questionnaire Design Chapter Nine. Chapter Nine Objectives To understand the role of the questionnaire in the data collection process. To become familiar.
Lecture 8A Designing and Conducting Formative Evaluations English Study Program FKIP _ UNSRI
Data Collection Methods
Chapter 29 conducting marketing research Section 29.1
1 The Patient Perspective: Satisfaction Survey Presented at: Disease Management Colloquium June 22, 2005 Shulamit Bernard, RN, PhD.
CONDUCTING A SURVEY Adapted from Del, Balso Michael, and Aland Lewis D. First Steps: A Guide to Social Research. Toronto: Nelson Thomson Learning
Teaching Today: An Introduction to Education 8th edition
1 Class 9 Analyzing Pretest Data, Modifying Measures, Keeping Track of Measures, Creating Scale Scores November 15, 2007 Anita L. Stewart Institute for.
1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for.
1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart.
1 Class 8 Pretesting Measures and Creating a Questionnaire November 6, 2008 Anita L. Stewart Institute for Health & Aging University of California, San.
VALIDITY AND VALIDATION: AN INTRODUCTION Note: I have included explanatory notes for each slide. To access these, you will probably have to save the file.
8. Observation Jin-Wan Seo, Professor Dept. of Public Administration, University of Incheon.
1 Qualitative Methods to Develop and Pretest Surveys in Diverse Groups Anna Nápoles- Springer, Ph.D. University of California San Francisco Center for.
CAHPS Clinician & Group Survey 2.0 Update Ron D. Hays RAND, Santa Monica, CA UCLA, Los Angeles, CA.
1 Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008 Anita L. Stewart Institute for Health & Aging University.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Evaluating Survey Items and Scales Bonnie L. Halpern-Felsher, Ph.D. Professor University of California, San Francisco.
1 Locating and Assessing the Usefulness of Health Measures for Health Disparities Research Anita L. Stewart, Ph.D. University of California, San Francisco.
Copyright 2010, The World Bank Group. All Rights Reserved. Testing and Documentation Part II.
WASHINGTON STATE UNIVERSITY EXTENSION Evaluation Based Program Development This material was developed in partnership with Rutgers's University and Lydia.
Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.
Chapter X Questionnaire and Form Design. Chapter Outline Chapter Outline 1) Overview 2) Questionnaire & Observation Forms i. Questionnaire Definition.
The effects of Peer Pressure, Living Standards and Gender on Underage Drinking Psychologist- Kanari zukoshi.
VERBAL COMMUNICATION II Health Science. COMMUNICATION.
Mood Disorder PHQ-9© PHQ-9-OV© SECTION D MOOD January 14, PM.
Self Care Instructions Exercise regularly Monitor your blood pressure Eat a healthful diet.
Chapter 14: Affective Assessment
Collecting data about service user satisfaction November 4th 2013 Workshop in London.
By Dr Hidayathulla Shaikh. Objectives  At the end of the lecture student should be able to –  Define survey  Mention uses of survey  Discuss types.
Developing a proposal Dónal O’Mathúna, PhD Senior Lecturer in Ethics, Decision-Making & Evidence
Survey Methodology Reliability and Validity
Research Methods for Business Students
You Can STOP Harassment
Interviewing Techniques
Data Collection Interview
Interviewing Techniques
QUESTIONNAIRE DESIGN AND VALIDATION
CCMH 535 RANK Career Begins/cchm535rank.com
BE MORE INVOLVED IN YOUR HEALTH CARE
What is the nature of descriptive measures to gather data?
Anita L. Stewart Institute for Health & Aging
Presentation transcript:

1 Class 7 Pretesting measures and considerations in modifying or adapting measures November 10, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2 Steps 5-7: Pretesting and Adapting 1. Specify context 2. Define concept for your study 3. Identify and review potential measures for a) conceptual and psychometric adequacy b) practicality and acceptability c) translation available if needed 4. Select best candidates 5. Pretest selected measures in your groups 6. Choose best ones based on pretest results, OR 7. Adapt if necessary to address problems this week

3 Purposes of Pretest by Context u In your search for measures, can look for evidence of pretesting –Evaluate whether measure was pretested and adequacy of methods u When creating a battery of measures –You can conduct pretest to help you decide between alternative measures u When using specific measures –You can conduct pretest to identify problems

4 Example of Measure Undergoing Cognitive Testing u Consumer Assessment of Health Plans Study (CAHPS TM ) –Harris-Kojetin et al., 1999 (in readings) u Describes extensive pretesting in developing and refining the CAHPS TM u Provides example of good pretesting

5 Step 5: Pretest in Target Population u Pretesting essential for measures being applied to any new population group –Especially priority measures (e.g., outcomes) u Pretest is to identify: –problems with method of administration, respondent burden, procedures –problems with questions or response choices

6 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

7 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

8 General Debriefing Pretest u Goal –Identify problems with the procedures –Estimate time needed to complete instrument –Identify serious problems with items u Procedures –Subjects answer entire questionnaire –At end, debrief –Close to true task

9 Debriefing Questions After Administration of Survey.. Ask respondents: u Were any questions confusing? u Which words were hard to understand? u Which questions were difficult to answer? caused distress? u Was questionnaire too long? u Confusing instructions?

10 Patient Acceptance Scale u 6-item scale about burden of a survey u Negative affect burden –Feeling embarrassed, upset, annoyed, uncomfortable u Questionnaire length u Ease of answering questionnaire Zimmerman M et al., Med Care, 1994;32:

11 Example of General Pretest to Select Best Measure of Patient Satisfaction u Compared 4 questionnaires on –Response rates, missing data, completion time, patient ratings of the questionnaire u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., A randomized trial of four patient satisfaction questionnaires Med Care, 2003;41(12):

12 Example of General Pretest to Select Best Measure of Patient Satisfaction u Compared 4 questionnaires on –Response rates, missing data, completion time, patient ratings of the questionnaire u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., A randomized trial of four patient satisfaction questionnaires Med Care, 2003;41(12):

13 Problems with General Pretests Respondents… u often don’t understand the task. u don’t want to appear as if they didn’t understand u have a hard time telling you anything was wrong u easier to say everything was fine

14 In-Depth Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions

15 Cognitive Interviews Examine 4 Steps in Answering Questions u Comprehend the question (as intended) u Retrieve the information –various strategies used to access memory u Judgment formation - formulate an answer –calculate or judge the correct information u Edit response - decide what to report –is answer embarrassing, socially undesirable? Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996

16 Purpose of Cognitive Interviews To learn.. u.. if respondents understand words and phrases as intended (meaning) u.. about the process of answering the questions u.. whether items are unacceptable u.. about the usefulness of response choices –whether response choices are adequate –how they use the response choices

17 Two Types of Cognitive Interviews u Think aloud interviews –Respondent asked to think aloud as they answer question u Probe interviews –Interviewer asks specific questions to elicit how respondent answered question –Scripted and spontaneous probing

18 Think Aloud Interviews u Thorough examination of the entire thought process of creating answers u Require that respondents verbalize their thought processes as they are answering items –Hard task for most people u Mostly done at “cognitive interviewing” labs, e.g., at National Institutes of Health

19 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

20 Probe Questions to Explore Meaning u What does the word _______ mean to you? u What does the phrase ________ mean to you?

21 Example: Probing the Meaning of a Phrase I asked you about how the office staff treated you personally …. What does the phrase “office staff” mean to you?

22 Example: Probing the Meaning of a Phrase What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office”

23 Example: Probing the Meaning of a Phrase What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office” u We changed the question to receptionist and appointment staff

24 Types of Probes (cont.) u Retrieval –How did you remember that? u Judgment –Why did you pick that number for your answer? u Response –Do you think that most people answer this question honestly? Collins D. Quality of Life Research :

25 Types of Probes (cont.) u Redundancy –How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”?

26 Types of Cognitive Probes (cont.) u Cultural appropriateness and meaning: –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?”

27 Types of Cognitive Probes (cont.) u Cultural appropriateness and meaning: –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?” u We changed the question to “personal beliefs about your health”

28 Overcoming “Social Desirability” in Cognitive Interviews u Ask respondents whether they think others… –would have difficulty answering a question –would answer the question honestly

29 Example of Probe on Difficulty: CES- D Item “During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends” u Probe: Do you feel this is a question that people would or would not have difficulty understanding? –Latinos more likely than other groups to report people would have difficulty TP Johnson, Health Survey Research Methods, 1996

30 Example: Probing the Process of Answering u I asked you _____ and you answered____. –Why did you pick this answer? –What were you thinking of when you picked this answer? –Can you tell me what you were thinking when you answered this way? »Can you give me some examples? u What came to mind when I asked you _____?

31 Example: Use of Response Scale u Do diverse groups use the response scale in similar ways? u Re questions about cultural competence of providers –Interviewers reported that Asian respondents who were completely satisfied did not like to use the highest score on the rating scale CPEHN Report, 2001

32 Sample Result: Use of Response Scale u In an exercise class of Samoans, instructor asked them to rate the difficulty of the exercise he just did on a 1-10 scale u They did not understand what he meant by a 1-10 scale –“Western” metric?

33 Cognitive Interviewing: Example of Probing Questions Acceptability of Questions u I asked you _____. –Did you find this question offensive? –Was it distressing to answer this question?

34 Example of Probe for Acceptability u When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). –Were you offended by this question?

35 Interviewer’s Role: Spontaneous Probing u Probe on items that appear to be problematic u If a long pause in answering.. –“I noticed you pausing - what came to mind when you I asked you that question?”

36 Other Cues to Problems in Face-to- Face Pretests When administering the survey… u Be aware of behavioral cues related to specific items or to questions in general –Discomfort –Yawning –Looking at their watch

37 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

38 Two Approaches to Probing u Concurrent probing –Ask probes immediately after respondent has given answer to survey item –Advantage-information is fresh in respondent’s mind u Retrospective probing –Ask probes after entire interview –Advantage-able to assess standard administration of items

39 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

40 Special Script for Recruiting People for Cognitive Interviewing u Respondents often do not understand their role u Explain to them how their help fits into the larger study, goal of main study, process of creating questions u Explain their role clearly: –“help us learn how to ask better questions” –“help us make questions clearer for others”

41 Recruiting Subjects for Cognitive Interviewing u Explain who you are, purpose of pretest, their role, expected duration (1-1.5 hours) –Subjects told it is a pretest and that their task is to help by identifying problems u Pay subjects since it is a special demand on them (if you can) u If survey is long, may have to pretest different sections on different subjects  Requires more total subjects

42 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

43 Administer Questionnaire and Probes u Administer entire questionnaire –Or break into sections u Administer probes u Audiotape entire interview including probes –Transcribe audiotapes RESULT: information on the entire process of administering the questionnaire

44 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

45 Summarize Data on Pretest Interviews u Summarize problems and nature of problems for each item u Determine how important problems are u Data become basis for possible revisions/adaptations

46 Behavioral Coding u Review transcripts to identify problems u Two types of problems: –Problems with standard administration of items »respondent and interviewer “problem” behaviors –Problems identified with specific probe questions

47 Examples of Behavioral Codes for Standard Administration Interviewer behavior u Hard to read - interviewer experiences difficulty reading question Respondent behavior u Repeat question - respondent asks to have the question repeated u Asks for clarification

48 Summarize Behavioral Coding For Each Item u Proportion of interviews with each problematic behavior u For standard administration –# of occurrences of each problem divided by N »e.g., 7/48 respondents requested clarification u For probes –# of occurrences of each problem divided by N for probe »e.g., 2/15 respondents did not understand meaning based on response to probe

49 Behavioral Coding Summary Sheet: Standard Administration (N=20) Item # Interviewer: difficulty reading Subject: asks to repeat Q Subject: asks for clarification 12/2001/ /202/20 401/200

50 Examples of Behavioral Codes Based on Probes u Probe on meaning –Open-ended responses indicate lack of understanding u Probe on use of response options –Open-ended responses indicate response options are problematic

51 Example: Probing the Meaning of a Phrase I asked you how often doctors asked you about your health beliefs. What does the term “health beliefs” mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?” Behavioral coding: Count number of times a subject did NOT understand meaning as intended

52 Behavioral Coding Summary Sheet: Standard Administration (N=20) + Probes Item #Probes Meaning unclear Interviewer -difficulty reading Subject: asks to repeat Q Subject: asks for clarification 1102/102/2001/ /151/203/202/ /200

53 Summarize Behavioral Coding u Determine if problem is common –Items with only a few problems may be fine –Items are questionable when »several types of problems were found »several subjects experienced the same problem

54 Behavioral Coding Summary Sheet: Standard Administration (N=20) + Probes Item #Probes Meaning unclear Interviewer -difficulty reading Subject: asks to repeat Q Subject: asks for clarification 1102/102/2001/ /151/203/202/ /200

55 Summarize Behavioral Coding (cont.) u Determine if common problems with an item are serious –Gross misunderstanding of the question –Yields completely erroneous answer –Couldn’t answer the question at all u Some less serious “common” problems can be addressed by improved instructions or a slight modification

56 Now What! u Issues in adapting measures based on pretest results

57 Options for Adapting or Revising u Depend on whether you are using a standard measure or a new measure

58 Writing New Items u One approach to finding problems with a standard measure in diverse groups –Write some new items that you think will be better for this group –Write them in the same format as the standard measure u Can test the adequacy of the original measure and a “modified” measure that includes your new items –A way to begin to develop new items

59 Adapting Measures u If a measure needs modification in your opinion –Can contact author(s) to see if they would like to work with you –Be sure your opinion is based on extensive pretests in several groups by your group and/or other researchers –Work with a measurement specialist to assure that your proposed modifications are likely to solve the problem

60 Internal versus External Validity u Internal validity - your study is able to answer your research questions with a measure u External validity - your results on a measure can be compared to other studies using the same measure

61 Tradeoffs of Using Adapted Measures u Modifying measures –Can improve internal validity - able to answer your question with this measure –You lose external validity - can’t compare your scores to other studies u Can test measure in original and adapted form –If “adapted” measure works better, use it u If “adapted” measure does not work better –Can still use original measure (it may not work either)

62 Handouts: Examples of Probe Questions u Sample Cognitive Interview Protocol –Ron Hays and Leo Morales u Selected probes from Interpersonal Processes of Care study u Summary of NCHS working paper series on cognitive methods (website)

63 Homework for Next Class (Dec 2) u Identify 4-5 questions in measure you have selected that might be a problem for your target population –Phrases, length, response choices u Write one “probe” question for each problem u Conduct 2 pretests on persons similar to those you are interested in u Summarize briefly your results