1. 1 Class 8 Pretesting Measures and Creating a Questionnaire November 6, 2008 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2. 2 Overview of Class 8 u Choosing best measure(s) for pretesting u Creating a questionnaire u Methods for pretesting measures

3. 3 Process of Selecting Good Measures for Your Studies Define concept (variable) Identify potential measures Review measures’ properties --conceptual adequacy --psychometric adequacy Pretest best 1-2 measures Select final measure

4. 4 Select Best Candidate Measures to Pretest u Select best measures for all concepts in your conceptual framework u Pretest measures you plan to use as outcome measures or key independent variables –Pretest 2 measures if possible so you can choose the best one

5. 5 Strategies for Selecting Best Measure of “Unidimensional” Concept u Select existing measure in its entirety u Select two existing measures –Have a choice later based on psychometric results

6. 6 Strategies for Selecting Best Measure of “Multidimensional” Concept u Select existing instrument in its entirety u Select subscales of only relevant domains from existing instruments –those that meet your needs u Supplement either of the above with subscales from other measures

7. 7 What if There is No “Best” Measure? u For priority concepts with inadequate measure, select the one that is the closest in your review –Use pretest or other qualitative studies as basis to adapt or modify u For priority concepts with no measure that meets your needs, may require development of new measure

8. 8 “Hybrid” Measures u Combine items from more than one published measure (or combining subscales) u Add new items to existing measures –Good when you find missing concepts from an otherwise good measure

9. 9 “Modified” or “Hybrid” Measures u Re-read Switzer 1999 (from class 3 readings) on “modified measures” and “hybrid measures” (p. 405-406) u Note limitations of modifying: –“substantial psychometric work is needed” – depends on magnitude of modifications

10. 10 Overview of Class 8 u Choosing measures for pretesting u Creating a questionnaire u Methods for pretesting measures

11. 11 Considerations in Creating a Questionnaire A. Introduction B. Statement of confidentiality C. Length D. Sections, section headings to break it up E. Order of questions, measures F. Formatting G. Conclusion

12. 12 A. Introduction: Inform respondent … u Purpose of overall study and of this questionnaire u Who is conducting the study u Topics included u Expected time to complete u Assurance that participation is voluntary, can skip any questions u Specific instructions for completing

13. 13 B. Statement of Confidentiality u Purpose: inform respondents of extent to which their answers are protected  if there are risks to confidentiality, state them u Can us section from consent form pertaining to survey data

14. 14 Sample Statement u “All information that would permit identification of individuals will be regarded as strictly confidential, will be used for purposes of evaluating the study, and will not be disclosed or released for any other purposes without prior consent, except as required by law.”

15. 15 Statement of Confidentiality (cont.) u Note that this is a very high reading level –Would need to simplify for lower SES group u “All information about you will be kept as confidential as possible. Your name will not be used in any published reports about this study.”

16. 16 C. Length of Questionnaire  Acceptable length depends on:  Interest of respondents in topic area  Health  Free time of respondents

17. 17 D. Create Sections of Questionnaire u Break up with sections, grouped by topic (minimizes psychological burden) u Introduce new topics with –Phrases - “the next questions are about how you’ve been feeling” –Simple section headings, e.g., How You’ve Been Feeling

18. 18 E. Question Order u Order can affect willingness to complete survey u Begin with general questions –Easy, non-threatening, interesting, related to purpose of study u Proceed with more specific questions, personal and sensitive questions u Conclude with demographics (least interesting, sensitive)

19. 19 F. Formatting Self-Administered Questionnaires u Make the tasks of reading questions, following instructions, and recording answers as easy as possible for respondents –Clear instructions for indicating their answer –Easy to track separate questions

20. 20 Principles of Formatting u Create a lot of space on page u Use very light background for best contrast –Minimize use of color as background u Number all questions (unique number for each one helps data entry) u Allow sufficient space to write answers to open-ended questions u Special issues for older adults: –Larger font size (14) –Higher contrast (black on white)

21. 21 Indicate How to Respond u Give specific instructions for answering questions, e.g., –Circle all that apply –Circle one number –Check the box that best represents how you feel u If no instructions, subjects will have to figure out what to do and may not do it correctly

22. 22 Types of Formats: Horizontal How would you rate your health in general? (check one box)  Excellent  Good  Fair  Poor (Easy to get confused)

23. 23 Types of Formats: Respondent Writes Response Choice For each question, write in one of the following numbers: 1 = poor 2 = fair 3 = good 4 = excellent ___ 1. How would you rate the availability of medical information by phone? ___ 2. How would you rate the length of time spent waiting at the office to see a provider?

24. 24 Types of Formats: Vertical (circle one number) No, not limited at all...........1 Yes, limited a little.......…..2 Yes, limited a lot...........….3 During the past 4 weeks, did your health limit you in walking one block..

25. 25 Types of Formats: Matrix (Circle one number on each line) How often during the past 4 weeks did your health limit you …. No, not limited at all Yes, limited a little Yes, limited a lot 1. Walking one block123 2. Walking several blocks123 3. Climbing one flight of stairs123

26. 26 Types of Formats: Matrix with Boxes (Check one box on each line) How often during the past 4 weeks did your health limit you …. No, not limited at all Yes, limited a little Yes, limited a lot 1. Walking one block 2. Walking several blocks 3. Climbing one flight of stairs

27. 27 G. Conclude Questionnaire u Add a brief thank you for completing the questionnaire (for their time and effort) u If questionnaire is to be mailed back, include instructions for mailing –Always include a pre-stamped, preaddressed return envelope

28. 28 Sample Questionnaires u See sample questionnaires –(sample formats)

29. 29 Overview of Class 8 u Choosing measures for pretesting u Creating a questionnaire u Methods for pretesting measures

30. 30 Pretest in Target Population u Pretesting essential for measures being applied to any new population group –Especially priority measures (e.g., outcomes) u Pretest is to identify: –problems with procedures »method of administration, respondent burden –problems with questions »Item stems, response choices, and instructions

31. 31 Problems with Questions or Response Choices u Are all words/phrases understood as intended? u Are questions interpreted similarly by all respondents? u Are some questions not answered? u Are any questions offensive or irrelevant? u Does each closed-ended question have an answer that applies to each respondent? –Are the response choices adequate?

32. 32 Types of Pretests u General debriefing pretest (N=10) u Large pretest (N=100) –test measurement properties prior to major study u In-depth cognitive interviewing pretest (N=5-10 each group)

33. 33 Types of Pretests u General debriefing pretest (N=10) u Large pretest (N=100) –test measurement properties prior to major study u In-depth cognitive interviewing (N=5-10 each group)

34. 34 General Debriefing Pretest u Goal –Identify problems with the procedures –Estimate time needed to complete instrument –Identify serious problems with items u Procedures –Subjects answer entire questionnaire –At end, debrief –Close to true task

35. 35 Sample Debriefing Questions After Administration of Survey.. Ask respondents: u Were any questions confusing? u Which words were hard to understand? u Which questions were difficult to answer? caused distress? u Was questionnaire too long? u Were instructions confusing?

36. 36 Problems with General Pretests u Respondents… –often don’t understand the task. –don’t want to appear as if they didn’t understand –have a hard time telling you anything was wrong –easier to say everything was fine

37. 37 Types of Pretests u General debriefing pretest (N=10) u Large pretest (N=100) –test measurement properties prior to major study u In-depth cognitive interviewing pretest (N=5-10 each group)

38. 38 Large Pretest – N=100 u Test psychometric properties and procedures for large-scale administration –Administer surveys –Examine item distributions, missing data –Calculate internal-consistency reliability –Conduct some preliminary validity studies u Only large-scale studies have this luxury

39. 39 “Questionnaire” Asking Opinions About another Questionnaire/Survey u 6-items about burden of a survey –Questionnaire length –Ease of answering questionnaire –Negative affect burden (4 items) »Feeling embarrassed, upset, annoyed, uncomfortable M Zimmerman et al., Med Care, 1994;32:603-608

40. 40 4-item Negative Affect Burden Scale u How much were you annoyed by being asked these questions? u How much were you embarrassed by being asked these questions? u How much did it upset you to be asked these questions? u How uncomfortable did it make you feel to be asked these questions? u Responses: very much, somewhat, a little, not at all Zimmerman M et al.1994

41. 41 Large Pretest of 4 Patient Satisfaction with Hospital Care Measures (N=2,850) u Compared 4 questionnaires on –Response rates, missing data, completion time u 10 evaluation items at end of questionnaire (1-5 agree-disagree scale) –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions TV Perneger et al., Med Care 2003;41:1343-1352

42. 42 Large Pretest of 4 Patient Satisfaction with Hospital Care Measures (N=2,850) u Compared 4 questionnaires on –Response rates, missing data, completion time u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., Med Care 2003;41:1343-1352

43. 43 Missing Concepts in H-CAHPS Pretest u Pretest included 2 open-ended questions: –What did you like most about the care you received? –What aspect of care would you change? u Good way to examine “content validity” or conceptual relevance u Analysis of responses: closed-ended items covered most aspects of care that patients mentioned MP Hurtado et al. Health Serv Res, 2005;40-6, Part II:2140-2161

44. 44 Medical Outcomes Study Measures u On long-form measures, conducted 9 full- scale pilot studies over a 1-year period –Physical functioning, health perceptions, energy/fatigue, sleep, role functioning, pain, symptoms, family functioning, sexual functioning u Samples ranged from 50-100 each Stewart AL and Ware JE, Measuring Functioning and Well-Being, 1992, p. 72..

45. 45 Example: 3 Pilot Studies of MOS Health Perceptions Measures u Measures based on historic fairly long measures from earlier studies u Tested new items measuring health distress (old item – worry about health) u Eliminated several items to shorten long scales

46. 46 Types of Pretests u General debriefing pretest (N=10) u Large pretest (N=100) –test measurement properties prior to major study u In-depth cognitive interviewing pretest (N=5-10 each group)

47. 47 In-Depth Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions u Goal: understand thought processes used to answer questions u Can help write/adapt questions

48. 48 Cognitive Interviews Examine 4 Steps in Answering Questions u Comprehend question (as intended) u Retrieve information - various strategies used to recall information u Judgment formation - calculate the correct response u Response - decide what to report Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996

49. 49 Comprehend Questions as Intended u Bulk of cognitive interview pretests assess whether your items are interpreted by respondents as you intended

50. 50 Retrieve Information u Information needed to answer the question must be retrieved from memory –Episodic data – stored in detail –Schemas – generalizations about events u Frequent, routine events more likely to be stored as “schema” with no detail

51. 51 Example of Retrieval Schema u People with chronic conditions and a lot of doctor visits tend to have “generic” memories of these visits –Multiple visits blurred together –Could describe a typical visit but not any visit in particular –Hard to recall how many visits

52. 52 Judgment Formation u Involves using the recalled information to estimate frequency of an event, or to decide whether one agrees or disagrees with a statement u Cues taken from the item response choices

53. 53 Response u After respondent has judged what the best answer is, decides how to respond –Accurately? –To improve social desirability? –To minimize embarrassment?

54. 54 Two Basic Types of Cognitive Interviews u Think aloud interviews –Respondent asked to think aloud as they answer each question u Probe interviews –Interviewer asks specific questions to determine how respondent answered question –Scripted and spontaneous probing

55. 55 Think Aloud Interviews u Thorough examination of the entire thought process of creating answers u Require that respondents verbalize their thought processes as they are answering items –Hard task for most people u Mostly done at “cognitive interviewing” labs –e.g., at NIH or Census Bureau

56. 56 Probe Interviews: Types of Probes u Interpret meaning as you intended u Ease of answering – appropriate response choices u Acceptability of questions –Culturally appropriate u Clarity, difficulty u Elicit socially desirable responding

57. 57 Determining if Respondents Understand Meaning as Intended u What do you think we meant by the phrase ________? u Threatening phrasing – implies there is a right answer

58. 58 More Appropriate Probe Questions to Explore Meaning u What does the word _______ mean to you? u What do you understand by _______ ? u What came to mind when I asked you _____? u Ask person to paraphrase a question –Repeat question in your own words

59. 59 Probing Meaning of a Phrase u I asked you about how the office staff treated you personally …. –What does the phrase “office staff” mean to you? u I asked you how often doctors asked you about your health beliefs? –What does the term ‘health beliefs’ mean to you?

60. 60 Probing Meaning of Phrase u After a series of questions about the person’s main job –You mentioned earlier that you had more than one job. How did you decide which job was your main job? DeMaio TJ et al., Answering Questions, Jossey-Bass, 1996, p. 177..

61. 61 Probing Meaning of Entire Question u Do you go to the doctor for checkups even when you’re well? –What does the word “checkup” mean to you? –Can you rephrase this question in your own words? Pasick RJ et al., Pub Health Rep, 2001;116:223.

62. 62 Probing Meaning of Entire Question u In the last 6 months, have you had problems getting to see a specialist? (yes/no) –Tell me more about that. –What kind of doctor do you think of as a specialist? –If yes - What kinds of problems have you had?

63. 63 Confidence in Answer u After respondent answers a question –How confident are you in the answer you gave? »Very confident, somewhat confident, not confident at all u Lack of confidence is a flag for a problem

64. 64 Probing Use of Response Scale u I asked you _____ and you answered____. u What were you thinking of when you picked this answer? u Can you tell me why you picked that number for your answer? u How did you decide to choose that answer? u Were you able to find the right answer to the question from the response options?

65. 65 Probing Retrieval u How did you remember that? u Did you have a particular time period in mind? u How did you calculate your answer? u Can you tell me what you were thinking when you answered this way? »Can you give me some examples?

66. 66 Probing Questions to Determine Acceptability I asked you _____. –Did you find this question offensive? –Was it distressing to answer this question?

67. 67 Overcoming “Social Desirability” in Cognitive Interviews u Ask respondents whether they think others –would have difficulty answering a question –would answer honestly u Example: –Do you think that most people would answer this question honestly?

68. 68 Probe on Difficulty: CES-D Item “During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends” –Do you feel this is a question that people would or would not have difficulty understanding? TP Johnson, Health Survey Research Methods, 1996

69. 69 Probe for Acceptability u When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). –Were you offended by this question? –Do you think people would be offended by this question?

70. 70 Steps in Cognitive Interview Pretesting u Identify problematic items, develop scripted “probes” u Decide whether to probe after each question or at the end u Recruit sample for cognitive interviews u Administer questionnaire and probe questions u Tape record and transcribe u Summarize results

71. 71 Recruiting for Cognitive Interviewing u Your respondent is assuming two tasks: –Answering questions (normal interview) –Being debriefed OR answering questions about their answers (cognitive interviewing) u Difficult to transition from one to the other

72. 72 Recruiting for Cognitive Interviewing (cont) u Explain how their help fits into the process of creating questions u Ways to explain –“help us learn how to ask better questions” –“help us make questions clearer for others” –“help us learn how we can improve this questionnaire” –“help us find problems with the questions”

73. 73 Recruiting for Cognitive Interviewing (cont.) u Pay subjects (if you can) –Special demand on them u If survey is long, may have to pretest different sections on different subjects –Requires more total subjects

74. 74 Administer Questionnaire and Probes u Administer entire questionnaire –Or break into sections u Administer probes u Audiotape entire interview including probes –Transcribe audiotapes RESULT: information on the entire process of administering the questionnaire

75. 75 Two Sequences to Probing u Concurrent probing –Ask probes immediately after respondent has given answer to survey item –Advantage-information is fresh in respondent’s mind u Retrospective probing –Ask probes after entire interview –Advantage-able to assess standard administration of items

76. 76 Interviewer’s Role: Spontaneous Probing u Probe on items that appear to be problematic u If a long pause in answering.. –“I noticed you pausing - what came to mind when you I asked you that question?”

77. 77 Other Cues to Problems in Face-to- Face Pretests When administering the survey… u Be aware of behavioral cues related to specific items or to questions in general –Discomfort –Yawning –Looking at their watch

78. 78 Handouts: Sample Cognitive Interviews u Sample Cognitive Interview Protocol –Ron Hays and Leo Morales (UCLA) »Experts at cognitive interviewing pretests u Cognitive interview protocols for Interpersonal Processes of Care study –Divided into two parts (Part I and Part II) –Each part: a questionnaire and the cognitive interview questions accompanying it

79. 79 Other Handouts u Using cognitive interviews to develop structured surveys (May 2007) –Annotated bibliography of articles describing methods of cognitive interviews u Sample guide to measures used in questionnaire/survey

80. 80 Homework for Class 9 u Create “mini questionnaire” of one measure –Formatted for self-report, including instructions u Write 4 “probe” questions of selected items –Include some on the meaning of a word/phrase and some on adequacy of response choices u Identify source of 2 pretest “subjects” similar to those you would be including in your research