1. 1 Class 8 Choosing Measures to Pretest, Creating a Questionnaire and Questionnaire Guide, Methods for Pretesting Measures November 9, 2006 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2. 2 Basic Steps in Selecting Appropriate Measures 1. Specify context 2. Define concept for your study 3. Identify and review potential measures for a) conceptual and psychometric adequacy b) practicality and acceptability c) translation available if needed 4. Select best candidates 5. Pretest selected measures in your target population 6. Choose best ones based on pretest results OR 7. Adapt if necessary to address problems

3. 3 Overview of Class 8 u Choosing measures for pretesting u Creating a questionnaire and a guide to measures in your questionnaire u Methods for pretesting measures

4. 4 4. Select Best Candidate Measures of Your Concept to Pretest u Select best measures for all concepts in your conceptual framework –Preferably at least 2 measures per concept u Pay special attention to priority concepts –Outcome measures or main independent variables

5. 5 Strategies for Selecting Best Measures u Select existing instrument in its entirety u Select subscales of relevant domains from existing instruments –only those that meet your needs u Supplement existing instrument with additional measures –core plus modules approach

6. 6 What if no “Best” Measure? u For priority concepts with inadequate measure, could select optimal measure and adapt or modify u For priority concepts with no measure, may require development of new measure

7. 7 Overview of Class 8 u Choosing measures for pretesting u Creating a questionnaire and a guide to measures in your questionnaire u Methods for pretesting measures

8. 8 Process of Creating Questionnaire A. Introduction B. Statement of confidentiality C. Length D. Sections, section headings to break it up E. Order of questions, measures F. Formatting G. Conclusion

9. 9 Process A. Introduction u Inform respondent of purpose of overall study and of this questionnaire u Who is conducting the study u Topics included u Expected time to complete u Assurance that participation is voluntary, can skip any questions u Specific instructions for completing

10. 10 Process B. Statement of Confidentiality u Purpose: inform respondents of extent to which their answers are protected  if there are risks to confidentiality, state them u Sample Statement: “All information that would permit identification of individuals will be regarded as strictly confidential, will be used for purposes of evaluating the study, and will not be disclosed or released for any other purposes without prior consent, except as required by law.”

11. 11 Statement of Confidentiality (cont.) u Note that this is a very high reading level u Would need to simplify for lower SES group

12. 12 Process C. Length of Questionnaire Acceptable length depends on:  Commitment of respondents to topic area  Health, free time of respondents  Short forms of most measures can help  For those too ill or busy to complete long survey  Minimizes bias due to systematic nonresponse of those more ill

13. 13 Process D. Sections of Questionnaire u Break up with sections, grouped by topic (minimizes psychological burden) u Introduce new topics with –Phrases - “the next questions are about how you’ve been feeling” –Simple section headings, e.g., How You’ve Been Feeling

14. 14 Process E. Question Order u Order can affect willingness to complete survey u Begin with general questions –Easy, non-threatening, interesting, related to purpose of study u Proceed with more specific questions, more personal and sensitive questions u Conclude with demographics (least interesting, sensitive)

15. 15 Process F. Formatting Self-Administered Questionnaires u Goal - make the tasks of reading questions, following instructions, and recording answers as easy as possible for respondents –Clear instructions for indicating their answer –Easy to track separate questions

16. 16 Principles of Formatting u Create a lot of space on page u Use very light background for best contrast u Number all questions u Allow sufficient space for open-ended questions –Lines far enough apart for large script u Special issues for older adults: –Larger font size (14) –Higher contrast (black on white)

17. 17 Indicate How to Respond u Give specific instructions for answering questions, e.g., –Circle all that apply –Circle one number –Put a check in the box that best represents how you feel u If no instructions, subjects will have to figure out what to do and may not do it correctly

18. 18 Types of Formats: Horizontal u How would you rate your health in general? (circle one number) 1 Excellent 2 Good 3 Fair 4 Poor (easy to get confused)

19. 19 Types of Formats: Horizontal u How would you rate your health in general? (check one box)  Excellent  Good  Fair  Poor (Still easy to get confused)

20. 20 Types of Formats: Vertical (circle one number) No, not limited at all...........1 Yes, limited a little.......…..2 Yes, limited a lot...........….3 During the past 4 weeks, did your health limit you in walking one block..

21. 21 Types of Formats: Matrix (circle one number) How often during the past 4 weeks did No, not Yes, Yes, your health limit limited limited limited you in... at all a little a lot Walking one block 1 23 Walking several blocks 1 23 Climbing 1 flight of stairs 1 2 3

22. 22 Types of Formats: Matrix with Boxes (Check one box on each row) How often during the past 4 weeks did No, not Yes, Yes, your health limit limited limited limited you in... at all a little a lot Walking one block Walking several blocks Climbing 1 flight of stairs Easier to administer, but harder for data entry

23. 23 Process G. Conclude Questionnaire u Add a brief thank you for completing the questionnaire (for their time and effort) u If questionnaire is to be mailed back, include instructions for mailing –Always include a pre-stamped, preaddressed return envelope

24. 24 Sample Questionnaires u See sample questionnaires –(sample formats)

25. 25 Questionnaire Guide: Organization of Survey Content u Identifies purpose of each measure in the survey (questionnaire) –Describes source of data, time points –Describes the number of scales for each measure u Documents sources of measures and items before you forget u For modified measures, helps you know exactly how you modified it

26. 26 See Organization of Survey Content Handout u Type of variable u Concept u Measure u Data source u Data points u Number of items/survey question numbers u Number of scores or scales for each measure u References for measure sources

27. 27 Overview of Class 8 u Choosing measures for pretesting u Creating a questionnaire and a guide to measures in your questionnaire u Methods for pretesting measures

28. 28 Step 5: Pretest in Target Population u Pretesting essential for measures being applied to any new population group –Especially priority measures (e.g., outcomes) u Pretest is to identify: –problems with method of administration, respondent burden, procedures –problems with questions or response choices

29. 29 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

30. 30 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

31. 31 General Debriefing Pretest u Goal –Identify problems with the procedures –Estimate time needed to complete instrument –Identify serious problems with items u Procedures –Subjects answer entire questionnaire –At end, debrief –Close to true task

32. 32 Debriefing Questions After Administration of Survey.. Ask respondents: u Were any questions confusing? u Which words were hard to understand? u Which questions were difficult to answer? caused distress? u Was questionnaire too long? u Confusing instructions?

33. 33 Patient Acceptance Scale u 6-item scale about burden of a survey u Negative affect burden –Feeling embarrassed, upset, annoyed, uncomfortable u Questionnaire length u Ease of answering questionnaire Zimmerman M et al., Med Care, 1994;32:603-608

34. 34 Example of General Pretest to Select Best Measure of Patient Satisfaction u Compared 4 questionnaires on –Response rates, missing data, completion time, patient ratings of the questionnaire u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., A randomized trial of four patient satisfaction questionnaires Med Care, 2003;41(12):1343-1352

35. 35 Example of General Pretest to Select Best Measure of Patient Satisfaction u Compared 4 questionnaires on –Response rates, missing data, completion time, patient ratings of the questionnaire u 10 evaluation items at end of questionnaire –Clarity of questions, ease of finding an answer, questions about unimportant issues, ease of completion, too long, layout confusing, lacked important questions Perneger TV et al., A randomized trial of four patient satisfaction questionnaires Med Care, 2003;41(12):1343-1352

36. 36 Problems with General Pretests Respondents… u often don’t understand the task. u don’t want to appear as if they didn’t understand u have a hard time telling you anything was wrong u easier to say everything was fine

37. 37 Types of Pretests u General debriefing pretest (N=10) u In-depth cognitive interviewing (N=5-10 each group) u Large pretest (N=100) –test measurement properties prior to major study

38. 38 In-Depth Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions

39. 39 Cognitive Interviews Examine 4 Steps in Answering Questions u Comprehend the question (as intended) u Retrieve the information –various strategies used to access memory u Judgment formation - formulate an answer –calculate or judge the correct information u Edit response - decide what to report –is answer embarrassing, socially undesirable? Sudman S et al., Thinking About Answers, Jossey-Bass, San Francisco, 1996

40. 40 Purpose of Cognitive Interviews To learn.. u.. if respondents understand words and phrases as intended (meaning) u.. about the process of answering the questions u.. whether items are unacceptable u.. about the usefulness of response choices –whether response choices are adequate –how they use the response choices

41. 41 Two Types of Cognitive Interviews u Think aloud interviews –Respondent asked to think aloud as they answer question u Probe interviews –Interviewer asks specific questions to elicit how respondent answered question –Scripted and spontaneous probing

42. 42 Think Aloud Interviews u Thorough examination of the entire thought process of creating answers u Require that respondents verbalize their thought processes as they are answering items –Hard task for most people u Mostly done at “cognitive interviewing” labs, e.g., at National Institutes of Health

43. 43 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

44. 44 Probe Questions to Explore Meaning u What does the word _______ mean to you? u What does the phrase ________ mean to you?

45. 45 Example: Probing the Meaning of a Phrase I asked you about how the office staff treated you personally …. What does the phrase “office staff” mean to you?

46. 46 Example: Probing the Meaning of a Phrase What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office”

47. 47 Example: Probing the Meaning of a Phrase What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office” u We changed the question to receptionist and appointment staff

48. 48 Types of Probes (cont.) u Retrieval –How did you remember that? u Judgment –Why did you pick that number for your answer? u Response –Do you think that most people answer this question honestly? Collins D. Quality of Life Research 2003. 12:229-38.

49. 49 Types of Probes (cont.) u Redundancy –How is the phrase “give you advice about your diet and exercise” different from the phrase “talk to you about your diet and exercise”?

50. 50 Types of Cognitive Probes (cont.) u Cultural appropriateness and meaning: –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?”

51. 51 Types of Cognitive Probes (cont.) u Cultural appropriateness and meaning: –I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?” u We changed the question to “personal beliefs about your health”

52. 52 Overcoming “Social Desirability” in Cognitive Interviews u Ask respondents whether they think others… –would have difficulty answering a question –would answer the question honestly

53. 53 Example of Probe on Difficulty: CES- D Item “During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends” u Probe: Do you feel this is a question that people would or would not have difficulty understanding? –Latinos more likely than other groups to report people would have difficulty TP Johnson, Health Survey Research Methods, 1996

54. 54 Example: Probing the Process of Answering u I asked you _____ and you answered____. –Why did you pick this answer? –What were you thinking of when you picked this answer? –Can you tell me what you were thinking when you answered this way? »Can you give me some examples? u What came to mind when I asked you _____?

55. 55 Example: Use of Response Scale u Do diverse groups use the response scale in similar ways? u Re questions about cultural competence of providers –Interviewers reported that Asian respondents who were completely satisfied did not like to use the highest score on the rating scale CPEHN Report, 2001

56. 56 Sample Result: Use of Response Scale u In an exercise class of Samoans, instructor asked them to rate the difficulty of the exercise he just did on a 1-10 scale u They did not understand what he meant by a 1-10 scale –“Western” metric?

57. 57 Cognitive Interviewing: Example of Probing Questions Acceptability of Questions u I asked you _____. –Did you find this question offensive? –Was it distressing to answer this question?

58. 58 Example of Probe for Acceptability u When I asked you how often you felt discriminated against by doctors because of your race or ethnicity, you answered (read answer given). –Were you offended by this question?

59. 59 Steps in Cognitive Interview Pretesting u Identify most problematic items and develop scripted “probes” u Decide on order of administration –Probes after each question or at the end u Recruit sample for cognitive interviews u Administer entire questionnaire –Administer probe questions u Summarize results

60. 60 Handouts: Examples of Probe Questions u Sample Cognitive Interview Protocol –Ron Hays and Leo Morales u Selected probes from Interpersonal Processes of Care study u Summary of NCHS working paper series on cognitive methods (website)

61. 61 Homework for Week 9 u Create a short “questionnaire” of the measure, including instructions u Identify 4-5 questions in the measure you selected that might be a problem for your target population –Phrases, length, response choices u Write a “probe” question for each problem –Include some on the meaning of a word or phrase and some on the adequacy of the response choices.

62. 62 Vogt et al. u Role of focus groups in adapting measures for diverse populations u How focus groups led to expansion and addition of constructs, confirmation of existing items, and identification of relevant quotes that could be used for new items u Appendices provide examples of construct definitions and a focus group guide

63. 63 Harris-Kojetin Reading u Example of Measure Undergoing Cognitive Testing –Consumer Assessment of Health Plans Study (CAHPS TM ) u Describes extensive pretesting in developing and refining the CAHPS TM u Provides example of good pretesting

64. 64 Pasick et al. reading u Provides examples of what can be learned through various types of pretests u Methods included –interviewing investigators and interviewers –focus groups –cognitive interview testing u Suggests that for important measures, multiple approaches to pretesting are useful

65. 65 Switzer et al. reading u From class 3 section of class binder u p 405-406 – modifying measures