1. 1 Class 9 Steps in Creating and Testing Scale Scores, and Presenting Measurement Data December 1, 2005 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2. 2 Overview u Steps in creating and testing scales in your sample –Preparing raw data –Deriving scale scores –Testing scaling properties in your sample u Presenting measurement results u Presenting change scores

3. 3 Overview u Steps in creating and testing scales in your sample –Preparing raw data –Deriving scale scores –Testing scaling properties in your sample u Presenting measurement results u Presenting change scores

4. 4 Preparing Raw Data u Prepare surveys for data entry u Data entry u Range and consistency checks

5. 5 Preparing Surveys for Data Entry: 4 Steps u Review surveys for data quality u Reclaim missing and ambiguous data u Address ambiguities in the questionnaire prior to data entry u Code open-ended items

6. 6 Review Surveys for Data Quality u Examine each survey in detail as soon as it is returned, and mark any.. –Missing data –Inconsistent or ambiguous answers –Skip patterns that were not followed

7. 7 Reclaim Missing and Ambiguous Data u Go over problems with respondent –If survey returned in person, review then –If mailed, call respondent ASAP, go over missing and ambiguous answers –If you cannot reach by telephone, make a copy for your files and mail back the survey with request to clarify missing data

8. 8 Address Ambiguities in the Questionnaire Prior to Data Entry u When two choices are circled for one question, randomly choose one (flip a coin) u Clarify entries that might not be clear to data entry person

9. 9 Code Open-Ended Items u Open-ended responses have no numeric code –e.g., name of physician, reason for visiting physician u Goal of coding open-ended items –create meaningful categories from variety of responses –minimize number of categories for better interpretability –Assign a numeric score for data entry

10. 10 Example of Open-Ended Responses 1.What things do you think are important for doctors at this clinic to do to give you high quality care? u Listen to your patients more often u Pay more attention to the patient u Not to wait so long u Be more caring toward the patient u Not to have so many people at one time u Spend more time with the patients u Be more understanding

11. 11 Process of Coding Open-Ended Data u Develop classification scheme –Review responses from 25 or more questionnaires –Begin a classification scheme –Assign unique numeric codes to each category –Maintain a list of codes and the verbatim answers for each –Add new codes as new responses are identified u If a response cannot be classified, assign a unique code and address it later

12. 12 Example of Open-Ended Codes Communication = 1 u Listen to your patients more often = 1 u Pay more attention to the patient = 1 Access to care = 2 u Not to wait so long = 2 u Not to have so many people at one time = 2 Allow more time = 3 u Spend more time with the patients = 3 Emotional Support = 4 u Be more understanding = 4 u Be more caring toward the patient

13. 13 Verify Assigned Codes u Ideally, have a second person independently classify each response according to final codes u Investigator can review a small subset of questionnaires to assure that coding assignment criteria are clear and are being followed

14. 14 Reliability of Open-Ended Codes u Depends on quality of question, of codes assigned, and the training and supervision of coders u Initial coder and second coder should be concordant in over 90% of cases

15. 15 Data Entry u Set up file u Double entry of about 10% of surveys –SAS or SPSS will compare two for accuracy »Acceptable 0-5% error »If 6% or greater – consider re-entering data

16. 16 Item Naming Conventions u Optimal coding is to assign raw items their questionnaire number –Can always link back to questionnaire easily u Some people assign a variable name to the questionnaire item –This will drive you crazy

17. 17 Print Frequencies of Each Item and Review: Range Checks u Verify that responses for each item are within acceptable range –Out of range values can be checked on original questionnaire »corrected or considered “missing” –Sometimes out of range values mean that an item has been entered in the wrong column »a check on data entry quality

18. 18 Print Frequencies of Each Item and Review: Consistency Checking u Determine that skip patterns were followed u Number of responses within a skip pattern need to equal number who answered “skip in” question appropriately

19. 19 Print Frequencies of Each Item and Review: Consistency Checking (cont.) 1. Did your doctor prescribe any medications? (yes, no) 1a. If yes, did your doctor explain the side effects of the medication? u If 75 respondents (of 90) said yes to 1, expect 75 responses to question 1a. –Often will find that more people(e.g., 80) answered the second question than were supposed to

20. 20 Print Frequencies of Each Item and Review: Consistency Checking (cont.) u Need to go back to a questionnaires of those with problems –check whether initial “filter” item was incorrectly answered or whether respondent inadvertently answered subset –sometimes you won’t know which was correct u Hopefully this was caught during initial review of questionnaire and corrected by asking respondent

21. 21 Overview u Interpreting cognitive interviewing results u Steps in creating and testing scales in your sample –Preparing raw data –Deriving scale scores –Testing scaling properties in your sample u Presenting measurement results u Presenting change scores

22. 22 Deriving Scale Scores u Develop a “codebook” of scoring rules –Handout – Summary of variables and variable coding u Create scores with computer algorithms in SAS, SPSS, or other program u Review scores to detect programming errors u Revise computer algorithms as needed u Review final scores

23. 23 Codebook of Scoring Rules (Handout) u Codebook is a scoring guide for entire instrument –Scale or subscale name, description of scale, item numbers, item scoring (e.g., reverse some items if needed), what a high score means, missing data rules –Special coding of certain items u Sometimes rules conform to published scoring rules

24. 24 Variable Naming Conventions (Variables of Composite of Items) u Assigning variable names is an important step –make them as meaningful as possible –plan them for all questionnaires at the beginning u For study with more than one source of data, a suffix can indicate which point in time and which questionnaire –B for baseline, 6 for 6-month, Y for one year –M for medical history, L for lab tests

25. 25 Variable Naming Conventions (Variables of Composite of Items) (cont.) Medical History Questionnaire HYPERTMB HYPERTM6 Baseline 6 months

26. 26 Variable Naming Conventions (cont.) u A prefix can help sort variable groupings alphabetically –e.g., S for symptoms SPAINB, SFATIGB, SSOBB

27. 27 Creating Likert Scale Scores u Translate codebook scoring rules into program code (SAS, SPSS): –Determine direction of scoring of final measure –Reverse all items that are not already in that direction –Average remaining items –Apply missing data rule u e.g., if more than 50% missing

28. 28 Review Summary Statistics of Derived Scores to Detect Programming Errors u Run raw data through program –can be a preliminary subset of raw data to debug program u Review summary statistics of scores to determine accuracy of program –Do the mean, SD make sense? –Is the observed range appropriate? Are there any cores outside the possible range? –Does the % missing seem about right?

29. 29 Revise Computer Algorithms As Needed u For those that don’t make sense, review programming statements u Locate errors and correct

30. 30 Review Final Scores u Review scores again u Repeat process until you are satisfied that the computer algorithm is producing accurate scores –For a complete test of programming accuracy, calculate a few scores by hand from one or two questionnaires »Make sure those respondents’ scores match what you get

31. 31 Overview u Steps in creating and testing scales in your sample –Preparing raw data –Deriving scale scores –Testing scaling properties in your sample u Presenting measurement results u Presenting change scores

32. 32 Testing Scaling Properties and Reliability in Your Sample for Multi-Item Scales u Obtain item-scale correlations –Part of internal consistency reliability program u Calculate reliability in your sample (regardless of known reliability in other studies) –internal-consistency for multi-item scales –test-retest if you obtained it

33. 33 Review Results (Handout) u Item-scale correlations –Be sure each item correlates at least.30 and preferably.40 with the total scale (corrected for overlap) u Internal consistency (Cronbach’s alpha) –Should be at least.70 –If lower, see if modifying items (above) will improve it u Test-retest reliability –Should meet standards

34. 34 Overview u Steps in creating and testing scales in your sample –Preparing raw data –Deriving scale scores –Testing scaling properties in your sample u Presenting measurement results u Presenting change scores

35. 35 Presenting Measurement Results (Handout) u Present for each final scale: –% missing –Mean, standard deviation –Observed range, possible range –Floor and ceiling effects, skewness statistic –Reliability information »Internal consistency reliability »Range of item-scale correlations »Number of item-scale correlations >.30

36. 36 Overview u Steps in creating and testing scales in your sample –Preparing raw data –Deriving scale scores –Testing scaling properties in your sample u Presenting measurement results u Presenting change scores

37. 37 Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change divided by baseline score u Perceived change –Single question asking respondent whether and how much they think they changed (from some prior time period)

38. 38 Measured Change u Difference in scores from baseline to follow-up u Example: Pain during the past 2 weeks, 0-10 cm Visual Analog Scale –Time 1 (baseline) - score of 5 –Time 2 (follow-up) - score of 8 –Difference = +3 or -3 depending on which way we subtract

39. 39 Measured Change: What is Missing? u How should we interpret a change score of +3 or -3? u Depends on: –Direction of scores (is higher score better or worse) –Which was subtracted from which? »Follow-up minus baseline? (T2 - T1) »Baseline minus follow-up? (T1 - T2)

40. 40 Measured Change: What is Missing? u How to calculate depends on what you want the change score to indicate –positive score is improvement or worsening u Positive score to indicate improvement: –high score is better »Subtract time 2 from time 1 »Positive change score = improvement –high score is worse »Subtract time 1 from time 2 »Positive change score = improvement

41. 41 Example of Change Score u You want a positive change to indicate improvement –high score is better u Subtract score nearest “worst” end from score nearest “best” end (worst) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (best) time 1 time 2 Time 2 minus Time 1 = change of +4 (improved by 4 points)

42. 42 Example of Change Score u You want a positive change to indicate improvement –high score is worse u Subtract score nearest “best” end from score nearest “worst” end (best) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (worst) time 2 time 1 Time 1 minus Time 2 = change of +4 (improved by 4 points)

43. 43 Interpreting Change Scores: What is Wrong? u A study predicting utilization of health care (outpatient visits) over a 1-year period as a function of self-efficacy u A results sentence: –“Reduced utilization at one year was associated with level of self efficacy at baseline (p <.01) and with 6-month changes in self efficacy (p <.05).”

44. 44 Interpreting Change Scores: Making it Clearer u “Reduced outpatient visits at one year were associated with lower levels of self efficacy at baseline (p <.01) and with 6-month improvements in self efficacy.” u Old way: –“Reduced utilization at one year was associated with level of self efficacy at baseline (p <.01) and with 6-month changes in self-efficacy.”

45. 45 Presenting Change Scores in Tables: What is Wrong? u Change in anxiety over a 1-year period for two groups 1 year change in anxiety p Exercise group - 40 <.001 Education group +4 ns

46. 46 Presenting Change Scores in Tables: Making it Clearer u Change in anxiety over a 1-year period for two groups 1 year change* in anxiety p Exercise group - 40 <.001 Education group +4 ns *Change scores are 1-year minus baseline; negative score indicates decrease in anxiety

47. 47 Percentage Change u Measured change divided by baseline score u Example: pain measure, higher is more pain –change score of -2, baseline score of 6 –2/6 = 33% reduction in pain

48. 48 Example of Percentage Change: Problem with Likert Scales u You want a positive change to indicate improvement (and high score is better) u Subtract score nearest “worst” end from score nearest “best” end (worst) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (best) time 1 time 2 Time 2 minus Time 1 = change of +4 (improved by 4 points) 4 / 8 = 50% improvement

49. 49 Example of Percentage Change: Problem with Likert Scales (cont.) u You want a positive change to indicate improvement –high score is worse u Subtract score nearest “best” end from score nearest “worst” end (best) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (worst) time 2 time 1 Time 1 minus Time 2 = change of +4 (improved by 4 points) 4 / 16 = 25% improvement

50. 50 Percentage Change Scores Only Work for Ratio-Level Measures u Can do percentage change only on scales with a –True zero (zero represents the absence of the trait in question) u Ratio scores - weight in pounds u Person weighs 150 pounds –Gains 10, gained 15% of original weight –Loses 10, lost 15% of original weight

51. 51 Perceived Change (Retrospective Change) u How much has your physical functioning changed since your surgery? 1 - very much worse 2 - much worse 3 - worse 4 - no change 5 - better 6 - much better 7 - very much better

52. 52 Perceived/Retrospective Change u Perceived change enables respondent to define physical functioning in terms of what it means to them u Measured change is a change on specific questions that were contained in the particular measure, e.g. –Difficulty walking –Difficulty climbing stairs u If the person had no change in these particular items, their measured change score will be 0 (no change) u If the same person became much worse in terms of bending over, they will report that they became worse

53. 53 Perceived/Retrospective Change u Recommend including both types of measures to assess change –Measured change enables »comparison with other studies »May be more sensitive because has more scale levels (if multi-item measure) –Perceived/Retrospective change enables »Person to report on domain using their own definition »Picks up changes “unmeasured” by particular measure

54. 54 Next Week: Class 10 u Future directions u Factor analysis

55. 55 Homework for Next Week u Final paper (see outline - Handout)