1. 1 Class 10 Creating Scores and Change Scores, Presenting Measurement Data, Selecting Standard Survey Items November 29, 2007 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2. 2 Overview of Class 10 u Creating summated scales and presenting measurement information u Creating and presenting change scores u “The rest of the survey” –Locating standard survey questions

3. 3 Creating Likert Scale Scores u Translate codebook scoring rules into program code (SAS, SPSS): –Reverse all items that are not already in desired direction (e.g., higher = better) –Average all items »Allows score if 1 item is answered –Apply missing data rule if different »e.g., if more than 50% items missing

4. 4 Program Statements: Creating Summated Scale Scores u From last week (SAS statements)

5. 5 Review Summated Scores u Review scores for out-of-range values, outliers, expected mean u For scores with problems, review programming statements, locate errors and correct u Repeat process until computer algorithm is producing accurate scores –To test programming accuracy, calculate scores by hand from 2 questionnaires »Check that they match computer generated scores

6. 6 Testing Scaling Properties in Your Sample for Multi-Item Scales u Obtain item-scale correlations –Part of reliability program –Each item correlates at least.30 with the total scale (corrected for overlap)

7. 7 Testing Scaling Properties in Your Sample for Multi-Item Scales (cont) u Calculate internal-consistency reliability (Cronbach’s alpha) for multi-item scales in your sample –Regardless of reliability in other studies u Internal consistency should be at least.70 –If lower, see if deleting items <.30 will improve it

8. 8 Presenting Measurement Results (Handout) u Present for each final scale: –% missing –Mean, standard deviation –Observed range, possible range –Floor and ceiling effects, skewness statistic –Range of item-scale correlations »Number of item-scale correlations >.30 –Internal consistency reliability

9. 9 Overview of Class 10 u Creating summated scales and presenting measurement information u Creating and presenting change scores u “The rest of the survey” –Locating standard survey questions

10. 10 Change Scores are Important Variables! u Creating change score variables is complex –Requires thought ahead of time u Don’t rely on your programmer u Include specification of change scores in your codebook

11. 11 Three Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change as percent of baseline score u Perceived change –How much change respondent reports (from some prior time period)

12. 12 Measured Change u Difference in scores from baseline to follow-up u Example measure administered at baseline and 1 month after treatment –Pain in past 2 weeks, 0-10 numeric scale, 10 = worst pain

13. 13 Measured Change (cont) u Hypothetical results –Time 1 (baseline) - score of 5 –Time 2 (one month) - score of 8 u How should change be measured?

14. 14 Measured Change (cont) Time 1 (baseline) - score of 5 Time 2 (one month) - score of 8 u How should change be measured? u Two options: –Time 2 minus time 1 –Time 1 minus time 2

15. 15 Measured Change (cont) Time 1 (baseline) - score of 5 Time 2 (one month) - score of 8 u Option one: time 2 minus time 1= +3 u Option two: time 1 minus time 2 = -3 u Interpretation of change score?

16. 16 Interpretation of Change Score u What do you want the change score to indicate? –Positive change score = improving? –Positive change score = worsening? u Scoring thus depends on: –Direction of scores on original measure (is higher score better or worse?) –Which was subtracted from which?

17. 17 Define Change Score Before Calculation: Algorithms You want positive score = improvement u If high score on measure is better –Time 2 minus time 1 u If high score on measure is worse –Time 1 minus time 2 You want positive score = decline u If high score on measure is better –Time 1 minus time 2 u If high score on measure is worse –Time 2 minus time 1

18. 18 Example: You Want Positive Score To Indicate Improvement u Hypothetical subject: Improved u Subtract score nearest “worst” end from score nearest “best” end (worst) 0 1 2 3 4 5 6 7 8 9 10 (best) time 1 time 2

19. 19 Example: You Want Positive Score To Indicate Improvement u Subtract score nearest “worst” end from score nearest “best” end (worst) 0 1 2 3 4 5 6 7 8 9 10 (best) time 1 time 2 Time 2 minus time 1 = +4 (improved by 4 points)

20. 20 Example: You Want Positive Score To Indicate Improvement (Scale Reversed) u Hypothetical subject: Improved u Subtract score nearest “worst” end from score nearest “best” end (best) 0 1 2 3 4 5 6 7 8 9 10 (worst) time 2 time 1

21. 21 Example: You Want Positive Score To Indicate Improvement (Scale Reversed) u Subtract score nearest “worst” end from score nearest “best” end (best) 0 1 2 3 4 5 6 7 8 9 10 (worst) time 2 time 1 Time 1 minus time 2 = +4 (improved by 4 points)

22. 22 Recommendation: Make Change Score Intuitively Meaningful u If high score on measure = better u Calculate change score so positive change score = improved –Time 2 minus time 1 u If high score on measure = worse u Calculate change scores so positive change score = improved –Time 1 minus time 2

23. 23 Interpreting “Measured Change” Scores: What is Wrong? u In a study predicting utilization of health care (outpatient visits) over a 1-year period as a function of self-efficacy… u A results sentence: –“Reduced utilization at one year was associated with level of self efficacy at baseline (p <.01) and with 6-month changes in self efficacy (p <.05).”

24. 24 Interpreting “Measured Change” Scores: Making it Clearer u “Reduced outpatient visits at one year were associated with lower levels of self efficacy at baseline (p <.01) and with 6-month improvements in self efficacy.” u Old way: –“Reduced utilization at one year was associated with level of self efficacy at baseline (p <.01) and with 6-month changes in self-efficacy.”

25. 25 Three Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change as percent of baseline score u Perceived change –How much change respondent reports (from some prior time period)

26. 26 Presenting Change Scores in Tables: What is Wrong? u Change in anxiety over a 1-year period for two groups 1 year change in anxiety p Exercise group - 40 <.001 Education group +4 ns

27. 27 Presenting Change Scores in Tables: Making it Clearer u Change in anxiety over a 1-year period for two groups 1 year change in anxiety p Exercise group - 40 <.001 Education group +4 ns *Negative score indicates decreased anxiety (change scores are 1-year minus baseline)

28. 28 Reliability of Change Score u Difference scores have been criticized as having low reliability u Nunnally (1994) considers alternatives and suggests this may not be as large a problem as previously thought (p. 247) Nunnally JC and Bernstein IH. Psychometric Theory, Third Edition, McGraw-Hill, New York, 1994.

29. 29 Percentage Change u Measured change divided by baseline score u Example: pain measure, higher is more pain –change score of -2, baseline score of 6 –2/6 = 33% reduction in pain

30. 30 Example of Percentage Change Problem with Likert Scales u You want a positive change to indicate improvement (and high score is better) u Subtract score nearest “worst” end from score nearest “best” end (worst) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (best) time 1 time 2 Time 2 minus Time 1 = change of +4 (improved by 4 points) Change of 4 / baseline score of 8 = 50% improvement

31. 31 Example of Percentage Change Problem with Likert Scales (cont.) u You want a positive change to indicate improvement –high score is worse u Subtract score nearest “best” end from score nearest “worst” end (best) 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (worst) time 2 time 1 Time 1 minus Time 2 = change of +4 (improved by 4 points) Change of 4 / baseline score of 16 = 25% improvement

32. 32 Percentage Change Scores Only Work for Ratio-Level Measures u Can do percentage change only on scales with a true zero –zero represents the absence of the trait in question u Ratio scores - weight in pounds u Person weighs 150 pounds –Gains 10, gained 15% of original weight –Loses 10, lost 15% of original weight

33. 33 Three Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change as percent of baseline score u Perceived change –How much change respondent reports (from some prior time period)

34. 34 Perceived Change (Retrospective Change) u How much has your physical functioning changed since your surgery? 1 - very much worse 2 - much worse 3 - worse 4 - no change 5 - better 6 - much better 7 - very much better

35. 35 Perceived Change (Retrospective Change) – Better Response Choice? u How much has your physical functioning changed since your surgery? -3 Very much worse -2 Much worse -1 Worse 0 No change 1 Better 2 Much better 3 Very much better

36. 36 Perceived/Retrospective Change u Perceived change enables respondent to define physical functioning in terms of what it means to them u Measured change is a change on specific questions that were contained in the particular measure

37. 37 Example of Measured Change u Baseline and 6-month limitations: –Difficulty walking –Difficulty climbing stairs u Measured change: change on these 2 physical functions u If person had no change walking or climbing stairs –Score would be “no change”

38. 38 Example of Perceived Change u To what extent did your physical functioning change over the past 6 months? –Much worse –Worse –No change –Better –Much better u If person has more trouble bending over, and considers this as part of physical functioning, they will report becoming worse

39. 39 Perceived/Retrospective Change u Recommend including both types of measures to assess change –Measured change enables »Comparison with other studies »May be more sensitive because has more scale levels (if multi-item measure) »Investigator defines clinically relevant outcomes –Perceived/Retrospective change enables »Person to report on domain using their own definition »Picks up changes “unmeasured” by particular measure

40. 40 Overview of Class 10 u Creating summated scales and presenting measurement information u Creating and presenting change scores u “The rest of the survey” –Locating standard survey questions

41. 41 Locating “Standard” Survey Questions u MD characteristics u Comorbidity, chronic conditions u Medical history, family history u Health behaviors

42. 42 Demographics – Just About Everywhere u Basic demographics u Socioeconomic status u Financial information (assets, income, wealth) u Employment, occupation u Retirement u Health insurance

43. 43 Take Away Point: u Don’t write these yourself u Use standard questions from appropriate existing surveys

44. 44 National and State Surveys u Population surveys u Tend to have single-item measures rather than multi-item scales –Good for “standardized” survey items

45. 45 State Surveys u http://www.chis.ucla.edu/ http://www.chis.ucla.edu/ u California Health Interview Survey (CHIS) u “Questionnaires” u See contents of 2006 CHIS: adults and adolescents

46. 46 National Surveys u Behavioral Risk Factor Surveillance System Questionnaires u http://www.cdc.gov/brfss/questionnaires/ questionnaires.htm http://www.cdc.gov/brfss/questionnaires/ questionnaires.htm u See contents of 2006 BRFSS –English and Spanish

47. 47 MacArthur Research Network on Socioeconomic Status and Health u Measures of economic status, occupational status, education, and perceived social status –Includes rationale u http://www.macses.ucsf.edu/Research/Social% 20Environment/notebook/economic.html http://www.macses.ucsf.edu/Research/Social% 20Environment/notebook/economic.html u Also basic demographics

48. 48 Center for Aging in Diverse Communities (CADC) u Recommends items measuring socioeconomic status –Education, income, race/ethnicity, place of birth/generation, English language proficiency, financial hardship u Main website: http://medicine.ucsf.edu/cadc/cores/mea surement/index.html

49. 49 Cancer Research Measures u The Division of Cancer Epidemiology and Genetics u Demographics, medical history, family history, other risk factors http://dceg.cancer.gov/QMOD/

50. 50 Non-English Language? u California Health Interview Survey –Numerous languages u Spanish language surveys –SALSA –Hispanic Health and Nutrition Examination Survey (HHANES) –National Mexican Health and Aging Study –Behavioral Risk Factor Surveillance System (CDC)

51. 51 NCHS National Health Care Surveys: Surveys of Physicians u Family of provider-based surveys u Provide information about –organizations and providers –services rendered –patients they serve u Measures of physician variables including practice characteristics http://www.cdc.gov/nchs/nhcs.htm

52. 52 Basic Measures About Children? u Commonwealth Fund Survey of Parents with Young Children –Parent administered u CHIS for adolescents (self-administered) and children (parent-administered)

53. 53 End of Class! Thank you