1 Class 10 Creating Scores and Change Scores, Presenting Measurement Data, Selecting Standard Survey Items November 29, 2007 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2 Overview of Class 10 u Creating summated scales and presenting measurement information u Creating and presenting change scores u “The rest of the survey” –Locating standard survey questions

3 Creating Likert Scale Scores u Translate codebook scoring rules into program code (SAS, SPSS): –Reverse all items that are not already in desired direction (e.g., higher = better) –Average all items »Allows score if 1 item is answered –Apply missing data rule if different »e.g., if more than 50% items missing

4 Program Statements: Creating Summated Scale Scores u From last week (SAS statements)

5 Review Summated Scores u Review scores for out-of-range values, outliers, expected mean u For scores with problems, review programming statements, locate errors and correct u Repeat process until computer algorithm is producing accurate scores –To test programming accuracy, calculate scores by hand from 2 questionnaires »Check that they match computer generated scores

6 Testing Scaling Properties in Your Sample for Multi-Item Scales u Obtain item-scale correlations –Part of reliability program –Each item correlates at least.30 with the total scale (corrected for overlap)

7 Testing Scaling Properties in Your Sample for Multi-Item Scales (cont) u Calculate internal-consistency reliability (Cronbach’s alpha) for multi-item scales in your sample –Regardless of reliability in other studies u Internal consistency should be at least.70 –If lower, see if deleting items <.30 will improve it

8 Presenting Measurement Results (Handout) u Present for each final scale: –% missing –Mean, standard deviation –Observed range, possible range –Floor and ceiling effects, skewness statistic –Range of item-scale correlations »Number of item-scale correlations >.30 –Internal consistency reliability

9 Overview of Class 10 u Creating summated scales and presenting measurement information u Creating and presenting change scores u “The rest of the survey” –Locating standard survey questions

10 Change Scores are Important Variables! u Creating change score variables is complex –Requires thought ahead of time u Don’t rely on your programmer u Include specification of change scores in your codebook

11 Three Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change as percent of baseline score u Perceived change –How much change respondent reports (from some prior time period)

12 Measured Change u Difference in scores from baseline to follow-up u Example measure administered at baseline and 1 month after treatment –Pain in past 2 weeks, 0-10 numeric scale, 10 = worst pain

13 Measured Change (cont) u Hypothetical results –Time 1 (baseline) - score of 5 –Time 2 (one month) - score of 8 u How should change be measured?

14 Measured Change (cont) Time 1 (baseline) - score of 5 Time 2 (one month) - score of 8 u How should change be measured? u Two options: –Time 2 minus time 1 –Time 1 minus time 2

15 Measured Change (cont) Time 1 (baseline) - score of 5 Time 2 (one month) - score of 8 u Option one: time 2 minus time 1= +3 u Option two: time 1 minus time 2 = -3 u Interpretation of change score?

16 Interpretation of Change Score u What do you want the change score to indicate? –Positive change score = improving? –Positive change score = worsening? u Scoring thus depends on: –Direction of scores on original measure (is higher score better or worse?) –Which was subtracted from which?

17 Define Change Score Before Calculation: Algorithms You want positive score = improvement u If high score on measure is better –Time 2 minus time 1 u If high score on measure is worse –Time 1 minus time 2 You want positive score = decline u If high score on measure is better –Time 1 minus time 2 u If high score on measure is worse –Time 2 minus time 1

18 Example: You Want Positive Score To Indicate Improvement u Hypothetical subject: Improved u Subtract score nearest “worst” end from score nearest “best” end (worst) (best) time 1 time 2

19 Example: You Want Positive Score To Indicate Improvement u Subtract score nearest “worst” end from score nearest “best” end (worst) (best) time 1 time 2 Time 2 minus time 1 = +4 (improved by 4 points)

20 Example: You Want Positive Score To Indicate Improvement (Scale Reversed) u Hypothetical subject: Improved u Subtract score nearest “worst” end from score nearest “best” end (best) (worst) time 2 time 1

21 Example: You Want Positive Score To Indicate Improvement (Scale Reversed) u Subtract score nearest “worst” end from score nearest “best” end (best) (worst) time 2 time 1 Time 1 minus time 2 = +4 (improved by 4 points)

22 Recommendation: Make Change Score Intuitively Meaningful u If high score on measure = better u Calculate change score so positive change score = improved –Time 2 minus time 1 u If high score on measure = worse u Calculate change scores so positive change score = improved –Time 1 minus time 2

23 Interpreting “Measured Change” Scores: What is Wrong? u In a study predicting utilization of health care (outpatient visits) over a 1-year period as a function of self-efficacy… u A results sentence: –“Reduced utilization at one year was associated with level of self efficacy at baseline (p <.01) and with 6-month changes in self efficacy (p <.05).”

24 Interpreting “Measured Change” Scores: Making it Clearer u “Reduced outpatient visits at one year were associated with lower levels of self efficacy at baseline (p <.01) and with 6-month improvements in self efficacy.” u Old way: –“Reduced utilization at one year was associated with level of self efficacy at baseline (p <.01) and with 6-month changes in self-efficacy.”

25 Three Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change as percent of baseline score u Perceived change –How much change respondent reports (from some prior time period)

26 Presenting Change Scores in Tables: What is Wrong? u Change in anxiety over a 1-year period for two groups 1 year change in anxiety p Exercise group - 40 <.001 Education group +4 ns

27 Presenting Change Scores in Tables: Making it Clearer u Change in anxiety over a 1-year period for two groups 1 year change in anxiety p Exercise group - 40 <.001 Education group +4 ns *Negative score indicates decreased anxiety (change scores are 1-year minus baseline)

28 Reliability of Change Score u Difference scores have been criticized as having low reliability u Nunnally (1994) considers alternatives and suggests this may not be as large a problem as previously thought (p. 247) Nunnally JC and Bernstein IH. Psychometric Theory, Third Edition, McGraw-Hill, New York, 1994.

29 Percentage Change u Measured change divided by baseline score u Example: pain measure, higher is more pain –change score of -2, baseline score of 6 –2/6 = 33% reduction in pain

30 Example of Percentage Change Problem with Likert Scales u You want a positive change to indicate improvement (and high score is better) u Subtract score nearest “worst” end from score nearest “best” end (worst) (best) time 1 time 2 Time 2 minus Time 1 = change of +4 (improved by 4 points) Change of 4 / baseline score of 8 = 50% improvement

31 Example of Percentage Change Problem with Likert Scales (cont.) u You want a positive change to indicate improvement –high score is worse u Subtract score nearest “best” end from score nearest “worst” end (best) (worst) time 2 time 1 Time 1 minus Time 2 = change of +4 (improved by 4 points) Change of 4 / baseline score of 16 = 25% improvement

32 Percentage Change Scores Only Work for Ratio-Level Measures u Can do percentage change only on scales with a true zero –zero represents the absence of the trait in question u Ratio scores - weight in pounds u Person weighs 150 pounds –Gains 10, gained 15% of original weight –Loses 10, lost 15% of original weight

33 Three Types of Change Scores u Measured change –Difference in scores between baseline and follow-up u Percentage change –Measured change as percent of baseline score u Perceived change –How much change respondent reports (from some prior time period)

34 Perceived Change (Retrospective Change) u How much has your physical functioning changed since your surgery? 1 - very much worse 2 - much worse 3 - worse 4 - no change 5 - better 6 - much better 7 - very much better

35 Perceived Change (Retrospective Change) – Better Response Choice? u How much has your physical functioning changed since your surgery? -3 Very much worse -2 Much worse -1 Worse 0 No change 1 Better 2 Much better 3 Very much better

36 Perceived/Retrospective Change u Perceived change enables respondent to define physical functioning in terms of what it means to them u Measured change is a change on specific questions that were contained in the particular measure

37 Example of Measured Change u Baseline and 6-month limitations: –Difficulty walking –Difficulty climbing stairs u Measured change: change on these 2 physical functions u If person had no change walking or climbing stairs –Score would be “no change”

38 Example of Perceived Change u To what extent did your physical functioning change over the past 6 months? –Much worse –Worse –No change –Better –Much better u If person has more trouble bending over, and considers this as part of physical functioning, they will report becoming worse

39 Perceived/Retrospective Change u Recommend including both types of measures to assess change –Measured change enables »Comparison with other studies »May be more sensitive because has more scale levels (if multi-item measure) »Investigator defines clinically relevant outcomes –Perceived/Retrospective change enables »Person to report on domain using their own definition »Picks up changes “unmeasured” by particular measure

40 Overview of Class 10 u Creating summated scales and presenting measurement information u Creating and presenting change scores u “The rest of the survey” –Locating standard survey questions

41 Locating “Standard” Survey Questions u MD characteristics u Comorbidity, chronic conditions u Medical history, family history u Health behaviors

42 Demographics – Just About Everywhere u Basic demographics u Socioeconomic status u Financial information (assets, income, wealth) u Employment, occupation u Retirement u Health insurance

43 Take Away Point: u Don’t write these yourself u Use standard questions from appropriate existing surveys

44 National and State Surveys u Population surveys u Tend to have single-item measures rather than multi-item scales –Good for “standardized” survey items

45 State Surveys u u California Health Interview Survey (CHIS) u “Questionnaires” u See contents of 2006 CHIS: adults and adolescents

46 National Surveys u Behavioral Risk Factor Surveillance System Questionnaires u questionnaires.htm questionnaires.htm u See contents of 2006 BRFSS –English and Spanish

47 MacArthur Research Network on Socioeconomic Status and Health u Measures of economic status, occupational status, education, and perceived social status –Includes rationale u 20Environment/notebook/economic.html 20Environment/notebook/economic.html u Also basic demographics

48 Center for Aging in Diverse Communities (CADC) u Recommends items measuring socioeconomic status –Education, income, race/ethnicity, place of birth/generation, English language proficiency, financial hardship u Main website: surement/index.html

49 Cancer Research Measures u The Division of Cancer Epidemiology and Genetics u Demographics, medical history, family history, other risk factors

50 Non-English Language? u California Health Interview Survey –Numerous languages u Spanish language surveys –SALSA –Hispanic Health and Nutrition Examination Survey (HHANES) –National Mexican Health and Aging Study –Behavioral Risk Factor Surveillance System (CDC)

51 NCHS National Health Care Surveys: Surveys of Physicians u Family of provider-based surveys u Provide information about –organizations and providers –services rendered –patients they serve u Measures of physician variables including practice characteristics

52 Basic Measures About Children? u Commonwealth Fund Survey of Parents with Young Children –Parent administered u CHIS for adolescents (self-administered) and children (parent-administered)

53 End of Class! Thank you