1. 1 Identifying and Selecting Measures for Health Disparities Research Anita L. Stewart, Ph.D. University of California, San Francisco Clinical Research with Diverse Communities EPI 222, Spring April 22, 2010

2. 2 Inappropriate Measures can Result in: u Conceptual inadequacy –Measuring wrong concept u Poor data quality (e.g. missing data) u Poor variability u Poor reliability and validity u Inability to detect true associations –e.g., no measured change in outcome when change occurred

3. 3 Measure, Scale, Index, Item u Measure - single- or multi-item scale or index (umbrella term) u Scale - aggregation of items from one concept, scored using accepted scaling method u Index - aggregation of 2 or more scales into a summary score u Item - a single question or statement including its response scale

4. 4 Composition of an Item During the past month, how much of the time have you felt tired? 1 Never 2 A little of the time 3 Some of the time 4 Most of the time 5 All of the time Response scale Item stem

5. 5 Measure, Instrument, Battery, Questionnaire, Survey, Tool, Inventory u Measure - single- or multi-item scale or index (umbrella term) u Instrument - a published, named measure or set of measures u Battery - collection of measures from diverse sources

6. 6 Single-item Measures - Usually Ordinal u Advantages –Response choices interpretable u Disadvantages –Impossible to assess complex concept –Very limited variability, often skewed –Reliability usually low

7. 7 Multi-Item Measures or Scales Multi-item scales are created by combining two or more items into an overall measure or scale score Sometimes called s ummated ratings scales

8. 8 Advantages of Multi-item Measures (Over Single Items) u More scale values (improves score distribution) u Reduces # of scores to measure a concept u Improves reliability (reduces random error) u Reduces % with missing data (can estimate score if items are missing) u More likely to reflect concept (content validity)

9. 9 Interpersonal Processes of Care: Multi- item Measures I. COMMUNICATION Hurried communication(5) Elicited concerns, responded (3) Explained results, medications (4) II. DECISION MAKING Patient-centered decision-making (4) III. INTERPERSONAL STYLE Compassionate, respectful (5) Discriminated (4) Disrespectful office staff (4)

10. 10 Selecting Measures for Your Research u Goal: find a measure of your concept that has been developed using stringent measurement development methods u Your task: find measures and review them for all steps in measurement development process

11. 11 Process of Selecting Good Measures for Your Studies Define concept (variable) Identify potential measures Review measures’ properties --conceptual adequacy --psychometric adequacy Pretest best 1-2 measures Select final measure

12. 12 Process of Selecting Good Measures for Your Studies Define concept (variable) Identify potential measures Review measures’ for --conceptual adequacy --psychometric adequacy Pretest best 1-2 measures Select final measure

13. 13 Concept/Construct u A variable that is relatively abstract as opposed to concrete –e.g. health status, stress, acculturation vs. height, body temp u An abstraction based on observations of certain behaviors or characteristics u Cannot be assessed directly

14. 14 Measures of Concepts u Concepts are defined and operationalized in terms of observed indicators or measures u Measures are “proxies” for the latent variables we cannot directly observe

15. 15 Define Concept For Your Study u Define concept from your perspective, taking into account your… –study questions –target population u For outcomes, describe: –how intervention or independent variables might affect it –specific types of changes you expect

16. 16 Example of Concept: Interpersonal Processes of Care u Interpersonal processes between physicians and patients –Communication, decision making, respectfulness, empathy –Emphasis on processes relevant to vulnerable patients

17. 17 Purpose of Measuring Concept u Describe how your concept fits into your research question u Outcome measure? u Determinant of health? u Identify need for intervention?

18. 18 Example: Interpersonal Processes of Care as an Outcome u Interpersonal processes of care may influence patient outcomes –Proximal outcomes »Knowledge of condition and recommendations »Adherence to recommendations »Patient satisfaction –Ultimate outcomes »Health status

19. 19 Specify Context for Measures u Study characteristics affecting choice of measures –Nature of target population –Practical constraints u Nature of population (patients) –Lower educational level? Limited literacy? –Healthy or ill?

20. 20 Process of Selecting Good Measures for Your Studies Define concept (variable) Identify potential measures Review measures’ for --conceptual adequacy --psychometric adequacy Pretest best 1-2 measures Select final measure

21. 21 Locate Potential Measures u Identify candidate measures for all concepts u Redundancy OK for now u DO NOT develop your own questions unless it is absolutely necessary

22. 22 Locating Potential Measures u Multi-item measures with known psychometric properties u Most good measures have been published –Original development and testing information –Information on application of measure in subsequent publications

23. 23 Locating Potential Measures u Electronic databases u Compendia (reviews of measures) u Organizations and research centers u Government agencies u National and state surveys u Universities and individual researchers u Wikipedia (new)

24. 24 Why Not Just Search on PubMed? u Can search PubMed using your concept term and MeSH terms questionnaires, measures, etc. –Terminology is not always intuitive –Articles are indexed by most specific MeSH term available u Cumbersome!

25. 25 Measurement and Methods Cores Resources CADC (see handout): u http://dgim.ucsf.edu/cadc/cores/measurem ent/index.html http://dgim.ucsf.edu/cadc/cores/measurem ent/index.html RCMAR coordinating center u http://www.rcmar.ucla.edu/wiki/index.php /Measurement_Workgroup http://www.rcmar.ucla.edu/wiki/index.php /Measurement_Workgroup

26. 26 Locating Published Measures: Compendia u Specific measures of various concepts are compiled, reviewed, listed, or otherwise provided –Many books and some special journal issues review and critique various measures u Reviews of measures are extremely helpful –Try to find these if possible

27. 27 Best Compendium u McDowell I and Newell C, Measuring Health: A Guide to Rating Scales and Questionnaires. Third Edition, Oxford University Press, 2006. –Reviews measures of physical disability, social health, psychological well-being, anxiety, depression, mental status, pain, and general health status and quality of life –Includes actual instruments where possible and selected items in some

28. 28 Compendia by Web u National Cancer Institute website u Health behavior constructs: theory, measurement, and research –Reviews concepts and measures of constructs such as perceived control, social support, and perceived vulnerability http://dccps.cancer.gov/brp/constructs/index.html

29. 29 CINAHL (Nursing and Allied Health Literature) u CINAHL - on UCSF library home page u To find several potential measures –Use “questionnaires, scales, self-report, or surveys” in Major Subject Heading field AND your topic area in another field (e.g., Abstract) u To find articles using a specific measure –Use name of measure and Instrumentation field

30. 30 Google Scholar u http://scholar.google.com/ http://scholar.google.com/ u More practical for searching on vague terms u Follow up on PubMed

31. 31 Consider Alternative Labels to Search u Search on your term – see what other terms come up u Housing instability –Homelessness u Clinical assertiveness –Interpersonal behavior

32. 32 Compendia by Web u National Cancer Institute u Health behavior constructs: theory, measurement, and research –Reviews concepts and measures of constructs such as perceived control, social support, and perceived vulnerability http://dccps.cancer.gov/brp/constructs/index.html

33. 33 Locating Measures: Organizations and Research Centers u Some organizations and centers specialize in measurement development –RAND –MacArthur Research Network on Socioeconomic Status and Health

34. 34 RAND Health Program u Measures, scoring manuals, and publications for measures of: –Health-related quality of life –Quality of care, patient satisfaction –Mental health –Maternal, child, and adolescent health –Medical Outcomes Study (MOS) measures of functioning and well-being http://www.rand.org/health/surveys_tools.html

35. 35 MacArthur Research Network on Socioeconomic Status and Health u Reviews measures in several domains: –Psychosocial –Social and physical environment –Socioeconomic status (SES) »SES across the lifecourse http://www.macses.ucsf.edu/Research/overview.htm

36. 36 Examples of MacArthur Network Psychosocial Measures u Anxiety u Coping u Depression u Discrimination u Hostility u Optimism/pessimism u Personal control u Psychological stress u Purpose in life u Self-esteem u Social support u Vitality and vigor http://www.macses.ucsf.edu/Research/wgps.htm

37. 37 Society of General Internal Medicine u Dataset compendium – has links to numerous datasets such as: –National Survey of Children’s Health (NSCH) –California Health Interview Survey (CHIS) –Coronary Artery Risk Development in Young Adults (CARDIA) u Each link has section for data collection forms (questionnaires) –Sometimes involves several layers http://www.sgim.org/index.cfm?pageId=864

38. 38 Ottawa Health Decision Centre u Patient and physician decision aids –http://www.ohri.ca/centres/DecisionAids/default.asp u Patient measures –Decisional conflict, decisional regret, stage of decision making, decision self-efficacy, and others –http://decisionaid.ohri.ca/eval.html

39. 39 Commonwealth Fund Surveys u Health insurance, Medicare u Health system performance u Health care quality, patient centered care, underserved populations u Child health, care of the elderly http://www.commonwealthfund.org/surveys/

40. 40 Locating Measures: Government Agencies u Several federal and state government agencies provide measures –Information about measures (e.g. clearinghouse) –Actual measures

41. 41 National Center for Health Statistics (NCHS) u Surveys and data collection systems u Can download –Any survey or portion of survey u Handouts: –Summary of surveys August 2008 http://www.cdc.gov/nchs

42. 42 NCHS National Health Care Surveys: Surveys of Physicians u Family of provider-based surveys u Provide objective, reliable information about –organizations and providers –services rendered –patients they serve http://www.cdc.gov/nchs/surveys.htm

43. 43 Centers for Disease Control and Prevention u Behavioral Risk Factor Surveillance System (BRFSS) –Surveys of health behaviors www.cdc.gov/brfss/questionnaires/index.htm

44. 44 CDC: National School-Based Youth Risk Behavior Survey (YRBS) u Survey conducted every other year –random national sample of youth in grades 9-12 –Most states conduct survey u Measures of substance use, risky sexual behaviors, diet, physical activity, overweight http://www.cdc.gov/HealthyYouth/yrbs/index.htm

45. 45 California Health Interview Survey (CHIS) u A behavioral risk surveillance survey modeled after the CDC BRFSS u Adults, adolescents, and children u Numerous languages http://www.chis.ucla.edu/questionnaires.html

46. 46 U.S. Dept of Veterans Affairs u National Center for Post Traumatic Stress Disorder –http://www.ptsd.va.gov/http://www.ptsd.va.gov/ u Assessment/Instruments –Trauma Exposure MeasuresTrauma Exposure Measures –PTSD ScreensPTSD Screens –Adult Self ReportAdult Self Report

47. 47 Agency for Healthcare Research and Quality (AHRQ) u National Quality Measures Clearinghouse u Consumer Assessment of Health Plans Survey (CAHPS) http://www.ahrq.gov/qual/

48. 48 National Cancer Institute u Special initiatives on measures u Health Information National Trends Survey (HINTS) –Measures of cancer knowledge, cancer communication, risk perception http://hints.cancer.gov/

49. 49 Locating Measures: Universities and Individual Researchers u Individual investigators who specialize in measurement and some universities post measures u Some are large research studies on university websites

50. 50 Locating Measures: Large Research Studies u Many large-scale, multi-center and longitudinal studies have developed and used measures on health-related topics u Increasingly, they are posting these on “study” websites

51. 51 Sacramento Area Latino Study on Aging (SALSA) Study u NIA funded longitudinal study of Latinos in the Sacramento region u Started in 1996 –each person followed for up to 5 years http://sitemaker.umich.edu/salsa.study/home

52. 52 Locating Measures: Finding Authors of Measures u Published research using measure you are interested in –Unpublished measures often described in methods –Authors may provide measures

53. 53 Process of Selecting Good Measures for Your Studies Define concept (variable) Identify potential measures Review measures’ for --conceptual adequacy --psychometric adequacy Pretest best 1-2 measures Select final measure

54. 54 Conceptual Adequacy for Your Study u Concept being measured “matches” the concept you defined –Sometimes can only be determined by reviewing items u If not a perfect match –How close is it to your concept? –Can it be modified to get at missing components?

55. 55 Psychometric Adequacy for Your Study u In samples similar to your target group: –good variability –low percent of missing data –good reliability –good validity u As an outcome for planned intervention –responsive, sensitive to change in similar population –able to detect expected magnitude of change

56. 56 Reliability u Extent to which an observed score is free of random error u Population-specific; reliability increases with: –sample size –variability in scores (dispersion) –a person’s level on the scale

57. 57 Internal Consistency Reliability: Cronbach’s Alpha u Requires multiple items measuring same construct u Extent to which items measure same construct (same latent variable) u It is a function of: –Number of items –Average correlation among items –Variability in your sample

58. 58 Minimum Standards for Internal Consistency Reliability u For group comparisons (e.g., regression, correlational analyses) –.70 or above is minimum –.80 is optimal JC Nunnally, Psychometric Theory 3 rd ed, McGraw-Hill, 1994

59. 59 Validity u Does a measure (or instrument) measure what it is supposed to measure? u And… Does a measure NOT measure what it is NOT supposed to measure?

60. 60 Validation of Measures is an Iterative, Lengthy Process u Validity is not a property of the measure –Validity is a property of a measure for particular purpose and sample –Validation studies for one purpose and sample may not serve another purpose or sample u Accumulation of evidence: –Different samples –Longitudinal designs

61. 61 Construct Validity Basics A process of answering the following questions: u What is the hypothesis? u What are the results? u Do the results support (confirm) the hypothesis?

62. 62 Practical Considerations: Match to Your Context Study context u Need permission to use? Any cost of using? u Scoring rules available? u Method of administration appropriate? u Short forms if needed? Appropriate for your sample u Reading level u Translations u Acceptability, respondent burden

63. 63 Practical - Obtaining Permission u Need permission to use or to adapt? u Public domain –If items are published or in the public domain, usually don’t need permission u Private or proprietary –Need to write to author or distributor –Allow 4-6 weeks to obtain measure and/or permission

64. 64 Practical - Cost to Use or to Score Measures u Cost of administering and scoring –Fee for each “instrument” purchased –Cost of any needed scoring software? –Cost to have it scored by source? »Cost per “instrument”?

65. 65 Practical - Scoring u Are scoring instructions clearly documented? –Do you have a scoring codebook? u Are computer scoring programs available? u (Cost of scoring)

66. 66 Direction of High Score: Interpersonal Processes of Care Measures I. COMMUNICATION Hurried communication(-) Elicited concerns, responded (+) Explained results, medications (+) II. DECISION MAKING Patient-centered decision-making (+) III. INTERPERSONAL STYLE Compassionate, respectful (+) Discriminated (-) Disrespectful office staff (-)

67. 67 Reading Level u Is reading level appropriate for your target population? –Special concern in lower SES, limited English proficiency groups u If reading level not known –Make your own judgment –Pretest with target population

68. 68 Availability of Translations if Needed u If you need measure in another language, are there translations available? –Official (published and tested) –Unofficial (by some other researcher)

69. 69 Respondent Burden u Real burden –Length, convenience, time needed to complete u Perceived burden –a function of item difficulty, distress due to content, perceived value of survey, expectations of length u Some population subgroups may have more difficulty, take longer to complete

70. 70 Process of Selecting Good Measures for Your Studies Define concept (variable) Identify potential measures Review measures’ for --conceptual adequacy --psychometric adequacy Pretest best 1-2 measures Select final measure

71. 71 Pretest Potential Measures in Your Target Population u Select best measures for all concepts in your conceptual framework –existing instrument in its entirety –subscales of relevant domains (e.g., only those that meet your needs)

72. 72 Pretest in Target Population u Pretesting essential for measures being applied to any new population group –Especially priority measures (e.g., outcomes) u Pretest is to identify: –problems with procedures »method of administration, respondent burden –problems with questions »Item stems, response choices, and instructions

73. 73 Types of Pretests u General debriefing pretest (N=10) u Large pretest (N=100) –test measurement properties prior to major study u In-depth cognitive interviewing pretest

74. 74 In-Depth Cognitive Interviews u Derived from social and cognitive psychology to explore processes respondents use to answer survey questions u Goal: understand thought processes used to answer questions u Can help write/adapt questions

75. 75 Types of Problems with Questions or Response Choices u Are all words/phrases understood as intended? u Are questions interpreted similarly by all respondents? u Are some questions not answered? u Are any questions offensive or irrelevant? u Does each closed-ended question have an answer that applies to each respondent? –Are the response choices adequate?

76. 76 Summary u Methods for selecting/reviewing measures described here are “ideal” u Apply these methods to your most important measures –e.g., outcomes, key independent variables

77. 77 Homework u See handout for class 4 and 5 homework u Complete rows 1-11 in matrix –Use form posted on the website –Remaining rows are for the next measurement lecture (class 5) u Include your name in the filename u Email by Monday April 26 to Anita.Stewart@ucsf.edu