Comparison of Self-Reported and Medical Record Health Care Utilization measures Concurrent criterion validation of a domain on health care utilization as part of a QOL instrument by comparing it with charts from a centralized medical chart center for the area

Information not provided: Abstract 1992 Evaluative tool QOL Benign prostatic hyperplasia Random selection of 110 men in male across the span of severity of BPH (no to severe symptoms) 6 domains: symptoms, degree of bother, BPH specific interference with activities, psycho, worries-concerns, sexual satisfaction tested for test-retest reliability, responsiveness to change, internal consistency, validity

Measurement Pre-existing self-reported questionnaire on QOL for benign prostate hyperplasia, with a 5-item domain on health care utilization domains: same 6 + health care use, medical Hx, smoking, sociodemo info

Proposed use of the measurement Predictive tool for research for administrative (health care planning, policy) purposes

Research question Not clearly stated Pop: male 40-79 community-dwelling, Olmsted county Intervention: self-administered questionnaire QOL Outcomes: comparison with centralized medical records of the county, # in-pt nights in last yr # in-pt nights for mental reasons, # visits to MD in last 2 wks, # visits to MD in 1 yr, visits to MD for any of 12 symptoms of HBP (yes/no) Direction: ?

Aspect of validity Concurrent criterion validity

Study design and methods Question not clear Pop: random sample (size 500,10% hospitalized in last yr) exclusion criteria not clear Meaningful criterion: yes Appropriate comparison: independant, blind? Tool: ideally only measuring health care use scaling of answers not mentionned, questions not stated Statistical analysis: no pre-set kappa or phi

Bias Selection bias Information bias Response bias (55% returned the quest) Information bias Misclassification related to extraction data chart Recall bias Social desirability (underreporting) Reporting bias ( in-hosp for mental prob)

Answered the research question? What was the research question? They answered the question we guessed they addressed Tool: 1 question: did not report any result 4 questions: reported as reliability!

Implications Administrative, planning, research use for male 40-79 validity mostly for # nights in-pt in last yr recalling seeing a MD in last 2 wks Clinical use: no application foreseen

Analysis of criterion validity Dichotomous: phi () Continuous: weighted kappa, Pearson, ICC Ordinal: consider as if continuous or use Wilcoxon signed rank test

Analysis criterion validity Dichotomous: based on 2X2 table Φ=׀BC-AD׀ /[(A+B)(C+D)(A+C)(B+D)]1/2 a b c d

In the last year, have you seen a physician for any of the above symptoms? Computerized charts Y N Questionnaire Y 50 20 N 10 30  = 0.45 However, not reported in methods or results

Analysis of ordinal data Wilcoxon signed sum rank

How many in-pt nights in last yr? Quest 0 2 5 5 5 7 15 Charts 0 0 5 7 10 3 12 sign 0 2 0 -2 -5 4 3 signRank 1.5 3.5 1.5 -3.5 -7 6 5 add the positives: (1.5 + 3.5 + 1.5 + 6 +5) = 17.5 add the negatives: (-3.5 - 7) = -10.5 Ho sumpositive = sum negative then use a table to find p-value or use an approximate z test then find p-value appropriate use of Wilcoxon in the statistical analysis

Ordinal: treat as continuous Choice of weighted kappa, Pearson, ICC used Weighted kappa kappa= (Po- Pe) / (1-Pe) quadratic weight: use ( exact disagreement)2 Here, weight: use (exact agreement)2 wKappa= 1- {wij X Poij /(wij X Peij)} why use simple agreement, kappa and Wilcoxon?

a word on reliability Pearson vs ICC vs kappa If use quadratic weight (disagreement)2 then kappa = ICC kappa: only for dichotomous variables Pearson: linear regression ANOVA measures how well you may fit the obs on a straight line

Pearson vs ICC Pearson compares 2 obs at a time or only 1 ICC if 10 raters: 45 Pearson correlation coefficients! or only 1 ICC

References Health Measurement Scales. A practical guide to their development and use DL Steiner, GR Norman Statistics The Bare Essentials HRM 727 Measurement Tool, course package G Norman