Outcome Measurements Study Design Liz Lingard BPhty, MPhil, MPH Research Coordinator Department of Orthopaedics Freeman Hospital

Orthopaedic Clinical Research Traditionally focused on reporting clinical and radiographic results Traditionally focused on reporting clinical and radiographic results Rates of complications following procedures Rates of complications following procedures Survivorship analysis of the life of a prosthesis Survivorship analysis of the life of a prosthesis Many studies are observational Many studies are observational

Need For Outcome Measures Technological advances, epidemiologic and demographic trends and rising public demand had rapidly increased health care expenditure over the past three decades Technological advances, epidemiologic and demographic trends and rising public demand had rapidly increased health care expenditure over the past three decades As health care resources are limited they need to be allocated according to clinical effectiveness and cost-effectiveness As health care resources are limited they need to be allocated according to clinical effectiveness and cost-effectiveness

Orthopaedic Interventions Surgical treatment for one or more of the following musculoskeletal problems Surgical treatment for one or more of the following musculoskeletal problems  Pain  Loss of range of motion  Instability  Functional limitation  Deformity  Trauma  Tumour

International Classification of Functioning, Disability and Health (ICF) Functioning and Disability Functioning and Disability  Body function and structures  Activities and participation Contextual Factors Contextual Factors  Environmental factors  Personal factors

Interaction Between ICF Components

Uses of Outcome Measures Clinical research Clinical research Audit and quality assurance Audit and quality assurance Assess health care needs of populations Assess health care needs of populations Assist in individual patient care Assist in individual patient care

Outcome Measures Assessment of disease or injury severity Assessment of disease or injury severity Measure change over time due to: Measure change over time due to:  Natural history  Intervention Supplement mortality, clinical, radiological and laboratory measures Supplement mortality, clinical, radiological and laboratory measures Outcome instruments Outcome instruments  Observer administered  Patient administered

Observer Administered Functional Tests Functional Tests  Timed Up & Go  6 minute walk  Activity Monitoring Global rating scores Global rating scores  Knee Society Score  Harris Hip Score  Foot and Ankle Score

ActivPal Monitor Ambulatory activity monitor worn for 7 days Activity Monitoring

matched walking periods Subject A: 72 year old THR patientSubject B: 65 year old spouse

Activity Summary

Stepping Summary

Global Rating Scores May combine measures of pain, joint range of motion, stability, muscle power, deformity, function and even complications and investigations May combine measures of pain, joint range of motion, stability, muscle power, deformity, function and even complications and investigations Weight of scoring of each item varies Weight of scoring of each item varies Inter / intra-rater reliability & observer bias Inter / intra-rater reliability & observer bias

Pitfalls of global scores Different scoring systems assign different weights to parts of the score and the summary results are often inconsistent giving contrary measures of success in the same patient. (Bryant et al. JBJS-Br 1993) Different scoring systems assign different weights to parts of the score and the summary results are often inconsistent giving contrary measures of success in the same patient. (Bryant et al. JBJS-Br 1993)

Patient Administered Report summary scores for different health dimensions Report summary scores for different health dimensions  Physical Function  Symptoms – pain  Global judgements of health  Psychological well-being  Social well-being  Cognitive functioning  Role activities  Personal constructs  Satisfaction

Patient Administered Disease-specific (WOMAC, AIMS) Disease-specific (WOMAC, AIMS) Site or region-specific (KOOS, DASH, ODI) Site or region-specific (KOOS, DASH, ODI) Dimension-specific (McGill Pain Questionnaire) Dimension-specific (McGill Pain Questionnaire) General health status (SF-36, NHP) General health status (SF-36, NHP) Health state utility (EQ-5D) Health state utility (EQ-5D) Satisfaction (Management / Outcome) Satisfaction (Management / Outcome)

WOMAC Western Ontario & McMaster University OA index Western Ontario & McMaster University OA index Tri-dimensional scale (Likert and VAS versions): Tri-dimensional scale (Likert and VAS versions):  Pain (5 items)  Stiffness (2 items)  Function (17 items) Validated on patients with OA of hip and knee Validated on patients with OA of hip and knee  Sensitive and responsive for detecting change over time due to NSAIDs and joint arthroplasty Translated into over 50 different languages Translated into over 50 different languages

WOMAC Responses for each item are none, mild, moderate, severe or extreme (pain, stiffness or difficulty) Responses for each item are none, mild, moderate, severe or extreme (pain, stiffness or difficulty) Scores transformed to (100 best) Scores transformed to (100 best)  100 = none  75 = mild  50 = moderate  25 = severe  0 = extreme

Preoperative WOMAC None Mild Moderate Severe Extreme Note: Data on Primary Joints – 1,061 TKA, 760 THA & 174 BHR

1-Year WOMAC None Mild Moderate Severe Extreme Note: Data on Primary Joints TKA, 387 THA & 75 BHR

Developed and validated by Ewa Roos Developed and validated by Ewa Roos Meniscus injury, ACL injury and/or OA Meniscus injury, ACL injury and/or OA Adds dimensions to the WOMAC scale: Adds dimensions to the WOMAC scale:  Sport and recreation function  Knee-related quality of life Improves sensitivity of evaluation of younger or more physically active persons Improves sensitivity of evaluation of younger or more physically active persons Knee injury and Osteoarthritis Outcome Score (KOOS)

Squatting Squatting Running Running Jumping Jumping Twisting/pivoting Twisting/pivoting Kneeling Kneeling KOOS – Sports/Recreation

How often are you aware of your knee problems? How often are you aware of your knee problems? Have you modified your lifestyle to avoid potentially damaging activities to your knee? Have you modified your lifestyle to avoid potentially damaging activities to your knee? How much are you troubled with lack of confidence in your knee? How much are you troubled with lack of confidence in your knee? In general, how much difficulty do you have with your knee? In general, how much difficulty do you have with your knee? KOOS – Knee Related QOL

KOOS Profile Roos EM et al, Osteoarthritis & Cartilage 1999; 7: WOMAC KOOS

KOOS Profile Roos EM et al, JOSPT 1998; 78:88-96

WOMAC Pain and Function None Mild Moderate Severe Extreme Patients at 10 years after primary TKR

How often are you aware of your knee problem? Patients at 10 years after primary TKR

In general, how much difficulty do you have with your knee? Patients 10 years after primary TKR

Short-Form 36 (SF-36) Reports on 8 health dimensions Reports on 8 health dimensions  Physical Functioning  Bodily Pain  Role Physical  General Health Perception  Mental Health  Role Emotional  Vitality  Social Functioning Each subscale given a score (100 best) Each subscale given a score (100 best) Physical component score Mental component score

SF-36 – Primary TKA Note: Data on 1,061 at preop, year & 243 at 2-years

SF-36 – Primary THA Note: Data on 760 at preop, 387 at 1-year & 186 at 2-years

SF-36 – Primary BHR Note: Data on 174 at preop, 75 at 1-year & 35 at 2-years

Satisfaction with outcome Asked on follow-up questionnaires Asked on follow-up questionnaires Patient rates how satisfied they are with the results of surgery: Patient rates how satisfied they are with the results of surgery:  Overall  Pain relief  Ability to do ADL  Ability to do recreational activities Responses are very satisfied, somewhat satisfied, somewhat dissatisfied and very dissatisfied Responses are very satisfied, somewhat satisfied, somewhat dissatisfied and very dissatisfied

Satisfaction – Primary TKA Note: Data on 490 primary TKA patients at 1-Year

Satisfaction – Primary THA Note: Data on 388 primary THA patients at 1-Year

ICF Quiz

Choosing an Outcome Measure Formulate a precise study question Formulate a precise study question Know your patient population Know your patient population Know what independent variables may affect the outcome Know what independent variables may affect the outcome Commonly used outcome measure does not necessarily mean it is the best measure to use for your study Commonly used outcome measure does not necessarily mean it is the best measure to use for your study

Independent Variable Examples Socio-demographics Socio-demographics Body mass index Body mass index Comorbid medical conditions Comorbid medical conditions Comorbid musculoskeletal conditions Comorbid musculoskeletal conditions Mental health status Mental health status Common geriatric problems Common geriatric problems

Appropriateness Disease / injury to be studied Disease / injury to be studied Patient population Patient population Intervention Intervention Dimension of health likely to be affected Dimension of health likely to be affected Acceptability Acceptability Feasibility Feasibility

Acceptability Minimise burden to patients: Minimise burden to patients:  Avoid distress to patients  Reduce refusal rate  Reduce incomplete data Need to consider: Need to consider:  Literacy of patients  Legibility and layout of questionnaire  Time taken to complete

Feasibility Impact on clinical staff / researchers Impact on clinical staff / researchers  Staff attitudes are key determinant of data collection Data collection Data collection  Brief questionnaires  Lengthy questionnaires with interviews  Postal / telephone Coding and entering data Coding and entering data Analysis of data Analysis of data

Validity Measure what it says it does Measure what it says it does Content – appropriate range of items Content – appropriate range of items Face – measures what they aim to measure Face – measures what they aim to measure Construct – correlates to other measures Construct – correlates to other measures Criterion – correlates to a ‘gold standard’ Criterion – correlates to a ‘gold standard’

Reliability Measures accurately Measures accurately Internal consistency Internal consistency Test – re-test Test – re-test Inter-rater Inter-rater Intra-rater Intra-rater

Responsiveness Sensitive to detecting changes over time that are important to the patient Sensitive to detecting changes over time that are important to the patient Correlates with changes in disease severity Correlates with changes in disease severity Effect size index = Effect size index = ( (mean 1 – mean 2 )/ pooled SD Standardised response mean = Standardised response mean = (SD of change in score (mean 1 – mean 2 )/ SD of change in score

Precision Ceiling and floor effects Ceiling and floor effects Same amount of change across different parts of the scale may not be equal Same amount of change across different parts of the scale may not be equal Instruments that make more distinctions are not more precise Instruments that make more distinctions are not more precise

Interpretation Outcome measures produce numeric scores Outcome measures produce numeric scores Score ranges need to be clear Score ranges need to be clear Need to know what is the smallest clinical meaningful changes that are perceptible to a patient for that particular instrument Need to know what is the smallest clinical meaningful changes that are perceptible to a patient for that particular instrument

Study Design

Retrospective Retrospective  Deliberate intervention  Observational studies Prospective Prospective  Deliberate intervention  Observational studies

Study Design Cross-sectional studies Cross-sectional studies  Disease description  Diagnosis and staging  Abnormal ranges  Disease severity  Disease process

Study Design Longitudinal studies Longitudinal studies  Prospective  Randomised  Non-randomised

Randomised vs. non-randomised RCT is seen as the ‘gold standard’ RCT is seen as the ‘gold standard’  Should ensures patients differ only by their exposure to the intervention  Often difficult to generalise results due to inclusion/exclusion criteria  Blinded trials in orthopaedics almost impossible introducing observer/patient bias Non-randomised trials Non-randomised trials  Cohort and case-controlled studies  More inclusive but may have selection bias

Study design For studies that compare health care interventions to be valid they must produce results that are: For studies that compare health care interventions to be valid they must produce results that are:  Generalisable to to other potential patients (high external validity)  Estimate outcome effects that can be reliably attributed to the intervention (high internal validity)

Solutions to threats of validity Internal validity Internal validity  Allocation bias (confounding) – risk adjustment and subgroup analysis  Patient preference – preference arms or adjustment for preference External validity External validity  Exclusions – expand inclusion criteria  Non-participation – multi-centre, pragmatic design  Non-participation – less rigorous consent

Potential Problems Recruitment Recruitment  Need to know what proportion of the reference population have been recruited  Reasons for non-participation – does this group of patients differ from participants. Knowing baseline details about this group strengthens your study  Volunteerism likely to be associated with age, gender, socioeconomic status, education etc. which may influence outcomes

Potential Problems Compliance Compliance  Depends on how well informed patients are at start of trial of what is expected  If intervention is ongoing (e.g. physical vs. surgical intervention) then poor compliance with intervention can affect statistical power  If patients are non-compliant with parts of study this is preferable to total non-compliance  Need to track all patients and document reasons for lost to follow-up

Statistical Support Speak to a statistician Speak to a statistician  After you have reviewed the literature, selected your primary outcome measure and drafted a protocol  Sample size calculation  Method of data collection  Advice about coding of data  Analysis of data BUT  Before you submit to either R&D or LREC