Constructing and Validating the Short Recovering Quality of Life (ReQoL) Measure for Use in a Mental Health Population. Anju Keetharuth, John Brazier, Janice Connell, Jill Carlton, Lizzie Taylor Buck, Tom Ricketts and Michael Barkham School of Health and Related Research Contact: reqol@sheffield.ac.uk Website: www.reqol.org.uk

Rationale for commissioning a new measure Concerns with limitations of EQ-5D in calculating QALYs for assessing cost-effectiveness in the areas of mental health Policy context Use of PROMs in commissioning value for money services Drive to expand use of PROMs Traditional measures focused on symptoms or too generic Recovery themes (CHIME) (Leamy, 2011) In terms of the rationale for developing a new measure, there are 3 main points: First, traditionally measures have focussed on symptoms, for example PHQ-9 and have also been through clinical assessment, for example HoNos. Second, in a recent report submitted to the DH, Jed Boardman, Mike Slade and Geoff Shepherd mentioned that existing measures do not adequately capture the wide range of mental health conditions and severity Finally a measure is needed that adequately captures what matters to service users. Such concerns include those identified by Leamy and colleagues. These are often referred as CHIME which stands for Connectedness, Hope, Identity, Meaning and Purpose, and Empowerment Connectedness Hope Identity Meaning and Purpose Empowerment

ReQoL (QoL literature) Stage 1: Identifying themes ReQoL (QoL literature) Connell et al. 2012 Recovery literature Leamy et al. 2011 Methods used 1. Belonging and Relationships Connectedness Systematic review of qualitative literature 2. Hope Hope 3. Self perception Identity Qualitative interviews with 19 service users 4. Activity Meaning 5. Autonomy, Choice and Control Empowerment Framework analysis to identify themes 6. Well-Being Well-being/Symptoms There are 5 main stages of the project. The first stage will involve the confirmation of themes for the measure and this runs for 3 months until the end of April 2014. Stage 2 is where we generate a long list of items from existing measures and the work in Stage 1. From Aug – Oct, we will be testing those items to see if they make sense to service users. In Stage 4a, we will ask service users to fill in the long measure to allow us to carry out analyses to construct a short measure. In Stage 4b we expect the brief measure to be filled routine in a number of NHS sites and other settings across England. We will test the psychometric properties of the measure. We are hoping to get a measure by October 2015 at the latest. Stage 5 is the ultimate stage where we will obtain preferences of the general public so that this measure can be used to calculate QALY which is how health benefits are valued in assessing cost-effectiveness. 7. Physical health

Stage 2: Generating items Methods used Identified items under each theme from: Manuscripts from qualitative interviews (mainly) Existing QoL measures - Existing Recovery measures 2. Applied Streiner and Norman criteria ReQoL team N = 87 Scientific Group N = 101 Expert User Group N = 180 ReQoL Sheffield team N = 122 Initial round of selection N = 1597

Stage 3: Face to face validity testing 59 adult service users interviewed across 4 NHS trusts 17 younger service users aged 16-18 22 service users from South Asian and Polish backgrounds Translatability assessment (MAPI Group) Methods used Participants were presented with a subset of 87 questions Commented on the questions Chose their favourite ones and rephrased items Wide range of self-reported diagnosis: Schizophrenia, Depression, Bipolar, Psychosis, OCD, Anxiety, Asperger’s syndrome, PTSD, Personality disorder, Eating disorder, Alcohol problems, Cognitive impairment, Agoraphobia, Tourette’s syndrome. Items falling in the following categories were eliminated: Potentially distressing Judgmental Difficult to respond to Not too relevant to everyone Difficulties with interpretation Potentially distressing (e.g. very negative items – I detested myself) Judgemental (e.g. advocating a way of life) Difficult to respond to (e.g. making inference about feelings of other people) Not too relevant to everyone (e.g. I felt loved versus I felt cared for) Difficulties with interpretation (e.g. I felt guilty)

6514 service users required A subset were followed up (N = 1237) Stages 4 and 5: Psychometric testing 6514 service users required A subset were followed up (N = 1237) Methods used Recruitment took place in secondary care, primary care and voluntary sector 20 Trusts 64% Age Category 6 GP practices 20% 16 to 29 10% Casper Trial Cohort 6% 30 to 59 45% Service users completed item pools Online panels 8% 60 to 79 23% Voluntary sector 2% 79 and over 22% Self report main diagnosis Mode of administration Face to face in clinics Post Online Depression 44% Anxiety 28% Psychotic disorders 17% Potentially distressing (e.g. very negative items – I detested myself) Judgemental (e.g. advocating a way of life) Difficult to respond to (e.g. making inference about feelings of other people) Not too relevant to everyone (e.g. I felt loved versus I felt cared for) Difficulties with interpretation (e.g. I felt guilty) Bipolar 11% Personality disorder 6% Eating disorders 6%

Principles informing item selection All six mental health domains identified in Stage 1 are represented in the final measures Service user acceptability (qualitative data collected in Stages 3, 4 and 5) Clinician acceptability and usefulness (qualitative data based on focus groups with clinicians) Robust psychometric properties following the application of various analyses In terms of the rationale for developing a new measure, there are 3 main points: First, traditionally measures have focussed on symptoms, for example PHQ-9 and have also been through clinical assessment, for example HoNos. Second, in a recent report submitted to the DH, Jed Boardman, Mike Slade and Geoff Shepherd mentioned that existing measures do not adequately capture the wide range of mental health conditions and severity Finally a measure is needed that adequately captures what matters to service users. Such concerns include those identified by Leamy and colleagues. These are often referred as CHIME which stands for Connectedness, Hope, Identity, Meaning and Purpose, and Empowerment

Criteria to assess psychometric evidence Analyses Guidance/Judgement/rule Notes Missing data at item level If any item has ≥5% missing data, this item should be dropped None identified 2. Factor Analyses Identify items with high residual correlations (>0.1) 15 pairs identified (19 items)   3. IRT - Misfitting items Identify items with p values < 0.05 in at least 3 datasets. However, do not use the item level fit to exclude items. Acknowledge the misfit but retain item in the item pool. 5 items identified 4. IRT - Information functions Ensure that items cover the whole measurement range (i.e., intensity) by choosing items to balance maximization of information over the total range and content validity (including items from all themes) Items were chosen to cover the whole range   5.Differential Item Functioning Exclude items with DIF (age, ethnicity, gender, condition) 6. Sensitivity to change Ensure selected items show change in response over time Limited change: 2 items

Validation Test-retest reliability (2 weeks interval and those reporting same health) Intra-class coefficient ReQoL-10 = 0.85 (n = 488 general population; n = 279 patients) ReQoL-20 ICC = 0.90 (n = 249 general population) ICC = 0.87 (n = 100 patients) Cronbach alpha > 0.8 2. Convergent validity Correlation with SWEMWBS and CORE-10 (summative scores) > 0.88 3. Known Group validity using Standardised effect size (SES) General population versus: Patient population SES ReQoL -10 = 0.93 ReQoL-20 = 1.05 Common mental health SES ReQoL -10 = 1.17 ReQoL-20 = 1.33 Schizophrenia SES ReQoL -10 = 0.68 ReQoL-20 = 0.73 Bipolar SES ReQoL -10 = 1.00 ReQoL-20 = 1.11 Personality Disorder SES ReQoL -10 = 1.99 ReQoL-20 = 2.17 4. Responsiveness to change Standardised response mean (SRM)

Next steps Valuation – TTO Validation Implementation Cross-cultural adaptation work ReQoL licences obtained from: Oxford Innovation Ltd http://innovation.ox.ac.uk/outcome-measures/recovering-quality-life-reqol-questionnaire/ Link also from www.reqol.org.uk

The Meadowhead Acknowledgments Group Practice Funder: Policy Research Programme, Department of Health, UK Usual disclaimer applies. Members of the governance groups: Scientific, Advisory, Stakeholder and Expert Users All participants in the study, NIHR – CRN Mental health, staff at all participating trusts, charities and GP surgeries MAPI Group for translatability assessment The Meadowhead Group Practice

EXAMPLE – Item selection BELONGING & RELATIONSHIPS Top item:BEL2 – I felt lonely Relationships – requiring a judgement on other people   Relationship –self BEL3P WB5 BEL1 BEL4 BEL5P WB6 BEL2 I felt lonely Do we need another item in this theme? If so, which one of: BEL4 BEL3P BEL1 BEL5P WB5 and WB6 ACT2P ACT5P

Relationships with others 2nd BELONGING ITEM Relationships with others