How to Interpret Data: Critical appraisal Colette Smith UCL Research Department Infection and Population Health JUSTRI Skills Tool Kit Training 12 th December 2015

Putting study results into context Single studies cannot be interpreted in isolation – we also need to evaluate existing literature relating to the question of interest Basing judgment about a particular treatment (or ‘risk factor’) only on published material may lead to a biased picture because of publication bias Initiatives to compile information on all trials undertaken include: Cochrane collaboration - registers of trials and meta- analyses ( 2

Why do we need to appraise research? Research informs clinical care and public health policy. Evidence based medicine - ”Integrating individual clinical experience with best available evidence from systematic research” (Sackett). Vast numbers of research studies are carried out, presented and published each year. In order to understand whether we should apply the findings from a research study to clinical practice we need to assess its validity, quality and applicability. 3

Why do we need to appraise research? 1 Deliberate research fraud is unlikely but research may have: methodological problems or limitations inappropriate analysis or poor presentation of results insufficient explanation of methods/results misleading interpretation or conclusions problems with generalisability 4

Critical appraisal Important to consider and be aware of all potential biases, confounders and role of chance when interpreting results of studies However, it is easy to find fault with all medical research studies No study is ever perfect – all have strengths and weaknesses The important thing to consider is whether the study is sufficiently well performed that you believe the results 5

Hierarchy of evidence [Systematic review of RCTs] Randomised controlled trials (RCTs) Non-randomised intervention studies Cohort studies Case-control studies Cross-sectional studies Time series/ecological studies Expert opinion, consensus conference 6 Strongest evidence Weakest evidence

Components of RCT research appraisal 1.What question is the research asking? 2.Are the methods valid and appropriate? 3.What are the results? 4.How should the results be interpreted? 5.How does this study fit in with existing research? 6.What are the implications of the findings for clinical practice? 7

Components of observational appraisal 1 Although general appraisal issues same as for clinical trial, some additional appraisal issues Bias due to lack of randomisation: Differences between treatment/risk factor groups may be explained by many different factors Have known or possible confounding factors been considered? Are there any relevant results from RCTs? 8

Components of observational appraisal 2 Bias due to incomplete follow-up Observational data may be particularly prone to loss to follow-up. What proportion of subjects are lost to follow-up overall and in each group of interest? Lack of single pre-specified comparison Often observational databases are used to test many different research questions using a variety of different approaches Was the primary question of interest and analysis method clearly defined, prior to analysis? Have all relevant analyses been reported? Have multiple statistical tests been performed? 9

Tools for evaluating studies There are a number of guidelines/checklists for reporting and appraising different types of study designs: RCTs: Consolidated Standards of reporting trials (CONSORT) Observational studies: Strengthening the reportiong of observational studies in epidemiology (STROBE) Systematic reviews: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) All: Enhancing the QUAlity and Transparency Of health Research (EQUATOR)

Public health priorities Ideally based on ‘evidence’ - meta-analyses and systematic reviews Considerations of efficiency, cost-effectiveness and harm e.g. eradication of poverty for improving health? Must weigh up competing interests: Evidence – Values – Resources – Perceptions Different groups will have different priorities: Public Industry Government Scientific community 11

Summary Ideally much of clinical practice and health policy are based on research findings - therefore research appraisal is crucial. In appraising the results of clinical trials and observational studies we should consider the appropriateness of the methods and the interpretation and implications of the results. It is also important to put results from new studies into the context of existing research findings. 12

Socio-economic factors and late diagnosis of HIV in in the ABC Hospital Colette Smith on behalf of ABC HIV Hospital Cohort Disclaimer: All data are hypothetical, and do not necessarily represent real life! 13

Introduction Late diagnosis of HIV in the UK continues to be a major problem, with 42% diagnosed with CD4 count <350 cells/mm 3 in A timely diagnosis is important for slowing the progression of HIV 36 and therefore improving prognosis 37-39, reducing onwards transmission 40 and reducing costs to health services 41 Little is known about the association between late diagnosis and socio-economic factors such as housing, education and employment 14 1 Yin. Public health England pdf 36 Delpech. HIV med. 2013;14(S3):19-24; 37 Nakagawa. AIDS. 2012;26(3):335-43; 38 May. BMJ. 2011;343; 39 ARTCC Collaboration. Lancet. 2008;372(9635):293-9; 40 Marks. AIDS. 2006;20(10): ; 41 Mukolo. AIDS and Behavior. 2013;17(1):5-30; pdf

Methods – study design Patient registration forms have been used at ABC Hospital, London, UK in April 2001 which collected data on: – Presentation – HIV risk behaviour – Medical history – Demographic and socio-economic background 2003 met inclusion criteria: i)Attended hospital for first visit April May 2015 ii)Diagnosed with HIV within 1 year of this first visit iii)Had a first visit form completed (22% of all new diagnoses) Late diagnosis - CD4<350 cells/mm 3 within 12 months of diagnosis 15

Methods - explanatory variables Three measures of socio-economic status collected as self- reported on patient registration form: – Housing status: Home owner vs. Non-home owner – Current employment status: Employed vs. Not employed – Highest level of education: University or higher vs. Less than university Prevalence of late diagnosis assessed according to each socio- economic and demographic factor Associations between socio-economic factors and late diagnosis were assessed by logistic regression, adjusted for: – Gender/ sexual orientation (MSM; MSW; Women) – Age (continuous) 16

Percent diagnosed late by demographic factor 902/2003 (45%) were diagnosed with CD4<350 cells/mm 3 (29% with CD4<200 cells/mm 3 ) 17 P = ~ test for trend P =

Percent diagnosed late by socio-economic factor 18 P = ~ test for trend P = 0.104P = 0.786

Demographic factors associated with late diagnosis Unadjusted OR (95% CI) P Adjusted* OR (95% CI) P Gender/ sexual orientation MSM MSW3.67 (1.73, 7.79)3.38 (0.57, 10.23) Women3.16 (1.63, 6.15)3.11 (0.65, 8.09) Age<30 years years1.18 (0.47, 2.95)0.91 (0.35, 2.39) years2.17 (0.89, 5.27)1.52 (0.60, 3.87) >50 years2.47 (1.02, 6.72)1.76 (0.97, 3.19) Home ownerYes No3.26 (0.15, 7.83)1.00 (0.21, 4.75) EducationUniversity Non-university1.77 (0.88, 3.59)1.70 (0.56, 2.94) EmploymentEmployed Unemployed1.08 (0.50, 2.33)0.90 (0.40, 2.04) 19

Women who live in east London aged 31 – 57 years 20 Adjusted* OR (95% CI) P Home ownerYes1< No9.73 (3.64, 27.3) EducationUniversity Non-university5.06 (2.43, 12.11) EmploymentEmployed10.80 Unemployed1.02 (0.19, 5.04)

Conclusion Low socio-economic status has no impact on late diagnosis Age is an important predictor of late diagnosis whereas gender/sexual orientation group is not It is particularly important to monitor women who live in east London who are aged 31 to 57 years, as this is a high-risk group 21