Getting to grips with evidence that matters! NIHR Nottingham Hearing Biomedical Research Unit, UK Melanie Ferguson Helen Henshaw British Society of Hearing Aid Audiologists 13.9.14

Learning outcomes To explain the hierarchy of evidence To describe how to assess quality of research articles Wong and Hickson, 2012, Evidence-based practice in Audiology, Plural Publishing

Pie charts

Hierarchy of evidence Cox (2005) J Am Acad of Audiology, 16 (7) AAA (2010) APD guidelines Strength of evidence

Level of evidence – validity criteria Basis of systematic review on auditory training (Henshaw and Ferguson, 2013) Scientific study-specific Randomisation? Power calculation? For sample size (n) Blinding of participants and researchers? ----------- Outcome measure selection reporting Intervention-specific Ecologically valid training environment (at home)? Training performance feedback? Follow-up assessment? ----------- Compliance 0 = flawed or no information form which to make a judgement 1 = weak information or lack of detail 2 = appropriate use and reporting

Table 1: Study quality scores. Scoring: please circle 0 for flawed or no information from which to make a judgement, 1 = weak information or lack of detail, 2 = appropriate use and reporting, for scientific and intervention specific validity criteria. *Level of evidence (PTO): Study quality score of 0-3 = very low, 4-6 = low, 7-9 = moderate, 10-12 = high (adapted from GRADE Working Group, 2004). Article Scientific study validity criteria Intervention-specific study validity criteria Study quality score Level of evidence* Comments Randomisation? Power calculation to determine sample size? Blinding of participants and researchers? Ecologically valid (at- home) training environment? Training performance feedback provided? Follow-up to examine retention of training effects?   Paper #1 0 1 2 2 very-low Paper #2 10 high

Interactive session Two papers abstract, introduction, methods read Take table headings (e.g. randomisation) Search paper Make your judgement of quality (0,1,2) Do same with second paper Discussion at the end 0 = flawed or no information form which to make a judgement 1 = weak information or lack of detail 2 = appropriate use and reporting Rubbish Somewhere in between Good

Example - randomisdation Paper #1 Pg 920 “All the participants were submitted to the evaluation only after they were assigned to the Experimental group and Control group, and the individuals themselves pick a number to be randomized to which group they would be sent to”.

Level of evidence Study quality score   Level of evidence Confidence in estimation of effect 0-3 Very low The estimation of effect is uncertain 4-6 Low Further evidence is very likely to impact on our confidence in the estimation of effect and are likely to change the estimate 7-9 Moderate Further evidence is likely to impact on our confidence in the estimation of effect and may change the estimate 10-12 High Further evidence is very unlikely to change our confidence in the estimation of effect From GRADE working group 2004 End of session – what do you think the quality of the papers is

Purpose of the exercise Brief introduction to appraisal of the literature To highlight the factors that underlie “quality” of a paper to gain a broad understanding of what quality means Stop talking Get stuck in

Table 1: Study quality scores. Scoring: please circle 0 for flawed or no information from which to make a judgement, 1 = weak information or lack of detail, 2 = appropriate use and reporting, for scientific and intervention specific validity criteria. *Level of evidence (PTO): Study quality score of 0-3 = very low, 4-6 = low, 7-9 = moderate, 10-12 = high (adapted from GRADE Working Group, 2004). Article Scientific study validity criteria Intervention-specific study validity criteria Study quality score Level of evidence* Comments Randomisation? Power calculation to determine sample size? Blinding of participants and researchers? Ecologically valid (at- home) training environment? Training performance feedback provided? Follow-up to examine retention of training effects?   Paper #1 0 1 2 2 very-low Randomisation not clear No power calculation Double-blinding? Participants are not able to be blinded (intervention vs. no intervention) Training completed in lab No feedback mentioned No follow-up assessment mentioned General point: not repeatable because outcome measures are not clear/referenced. Paper #2 10 high Minimisation – best form of randomisation – adaptive stratified sampling that is used in clinical trials - aims to minimise the imbalance between participants in two groups based on pre-specified factors. (Pocock & Simon, 1975) Power calculation = 20 individuals per group to detect 2.5 dB SNR difference in digit triplets between the groups, Cohen’s d = .89  No blinding mentioned  Training took place @ home  Feedback (correct/incorrect response) was provided during training and at the end of each session  Follow-up 4 weeks post-training