1. Getting to grips with evidence that matters!
NIHR Nottingham Hearing Biomedical Research Unit, UK
Melanie Ferguson
Helen Henshaw
British Society of Hearing Aid Audiologists

2. Learning outcomes To explain the hierarchy of evidence
To describe how to assess quality of research articles
Wong and Hickson, 2012, Evidence-based practice in Audiology, Plural Publishing

3. Pie charts

4. Hierarchy of evidence Cox (2005) J Am Acad of Audiology, 16 (7)
AAA (2010) APD guidelines
Strength of evidence

5. Level of evidence – validity criteria
Basis of systematic review on auditory training (Henshaw and Ferguson, 2013)
Scientific study-specific
Randomisation?
Power calculation?
For sample size (n)
Blinding of participants and researchers?
Outcome measure
selection
reporting
Intervention-specific
Ecologically valid training environment (at home)?
Training performance feedback?
Follow-up assessment?
Compliance
0 = flawed or no information form which to make a judgement
1 = weak information or lack of detail
2 = appropriate use and reporting

7. Table 1: Study quality scores.
Scoring: please circle 0 for flawed or no information from which to make a judgement, 1 = weak information or lack of detail, 2 = appropriate use and reporting, for scientific and intervention specific validity criteria. *Level of evidence (PTO): Study quality score of 0-3 = very low, 4-6 = low, 7-9 = moderate, = high (adapted from GRADE Working Group, 2004).
Article
Scientific study validity criteria
Intervention-specific study validity criteria
Study quality score
Level of evidence*
Comments
Randomisation?
Power calculation to determine sample size?
Blinding of participants and researchers?
Ecologically valid (at- home) training environment?
Training performance feedback provided?
Follow-up to examine retention of training effects?
Paper #1 2
very-low
Paper #2 10
high

8. Interactive session Two papers abstract, introduction, methods
read
Take table headings (e.g. randomisation)
Search paper
Make your judgement of quality (0,1,2)
Do same with second paper
Discussion at the end
0 = flawed or no information form which to make a judgement
1 = weak information or lack of detail
2 = appropriate use and reporting
Rubbish
Somewhere in between
Good

9. Example - randomisdation
Paper #1
Pg 920
“All the participants were submitted to the evaluation only after they were assigned to the Experimental group and Control group, and the individuals themselves pick a number to be randomized to which group they would be sent to”.

10. Level of evidence
Study quality score
Level of evidence
Confidence in estimation of effect
0-3
Very low
The estimation of effect is uncertain
4-6
Low
Further evidence is very likely to impact on our confidence in the estimation of effect and are likely to change the estimate
7-9
Moderate
Further evidence is likely to impact on our confidence in the estimation of effect and may change the estimate
10-12
High
Further evidence is very unlikely to change our confidence in the estimation of effect
From GRADE working group 2004
End of session – what do you think the quality of the papers is

11. Purpose of the exercise
Brief introduction to appraisal of the literature
To highlight the factors that underlie “quality” of a paper
to gain a broad understanding of what quality means
Stop talking
Get stuck in

12. Table 1: Study quality scores.
Scoring: please circle 0 for flawed or no information from which to make a judgement, 1 = weak information or lack of detail, 2 = appropriate use and reporting, for scientific and intervention specific validity criteria. *Level of evidence (PTO): Study quality score of 0-3 = very low, 4-6 = low, 7-9 = moderate, = high (adapted from GRADE Working Group, 2004).
Article
Scientific study validity criteria
Intervention-specific study validity criteria
Study quality score
Level of evidence*
Comments
Randomisation?
Power calculation to determine sample size?
Blinding of participants and researchers?
Ecologically valid (at- home) training environment?
Training performance feedback provided?
Follow-up to examine retention of training effects?
Paper #1 2
very-low
Randomisation not clear
No power calculation
Double-blinding? Participants are not able to be blinded (intervention vs. no intervention)
Training completed in lab
No feedback mentioned
No follow-up assessment mentioned
General point: not repeatable because outcome measures are not clear/referenced.
Paper #2 10
high
Minimisation – best form of randomisation – adaptive stratified sampling that is used in clinical trials - aims to minimise the imbalance between participants in two groups based on pre-specified factors. (Pocock & Simon, 1975)
Power calculation = 20 individuals per group to detect 2.5 dB SNR difference in digit triplets between the groups, Cohen’s d = .89 
No blinding mentioned 
Training took home 
Feedback (correct/incorrect response) was provided during training and at the end of each session 
Follow-up 4 weeks post-training