Being Curious Or Making Reading More Worthwhile Critical Appraisal Being Curious Or Making Reading More Worthwhile

The Problem Vast and expanding literature. Limited time to read. Different reasons to read – mean different strategies. Keeping up to date. Answering specific clinical questions. Pursuing a research interest. Bruce Davies

Stages Clarify your reasons for reading. Specify your information need. Identify relevant literature. Critically appraise what you read. Bruce Davies

Clarify Your Reasons for Reading. Keeping up to date. Skimming the main journals and summary bulletins. Answering specific clinical questions. Finding good quality literature on subject. Pursuing a research interest. Extensive literature searching. Bruce Davies

Specify Your Information Need. What kind of reports do I want? How much detail do I need? How comprehensive do I need to be? How far back should I search? The answers to these questions should flow from the reasons for reading. Bruce Davies

Identify Relevant Literature. There are many ways of finding literature. Remember to ask a librarian – they are the experts. Selectivity is the key to successful critical appraisal. August 2001 Bruce Davies

Critically Appraise What You Read. Separating the wheat from the chaff. Time is limited – you should aim to quickly stop reading the dross. Others contain useful information mixed with rubbish. Simple checklists enable the useful information to be identified. August 2001 Bruce Davies

Questions to Ask Is it of interest? Why was it done? How was it done? What has been found? What are the implications? What else is of interest? August 2001 Bruce Davies

Questions to Ask Is it of interest? Why was it done? Title, abstract, source. Why was it done? Introduction. Should end with a clear statement of the purpose of the study. The absence of such a statement can imply that the authors had no clear idea of what they were trying to find out. Or they didn’t find anything but wanted to publish! Bruce Davies

Questions to Ask How was it done? Methods. Brief but should include enough detail to enable one to judge quality. Must include who was studied and how they were recruited. Basic demographics must be there. An important guide to the quality of the paper. Bruce Davies

Questions to Ask What has it found? Results. The data should be there – not just statistics. Are the aims in the introduction addressed in the results? Look for illogical sequences, bland statements of results. ? Flaws and inconsistencies. All research has some flaws – this is not nit picking, the impact of the flaws need to assessed. Bruce Davies

Questions to Ask What are the implications? Abstract / discussion. The whole use of research is how far the results can be generalised. All authors will tend to think their work is more important than the rest of us! What is new here? What does it mean for health care? Is it relevant to my patients? August 2001 Bruce Davies

Questions to Ask What else is of interest? Introduction / discussion. Useful references? Important or novel ideas? Even if the results are discounted it doesn’t mean there is nothing of value. Bruce Davies

NOT JUST A FAULT FINDING EXERCISE ! Bruce Davies

What Is the Method? The first task – alternative check lists for different methods. How was the study conducted to confirm the method? Authors sometimes use the wrong words to describe their work! Bruce Davies

Surveys Describe how things are now. Samples of populations or special groups. The samples must be randomly selected. Bruce Davies

Surveys Do not have separate control or comparison groups. Comparisons may be made between subgroups – but this is not control. Use of the term survey should identify the method – beware the use in what is really a cohort study. Bruce Davies

Surveys Cross-sectional is seldom used to describe other methods. There are many ways of selecting a sample: e.g: stratified, cluster and systematic. Bruce Davies

Cohort Studies Used to find out what happens to patients. A group is identified and then watched to see what events befall them. May have comparison or control groups – who must be identified from the start. Not an essential feature tho.’ August 2001 Bruce Davies

Cohort Studies Must have the element of time flowing forward, from the point at which they are identified. This is sometimes called a retrospective cohort study. The term cohort should be diagnostic. Bruce Davies

Clinical Trials Testing. Always concerned with effectiveness. The focus should always be on the outcome. The outcomes may not be beneficial – in other words side effect trials. Sometimes cohort trials are used to assess effectiveness. This is very poor research and can usually be dismissed. Bruce Davies

Clinical Trials When more than two things are compared it makes the study more complex and harder to get right. The key words to look for are: random allocation, double blind, single blind, placebo-controlled. The term outcome is sometimes used in cohort studies as well. Bruce Davies

Case-control Studies Ask what makes groups of patients different. Select a set of patients with a characteristic – eg a disease. The characteristics of this set are then compared with a control group who do not have the characteristic being studied – but all all other respects must be as identical as possible. Bruce Davies

Case-control Studies Case studies look backward – not forward as cohort studies do. Other terms used are: case-referrent, case-comparator, case-comparison. May also be called retrospective – as is used for some cohort studies. August 2001 Bruce Davies

The Results The major mental challenge. What do I think this really means? CAUTION. Large unexpected results are rare. Flawed studies and misleading findings are common. Bruce Davies

Size matters. Statistics A subject in itself. Some general thoughts are worth emphasising. Size matters. Bruce Davies

Probability Is just that. It is not proof. Think of horse racing and the lottery. Think of the odds. Bruce Davies

Pitfalls All statistical tests make assumptions about the raw data. If there is no raw data presented you cannot know if the tests are meaningful. Outliers. Bruce Davies

Pitfalls Skew. Non-independence. Serendipity masquerading as hypothesis – or data trawling! August 2001 Bruce Davies

Pitfalls Black box analyses. Modern computers make statistical testing easy – the authors may not know what they are doing! Bias – play devils advocate. Confounding. A very common problem in medicine. Colour televisions do not cause increases in hypertension. Bruce Davies

Checklists Checklists for particular types of literature are a quick and easy way of learning critical appraisal. They all have 3 stages: Basic questions. Essential appraisal. Detailed appraisal. Bruce Davies

BUT, BUT, BUT Checklists do not tell you about the quality or usefulness. This is still a subjective question. All the lists do is enable a more structured and thoughtful response to the subjective question. Bruce Davies

References The Pocket guide to critical appraisal. By: Iain Crombie. Pub BMJ Books, 1996. How to lie with statistics. By: Darrell Huff. Pub: Pelican, 1989. 2001, updated 2018 Bruce Davies