Why do we Need Randomised Controlled Trials? David Torgerson Director, York Trials Unit.

Slides:



Advertisements
Similar presentations
1 Chapter 4 The Designing Research Consumer. 2 High Quality Research: Evaluating Research Design High quality evaluation research uses the scientific.
Advertisements

Chapter 7 Flashcards. overall plan that describes all of the elements of a research or evaluation study, and ideally the plan allows the researcher or.
Lesson Overview 1.1 What Is Science?.
Validity (cont.)/Control RMS – October 7. Validity Experimental validity – the soundness of the experimental design – Not the same as measurement validity.
AP Statistics Section 5.2 B More on Experiments
Use of Placebos in Controlled Trials. Background The traditional ‘double-blind’ RCT uses a placebo to conceal allocation. There are a number of advantages.
Sample size issues & Trial Quality David Torgerson.
Robert Coe Neil Appleby Academic mentoring in schools: a small RCT to evaluate a large policy Randomised Controlled trials in the Social Sciences: Challenges.
Experimental evaluation in education Professor Carole Torgerson School of Education, Durham University, United Kingdom International.
Design and conduct of randomised controlled trials David Torgerson Director, York Trials Unit
Adapting Designs Professor David Torgerson University of York Professor Carole Torgerson Durham University.
The counterfactual logic for public policy evaluation Alberto Martini hard at first, natural later 1.
What makes a good quality trial? Professor David Torgerson York Trials Unit.
Reading the Dental Literature
Youth mentoring and the well-being of young people: Evidence from an Irish mixed- methods evaluation Dr Bernadine Brady Child & Family Research Centre.
Copyright © 2010, 2007, 2004 Pearson Education, Inc. Chapter 13 Experiments and Observational Studies.
Recruitment to Trials. Background Recruitment of participants is a VERY important issue. The general consensus is that most trials under recuit.
Introduction to the Design and Analysis of Trials can be found on: Before and After Studies: A Reminder.
Non-Experimental designs: Developmental designs & Small-N designs
Clinical trials methodology group Simon Gates 9 February 2006.
1 Psych 5500/6500 The t Test for a Single Group Mean (Part 5): Outliers Fall, 2008.
Pragmatic Randomised Trials. Background Many clinical trials take place in artificial conditions that do not represent NORMAL clinical practice. Often.
Session 8: Strategies to reduce violence
Chapter 13: Experiments and Observational Studies
Pre-randomisation consent (Zelen’s method)
Practicing Evidence Based Medicine
How to use Clinical Evidence to inform clinical decision making A case presentation using the CE review on acne.
Experiments and Observational Studies.  A study at a high school in California compared academic performance of music students with that of non-music.
BC Jung A Brief Introduction to Epidemiology - XI (Epidemiologic Research Designs: Experimental/Interventional Studies) Betty C. Jung, RN, MPH, CHES.
Are the results valid? Was the validity of the included studies appraised?
Before and After Studies in Injury Research Thomas Songer, PhD University of Pittsburgh
Module 1 Introduction to SRL. Aims of the Masterclass Understand the principles of self regulated learning (SRL) and how they apply to GP training Develop.
Outline of research activities – Poland Maciej Piotrowski Barcelona, January 2007.
Chapter 15 Current Concerns and Future Challenges.
Copyright © 2010 Pearson Education, Inc. Chapter 13 Experiments and Observational Studies.
Experiments and Observational Studies. Observational Studies In an observational study, researchers don’t assign choices; they simply observe them. look.
Copyright © 2007 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 13 Experiments and Observational Studies.
Epidemiology The Basics Only… Adapted with permission from a class presentation developed by Dr. Charles Lynch – University of Iowa, Iowa City.
Study Designs Afshin Ostovar Bushehr University of Medical Sciences Bushehr, /4/20151.
ECON ECON Health Economic Policy Lab Kem P. Krueger, Pharm.D., Ph.D. Anne Alexander, M.S., Ph.D. University of Wyoming.
EVIDENCE BASED MEDICINE Effectiveness of therapy Ross Lawrenson.
ARROW Trial Design Professor Greg Brooks, Sheffield University, Ed Studies Dr Jeremy Miles York University, Trials Unit Carole Torgerson, York University,
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
University of Durham D Dr Robert Coe University of Durham School of Education Tel: (+44 / 0) Fax: (+44 / 0)
H860 Reading Difficulties Week 7 Reading Interventions: How Do They Weigh Up?
Systematic Reviews By Jonathan Tsun & Ilona Blee.
Chapter 1 Statistical Thinking What is statistics? Why do we study statistics.
How to find a paper Looking for a known paper: –Field search: title, author, journal, institution, textwords, year (each has field tags) Find a paper to.
Quality First Teaching for All SENJIT 21 st May 2013.
Research Design ED 592A Fall Research Concepts 1. Quantitative vs. Qualitative & Mixed Methods 2. Sampling 3. Instrumentation 4. Validity and Reliability.
Chapter 10 Finding Relationships Among Variables: Non-Experimental Research.
1 Module 3 Designs. 2 Family Health Project: Exercise Review Discuss the Family Health Case and these questions. Consider how gender issues influence.
Introduction Studies are important for gathering information. In this lesson, you will learn how to effectively design a study so that it yields reliable.
Sifting through the evidence Sarah Fradsham. Types of Evidence Primary Literature Observational studies Case Report Case Series Case Control Study Cohort.
Southampton Education School Southampton Education School Can educational research really inform teaching in schools? Marcus 27 March.
1 Health and Disease in Populations 2002 Session 8 – 21/03/02 Randomised controlled trials 1 Dr Jenny Kurinczuk.
Compliance Original Study Design Randomised Surgical care Medical care.
EVALUATING u After retrieving the literature, you have to evaluate or critically appraise the evidence for its validity and applicability to your patient.
Types of Studies. Aim of epidemiological studies To determine distribution of disease To examine determinants of a disease To judge whether a given exposure.
CJ490: Research Methods in Criminal Justice UNIT #4 SEMINAR Professor Jeffrey Hauck.
LO: To be able to describe and evaluate the Cognitive Treatment for Schizophrenia.
1 Study Design Imre Janszky Faculty of Medicine, ISM NTNU.
P1) X has properties a, b, c, and z. P2) Y also has properties a, b, and c. C) By analogy, Y has property z. X: primary analogue Y: secondary analogue.
Experiments Textbook 4.2. Observational Study vs. Experiment Observational Studies observes individuals and measures variables of interest, but does not.
Day Care.
Statistics in Clinical Trials: Key Concepts
Robert West University College London London March 2008
Clinical Studies Continuum
11/20/2018 Study Types.
Starter Imagine - you did not do as well as you wanted to in a biology test, but your teacher praises you for working hard and trying your best. You feel.
Presentation transcript:

Why do we Need Randomised Controlled Trials? David Torgerson Director, York Trials Unit

What works? In most areas, education, health, criminal justice, etc, we want to know WHAT or WHETHER something works. »Do ‘bootcamps’ reduce criminal behaviour? »Are teaching volunteers effective? »Are computers effective at improving literacy and numeracy? Of secondary importance is HOW.

The WHAT question The ONLY way we can find out whether something works or not is by using a RANDOMISED CONTROLLED TRIAL. All other evaluative methods are INFERIOR ways of answering the WHAT question and some cannot answer it at all (e.g., qualitative research).

Structure of Session Randomised Controlled Trials ARE the ‘gold-standard’ evaluation method. »What is wrong with other research methods? »Why should we do trials

Before and After Methods

Clinical Practice in the 18 th Century "It is incident to physicians, I am afraid, beyond all other men, to mistake subsequence for consequence." Samuel Johnson, 1734

Background Traditionally most interventions have been evaluated using a pre-test post-test or before and after design. Participants are tested treated and then tested again any improvements are attributable to the intervention. Currently this is probably the most POPULAR evaluative method in most fields.

Who uses before and after? Policy makers Teachers assessing individual children. Action researchers. Parents Lecturers We all do.

Problems Problems include: »Temporal changes; »Regression to the mean.

Temporal Change Self-learning irrespective of teaching occurs. As children mature they will become better at learning. Any intervention or treatment is mixed up with these temporal changes difficult to disentangle.

Changes in Outcomes If we measured outcome on public examination results we will see an improvement. Is this because the intervention has worked? Or is it because exams have got easier? Or have children become more intelligent? Without a control group we CANNOT know.

Regression to the Mean As well as temporal changes before and after studies are confounded by a statistical phenomenon known as ‘Regression to or towards the mean’

Regression to the mean This is a GROUP phenomenon and occurs when the group are measured with an inexact measurement tool and then remeasured. Those individuals with ‘extreme’ values will have a high probability of regressing towards the mean on the second measurement.

History of RTM Galton’s work from 1869 started to provide the understanding of the phenomenon. By 1886 Galton had described the phenomenon among the heights of children and their parents (children of tall parents tend to be shorter and vice versa – regression to mediocrity).

Economists and RTM “I suspect that the regression fallacy is the most common fallacy in the statistical analysis of economic data” Milton Friedman 1992

Marking Exam Scripts For MSc in Health Sciences system of double marking markers are blind to student identity and the other marker’s mark. There is a tendency to disagree with marks at the extreme of the distribution. Explanation: Regression to mean.

RTM and exam scripts

Annual Increase in offences with firearms Amnesty

Did the Amnesty work? Unclear, the year preceeding the amnesty had a large, unexpected, increase in offences, we would expect through regression to the mean that in the following year the rate of increase would ‘regress’ back to towards the ‘average’ annual increase.

Education intervention Wheldall selected 40 pupils whose reading was at least 2 years behind their peers. Half were exposed to an intervention. Wheldall Educational Review 2000;52:29.

Before and after reading programme Difference highly statistically significant p < 0.001

Before and after reading programme Differences between groups NOT statistically significant

RTM misunderstanding “the mean gain scores translated to impressive effect sizes of 0.6.” “It could be argued that it is asking too much of any program to demonstrate enhanced efficacy on top of such high existing efficacy” “…control group gains were largely attributable to pre-existing …literacy programme..” Perhaps, BUT much of the gain will be due to RTM.

RTM and School Exclusions A qualitative and before and after evaluation of an intervention to reduce school exclusions said »“an RCT would not have been able to adequately address fundamental problems concerning the reliability and validity of quantitative data in relation to exclusions”

Flawed Methods Selected schools with HIGH exclusion rates on which to intervene. Therefore we would EXPECT exclusions to fall. They did by 15%. BUT schools with the fewest exclusions INCREASED exclusions by 55% whilst schools with the highest exclusions had a fall of 32%.

Mentoring In England, part of the KS3 Strategy Backed by Government and private funding ‘Mentoring’ means a lot of different things Research evidence is »Case studies »Feelings and perceptions of participants »Completely inadequate to infer impact

Neil Appleby’s Experiment A randomised controlled trial involving 20 underachieving Y8 (12-13 year-old) students Matched in pairs on ability and gender Randomly allocated: in each pair, one mentored, the other not Mentored group had 20 mins individually every two weeks (11 sessions) »‘It nearly killed me’ »Cost estimated at between £170 and £410 per mentored pupil, represents between 8-19% of the school’s annual per pupil funding for the whole of their education

What the teachers said about the mentored students … “**** is a changed person this year she has progressed greatly and is a superb helpful student.” “Better now, has achieved more, more confident.” “Generally a great improvement recently.” “****’s attitude and effort have improved over the year. He is a lot pleasanter and more willing to participate in lessons particularly oral work, he responds well to praise.”

What they said about the control group … “Has improved overall this term.” “****’s attitude and effort have improved over the last few months, she is now trying very hard to achieve her target. Great effort.” “Commended for attitude and progress.” “**** has settled since the beginning of the year.” “**** has undergone quite a transformation since September. Her attitude towards the teacher and her learning have improved drastically and she should be congratulated.”

Change in Teachers’ Ratings of progress, effort and attitude (English, maths and science combined)

What this proves If you identify a group of underachieving pupils at a particular time and then come back to them after a few months, many of them will have improved, whatever you did. Others (the ‘hard cases’) will not have improved, whether mentored or left alone. The interpretation of this would have been very different without a ‘control’ group

RTM and League Tables RTM GREAT for Governments to help the credulous into believing what they do works. In any league table those at the bottom will tend to ‘regress’ upwards to the mean whilst those at the top regress down. This lends support to naming and shaming or extra financial help to those at the bottom.

Dealing with RTM The only way to reliably deal with the problem is through randomised trials. Which is why before and after data are generally regarded, by the congnescenti, as almost USELESS.

History of Controlled Trials Because of temporal and regression to mean effects we MUST have a control group.

Background Many researchers over the centuries have seen the need for a ‘control’ group to avoid the inherent biases in the before and after study. Controlled trials have been conducted for several hundred years probably occasionally using randomisation.

Scurvy Scurvy was a very prevalent condition among sailors before the 19 th Century. A controlled trial in the middle of the 18 th Century of 12 sailors showed that the two sailors allocated to receive lime or orange juice recovered and were able to care for their ship mates allocated to vinegar or salt water.

Lack of Dissemination An even earlier trial in scurvy prevention used a ‘cluster’ design whereby a whole ship’s crew were allocated citrus fruit and were compared with two ships’ crews who were not. The treatment worked but lesson forgotten. After second trial took Navy 50 years to implement results

Agriculture Fisher is usually thought of as the originator of randomisation in the 1920s in agricultural experiments. He was concerned with the statistical properties of ‘randomness’ as well as the formation of unbiased groups.

Cambridge-Somerfield In 1937 a classic experiment – the Cambridge-Somerfield trial was launched. The aim was to show that social worker intervention among ‘delinquent’ boys would reduce ‘criminality’.

Design 650 boys were identified by their teachers as having delinquent behaviour that put them at later risk of criminal activity. 325 pairs were formed and one from each pair was allocated a social worker supported by psychiatrists.

Results – early follow-up % of boys indulging in crime. Green bar indicates intervention grop

Results later follow-up In 1975 ‘boys’ were followed up again when middle aged men. 58% of intervention group had NOT had a criminal conviction BUT 68% of control group had NOT had a conviction. If a control group had not been used success of the intervention would be assured.

Consequences of the Trial The social work profession largely ABANDONED the RCT as a method of evaluation as it failed to give the RIGHT results.

RCTs and education Lindquist writing about experimental methods in 1940 argued that advanced text books use “all illustrations given are in the field of agricultural experimentation and are concerned with “plots” “blocks” “yields” “treatments” etc, rather than with “schools” “classes” “scores” “methods” “pupils” etc.” Lindquist Statistical Analysis in Educational Research, 1940.

The Importance of Design in Educational Experiments (Lindquist) In 1940 in his book on statistics in educational research Lindquist quite clearly describes appropriate RCTs for educational research. His book is also the first description of the appropriate techniques to be used in analysing pupils scores in classes (I.e, cluster analysis), which was an advance on Fisher’s Design of experiments.

Cluster analysis In health statistics Lindquists statistical methods were largely ignored until the late 1980s when it became accepted to use the methods he advocated to analyse clustered data although even now most cluster trials are badly analysed. But 64 years on what about his descriptions on how to rigorously evaluate educational interventions?

Educational Trials: UK Not many trials in education have been undertaken in the UK. Most educational trials are from the USA. WHY? (my personal view) »Futility of the ‘paradigm war’; »Failure to understand their importance; »Trials often give the ‘wrong’ answer; »Lack of funding.

Opposition to Trials is widespread In health care many doctors will refuse to believe the results of a trial and argue the trial was faulty or poorly conducted if the result was ‘wrong’. Recent example: WHI study of hormone replacement – many doctors REFUSE to accept the findings of this study that it INCREASES risk of heart disease.

Opposition to Polio Trial “I found but one person who rigidly adherred to the idea of a placebo control and he is a bio-statistician who, if he did not adhere to this view, would have had to admit his own purposelessness in life” (Jonas Salk).

1950s to 1970s The use of trials expanded rapidly within and beyond medicine. In the social sciences experiments included: »Negative income tax; »Adoption; »Busing; »Public vs private schools; »Prevention of spousal abuse.

Health Care Trials Although ALL new medicines have to be evaluated using RCTs many medical treatments do not. HOWEVER, health care is ‘fortunate’ because we bury our disasters we KNOW how important trials are as a protection for patients.

Health Care Disasters Opposition to RCTs has declined over the years, partly due to a number of catastrophes, from unevaluated treatments. Harmful treatments are still in widespread use today – we just don’t know which ones.

Disasters among babies Routine practice in 40s and 50s to give premature infants pure oxygen. At the same time it was noted that there was an ‘epidemic’ of blindness among babies. Linked to oxygen use. Routine practice in 50s to give prophylactic antibiotics to premature infants, caused brain damage and death. BOTH of these problems only discovered AFTER an RCT was undertaken.

Trial sabotage Interestingly an early trial of pure oxygen for neonates was sabotaged by nurses who secretly gave oxygen to some of the controls because they KNEW that it was effective. Because of this ARROGANCE they contributed to the blinding of healthy babies.

Educational Disaster? On the basis of ‘before and after’ and anecdote widespread implementation of driver education (in the USA) among older pupils was implemented. It was thought that this would reduce car accidents. Did it? Fortunately, some ‘sceptics’ undertook a series of trials in the USA.

Driver Education - Results Roberts and colleagues (see Campbell Collaboration) reviewed these trials and undertook a meta-analysis. They found that driver education INCREASED the likelihood of deaths in car accidents as it increased the prevalence of young motorists.

UK Policy makers Have IGNORED these results and implemented driver education in some schools. This will directly increase deaths among young drivers.

Computers in Schools Introduction of computers into schools has not been preceded by large RCTs. The best evidence we have is from a ‘quasi- experiment’ from Israel, which showed that introduction of computers into half the state schools led to no change in Hebrew literacy but a DECLINE in maths. The Israeli Government has since introduced computers into all schools!!!

Volunteers in Schools The use of volunteers to help children learn to read is widespread – but are they effective? In a systematic review of RCTs only 7 trials could be identified with largest with ONLY 99 children. The effect of volunteering was very slight (0.19, to 0.68) and not statistically significant. Torgerson et al Ed Studies, 28 No 4.

Conclusions Virtually all new interventions need to be evaluated using RCTs. Unlike health care children are compelled to have education. Therefore it is even more urgent that they should not be exposed to ineffective educational interventions.

We need more trials