University of Durham D Dr Robert Coe University of Durham School of Education Tel: (+44 / 0) Fax: (+44 / 0) Making Comparisons Doctor of Education (EdD) Analysing, Interpreting and Using Educational Research (Research Methodology)

© 2007 Robert Coe, University of Durham 2 (standardised) Average score of person taught ‘normally’ Average score of person taught by experimental method Effect size

© 2007 Robert Coe, University of Durham 3

4 Effect Size is the difference between the two groups, relative to the standard deviation Effect Size = Mean of experimental group – Mean of control group Standard deviation

© 2007 Robert Coe, University of Durham 5 Examples of Effect Sizes: ES = 0.2 “Equivalent to the difference in heights between 15 and 16 year old girls” 58% of control group below mean of experimental group Probability you could guess which group a person was in = 0.54 Change in the proportion above a given threshold: from 50% to 58% or from 75% to 81%

© 2007 Robert Coe, University of Durham 6 “Equivalent to the difference in heights between 14 and 18 year old girls” 69% of control group below mean of experimental group Probability you could guess which group a person was in = 0.60 ES = 0.5 Change in the proportion above a given threshold: from 50% to 69% or from 75% to 88%

© 2007 Robert Coe, University of Durham 7 “Equivalent to the difference in heights between 13 and 18 year old girls” 79% of control group below mean of experimental group Probability you could guess which group a person was in = 0.66 ES = 0.8 Change in the proportion above a given threshold: from 50% to 79% or from 75% to 93%

© 2007 Robert Coe, University of Durham 8 Effect size The difference between the two means, expressed as a proportion of the standard deviation ES =(M e – M c ) / SD Issues Which standard deviation? Statistical significance? Margin of error? Normal distribution? Restricted range Reliability See

© 2007 Robert Coe, University of Durham 9 Exercise 3. On each plot, estimate the effect size

© 2007 Robert Coe, University of Durham 10 Bloom’s ‘two-sigma problem’ The search for methods as effective as one-to-one tutoring

© 2007 Robert Coe, University of Durham 11 Effect sizes for other interventions Effect ofonis Reducing class size from 24 to 15 Student achievement 0.15 Setting vs. mixed ability classes Student achievement for high achievers for low achievers Computer based instruction Student achievement in well-controlled studies School based drug education Substance use 0.12 Practice test takingTest scores0.32

© 2007 Robert Coe, University of Durham 12 Further examples of effect sizes

© 2007 Robert Coe, University of Durham 13 The importance of effect size What is the evidence? How big is the effect size? The next time somebody tries to tell you what to do… Ofsted? DfES? LEA? Standards Unit? Headteacher? Consultant? Advice, Policy

© 2007 Robert Coe, University of Durham 14 Effect Size vs Statistical Significance Emphasises amounts, not just directions Avoids inappropriate dichotomies Avoids confusion over ‘significance’ Draws attention to power Avoids ‘file drawer’ problem Promotes synthesis rather than disagreement Allows accumulation of knowledge

© 2007 Robert Coe, University of Durham 15 Comparing percentages Disorganised learners become exam failures … Medical students at the university were asked to provide a passport photo at the start of a module in paediatrics. A total of 366 (93 per cent) of students handed in the photo. Of the 29 who failed to do so, 13 went on to fail their end-of-year exams. (From THES, & BMJ)

© 2007 Robert Coe, University of Durham 16 Contingency table FailedPassedTotal Handed in photo Didn’t Total (55%) 0336 (100%) (55%) What percentage of the ‘photo’ group have to pass before it is clear there is a difference?

© 2007 Robert Coe, University of Durham 17  2 (Chi-squared) test If both groups were from the same population, you would expect approximately the same proportion passing in each But not exactly the same proportion Small differences are quite likely, large differences more unlikely  2 test tells you how unlikely p = probability of getting such a big difference purely by chance Depends on difference in percentages and the sample size

© 2007 Robert Coe, University of Durham 18 72% 28% Chi-squared test gives critical (p<0.05) numbers as…

© 2007 Robert Coe, University of Durham 19 Comparing percentage changes Has the gap got bigger or smaller?

© 2007 Robert Coe, University of Durham 20 Has the gap got bigger or smaller? Asian up 8pts, African Caribbean up 7pts, so Asian has increased more Asian increased by 8 from 30 (27%), African Caribbean by 7 from 19 (37%), so African Caribbean has increased more

© 2007 Robert Coe, University of Durham 21 Logit transformation

© 2007 Robert Coe, University of Durham 22 Odds ratios Asian Odds before: 30/70 = 0.43 Odds after: 38/62 = 0.63 Odds ratio: 0.63/0.43 = 1.4 African Caribbean Odds before: 19/81 = 0.23 Odds after: 26/74 = 0.35 Odds ratio: 0.35/0.23 =

© 2007 Robert Coe, University of Durham 23 Intergenerational mobility