2. University of Durham D Dr Robert Coe University of Durham School of Education Tel: (+44 / 0) 191 334 4184 Fax: (+44 / 0) 191 334 4180 E-mail: r.j.coe@dur.ac.uk http://www.dur.ac.uk/r.j.coe Causality and Experimental Design Doctor of Education (EdD) Analysing, Interpreting and Using Educational Research (Research Methodology)

3. © 2005 Robert Coe, University of Durham 2 Causal claims Examples “A causes B” “B is affected by A” “A influences B” “A improves B” “B benefits from A” “A results in B” “A prevents B” “The effect of A on B …” If you do A, B will result (provided X, Y, Z) A must be something you can choose to do (‘manipulable’) B would not result if you didn’t do A (How can you ever know this?)

4. © 2005 Robert Coe, University of Durham 3 Can we do without ‘causality’? Very hard to avoid (even in ‘interpretative’ writing) No causality  no predictability You can have it both ways There are general laws, but we can choose Causal laws are probabilistic – better applied to groups than individuals Just because we actively interact with our environment and interpret situations does not mean our behaviour is not (partially) predictable Human behaviour is unpredictable, but also predictable – the trick is to predict the bits you can

5. © 2005 Robert Coe, University of Durham 4 ImpaCT2: ICT and achievement What the report said: “evidence of a positive relationship between ICT use and achievement” (p2) “pupils characterised as high ICT users outperformed, on average, low ICT users in English and mathematics [at KS2]” (p11) What the press release said: “A new independent research report shows that computers can help to raise standards in schools”

6. © 2005 Robert Coe, University of Durham 5 Questions What is the difference between (1) “a positive relationship” and (2) “computers can help to raise standards”? How might the first be true and not the second? (List as many possible reasons as you can think of) How could you test (2) even if you knew that (1) was true? (To do this you will need to define exactly what you think (2) means)

7. © 2005 Robert Coe, University of Durham 6 What is ‘evidence-based’? Intervention, not description Evaluation, not common sense Knowledge about ‘what works’ must come from …

8. © 2005 Robert Coe, University of Durham 7 Effective Teachers (according to Hay McBer) … Set high expectations Are good at planning Employ a variety of teaching strategies Have a clear strategy for pupil management Manage time and resources wisely Employ a range of assessment methods Set appropriate homework Keep pupils on task (For which the DfEE paid £3m)

9. © 2005 Robert Coe, University of Durham 8 How to get rich (according to John D. Rockefeller) Go to bed early Rise early Strike oil (This advice was given for free)

10. © 2005 Robert Coe, University of Durham 9 The need for high expectations Everybody knows (Ofsted, school effectiveness research) teachers should have high expectations ‘Pygmalion’ effect (Rosenthal and Jacobsen, 1968) Hugely influential, but flawed Is it an effective intervention for teaching? No. For experienced teachers who know their students, ‘expectation effect’ is zero (Raudenbush, 1984). ‘High expectations’ may not be easily alterable, or may be the effect rather than the cause

11. © 2005 Robert Coe, University of Durham 10 Pre-experimental design 1 group:Outcome 1 1. You did A 2. B followed  3. A caused B ‘Treatment’

12. © 2005 Robert Coe, University of Durham 11 Example 1: Mentoring In England, part of the KS3 Strategy Backed by Government and private funding ‘Mentoring’ means a lot of different things Research evidence is Case studies Feelings and perceptions of participants Completely inadequate to infer impact

13. © 2005 Robert Coe, University of Durham 12 Neil Appleby’s Experiment A randomised controlled trial involving 20 underachieving Y8 (12-13 year-old) students Paired and split into two groups Mentored group had 20 mins individually every two weeks ‘It nearly killed me’ Cost estimated at £250 per mentored pupil Approx 20% of the school’s annual per pupil funding

14. © 2005 Robert Coe, University of Durham 13 What the teachers said about the mentored students … “**** is a changed person this year she has progressed greatly and is a superb helpful student.” “Better now, has achieved more, more confident.” “Generally a great improvement recently.” “****’s attitude and effort have improved over the year. He is a lot pleasanter and more willing to participate in lessons particularly oral work, he responds well to praise.”

15. © 2005 Robert Coe, University of Durham 14 What they said about the control group … “Has improved overall this term.” “****’s attitude and effort have improved over the last few months, she is now trying very hard to achieve her target. Great effort.” “Commended for attitude and progress.” “**** has settled since the beginning of the year.” “**** has undergone quite a transformation since September. Her attitude towards the teacher and her learning have improved drastically and she should be congratulated.”

16. © 2005 Robert Coe, University of Durham 15 What this proves If you identify a group of underachieving pupils at a particular time and then come back to them after a few months, many of them will have improved, whatever you did. Others (the ‘hard cases’) will not have improved, whether mentored or left alone. The interpretation of this would have been very different without a ‘control’ group

17. © 2005 Robert Coe, University of Durham 16 Quasi-experimental design Group 1: ‘Treatment’ Group 2: No ‘treatment’ Outcome 1 Outcome 2 Post-testIntervention 1. The two groups were the same at the beginning 2. You treated them differently 3. They were different at the end  4. The treatment caused the difference Experimental Control

18. © 2005 Robert Coe, University of Durham 17 Example 2: Class size Do pupils in small or large classes get better results? Overall, large classes do better, because Top sets are large, bottom sets are small Pupils who misbehave are put in small classes Socially advantaged schools are more popular and hence need to have larger classes Socially disadvantaged schools are given more money, so provide smaller classes Schools that are popular with pupils also attract the best teachers Pupils in large classes are not the same as those in small classes (and would have been different, regardless of class size)

19. © 2005 Robert Coe, University of Durham 18 Quasi-experimental design (with pre-test) Pre-test does not guarantee equivalence, it just tells you whether they are equivalent If they are not equivalent, interpretation is quite difficult Even if pre-test scores are well matched, the groups may not actually be the same ‘Treatment’ No ‘treatment’ Outcome 1 Outcome 2 Group 1: Pre-test Group 2: Pre-test

20. © 2005 Robert Coe, University of Durham 19 Example 3: Graduate earnings Graduates earn an extra £400 000 over a lifetime as a result of going to university 18 year olds Go to university Don’t go to university £££ £ Qualifications at 18 Pre-test measure

21. © 2005 Robert Coe, University of Durham 20 Randomised (true) experimental design Randomised Controlled Trial (RCT) ‘Treatment’ No ‘treatment’ Outcome 1 Outcome 2 1 Group Random allocation

22. © 2005 Robert Coe, University of Durham 21 Example 4: Support for ‘at-risk’ youngsters Cambridge-Somerville Youth Study (US, WWII) Aim: to reduce delinquency 650 ‘difficult’ & ‘average’ boys aged 5-11, half randomly allocated to receive support (home visits from social workers, psycholog(&iatr)ists, doctors, tutors, up to twice monthly for five years, summer camps, etc) At the end of the project, most said it was helpful, but no clear differences between the two groups

23. © 2005 Robert Coe, University of Durham 22 30 years later … Two thirds said it was helpful; e.g. kept them off the streets/out of jail, made them more understanding, showed someone cared. 57 ‘objective’ comparisons of criminal behaviour, health, family, work and leisure time, beliefs and attitudes. Those who had received the support were no better on any of these outcomes 7 comparisons showed significant advantage to the control group

24. © 2005 Robert Coe, University of Durham 23 Percentage who had experienced ‘undesirable outcomes’ (eg death, criminal conviction, psychiatric disorder): Those ‘supported’ were more dissatisfied with life, work and marriage (McCord, 1981)

25. © 2005 Robert Coe, University of Durham 24 The logic of experimental design 1. The two groups were the same at the beginning 2. You treated them differently 3. They were different at the end Hence, the treatment caused the difference 4. A similar result would be found elsewhere Internal validity External validity

26. © 2005 Robert Coe, University of Durham 25 Threats to causal inference 1: Lack of valid comparison No sense of what would have happened otherwise No control/comparison group Non-randomised control group Non-equivalence initially Volunteer / selection effects Statistical regression Overcompensation by covariance analysis Random allocation that has not worked

27. © 2005 Robert Coe, University of Durham 26 Threats to causal inference 2: Unintended treatments The effect (or lack of it) was due to something other than the intended treatment Reactivity (to pre-test or other measures) Experiment effects (Hawthorne effects, competition/demoralisation between groups, contamination of treatments, expectation/apprehension effects) Implementation fidelity (treatment not delivered as intended) Control group intervention not specified Extraneous factors (history, maturation)

28. © 2005 Robert Coe, University of Durham 27 Threats to causal inference 3: Incorrect interpretation An apparent effect (or lack of it) is not what it seems Inappropriate timing of outcome measures Invalid outcome measures (inappropriate, biased, too specific, too broad, wrong interpretation) Measurement issues (reliability, ceiling/floor effects, interval scales) Attrition (missing data, drop-out, loss to follow-up) Statistical power (sampling variation) Post hoc bias (fishing/opportunism, selective analysis and reporting)

29. © 2005 Robert Coe, University of Durham 28 Threats to causal inference 4: Failure to generalise Similar results would/could not be found again in other contexts Representativeness (context & population must be well-defined; sample - intended and actual - must be representative of this population) Replicability (intervention must be feasible, well- defined, replicable) Scale (differences between small-scale experiment and large-scale policy) Transfer (results may not apply in other contexts)

30. © 2005 Robert Coe, University of Durham 29 Experiments are unethical? It is unethical not to evaluate practices and policies Teachers ‘experiment’ anyway Need not have ‘no treatment’ control group Can compensate control group in other ways Sometimes treatments are withheld anyway Only randomise at the borderline

31. © 2005 Robert Coe, University of Durham 30 Making evidence-based decisions The problem (and possible solutions) must be ‘generalisable’ (= ‘policy’) Agreement about outcomes? What evidence exists already? Theory (formal and informal) Experience Research Systematic reviews of research Conduct an experiment

32. © 2005 Robert Coe, University of Durham 31 Teaching is more complex