1. Observational studies-the broken scientific method?
Group 1: Mei Ying (Presenter), Boon Xuan and Fatin
Young, Stanley S. and Karr, Alan (2011). Deming, data and
observational studies: A process out of control and needing fixing. Significance, 8 (3), 116–120.

2. Overview What is an observational study? Background The problem
Multiple testing
Bias
Multiple modelling
A new strategy
Conclusion

3. What is an observational study?
A study where the values of the independent and dependent variables are recorded based on observation of the sample.
When is it used?
When experimental studies are not possible.
Used in studying a rare disease/occurrence.
Disadvantages:
Non-random assignment of sample.
Prone to confounding variables.
Results cannot be used to establish causal relationships.

4. Background
“Any claim coming from an observational study is most likely to be wrong.”
(Young & Karr, 2011)
Coffee causes cancer
Eating cereal helps women give birth to boys
People with Type A personalities likely to get heart attacks
Etc etc...

5. Background
Many of these claims made by observational studies are not replicable:
In 2005, 49 claims made were randomly sampled→ 6 of these were observational studies.
5 out of 6 of these observational studies failed to reproduce the same result→83% failure rate.

6. Background More proof: (Young & Karr, 2011)
None of the claims made by the observational studies were reproducible. Some claims were even countered against.
More proof:
100% failure rate! Shocking ...
(Young & Karr, 2011)

8. The problem Lets consult the expert! ‘Father of Quality Management’
‘14 Points for Management’
3rd point: Cease dependence on inspection to achieve quality
Explanation: To achieve a product of quality, we should not do a quality check only after the product is completed (inspection), but rather control the processes at each step of the way.
W. Edwards Deming,
Statistician

9. The problem The comparison: Step 1 Step 2 Step 3 Step 4 End product
Many erroneous products
The comparison:
Inspection here
Product control
Step 1
Step 2
Step 3
Step 4
End product
Step 1
Step 2
Step 3
Step 4
Process control
End product
Managerial Control
Managerial Control
Managerial Control
Managerial Control
Few erroneous products

10. Does this remind you of something?
The problem
Journal editors and reviewers decide which papers are worthy to be published
Workflow of an observational study:
Data collection
Data cleaning
Statistical analysis
Interpretation of results
Report/paper writing
Does this remind you of something?
Many erroneous products
Inspection here
Product control
Step 1
Step 2
Step 3
Step 4
End product

11. Multiple testing The 1st limitation of observational studies
In real life, observational studies employ multiple testing.
‘Cereal helps women give birth to boys’ example:
Women were asked questions about the consumption of 133 different foods.
The genders of mother’s child recorded for each type of food.
‘Cereal’ gave a positive result of having more male children associated with it.

12. Multiple testing
Source: as cited in Young & Karr, 2011.

13. Multiple testing What happens when you do multiple testing?
You are doing many comparisons at the same time.
Assuming each test was carried out with 95% confidence (5% error rate), P(getting a significant result by chance)=1 - (1-0.05) = 0.05
If you carry out 10 comparisons at once, P(at least 1 significant result by chance)=1 - (1-0.05)10 = 0.40
Chance of committing a Type I error (reject null hypothesis when it is true i.e. a false positive) is at 40% when carrying out 10 comparisons.
Unless you have accounted for this error, chance of false positives are high.

14. Multiple testing What implications does this cause?
→ Your positive result in the observational study may be purely due to chance and not a real effect of the ‘treatment’.
‘Cereal’ gave a positive result of having more male children associated with it.
Due to chance maybe?

15. Bias The 2nd limitation of observational studies.
Many factors can lead to bias.
Example of experimental bias:
Doctors tend to assign HIV patients with higher risk of cardiovascular diseases (CVD) to receive the drug Abacavir.
When a study was done to assess the CVD risk of taking each drug, the study was bias against Abacavir since there was a higher proportion of patients with CVD risk under Abacavir treatment.

16. Abacavir

17. Bias Publication bias is a very real issue.
Journal editors and reviewers tend to favor research that show positive results rather than negative results.
‘No p-value<0.05, no chance of publication’ attitude and mindset.
Researchers are therefore enticed to manipulate experimental data/procedure to achieve a p-value of less than 0.05 (significant result).
Academic quality of research compromised.

18. Multiple modelling The 3rd limitation of observational studies.
Many different types of models can be created from 1 dataset.
However only 1 or a few models are accurate.
Models sometimes give a ‘p-value<0.05’ even though the model is not accurate, but researchers are too blinded by that ‘significant result’ to notice this/deliberately stick to this model that gives a positive result (model selection bias).
When people see ‘p-value<0.05’ they tend to focus only on the result and forgot about the methodology behind the result
Source: Tom Boulton

19. A new strategy
As you can see, the current scientific method is flawed. Authors S. Stanley Young and Alan Karr have therefore proposed a new way to carry out observational studies:

20. A new strategy
Allows more supervision over research results/claims.

21. A new strategy
One team assigned to do data cleanup, another team assigned to do data analysis.
Reason is that the analyst may be tempted to remove data points that will give a more favorable model and hence nicer results (model selection bias).
By separating the people who carry out data cleaning and analysis, we can prevent model selection bias.

22. A new strategy Total sample Sample A: Use to create the model
Sample B: Holdout data set

23. A new strategy
Analyst writes down, files statistical protocol and follows it.
Prevents analyst from being ‘too flexible’ during analysis.

24. A new strategy
The journal does not know the results from the holdout dataset.
Journal is forced to make a wise decision based on the data available and not be influenced by the holdout data results.

25. A new strategy Sample B: holdout data Sample A Create model
Fit data into model
Fits well: model is appropriate, claim is sound.
Do not fit: model is not appropriate, claim not sound.

26. Conclusion
Many claims made by observational studies are a sham, which gives people false information.
The problem is that the current observational study workflow is flawed.
A new strategy of carrying out observational studies is proposed which seeks to create quality research findings.
Cooperation from the managers of the system (the journals and funding agencies) are needed to make this strategy a success.

27. References
Young, Stanley S. and Karr, Alan (2011). Deming, data and observational studies: A process out of control and needing fixing. Significance, 8 (3), 116–120.
research-today-a-lot-thats-published-is-junk/#d22367e20b81
explained-implementation/