Observational studies-the broken scientific method? Group 1: Mei Ying (Presenter), Boon Xuan and Fatin Young, Stanley S. and Karr, Alan (2011). Deming, data and observational studies: A process out of control and needing fixing. Significance, 8 (3), 116–120.

Overview What is an observational study? Background The problem Multiple testing Bias Multiple modelling A new strategy Conclusion

What is an observational study? A study where the values of the independent and dependent variables are recorded based on observation of the sample. When is it used? When experimental studies are not possible. Used in studying a rare disease/occurrence. Disadvantages: Non-random assignment of sample. Prone to confounding variables. Results cannot be used to establish causal relationships.

Background “Any claim coming from an observational study is most likely to be wrong.” (Young & Karr, 2011) Coffee causes cancer Eating cereal helps women give birth to boys People with Type A personalities likely to get heart attacks Etc etc... http://abidinginthevine.net/wp-content/uploads/2015/07/can-you-prove-it.jpg

Background Many of these claims made by observational studies are not replicable: In 2005, 49 claims made were randomly sampled→ 6 of these were observational studies. 5 out of 6 of these observational studies failed to reproduce the same result→83% failure rate.

Background More proof: (Young & Karr, 2011) None of the claims made by the observational studies were reproducible. Some claims were even countered against. More proof: 100% failure rate! Shocking ... (Young & Karr, 2011)

http://www.forbes.com/sites/henrymiller/2014/01/08/the-trouble-with-scientific-research-today-a-lot-thats-published-is-junk/#d22367e20b81

The problem Lets consult the expert! ‘Father of Quality Management’ ‘14 Points for Management’ 3rd point: Cease dependence on inspection to achieve quality Explanation: To achieve a product of quality, we should not do a quality check only after the product is completed (inspection), but rather control the processes at each step of the way. W. Edwards Deming, Statistician https://www.census.gov/history/img/deming.jpg

The problem The comparison: Step 1 Step 2 Step 3 Step 4 End product Many erroneous products The comparison: Inspection here Product control Step 1 Step 2 Step 3 Step 4 End product Step 1 Step 2 Step 3 Step 4 Process control End product Managerial Control Managerial Control Managerial Control Managerial Control Few erroneous products

Does this remind you of something? The problem Journal editors and reviewers decide which papers are worthy to be published Workflow of an observational study: Data collection Data cleaning Statistical analysis Interpretation of results Report/paper writing Does this remind you of something? Many erroneous products Inspection here Product control Step 1 Step 2 Step 3 Step 4 End product

Multiple testing The 1st limitation of observational studies In real life, observational studies employ multiple testing. ‘Cereal helps women give birth to boys’ example: Women were asked questions about the consumption of 133 different foods. The genders of mother’s child recorded for each type of food. ‘Cereal’ gave a positive result of having more male children associated with it.

Multiple testing Source: https://xkcd.com/882/ as cited in Young & Karr, 2011.

Multiple testing What happens when you do multiple testing? You are doing many comparisons at the same time. Assuming each test was carried out with 95% confidence (5% error rate), P(getting a significant result by chance)=1 - (1-0.05) = 0.05 If you carry out 10 comparisons at once, P(at least 1 significant result by chance)=1 - (1-0.05)10 = 0.40 Chance of committing a Type I error (reject null hypothesis when it is true i.e. a false positive) is at 40% when carrying out 10 comparisons. Unless you have accounted for this error, chance of false positives are high.

Multiple testing What implications does this cause? → Your positive result in the observational study may be purely due to chance and not a real effect of the ‘treatment’. ‘Cereal’ gave a positive result of having more male children associated with it. Due to chance maybe?

Bias The 2nd limitation of observational studies. Many factors can lead to bias. Example of experimental bias: Doctors tend to assign HIV patients with higher risk of cardiovascular diseases (CVD) to receive the drug Abacavir. When a study was done to assess the CVD risk of taking each drug, the study was bias against Abacavir since there was a higher proportion of patients with CVD risk under Abacavir treatment.

Abacavir

Bias Publication bias is a very real issue. Journal editors and reviewers tend to favor research that show positive results rather than negative results. ‘No p-value<0.05, no chance of publication’ attitude and mindset. Researchers are therefore enticed to manipulate experimental data/procedure to achieve a p-value of less than 0.05 (significant result). Academic quality of research compromised.

Multiple modelling The 3rd limitation of observational studies. Many different types of models can be created from 1 dataset. However only 1 or a few models are accurate. Models sometimes give a ‘p-value<0.05’ even though the model is not accurate, but researchers are too blinded by that ‘significant result’ to notice this/deliberately stick to this model that gives a positive result (model selection bias). When people see ‘p-value<0.05’ they tend to focus only on the result and forgot about the methodology behind the result Source: Tom Boulton

A new strategy As you can see, the current scientific method is flawed. Authors S. Stanley Young and Alan Karr have therefore proposed a new way to carry out observational studies:

A new strategy Allows more supervision over research results/claims.

A new strategy One team assigned to do data cleanup, another team assigned to do data analysis. Reason is that the analyst may be tempted to remove data points that will give a more favorable model and hence nicer results (model selection bias). By separating the people who carry out data cleaning and analysis, we can prevent model selection bias.

A new strategy Total sample Sample A: Use to create the model Sample B: Holdout data set

A new strategy Analyst writes down, files statistical protocol and follows it. Prevents analyst from being ‘too flexible’ during analysis.

A new strategy The journal does not know the results from the holdout dataset. Journal is forced to make a wise decision based on the data available and not be influenced by the holdout data results.

A new strategy Sample B: holdout data Sample A Create model Fit data into model Fits well: model is appropriate, claim is sound. Do not fit: model is not appropriate, claim not sound.

Conclusion Many claims made by observational studies are a sham, which gives people false information. The problem is that the current observational study workflow is flawed. A new strategy of carrying out observational studies is proposed which seeks to create quality research findings. Cooperation from the managers of the system (the journals and funding agencies) are needed to make this strategy a success.

References Young, Stanley S. and Karr, Alan (2011). Deming, data and observational studies: A process out of control and needing fixing. Significance, 8 (3), 116–120. http://www.stat.berkeley.edu/~mgoldman/Section0402.pdf http://www.forbes.com/sites/henrymiller/2014/01/08/the-trouble-with-scientific- research-today-a-lot-thats-published-is-junk/#d22367e20b81 https://www.smartersolutions.com/business-system-iee/demings-14-points- explained-implementation/