MUSC Biomedical Trainee Retreat on the Responsible Conduct of Research “Data Management, Record Keeping, and Experimental Rigor” Ed Krug BE101 876-2404 krugel@musc.edu 12/07/16

Which of these 3 issues has been the greatest frustration for you in the following – give an example: maintaining your own research records troubleshooting problems in your own experiments/studies reproducing the work of others (either from your lab or from the literature)

Studies show a very low reproducibility for articles published in scientific journals The biotech company Amgen had a team of about 100 scientists trying to reproduce the findings of 53 “landmark” articles in cancer research published by reputable labs in top journals. Only 6 of the 53 studies were reproduced. [Nature 483:531-33 (2012)] Scientists at the pharmaceutical company, Bayer, examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine.  Published results were reproduced in only 14 out of 67 projects.[Nature Reviews Drug Discovery 10:712 (2011)] However, lack of reproducibility does not necessarily mean results are false. http://blog.jove.com/2012/05/03/studies-show-only-10-of-published-science-articles-are-reproducible-what-is-happening

According to the NIH … (RFA-GM-15-006) “The lack of reproducibility is not due primarily to intentional fabrication or falsification of data. Rather, in many cases there is a lack of awareness or adherence to sufficiently high standards in the planning and execution of scientific experiments, and in transparency in the reporting of science.  Examples include inherently weak experimental designs and over-interpretation of statistically marginal differences and variability of materials, differing brands or even lots of reagents, differing or drifting strains of organisms and cells in culture, or other variables that have not been adequately controlled.”  

According to the NIH … (RFA-GM-15-006) “Graduate students were often significantly dependent on the mentor or the mentor's lab for the training received, and postdoctoral fellows were primarily dependent on the mentor or mentor's lab at all institutions.  Rather than being learned in prescribed curricula, training in good laboratory practices that influence data reproducibility appears to be largely passed down from generation to generation of working scientists, with substantial variation from laboratory to laboratory.”  

https://www.nih.gov/research-training/rigor-reproducibility/training

Suggested Best Practices to Enhance Rigor and Reproducibility Steward and Balice-Gordon. Neuron 84: 572-582 (2014)

Suggested Best Practices to Enhance Rigor and Reproducibility Steward and Balice-Gordon. Neuron 84: 572-582 (2014)

Essential Characteristics of a Research Notebook Bound pages Number every page - index in front Table of Contents Abbreviations list Relevant sections - title, purpose, materials & methods, results, conclusions, notes Purpose and M&M written before starting the experiment Includes reagent details, e.g. mfr, lot number, location, etc. Cross-referenced to computer files, images, etc. Cross-referenced with collaborator’s notebook Sufficient clarity that your mentor and another trainee could understand and replicate what you have done! Notebooks are university property! stays with PI – 3 years past acceptance of final financial statement (6 years from end of grant to be safe)

“Data” is essentially all that would be required to replicate your study 1) Physical records lab notebook instrument printouts images computer analysis e-mail lab meeting discussion probes and cells 2) Analysis of primary data data selection statistical analysis calculations normalization 3) Presentation of results graphs tables pictures Figures 4) Publication posters seminars manuscripts funding requests

Give a copy of this inventory to your mentor and any collaborators. I strongly suggest a listing of all notebook and page numbers corresponding to any data used for a poster, manuscript or funding application! Give a copy of this inventory to your mentor and any collaborators.

NIH Data Sharing Policy “The NIH expects and supports the timely release and sharing of final research data from NIH-supported studies for use by other researchers.” NOT-OD-03-032 “Definition of Research Data: Recorded factual material commonly accepted in the scientific community as necessary to validate research findings. It does not include preliminary analyses; drafts of scientific papers; plans for future research; peer reviews; communications with colleagues; physical objects (e.g., laboratory samples, audio or video tapes); trade secrets; commercial information; materials necessary to be held confidential by a researcher until publication in a peer-reviewed journal; information that is protected under the law (e.g., intellectual property); personnel and medical files and similar files, the disclosure of which would constitute an unwarranted invasion of personal privacy; or information that could be used to identify a particular person in a research study.” http://grants1.nih.gov/grants/policy/data_sharing/index.htm

Image Processing Issues “Data may be excluded from the experimental results only if you have a sound reason to do so!” Mother Nature indicate justification in your notebook for any observations excluded for an experiment

Availability of image processing tools has led to increase in falsification of image data Gilbert. Nature doi:10.1038/news.3009.991

Be transparent in including image processing details in reporting your research The Journal of Cell Biology: 25% of accepted papers have at least 1 figure with undocumented processing (applied to both controls and experimental images) revokes the acceptance of about 1% of its papers due to inappropriate image manipulation (differential modifications with intention to deceive) Image manipulation precludes alternative interpretation of the data in the future. Rossner, M. (2006). The Scientist 20:24-25.

Brightness -Contrast Adjustment Journals routinely scan all images in a manuscript As photographed Brightness -Contrast Adjustment

Manipulated pixels can be easily detected http://resources.infosecinstitute.com/error-level-analysis-detect-image-manipulation/

Was this manipulation necessary? Rossner, M. and Yamada, K. (2004). J. Cell Biol. 166:11-15.

All data have signatures – even gels

“Innovative approaches” to data presentation are not allowed!

Sometimes image manipulations are not so easily detected As photographed Find the changes (3)

Nature, Instructions to Authors What CAN I do? Nature, Instructions to Authors List all image acquisition and processing tools and software Document key image-gathering setting and manipulations in the Supplemental materials Clearly demarcate borders between images collected at different times Avoid use of touch up tools or deliberately obscuring parts of an image Processing is acceptable only if applied across the entire image Be prepared to deliver the original, unprocessed images to the editor

Brightness and contrast adjustments must be applied equally across all images being compared Rossner, M. (2006). The Scientist 20:24-25.

You can only adjust brightness and contrast to the point where you do not lose data Rossner, M. and Yamada, K. (2004). J. Cell Biol. 166:11-15.

http://swehsc.pharmacy.arizona.edu/micro/digital-image-ethics

https://www.nih.gov/research-training/rigor-reproducibility/training