The reusable holdout: Preserving validity in adaptive data analysis

Slides:

Advertisements

Similar presentations

Tweet Classification for Political Sentiment Analysis Micol Marchetti-Bowick.

Advertisements

Figure 1.1 The observer in the truck sees the ball move in a vertical path when thrown upward. (b) The Earth observer views the path of the ball as a parabola.

Using Data Privacy for Better Adaptive Predictions Vitaly Feldman IBM Research – Almaden Foundations of Learning Theory, 2014 Cynthia Dwork Moritz Hardt.

Food Security: The Challenge of Feeding 9 Billion People by H. Charles J. Godfray, John R. Beddington, Ian R. Crute, Lawrence Haddad, David Lawrence, James.

Preserving Statistical Validity in Adaptive Data Analysis Vitaly Feldman IBM Research - Almaden Cynthia Dwork Moritz Hardt Toni Pitassi Omer Reingold Aaron.

Comment on “A Complete Skull from Dmanisi, Georgia, and the Evolutionary Biology of Early Homo” by Jeffrey H. Schwartz, Ian Tattersall, and Zhang Chi Science.

Early-Life Blockade of the 5-HT Transporter Alters Emotional Behavior in Adult Mice by Mark S. Ansorge, Mingming Zhou, Alena Lira, René Hen, and Jay A.

The Recent Increase in Atlantic Hurricane Activity: Causes and Implications by Stanley B. Goldenberg, Christopher W. Landsea, Alberto M. Mestas-Nuñez,

Comment on “Single-trial spike trains in parietal cortex reveal discrete steps during decision-making” by Michael N. Shadlen, Roozbeh Kiani, William T.

Current Problems in the Management of Marine Fisheries by J. R. Beddington, D. J. Agnew, and C. W. Clark Science Volume 316(5832): June 22, 2007.

A global quantification of “normal” sleep schedules using smartphone data by Olivia J. Walch, Amy Cochran, and Daniel B. Forger Science Volume 2(5):e

Assessing faculty professional development in STEM higher education: Sustainability of outcomes by Terry L. Derting, Diane Ebert-May, Timothy P. Henkel,

Rigorous Data Dredging Theory and Tools for Adaptive Data Analysis.

Table 2. Correlation matrix for variables in this study

by James E. Cresswell, and Helen M. Thompson

Computation and Society: The Case of Privacy and Fairness

Fig. 1. (A) Development of the free calcium ions measured by the calcium ion selective electrode (black line) at pH = 9.25 in comparison with the dosed.

Movie 7 Rapid movements of a single-cell protozoan T

Understanding Generalization in Adaptive Data Analysis

Algorithmic Approaches to Preventing Overfitting in Adaptive Data Analysis Part 1 Aaron Roth.

Vitaly (the West Coast) Feldman

by Alexei A. Aravin, Gregory J. Hannon, and Julius Brennecke

Preserving Validity in Adaptive Data Analysis

Cities are the Future by Nicholas S. Wigginton, Julia Fahrenkamp-Uppenbrink, Brad Wible, and David Malakoff Science Volume 352(6288): May 20, 2016.

Rethinking the global supply chain

by Yue Shen, Wang Zhang, Shu-Lei Chou, and Shi-Xue Dou

by Chuan-Chao Wang, Qi-Liang Ding, Huan Tao, and Hui Li

Fig. 1 Number of somatic mutations in representative human cancers, detected by genome-wide sequencing studies. Number of somatic mutations in representative.

Social-Ecological Resilience to Coastal Disasters

Combining satellite imagery and machine learning to predict poverty

Supervised vs. unsupervised Learning

Privacy-preserving Prediction

Data science online training.

by A. L. Westerling, H. G. Hidalgo, D. R. Cayan, and T. W. Swetnam

Technology-driven layer-by-layer assembly of nanofilms

Models of education in medicine, public health, and engineering

Graphs in Science.

The Antisense Transcriptomes of Human Cells

Fig. 3 ROC curves of mCCNA1 and mVIM assayed in esophageal cytology brushings from control normal-appearing GE junctions versus BE and EAC cases. ROC curves.

Evaluating replicability of laboratory experiments in economics

Comment on “High-resolution global maps of 21st-century forest cover change” by Robert Tropek, Ondřej Sedláček, Jan Beck, Petr Keil, Zuzana Musilová, Irena.

Classification accuracy for each taxon at different ranks of the NCBI taxonomy. Classification accuracy for each taxon at different ranks of the NCBI taxonomy.

Technical reproducibility and biological variability.

by Thomas R. Karl, Anthony Arguez, Boyin Huang, Jay H

ECOM method recovers time correlation with 2-ms precision from 219-ms imaging frames. ECOM method recovers time correlation with 2-ms precision from 219-ms.

Maternal mental illness

Fig. 6 Relation between Q and other impact indicators.

Fig. 1 Patterns of productivity during a scientific career.

Congratulations! Now Get to Work

Fig. 1. Reports rising. Reports rising. Number of publications recorded in Scopus that have, in the title or abstract, at least one of the following expressions:

Graphs in Science Chapter 1, Section 5 Page 34.

by Laura M. Wallace, Spahr C

Fig. 7. Motion adaptation increases time-dependent response modulations (TDRM) relatively to the average cell response.TDRM normalized to the value obtained.

© The Author(s) Published by Science and Education Publishing.

Examples of the different position of the centre of head rotation (CR)

(A) Mean CoP and maximum XcoM during the stride as a function of push magnitude. (A) Mean CoP and maximum XcoM during the stride as a function of push.

Mean and s.d. values of the observed vertical impulse ratio (Robs) and predicted vertical impulse ratio (Rpred). Mean and s.d. values of the observed vertical.

Fig. 3. Mean force and velocity during jumping

Leeches require closed loop sensory feedback to localize stimuli.

Fig. 5 Stability of the Q parameter.

© The Author(s) Published by Science and Education Publishing.

by Cathryn A. Manduca, Ellen R. Iverson, Michael Luxenberg, R

by Hiro Nimiya, Tatsunori Ikeda, and Takeshi Tsuji

by Peter Bacchetti, Steven G. Deeks, and Joseph M. McCune

Fig 2 Predicted probabilities of being in each of the five change in substance use categories (change from baseline in average daily amount of main substance)

Fig. 1 Example of diversified propagation patterns of the MJO.

© The Author(s) Published by Science and Education Publishing.

Average Execution Time in seconds

Presentation transcript:

The reusable holdout: Preserving validity in adaptive data analysis by Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth Science Volume 349(6248):636-638 August 7, 2015 Published by AAAS

Fig. 1 Learning uncorrelated label. Learning uncorrelated label. (A) Using the standard holdout. (B) Using Thresholdout. Vertical axes indicates average classification accuracy over 100 executions (margins are SD) of the classifier on training, holdout, and fresh sets. Horizontal axes show the number of variables selected for the classifier. Cynthia Dwork et al. Science 2015;349:636-638 Published by AAAS

Fig. 2 Learning partially correlated label with standard holdout. Learning partially correlated label with standard holdout. (A) Using the standard holdout algorithm. (B) Using Thresholdout. Axes are as in Fig. 1. Cynthia Dwork et al. Science 2015;349:636-638 Published by AAAS