Optimal Learning in the Laboratory Sciences

Slides:

Advertisements

Similar presentations

Estimating Identification Risks for Microdata Jerome P. Reiter Institute of Statistics and Decision Sciences Duke University, Durham NC, USA.

Advertisements

Modeling of Data. Basic Bayes theorem Bayes theorem relates the conditional probabilities of two events A, and B: A might be a hypothesis and B might.

Computational Stochastic Optimization:

AP STATISTICS Simulating Experiments. Steps for simulation Simulation: The imitation of chance behavior, based on a model that accurately reflects the.

Statistical Issues in Research Planning and Evaluation

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences Richer belief models December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton University.

Model Assessment and Selection

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences A case application – Growing carbon nanotubes December 10, 2014 Warren B. Powell Kris Reyes.

AGC DSP AGC DSP Professor A G Constantinides© Estimation Theory We seek to determine from a set of data, a set of parameters such that their values would.

An Optimal Learning Approach to Finding an Outbreak of a Disease Warren Scott Warren Powell

Chapter 28 Design of Experiments (DOE). Objectives Define basic design of experiments (DOE) terminology. Apply DOE principles. Plan, organize, and evaluate.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences Working with nonlinear belief models December 10, 2014 Warren B. Powell Kris Reyes Si Chen.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences The power of interactivity December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences Building a belief model December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton University.

CHAPTER 15 S IMULATION - B ASED O PTIMIZATION II : S TOCHASTIC G RADIENT AND S AMPLE P ATH M ETHODS Organization of chapter in ISSO –Introduction to gradient.

Determining Sample Size

Computational Stochastic Optimization: Bridging communities October 25, 2012 Warren Powell CASTLE Laboratory Princeton University

HIT241 - RISK MANAGEMENT Introduction

Minimal Test Collections for Retrieval Evaluation B. Carterette, J. Allan, R. Sitaraman University of Massachusetts Amherst SIGIR2006.

Adaptive Importance Sampling for Estimation in Structured Domains L.E. Ortiz and L.P. Kaelbling.

Elementary Statistical Methods André L. Souza, Ph.D. The University of Alabama Lecture 22 Statistical Power.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences The knowledge gradient December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton University.

Kernel Classifiers from a Machine Learning Perspective (sec ) Jin-San Yang Biointelligence Laboratory School of Computer Science and Engineering.

IT Project Management, Third Edition Chapter 11 1 Chapter 6: Project Risk Management.

ECE 8443 – Pattern Recognition ECE 8423 – Adaptive Signal Processing Objectives: Deterministic vs. Random Maximum A Posteriori Maximum Likelihood Minimum.

Simulation is the process of studying the behavior of a real system by using a model that replicates the behavior of the system under different scenarios.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences Overview December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton University

Statistical Decision Theory Bayes’ theorem: For discrete events For probability density functions.

What is a DOE? Design of Experiment Adapted from experiments.cfm#preparation.

A Semi-Blind Technique for MIMO Channel Matrix Estimation Aditya Jagannatham and Bhaskar D. Rao The proposed algorithm performs well compared to its training.

Machine Design Under Uncertainty. Outline Uncertainty in mechanical components Why consider uncertainty Basics of uncertainty Uncertainty analysis for.

© 2009 Ilya O. Ryzhov 1 © 2008 Warren B. Powell 1. Optimal Learning On A Graph INFORMS Annual Meeting October 11, 2009 Ilya O. Ryzhov Warren Powell Princeton.

- 1 - Overall procedure of validation Calibration Validation Figure 12.4 Validation, calibration, and prediction (Oberkampf and Barone, 2004 ). Model accuracy.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences Forming the decision set December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton.

Université d’Ottawa / University of Ottawa 2001 Bio 8100s Applied Multivariate Biostatistics L1a.1 Lecture 1a: Some basic statistical concepts l The use.

1 1 Slide Simulation Professor Ahmadi. 2 2 Slide Simulation Chapter Outline n Computer Simulation n Simulation Modeling n Random Variables and Pseudo-Random.

Slide 1 of 48 Measurements and Their Uncertainty

Machine Learning 5. Parametric Methods.

Nonlinear differential equation model for quantification of transcriptional regulation applied to microarray data of Saccharomyces cerevisiae Vu, T. T.,

1 Foundations of Research Cranach, Tree of Knowledge [of Good and Evil] (1472) Click “slide show” to start this presentation as a show. Remember: focus.

1 Chapter 8: Model Inference and Averaging Presented by Hui Fang.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences Searching a two-dimensional surface December 10, 2014 Warren B. Powell Kris Reyes Si Chen.

Application of Dynamic Programming to Optimal Learning Problems Peter Frazier Warren Powell Savas Dayanik Department of Operations Research and Financial.

Hypothesis Testing. Statistical Inference – dealing with parameter and model uncertainty  Confidence Intervals (credible intervals)  Hypothesis Tests.

Sequential Off-line Learning with Knowledge Gradients Peter Frazier Warren Powell Savas Dayanik Department of Operations Research and Financial Engineering.

Richard Kass/F02P416 Lecture 6 1 Lecture 6 Chi Square Distribution (  2 ) and Least Squares Fitting Chi Square Distribution (  2 ) (See Taylor Ch 8,

RECONSTRUCTION OF MULTI- SPECTRAL IMAGES USING MAP Gaurav.

Outline Historical note about Bayes’ rule Bayesian updating for probability density functions –Salary offer estimate Coin trials example Reading material:

Reconciling Confidentiality Risk Measures from Statistics and Computer Science Jerry Reiter Department of Statistical Science Duke University.

Slide 1 Tutorial: Optimal Learning in the Laboratory Sciences The value of information December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

MathematicalMarketing Slide 3c.1 Mathematical Tools Chapter 3: Part c – Parameter Estimation We will be discussing  Nonlinear Parameter Estimation  Maximum.

Data Modeling Patrice Koehl Department of Biological Sciences

Lecture 1.31 Criteria for optimal reception of radio signals.

SAFEtalk Our new safety tool

Random Testing: Theoretical Results and Practical Implications IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 2012 Andrea Arcuri, Member, IEEE, Muhammad.

Chapter 17: Optimization In Simulation

1 Department of Engineering, 2 Department of Mathematics,

Chi Square Distribution (c2) and Least Squares Fitting

CSCI B609: “Foundations of Data Science”

1 Department of Engineering, 2 Department of Mathematics,

Discrete Event Simulation - 4

1 Department of Engineering, 2 Department of Mathematics,

CIS 488/588 Bruce R. Maxim UM-Dearborn

Slides for Introduction to Stochastic Search and Optimization (ISSO) by J. C. Spall CHAPTER 15 SIMULATION-BASED OPTIMIZATION II: STOCHASTIC GRADIENT AND.

Understanding Marketing Objectives

10701 / Machine Learning Today: - Cross validation,

5.2 Least-Squares Fit to a Straight Line

Lecture 15: Data Cleaning for ML

Optimal Learning in the Laboratory Sciences

Presentation transcript:

Optimal Learning in the Laboratory Sciences Tutorial: Optimal Learning in the Laboratory Sciences Evaluating risk December 10, 2014 Warren B. Powell Kris Reyes Si Chen Princeton University http://www.castlelab.princeton.edu Slide 1 1

Lecture outline Assessing and controlling risk

Assessing and controlling risk Questions facing any experimentalist: What is the risk that you will achieve specific goals (strength, reflectivity, fluorescence, …) with a specific experimental design? Measures of success: Relative success – How well might you do relative to the best that can be achieved? Absolute success – How well might you do relative to some absolute metric? What is the most cost-effective way of minimizing risk? Increasing the experimental budget Reducing the noise in a single experiment Do a better job of sequencing experiments

Sample application: Maximize current We start with a distribution of the possible true values of the unknown For For each possible truth, we find the design that produces the highest output. This represents the best you can possibly achieve with an unlimited experimental budget.

Managing risk Analysis approach: Estimate the probability of success Use “knowledge gradient” to simulate the decisions that would be made by a scientist. Simulate the process of running an experiment many times given an assumed budget and experimental noise. This has to be done over different values of the assumed truth. Estimate the probability of success Did you achieve absolute goal? How well did you do relative to the best that could have been achieved?

Managing risk Target level

Managing risk

Managing risk However, as we increase the target, there is a chance that the sampled truth cannot even achieve this value, let alone the possibility of the OL procedure of finding it.

Managing risk

Managing risk

Managing risk

Managing risk

Managing risk We can increase the likelihood of success by: Increasing the number of experiments Reducing the experimental noise Implications Managers can assess the risk from a research proposal. Experimentalists can find the least cost way to minimize risk. Increasing number of experiments