Bootstrapping (And other statistical trickery). Reminder Of What We Do In Statistics Null Hypothesis Statistical Test Logic – Assume that the “no effect”

Slides:



Advertisements
Similar presentations
Review of the Basic Logic of NHST Significance tests are used to accept or reject the null hypothesis. This is done by studying the sampling distribution.
Advertisements

Confidence Interval and Hypothesis Testing for:
Review: What influences confidence intervals?
Business 205. Review Sampling Continuous Random Variables Central Limit Theorem Z-test.
Resampling techniques Why resampling? Jacknife Cross-validation Bootstrap Examples of application of bootstrap.
Sample size computations Petter Mostad
Chapter 3 Hypothesis Testing. Curriculum Object Specified the problem based the form of hypothesis Student can arrange for hypothesis step Analyze a problem.
Sampling Distributions
Today Concepts underlying inferential statistics
Using Statistics in Research Psych 231: Research Methods in Psychology.
Getting Started with Hypothesis Testing The Single Sample.
Chapter 14 Inferential Data Analysis
Inferential Statistics
Standard error of estimate & Confidence interval.
Bootstrapping applied to t-tests
Bootstrap spatobotp ttaoospbr Hesterberger & Moore, chapter 16 1.
Chapter 12 Inferential Statistics Gay, Mills, and Airasian
INFERENTIAL STATISTICS – Samples are only estimates of the population – Sample statistics will be slightly off from the true values of its population’s.
Choosing Statistical Procedures
EVALUATION David Kauchak CS 451 – Fall Admin Assignment 3 - change constructor to take zero parameters - instead, in the train method, call getFeatureIndices()
AM Recitation 2/10/11.
Hypothesis Testing:.
Jeopardy Hypothesis Testing T-test Basics T for Indep. Samples Z-scores Probability $100 $200$200 $300 $500 $400 $300 $400 $300 $400 $500 $400.
Fundamentals of Data Analysis Lecture 4 Testing of statistical hypotheses.
Today’s lesson Confidence intervals for the expected value of a random variable. Determining the sample size needed to have a specified probability of.
1 Power and Sample Size in Testing One Mean. 2 Type I & Type II Error Type I Error: reject the null hypothesis when it is true. The probability of a Type.
Montecarlo Simulation LAB NOV ECON Montecarlo Simulations Monte Carlo simulation is a method of analysis based on artificially recreating.
PARAMETRIC STATISTICAL INFERENCE
Significance Tests: THE BASICS Could it happen by chance alone?
Education Research 250:205 Writing Chapter 3. Objectives Subjects Instrumentation Procedures Experimental Design Statistical Analysis  Displaying data.
University of Ottawa - Bio 4118 – Applied Biostatistics © Antoine Morin and Scott Findlay 08/10/ :23 PM 1 Some basic statistical concepts, statistics.
Learning Objectives In this chapter you will learn about the t-test and its distribution t-test for related samples t-test for independent samples hypothesis.
One-sample In the previous cases we had one sample and were comparing its mean to a hypothesized population mean However in many situations we will use.
Chapter 9 Power. Decisions A null hypothesis significance test tells us the probability of obtaining our results when the null hypothesis is true p(Results|H.
9 Mar 2007 EMBnet Course – Introduction to Statistics for Biologists Nonparametric tests, Bootstrapping
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Biostatistics, statistical software VII. Non-parametric tests: Wilcoxon’s signed rank test, Mann-Whitney U-test, Kruskal- Wallis test, Spearman’ rank correlation.
Hypothesis Testing A procedure for determining which of two (or more) mutually exclusive statements is more likely true We classify hypothesis tests in.
Stat 112: Notes 2 Today’s class: Section 3.3. –Full description of simple linear regression model. –Checking the assumptions of the simple linear regression.
1 Chapter 8 Introduction to Hypothesis Testing. 2 Name of the game… Hypothesis testing Statistical method that uses sample data to evaluate a hypothesis.
Experimental Design and Statistics. Scientific Method
Limits to Statistical Theory Bootstrap analysis ESM April 2006.
Statistical Inference for the Mean Objectives: (Chapter 9, DeCoursey) -To understand the terms: Null Hypothesis, Rejection Region, and Type I and II errors.
Hypothesis Testing An understanding of the method of hypothesis testing is essential for understanding how both the natural and social sciences advance.
Chapter 3: Statistical Significance Testing Warner (2007). Applied statistics: From bivariate through multivariate. Sage Publications, Inc.
Stats Lunch: Day 3 The Basis of Hypothesis Testing w/ Parametric Statistics.
Descriptive Statistics Used to describe a data set –Mean, minimum, maximum Usually include information on data variability (error) –Standard deviation.
Comparing Two Means Chapter 9. Experiments Simple experiments – One IV that’s categorical (two levels!) – One DV that’s interval/ratio/continuous – For.
Inferential Statistics Introduction. If both variables are categorical, build tables... Convention: Each value of the independent (causal) variable has.
Hypothesis Testing and the T Test. First: Lets Remember Z Scores So: you received a 75 on a test. How did you do? If I said the mean was 72 what do you.
AGENDA Review In-Class Group Problems Review. Homework #3 Due on Thursday Do the first problem correctly Difference between what should happen over the.
Statistics: Unlocking the Power of Data Lock 5 Section 6.4 Distribution of a Sample Mean.
1 Probability and Statistics Confidence Intervals.
Chapter 13 Understanding research results: statistical inference.
Lecture 8 Estimation and Hypothesis Testing for Two Population Parameters.
Inference for distributions: - Comparing two means.
Non-parametric Approaches The Bootstrap. Non-parametric? Non-parametric or distribution-free tests have more lax and/or different assumptions Properties:
Hypothesis Testing. Statistical Inference – dealing with parameter and model uncertainty  Confidence Intervals (credible intervals)  Hypothesis Tests.
BIOL 582 Lecture Set 2 Inferential Statistics, Hypotheses, and Resampling.
Fundamentals of Data Analysis Lecture 4 Testing of statistical hypotheses pt.1.
Statistical Inference for the Mean Objectives: (Chapter 8&9, DeCoursey) -To understand the terms variance and standard error of a sample mean, Null Hypothesis,
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
Bootstrapping and Randomization Techniques Q560: Experimental Methods in Cognitive Science Lecture 15.
Lecture 13 Dustin Lueker. 2  Inferential statistical methods provide predictions about characteristics of a population, based on information in a sample.
Chapter 22 Inferential Data Analysis: Part 2 PowerPoint presentation developed by: Jennifer L. Bellamy & Sarah E. Bledsoe.
AP Statistics Chapter 24 Comparing Means. Objectives: Two-sample t methods Two-Sample t Interval for the Difference Between Means Two-Sample t Test for.
Statistical Quality Control, 7th Edition by Douglas C. Montgomery.
Estimation & Hypothesis Testing for Two Population Parameters
When we free ourselves of desire,
Ch13 Empirical Methods.
Presentation transcript:

Bootstrapping (And other statistical trickery)

Reminder Of What We Do In Statistics Null Hypothesis Statistical Test Logic – Assume that the “no effect” case is true and then ask if our data is probable given that case. – If we accept the null hypothesis: Our data isn’t improbable if the null hypothesis were true – If we reject: Our data is improbable if the null hypothesis were true

Hypothesis Tests The Null Hypothesis: – This is the hypothesis that we are looking to disprove – Usually, that there is “No Difference” – i.e. My sample is the same as the population (in the Z test) In statistics the Null Hypothesis takes the form of the distribution of results that we would expect by chance More Likely Outcomes Less Likely Outcomes

Hypothesis Tests Remember, we have to take the upside down logic of how we would normally think about these things. We say, if the null hypothesis were true, is my sample probable? More Likely Outcomes Less Likely Outcomes

To Make it Work We have to make assumptions about the population from which we selected our data. – These usually take the form of parametric assumptions. In a t-test: We assume that the null population is normal In a multiple regression: we assume that the errors are normal In Poisson regression: we assume that the DV is Poisson

T Test (Independent Samples)

The Problem We are always having to make assumptions that we bend. – In multiple regression: errors are rarely exactly normal – In Poisson regression: the mean rarely equals the variance – Many statistical procedures assume Multivariate Normality – In path analysis: there are situations where even if the data were perfectly normal, the errors follow strange bimodal distributions

Example Skewed Distributions violate ALL typical parametric assumptions

Early Solutions The Monte Carlo Simulation: 1.Use the mean, variance and co-variance of your data to define a truly normal distribution 2.Sample repeatedly from these idealized distributions 3.Run your analyses using this simulated data 4.Your CI’s are the middle 95% of the distribution of parameters

Nate Silver Example 1.Makes his best prediction of a candidate’s share of the vote (say 42%) 2.Applies a standard error to that guess (maybe he thinks this is +-5% with 95% confidence)

3. Creates this distribution of possible outcomes for this candidate 42% 47%37%

4. Does this for each candidate in the nation 42% 51% 62% 67%31% 46%

6. Samples randomly from each of those distributions (which may represent a win loss for each candidate) And then determines who won the house or senate and by how many seats 7.Does this 1000 times and ends up with this:

Problems This method assumes that the original data is really multivariate normal, and that the obtained data just a messy approximation of this. This only solves situations where the standard errors do not follow a known distribution (but the data, in theory, do)

The Jackknife This is a good solution if your sample has outliers that are having undue influence on your data. 1.Recalculate estimate by leaving out 1 (or more) random cases from a dataset. 2.Repeat many times 3.New Parameter estimate is mean of all obtained parameters (usually B’s) 4.Std. Error is the variance of the distribution of B’s

Bootstrap This is generally agreed to be the best solutions of the sampling methods Idea is incredible simple (usually far easier than actually computing standard errors) Computationally intensive (by 1980’s standards). With modern day computing power you barely notice the added time.

Bootstrap Procedure 1.Sample cases from your dataset randomly with replacement to obtain a new sample (with duplicates) that matched the N-Size of your original 2.Calculate parameter estimates (don’t worry about standard errors) 3.Repeat steps 1 and times 4.Every parameter will now have estimates 1.Mean of this sample is you main parameter estimate 2.Middle 95% of this sample is your middle 95% CI for the parameter

From Hesterberg, Moore, Monaghan, Clipson and Epstein

Advantages Allows for non-symmetric, non-parametric distributions for variables And parameters You don’t need to even know what your distribution is

Disadvantages You are assuming that your sample accurately reflects the distribution of the population that you have drawn it from. This will be the case on average but various samples will deviate significantly from the population distribution Be careful using this in small sample (my guideline is less than 50)