1 Nonparametric Methods III Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University

Slides:



Advertisements
Similar presentations
1-Way Analysis of Variance
Advertisements

Statistical Analysis and Data Interpretation What is significant for the athlete, the statistician and team doctor? important Will Hopkins
CHAPTER 21 Inferential Statistical Analysis. Understanding probability The idea of probability is central to inferential statistics. It means the chance.
Inference for Regression
Model Adequacy Checking in the ANOVA Text reference, Section 3-4, pg
Design of Experiments and Analysis of Variance
10-1 Introduction 10-2 Inference for a Difference in Means of Two Normal Distributions, Variances Known Figure 10-1 Two independent populations.
Today Today: Chapter 10 Sections from Chapter 10: Recommended Questions: 10.1, 10.2, 10-8, 10-10, 10.17,
Analysis of Differential Expression T-test ANOVA Non-parametric methods Correlation Regression.
2008 Chingchun 1 Bootstrap Chingchun Huang ( 黃敬群 ) Vision Lab, NCTU.
Chapter 2 Simple Comparative Experiments
Experimental Evaluation
Chapter 2Design & Analysis of Experiments 7E 2009 Montgomery 1 Chapter 2 –Basic Statistical Methods Describing sample data –Random samples –Sample mean,
Today Concepts underlying inferential statistics
5-3 Inference on the Means of Two Populations, Variances Unknown
Simple Linear Regression and Correlation
Prelude of Machine Learning 202 Statistical Data Analysis in the Computer Age (1991) Bradely Efron and Robert Tibshirani.
Chapter 12 Inferential Statistics Gay, Mills, and Airasian
Nonparametric or Distribution-free Tests
Dennis Shasha From a book co-written with Manda Wilson
AM Recitation 2/10/11.
Estimation and Hypothesis Testing Faculty of Information Technology King Mongkut’s University of Technology North Bangkok 1.
1 STATISTICAL HYPOTHESES AND THEIR VERIFICATION Kazimieras Pukėnas.
Fundamentals of Data Analysis Lecture 7 ANOVA. Program for today F Analysis of variance; F One factor design; F Many factors design; F Latin square scheme.
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. Chapter 14 Comparing Groups: Analysis of Variance Methods Section 14.2 Estimating Differences.
Statistical Computing
More About Significance Tests
Fundamentals of Data Analysis Lecture 4 Testing of statistical hypotheses.
1 Nonparametric Methods I Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University
Copyright © 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 17 Inferential Statistics.
Education Research 250:205 Writing Chapter 3. Objectives Subjects Instrumentation Procedures Experimental Design Statistical Analysis  Displaying data.
9 Mar 2007 EMBnet Course – Introduction to Statistics for Biologists Nonparametric tests, Bootstrapping
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
1 Nonparametric Methods II Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University
Business Statistics for Managerial Decision Farideh Dehkordi-Vakil.
Multiple Regression and Model Building Chapter 15 Copyright © 2014 by The McGraw-Hill Companies, Inc. All rights reserved.McGraw-Hill/Irwin.
Inference for Regression Simple Linear Regression IPS Chapter 10.1 © 2009 W.H. Freeman and Company.
Chapter 4 Linear Regression 1. Introduction Managerial decisions are often based on the relationship between two or more variables. For example, after.
Confidence intervals and hypothesis testing Petter Mostad
Nonparametric Tests IPS Chapter 15 © 2009 W.H. Freeman and Company.
1 Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University Nonparametric.
Educational Research Chapter 13 Inferential Statistics Gay, Mills, and Airasian 10 th Edition.
1 Chapter 8 Introduction to Hypothesis Testing. 2 Name of the game… Hypothesis testing Statistical method that uses sample data to evaluate a hypothesis.
BPS - 3rd Ed. Chapter 161 Inference about a Population Mean.
Limits to Statistical Theory Bootstrap analysis ESM April 2006.
DOX 6E Montgomery1 Design of Engineering Experiments Part 2 – Basic Statistical Concepts Simple comparative experiments –The hypothesis testing framework.
Single-Factor Studies KNNL – Chapter 16. Single-Factor Models Independent Variable can be qualitative or quantitative If Quantitative, we typically assume.
Introducing Communication Research 2e © 2014 SAGE Publications Chapter Seven Generalizing From Research Results: Inferential Statistics.
IMPORTANCE OF STATISTICS MR.CHITHRAVEL.V ASST.PROFESSOR ACN.
Nonparametric Methods II 1 Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University
Sullivan – Fundamentals of Statistics – 2 nd Edition – Chapter 11 Section 1 – Slide 1 of 26 Chapter 11 Section 1 Inference about Two Means: Dependent Samples.
1 Estimation of Population Mean Dr. T. T. Kachwala.
Statistical Inference Statistical inference is concerned with the use of sample data to make inferences about unknown population parameters. For example,
1 Probability and Statistics Confidence Intervals.
Synthesis and Review 2/20/12 Hypothesis Tests: the big picture Randomization distributions Connecting intervals and tests Review of major topics Open Q+A.
Learning Objectives After this section, you should be able to: The Practice of Statistics, 5 th Edition1 DESCRIBE the shape, center, and spread of the.
Jump to first page Inferring Sample Findings to the Population and Testing for Differences.
Inferential statistics by example Maarten Buis Monday 2 January 2005.
Hypothesis Testing. Statistical Inference – dealing with parameter and model uncertainty  Confidence Intervals (credible intervals)  Hypothesis Tests.
BIOL 582 Lecture Set 2 Inferential Statistics, Hypotheses, and Resampling.
Fundamentals of Data Analysis Lecture 4 Testing of statistical hypotheses pt.1.
Statistical Inference for the Mean Objectives: (Chapter 8&9, DeCoursey) -To understand the terms variance and standard error of a sample mean, Null Hypothesis,
Copyright (c) 2004 Brooks/Cole, a division of Thomson Learning, Inc. Chapter 7 Inferences Concerning Means.
F73DA2 INTRODUCTORY DATA ANALYSIS ANALYSIS OF VARIANCE.
Appendix I A Refresher on some Statistical Terms and Tests.
STA248 week 121 Bootstrap Test for Pairs of Means of a Non-Normal Population – small samples Suppose X 1, …, X n are iid from some distribution independent.
Estimating standard error using bootstrap
Statistical Quality Control, 7th Edition by Douglas C. Montgomery.
Simulation: Sensitivity, Bootstrap, and Power
Presentation transcript:

1 Nonparametric Methods III Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University

2 PART 4: Bootstrap and Permutation Tests  Introduction  References  Bootstrap Tests  Permutation Tests  Cross-validation  Bootstrap Regression  ANOVA

3 References  Efron, B.; Tibshirani, R. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.  Companion/appendix-bootstrapping.pdf Companion/appendix-bootstrapping.pdf  project.org/bin/macosx/2.1/check/bootstr ap-check.ex project.org/bin/macosx/2.1/check/bootstr ap-check.ex  cat_080/pdf/moore14.pdf cat_080/pdf/moore14.pdf

4 Hypothesis Testing (1)  A statistical hypothesis test is a method of making statistical decisions from and about experimental data.  Null-hypothesis testing just answers the question of “ how well the findings fit the possibility that chance factors alone might be responsible. ”  This is done by asking and answering a hypothetical question.

5 Hypothesis Testing (2)  Hypothesis testing is largely the product of Ronald Fisher, Jerzy Neyman, Karl Pearson and (son) Egon Pearson. Fisher was an agricultural statistician who emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an (extended) hybrid of the Fisher vs. Neyman/Pearson formulation, methods and terminology developed in the early 20th century.

6 Hypothesis Testing (3)

7 Hypothesis Testing (4)

8 Hypothesis Testing (5)

9 Hypothesis Testing (7)  Parametric Tests:  Nonparametric Tests: Bootstrap Tests Permutation Tests

10 Confidence Intervals vs. Hypothesis Testing (1)  Interval estimation ("Confidence Intervals") and point estimation ("Hypothesis Testing") are two different ways of expressing the same information. c5_inferential_statistics/confidence_interv_hypo.html

11 Confidence Intervals vs. Hypothesis Testing (2)  If the exact p-value is reported, then the relationship between confidence intervals and hypothesis testing is very close. However, the objective of the two methods is different: Hypothesis testing relates to a single conclusion of statistical significance vs. no statistical significance. Confidence intervals provide a range of plausible values for your population. advancedStatistics/convidenceVsHypothesis.html

12 Confidence Intervals vs. Hypothesis Testing (3)  Which one? Use hypothesis testing when you want to do a strict comparison with a pre-specified hypothesis and significance level. Use confidence intervals to describe the magnitude of an effect (e.g., mean difference, odds ratio, etc.) or when you want to describe a single sample. advancedStatistics/convidenceVsHypothesis.html

13 P-value

14 Achieved Significance Level (ASL)

15 Bootstrap Tests  Methodology  Flowchart  R code

16 Bootstrap Tests  Beran (1988) showed that bootstrap inference is refined when the quantity bootstrapped is asymptotically pivotal.  It is often used as a robust alternative to inference based on parametric assumptions. bootstrapping.pdf

17 Hypothesis Testing by a Pivot

18  T statistics can be regarded as a pivot or an asymptotic pivotal when the data are normally distributed.  Bootstrap T tests can be applied when the data are not normally distributed. One Sample Bootstrap Tests

19 Bootstrap T tests  Flowchart  R code

20 Bootstrap B times Flowchart of Bootstrap T Tests

21 Bootstrap T Tests by R

22 An Example of Bootstrap T Tests by R

23 Bootstrap Tests by The “ BCa ”  The BCa percentile method is an efficient method to generate bootstrap confidence intervals.  There is a correspondence between confidence intervals and hypothesis testing.  So, we can use the BCa percentile method to test whether H 0 is true.  Example: use BCa to calculate p-value

24  Use R package “ boot.ci(boot) ”  Use R package “ bcanon(bootstrap) ”   BCa Confidence Intervals:

25

26 An Example of “ boot.ci(boot) ” in R

27

28 An example of “ bcanon(bootstrap) ” in R

29 BCa by

30 Use BCa to calculate p-value by R

31 Two Sample Bootstrap Tests  Flowchart  R code

32 Bootstrap B times Flowchart of Two-Sample Bootstrap Tests m+n=Ncombine

33 Two-Sample Bootstrap Tests by R

34 Output (1)

35 Output (2)

36 Permutation Tests  Methodology  Flowchart  R code

37 Permutation  In several fields of mathematics, the term permutation is used with different but closely related meanings. They all relate to the notion of (re-)arranging elements from a given finite set into a sequence.

38 Permutation Tests  Permutation test is also called a randomization test, re-randomization test, or an exact test.  If the labels are exchangeable under the null hypothesis, then the resulting tests yield exact significance levels.  Confidence intervals can then be derived from the tests.  The theory has evolved from the works of R.A. Fisher and E.J.G. Pitman in the 1930s.

39 Applications of Permutation Tests (1) We can use a permutation test only when we can see how to resample in a way that is consistent with the study design and with the null hypothesis. cat_080/pdf/moore14.pdf

40 Two-sample problems when the null hypothesis says that the two populations are identical. We may wish to compare population means, proportions, standard deviations, or other statistics. Matched pairs designs when the null hypothesis says that there are only random differences within pairs. A variety of comparisons is again possible. Relationships between two quantitative variables when the null hypothesis says that the variables are not related. The correlation is the most common measure of association, but not the only one. cat_080/pdf/moore14.pdf Applications of Permutation Tests (2)

41 Inference by Permutation Tests nn.Dahyot/453Bootstrap/05 _Permutation.pdf

42 Flowchart of The Permutation Test for Mean Shift in One Sample Partition 2 subset B times (treatment group) (control group) (treatment group) (control group)

43 An Example for One Sample Permutation Test by R EandTCh15a.txt

44

45 An Example of Output Results

46 Partition subset B times treatment subgroup control subgroup m+n=N combine Flowchart of The Permutation Test for Mean Shift in Two Samples treatment subgroup control subgroup

47 Bootstrap Tests vs. Permutation Tests  Very similar results between the permutation test and the bootstrap test.  is the exact probability when.  is not an exact probability but is guaranteed to be accurate as an estimate of the ASL, as the sample size B goes to infinity.

48 Cross-validation  Methodology  R code

49 Cross-validation  Cross-validation, sometimes called rotation estimation, is the statistical practice of partitioning a sample of data into subsets such that the analysis is initially performed on a single subset, while the other subset(s) are retained for subsequent use in confirming and validating the initial analysis. The initial subset of data is called the training set. the other subset(s) are called validation or testing sets.

50 Overfitting Problems  In statistics, overfitting is fitting a statistical model that has too many parameters.  When the degrees of freedom in parameter selection exceed the information content of the data, this leads to arbitrariness in the final (fitted) model parameters which reduces or destroys the ability of the model to generalize beyond the fitting data.  The concept of overfitting is important also in machine learning.  In both statistics and machine learning, in order to avoid overfitting, it is necessary to use additional techniques (e.g. cross-validation, early stopping, Bayesian priors on parameters or model comparison), that can indicate when further training is not resulting in better generalization. 

51 library(bootstrap) ?crossval

52 An Example of Cross-validation by R

53 output

54 Bootstrap Regression  Bootstrapping pairs: Resample from the sample pairs { }.  Bootstrapping residuals: 1. Fit by the original sample and obtain the residuals. 2. Resample from residuals.

55 Bootstrapping Pairs by R

56 Output

57 Bootstrapping Residuals by R

58 Bootstrapping residuals

59 ANOVA  When random errors follow a normal distribution:  When random errors do not follow a Normal distribution: Bootstrap tests: Permutation tests:

60 An Example of ANOVA by R (1)  Example Twenty lambs are randomly assigned to three different diets. The weight gain (in two weeks) is recorded. Is there a difference among the diets?  Reference otstrap.R otstrap.R

61 An Example of ANOVA by R (1)

62 An Example of ANOVA by R (2)

63 An Example of ANOVA by R (3)

64 Output (1)

65 Output (2)

66 Output (3)

67 Output (4)

68 Output (5)

69 Output (6)

70 Output (7)

71 The Second Example of ANOVA by R (1)  Data source kyphosis.html kyphosis.html  Reference  Kyphosis is a misalignment of the spine. The data are on 83 laminectomy (a surgical procedure involving the spine) patients. The predictor variables are age and age^2 (that is, a quadratic function of age), number of vertebrae involved in the surgery and start the vertebra number of the first vertebra involved. The response is presence or absence of kyphosis after the surgery (and perhaps caused by it).

72 The Second Example of ANOVA by R (2)

73 The Second Example of ANOVA by R (3)

74 The Second Example of ANOVA by R (4)

75 Output (1) Data = kyphosis

76 Output (2)

77 Output (3)

78 Output (4)

79 Output (5) #deviance #p-value

80 Output (6)

81 Exercises:  Write your own programs similar to those examples presented in this talk.  Write programs for those examples mentioned at the reference web pages.  Write programs for the other examples that you know.  Practice Makes Perfect! 81