1 BAYES and FREQUENTISM: The Return of an Old Controversy Louis Lyons Oxford University SLAC, January 2007.

Slides:



Advertisements
Similar presentations
Bayes rule, priors and maximum a posteriori
Advertisements

DO’S AND DONT’S WITH LIKELIHOODS Louis Lyons Oxford (CDF)
1 BAYES and FREQUENTISM: The Return of an Old Controversy Louis Lyons Oxford University LBL, January 2008.
Concluding Talk: Physics Gary Feldman Harvard University PHYSTAT 05 University of Oxford 15 September, 2005.
27 th March CERN Higgs searches: CL s W. J. Murray RAL.
1 Statistical Problems in Particle Physics Louis Lyons Oxford IPAM, November 2004.
Practical Statistics for LHC Physicists Bayesian Inference Harrison B. Prosper Florida State University CERN Academic Training Lectures 9 April, 2015.
BAYES and FREQUENTISM: The Return of an Old Controversy 1 Louis Lyons Imperial College and Oxford University CERN Latin American School March 2015.
1 Methods of Experimental Particle Physics Alexei Safonov Lecture #21.
1 LIMITS Why limits? Methods for upper limits Desirable properties Dealing with systematics Feldman-Cousins Recommendations.
BAYES and FREQUENTISM: The Return of an Old Controversy 1 Louis Lyons Imperial College and Oxford University Heidelberg April 2013.
Bayesian inference Gil McVean, Department of Statistics Monday 17 th November 2008.
1 BAYES versus FREQUENTISM The Return of an Old Controversy The ideologies, with examples Upper limits Systematics Louis Lyons, Oxford University and CERN.
Statistics In HEP 2 Helge VossHadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP 1 How do we understand/interpret our measurements.
8. Statistical tests 8.1 Hypotheses K. Desch – Statistical methods of data analysis SS10 Frequent problem: Decision making based on statistical information.
Statistics for Particle Physics: Intervals Roger Barlow Karlsruhe: 12 October 2009.
Slide 1 Statistics for HEP Roger Barlow Manchester University Lecture 1: Probability.
1 BAYES and FREQUENTISM: The Return of an Old Controversy Louis Lyons Oxford University CERN, October 2006 Lecture 4.
1 Practical Statistics for Physicists Dresden March 2010 Louis Lyons Imperial College and Oxford CDF experiment at FNAL CMS expt at LHC
Credibility of confidence intervals Dean Karlen / Carleton University Advanced Statistical Techniques in Particle Physics Durham, March 2002.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
I The meaning of chance Axiomatization. E Plurbus Unum.
1 BAYES and FREQUENTISM: The Return of an Old Controversy Louis Lyons Imperial College and Oxford University Vienna May 2011.
G. Cowan 2011 CERN Summer Student Lectures on Statistics / Lecture 41 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
1 Statistical Inference Problems in High Energy Physics and Astronomy Louis Lyons Particle Physics, Oxford BIRS Workshop Banff.
G. Cowan Lectures on Statistical Data Analysis 1 Statistical Data Analysis: Lecture 7 1Probability, Bayes’ theorem, random variables, pdfs 2Functions of.
G. Cowan Discovery and limits / DESY, 4-7 October 2011 / Lecture 3 1 Statistical Methods for Discovery and Limits Lecture 3: Limits for Poisson mean: Bayesian.
1 Practical Statistics for Physicists Stockholm Lectures Sept 2008 Louis Lyons Imperial College and Oxford CDF experiment at FNAL CMS expt at LHC
Statistical Analysis of Systematic Errors and Small Signals Reinhard Schwienhorst University of Minnesota 10/26/99.
1 Probability and Statistics  What is probability?  What is statistics?
Statistical Decision Theory
G. Cowan 2009 CERN Summer Student Lectures on Statistics1 Introduction to Statistics − Day 4 Lecture 1 Probability Random variables, probability densities,
Harrison B. Prosper Workshop on Top Physics, Grenoble Bayesian Statistics in Analysis Harrison B. Prosper Florida State University Workshop on Top Physics:
Dr. Gary Blau, Sean HanMonday, Aug 13, 2007 Statistical Design of Experiments SECTION I Probability Theory Review.
G. Cowan Lectures on Statistical Data Analysis Lecture 1 page 1 Lectures on Statistical Data Analysis London Postgraduate Lectures on Particle Physics;
Statistics Lectures: Questions for discussion Louis Lyons Imperial College and Oxford CERN Summer Students July
Practical Statistics for Particle Physicists Lecture 3 Harrison B. Prosper Florida State University European School of High-Energy Physics Anjou, France.
Statistics In HEP Helge VossHadron Collider Physics Summer School June 8-17, 2011― Statistics in HEP 1 How do we understand/interpret our measurements.
Frequentistic approaches in physics: Fisher, Neyman-Pearson and beyond Alessandro Palma Dottorato in Fisica XXII ciclo Corso di Probabilità e Incertezza.
1 Is there evidence for a peak in this data?. 2 “Observation of an Exotic S=+1 Baryon in Exclusive Photoproduction from the Deuteron” S. Stepanyan et.
Making sense of randomness
Confidence Intervals First ICFA Instrumentation School/Workshop At Morelia, Mexico, November 18-29, 2002 Harrison B. Prosper Florida State University.
Statistical Decision Theory Bayes’ theorem: For discrete events For probability density functions.
BAYES and FREQUENTISM: The Return of an Old Controversy 1 Louis Lyons Imperial College and Oxford University CERN Summer Students July 2014.
Experience from Searches at the Tevatron Harrison B. Prosper Florida State University 18 January, 2011 PHYSTAT 2011 CERN.
Practical Statistics for Particle Physicists Lecture 2 Harrison B. Prosper Florida State University European School of High-Energy Physics Parádfürdő,
1 Introduction to Statistics − Day 4 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Lecture 2 Brief catalogue of probability.
1 Methods of Experimental Particle Physics Alexei Safonov Lecture #24.
G. Cowan Lectures on Statistical Data Analysis Lecture 8 page 1 Statistical Data Analysis: Lecture 8 1Probability, Bayes’ theorem 2Random variables and.
1 Introduction to Statistics − Day 3 Glen Cowan Lecture 1 Probability Random variables, probability densities, etc. Brief catalogue of probability densities.
G. Cowan Lectures on Statistical Data Analysis Lecture 4 page 1 Lecture 4 1 Probability (90 min.) Definition, Bayes’ theorem, probability densities and.
Statistical NLP: Lecture 4 Mathematical Foundations I: Probability Theory (Ch2)
In Bayesian theory, a test statistics can be defined by taking the ratio of the Bayes factors for the two hypotheses: The ratio measures the probability.
G. Cowan Lectures on Statistical Data Analysis Lecture 12 page 1 Statistical Data Analysis: Lecture 12 1Probability, Bayes’ theorem 2Random variables and.
G. Cowan RHUL Physics Statistical Issues for Higgs Search page 1 Statistical Issues for Higgs Search ATLAS Statistics Forum CERN, 16 April, 2007 Glen Cowan.
1 BAYES and FREQUENTISM: The Return of an Old Controversy Louis Lyons Imperial College and Oxford University Amsterdam Oct 2009.
Statistical Issues in Searches for New Physics
BAYES and FREQUENTISM: The Return of an Old Controversy
What is Probability? Bayes and Frequentism
Probability and Statistics for Particle Physics
Statistical Issues for Dark Matter Searches
BAYES and FREQUENTISM: The Return of an Old Controversy
Confidence Intervals and Limits
Lecture 4 1 Probability (90 min.)
Statistical NLP: Lecture 4
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
Lecture 4 1 Probability Definition, Bayes’ theorem, probability densities and their properties, catalogue of pdfs, Monte Carlo 2 Statistical tests general.
CS639: Data Management for Data Science
Statistics for HEP Roger Barlow Manchester University
Presentation transcript:

1 BAYES and FREQUENTISM: The Return of an Old Controversy Louis Lyons Oxford University SLAC, January 2007

2

3 Commercial break #2 Wednesday 4pm, Jan 24 th : Brad Efron “The Bootstrap” Feb 21st, 22nd, 23th, 24 th : LL Learning to love the error matrix Maximum likelihood do’s and dont’s χ 2 for parameter determ + goodness of fit p-values and discovery

4 Topics Who cares? What is probability? Bayesian approach Examples Frequentist approach Systematics Summary

5 It is possible to spend a lifetime analysing data without realising that there are two very different fundamental approaches to statistics: Bayesianism and Frequentism.

6 How can textbooks not even mention Bayes/ Frequentism? For simplest case with no constraint onthen at some probability, for both Bayes and Frequentist (but different interpretations) See Bob Cousins “Why isn’t every physicist a Bayesian?” Amer Jrnl Phys 63(1995)398

7 We need to make a statement about Parameters, Given Data The basic difference between the two: Bayesian : Probability (parameter, given data) (an anathema to a Frequentist!) Frequentist : Probability (data, given parameter) (a likelihood function)

8 PROBABILITY MATHEMATICAL Formal Based on Axioms FREQUENTIST Ratio of frequencies as n  infinity Repeated “identical” trials Not applicable to single event or physical constant BAYESIAN Degree of belief Can be applied to single event or physical constant (even though these have unique truth) Varies from person to person *** Quantified by “fair bet”

9 Bayesian versus Classical Bayesian P(A and B) = P(A;B) x P(B) = P(B;A) x P(A) e.g. A = event contains t quark B = event contains W boson or A = I am in SLAC Auditorium B = I am giving a lecture Completely uncontroversial, provided…. P(A;B) = P(B;A) x P(A) /P(B)

10 Bayesian posteriorlikelihoodprior Problems: p(param) true or false “Degree of belief” Prior What functional form? Coverage Bayes’ Theorem p(parameter | data) α p(data | parameter) * p(parameter)

11 P(parameter) True or False “Degree of Belief” Credible interval Prior: What functional form? Uninformative prior: flat? In which variable? Unimportant if “data overshadows prior” Important for limits Subjective or Objective prior?

12

13 Prior

14 Prior = zero in unphysical region

15 Bayesian posterior  intervals Upper limit Lower limit Central interval Shortest

16 Ilya Narsky, FNAL CLW 2000

17 P (Data;Theory) P (Theory;Data) HIGGS SEARCH at CERN Is data consistent with Standard Model? or with Standard Model + Higgs? End of Sept 2000 Data not very consistent with S.M. Prob (Data ; S.M.) < 1% valid frequentist statement Turned by the press into: Prob (S.M. ; Data) 99% i.e. “It is almost certain that the Higgs has been seen”

18 P (Data;Theory) P (Theory;Data) Theory = male or female Data = pregnant or not pregnant P (pregnant ; female) ~ 3%

19 P (Data;Theory) P (Theory;Data) Theory = male or female Data = pregnant or not pregnant P (pregnant ; female) ~ 3% but P (female ; pregnant) >>>3%

20 Example 1 : Is coin fair ? Toss coin: 5 consecutive tails What is P(unbiased; data) ? i.e. p = ½ Depends on Prior(p) If village priest prior ~ δ ( 1/2 ) If stranger in pub prior ~ 1 for 0<p<1 (also needs cost function)

21 Example 2 : Particle Identification Try to separate π’s and protons probability (p tag; real p) = 0.95 probability ( π tag; real p) = 0.05 probability (p tag; real π ) = 0.10 probability ( π tag; real π ) = 0.90 Particle gives proton tag. What is it? If proton beam, very likely If general secondary particles, more even If pure π beam, ~ 0 Depends on prior = fraction of protons

22 Hunter and Dog 1)Dog d has 50% probability of being 100 m. of Hunter h 2)Hunter h has 50% probability of being within 100m of Dog d dh x River x =0River x =1 km

23 Given that: a) Dog d has 50% probability of being 100 m. of Hunter Is it true that b ) Hunter h has 50% probability of being within 100m of Dog d ? Additional information Rivers at zero & 1 km. Hunter cannot cross them. Dog can swim across river - Statement a) still true If dog at –101 m, hunter cannot be within 100m of dog Statement b) untrue

24

25 Classical Approach Neyman “confidence interval” avoids pdf for uses only P( x; ) Confidence interval : P( contains ) = True for any Varying intervals from ensemble of experiments fixed Gives range of for which observed value was “likely” ( ) Contrast Bayes : Degree of belief = is in

26

27 90% Classical interval for Gaussian σ = 1 μ ≥ 0 e.g. m 2 (ν e )

28 at 90% confidence and known, but random unknown, but fixed Probability statement about and Frequentist Bayesian and known, and fixed unknown, and random Probability/credible statement about

29 Coverage Fraction of intervals containing true value Property of method, not of result Can vary with param Frequentist concept. Built in to Neyman construction Some Bayesians reject idea. Coverage not guaranteed Integer data (Poisson)  discontinuities Ideal coverage plot C μ

30 Classical Intervals Problems Hard to understand e.g. d’Agostini Arbitrary choice of interval Possibility of empty range Nuisance parameters (systematic errors ) Advantages Widely applicable Well defined coverage

31

32 X obs = -2 now gives upper limit

33 Black lines Classical 90% central interval Red dashed: Classical 90% upper limit Flip-flop

34

35 Poisson confidence intervals. Background = 3 Standard Frequentist Feldman - Cousins

36

37

38 Importance of Ordering Rule Neyman construction in 1 parameter 2 measurements An aside: Determination of single parameter p via Acceptable level of Range of parameters given by 1)Values of for which data is likely i.e. p( ) is acceptable or 2) 2) is good 1) Range depends on [“Confidence interval coupled to goodness of fit”]

39 * * * Neyman Construction For given, acceptable (, ) satisfy = Defines cylinder in space Experiment gives interval Range depends on Range and goodness of fit are coupled

40 That was using Probability Ordering Now change to Likelihood Ratio Ordering For,no value of gives very good fit For Neyman Construction at fixed, compare: with where giving Cutting on Likelihood Ratio Ordering gives:

41 Therefore, range of is Constant Width Independent of Confidence Range and Goodness of Fit are completely decoupled

42 Bayesian Pros: Easy to understand Physical interval Cons: Needs prior Coverage not guaranteed Hard to combine

43 Standard Frequentist Pros: Coverage Widely applicable Cons: Hard to understand Small or empty intervals Difficult in many variables Needs ensemble

44 SYSTEMATICS For example Observed for statistical errors Physics parameter we need to know these, probably from other measurements (and/or theory) Uncertainties  error in Some are arguably statistical errors Shift Central Value Bayesian Frequentist Mixed

45 Simplest Method Evaluate using and Move nuisance parameters (one at a time) by their errors  If nuisance parameters are uncorrelated, combine these contributions in quadrature  total systematic Shift Nuisance Parameters

46 Bayesian Without systematics prior With systematics Then integrate over LA and b

47 If = constant and = truncated Gaussian TROUBLE! Upper limit on from Significance from likelihood ratio for and

48 Frequentist Full Method Imagine just 2 parameters and LA and 2 measurements N and M PhysicsNuisance Do Neyman construction in 4-D Use observed N and M, to give Confidence Region for LA and LA 68%

49 Then project onto axis This results in OVERCOVERAGE Aim to get better shaped region, by suitable choice of ordering rule Example: Profile likelihood ordering

50 Full frequentist method hard to apply in several dimensions Used in 3 parameters For example: Neutrino oscillations (CHOOZ) Normalisation of data Use approximate frequentist methods that reduce dimensions to just physics parameters e.g. Profile pdf i.e. Contrast Bayes marginalisation Distinguish “profile ordering” See Giovanni Punzi, PHYSTAT05 page 88

51 Talks at FNAL CONFIDENCE LIMITS WORKSHOP (March 2000) by: Gary Feldman Wolfgang Rolke hep-ph/ version 2 Acceptance uncertainty worse than Background uncertainty Limit of C. Lim. as σ  0 Need to check Coverage Lim σ

52 Method: Mixed Frequentist - Bayesian Bayesian for nuisance parameters and Frequentist to extract range Philosophical/aesthetic problems? Highland and Cousins (Motivation was paradoxical behaviour of Poisson limit when LA not known exactly)

53 Bayesian versus Frequentism Basis of method Bayes Theorem --> Posterior probability distribution Uses pdf for data, for fixed parameters Meaning of probability Degree of beliefFrequentist definition Prob of parameters? YesAnathema Needs prior?YesNo Choice of interval? YesYes (except F+C) Data considered Only data you have….+ other possible data Likelihood principle? YesNo Bayesian Frequentist

54 Bayesian versus Frequentism Ensemble of experiment NoYes (but often not explicit) Final statement Posterior probability distribution Parameter values  Data is likely Unphysical/ empty ranges Excluded by priorCan occur SystematicsIntegrate over priorExtend dimensionality of frequentist construction CoverageUnimportantBuilt-in Decision making Yes (uses cost function)Not useful Bayesian Frequentist

55 Bayesianism versus Frequentism “Bayesians address the question everyone is interested in, by using assumptions no-one believes” “Frequentists use impeccable logic to deal with an issue of no interest to anyone”

56 Next time : Discovery and p-values Hope: LHC moves us from era of ‘Upper Limits’ to that of DISCOVERY