Examining the balance between optimising an analysis for best limit setting and best discovery potential Gary C. Hill, Jessica Hodges, Brennan Hughey,

Slides:



Advertisements
Similar presentations
Trigger issues for KM3NeT the large scale underwater neutrino telescope the project objectives design aspects from the KM3NeT TDR trigger issues outlook.
Advertisements

Update on diffuse extraterrestrial neutrino flux search with 2000 AMANDA-II data Jessica Hodges, Gary Hill, Jodi Cooley This version of the presentation.
ICECUBE & Limits on neutrino emission from gamma-ray bursts IceCube collaboration Journal Club talk Alex Fry.
JNM Dec Annecy, France The High Resolution Fly’s Eye John Matthews University of Utah Department of Physics and High Energy Astrophysics Institute.
Combined Energy Spectra of Flux and Anisotropy Identifying Anisotropic Source Populations of Gamma-rays or Neutrinos Sheldon Campbell The Ohio State University.
London Collaboration Meeting September 29, 2005 Search for a Diffuse Flux of Muon Neutrinos using AMANDA-II Data from Jessica Hodges University.
High-energy particle acceleration in the shell of a supernova remnant F.A. Aharonian et al (the HESS Collaboration) Nature 432, 75 (2004) Nuclear Physics.
8. Statistical tests 8.1 Hypotheses K. Desch – Statistical methods of data analysis SS10 Frequent problem: Decision making based on statistical information.
EHE Neutrino Search with the IceCube Aya Ishihara The University of Wisconsin - Madison for the EHE Verification working group.
The presence of the South Pole Air Shower Experiment (SPASE) on the surface provides a set of externally tagged muon bundles that can be measured by AMANDA.
Sean Grullon For the IceCube Collaboration Searching for High Energy Diffuse Astrophysical Neutrinos with IceCube TeV Particle Astrophysics 2009 Stanford.
M. Kowalski Search for Neutrino-Induced Cascades in AMANDA II Marek Kowalski DESY-Zeuthen Workshop on Ultra High Energy Neutrino Telescopes Chiba,
A Search for Point Sources of High Energy Neutrinos with AMANDA-B10 Scott Young, for the AMANDA collaboration UC-Irvine PhD Thesis:
Energy Reconstruction Algorithms for the ANTARES Neutrino Telescope J.D. Zornoza 1, A. Romeyer 2, R. Bruijn 3 on Behalf of the ANTARES Collaboration 1.
Gary C. Hill, CCAPP Symposium 2009, Ohio State University, October 12th, 2009 Photograph: Forest Banks Gary C. Hill University of Wisconsin, Madison for.
Sensitivity of searches for new signals and its optimization
Statistical Analysis of Systematic Errors and Small Signals Reinhard Schwienhorst University of Minnesota 10/26/99.
EHE Search for EHE neutrinos with the IceCube detector Aya Ishihara for the IceCube collaboration Chiba University.
IceCube 40Point Source AnalysisResultsConclusions Search for neutrino point sources with the IceCube Neutrino Observatory Menlo Park, California TeVPA.
Astrophysics working group - CERN March, 2004 Point source searches, Aart Heijboer 1 Point Source Searches with ANTARES Outline: reconstruction news event.
Neutrino Point Source Searches with IceCube 22 String Configuration Michael Baker, for the IceCube Collaboration University of Wisconsin, Madison APS April.
Ronald Bruijn – 10 th APP Symposium Antares results and status Ronald Bruijn.
A statistical test for point source searches - Aart Heijboer - AWG - Cern june 2002 A statistical test for point source searches Aart Heijboer contents:
T.N. Ukwatta et al "Fermi Sensitivity to PBH Bursts" MG12 Paris July SENSITIVITY OF THE FERMI DETECTORS TO GAMMA-RAY BURSTS FROM EVAPORATING.
Point Source Search with 2007 & 2008 data Claudio Bogazzi AWG videconference 03 / 09 / 2010.
March 02, Shahid Hussain for the ICECUBE collaboration University of Delaware, USA.
Gus Sinnis Asilomar Meeting 11/16/2003 The Next Generation All-Sky VHE Gamma-Ray Telescope.
The ANTARES neutrino telescope is located on the bottom of the Mediterranean Sea, 40 km off the French coast. The detector is installed at a depth of 2.5.
AMANDA. Latest Results of AMANDA Wolfgang Rhode Universität Dortmund Universität Wuppertal for the AMANDA Collaboration.
AMANDA Per Olof Hulth The Wierdest wonder Is it good or is it bad?
Point source analysis: candidate source list, extended sources and energy Claudio Bogazzi AWG Videoconference 13/04/2011.
Status and Results Elisa Bernardini DESY Zeuthen, Germany VLVnT Workshop Amsterdam, Oct (
Searching for a Diffuse Flux of Neutrinos with AMANDA-II Jessica Hodges November 5, 2004 Prelim Exam.
1 Methods of Experimental Particle Physics Alexei Safonov Lecture #24.
Kirsten Münich University of Dortmund, Germany Analysis strategies and recent results from AMANDA-II.
High-energy gammas from the giant flare of SGR of December 2004 in AMANDA Juande D. Zornoza on behalf of the IceCube.
Longitude Latitude A recent analysis of the Fermi-LAT data by M. Su et al. [1] revealed two large spherical structures centered by our Galactic Center.
POINT SOURCE ANALYSIS WITH 5-LINE DATA SUMMARY AWG PHONE CALL Juan Pablo Gómez-Gozález 20 th April
Sebastian Kuch, Rezo Shanidze Preliminary Studies of the KM3NeT Physics Sensitivity KM3NeT Collaboration Meeting Pylos, Greece, April 2007.
Alexander Kappes (E. Strahler, P. Roth) ECAP, Universität Erlangen-Nürnberg for the IceCube Collaboration 2009 Int. Cosmic Ray Conf., Łódź,
2005 Unbinned Point Source Analysis Update Jim Braun IceCube Fall 2006 Collaboration Meeting.
A search for neutrinos from long-duration GRBs with the ANTARES underwater neutrino telescope arxiv C.W. James for the ANTARES collaboration.
In Bayesian theory, a test statistics can be defined by taking the ratio of the Bayes factors for the two hypotheses: The ratio measures the probability.
08/2006Julia Becker, Universität Dortmund TeV particle astrophysics, Madison Julia Becker for the IceCube collaboration August 2006 by the AMANDA experiment.
A New Upper Limit for the Tau-Neutrino Magnetic Moment Reinhard Schwienhorst      ee ee
AMANDA Per Olof Hulth The Wierdest wonder Is it good or is it bad?
Prospects of Identifying the Sources of the Galactic Cosmic Rays with IceCube Alexander Kappes Francis Halzen Aongus O’Murchadha University Wisconsin-Madison.
Confidence Intervals and Hypothesis Testing Mark Dancox Public Health Intelligence Course – Day 3.
Search for Ultra-High Energy Tau Neutrinos in IceCube Dawn Williams University of Alabama For the IceCube Collaboration The 12 th International Workshop.
First All-Sky Measurement of Muon Flux with IceCube IceCube REU Summer 2008 Kristin Rosenau Advisor: Teresa Montaruli.
Downgoing Muons in the IceCube experiment: Final presentation for Phys 735, Particle, Prof. Sridhara Dasu L.Gladstone 2008 Dec 3.
Imaging the Neutrino Universe with AMANDA and IceCube
Muons in IceCube PRELIMINARY
Jessica Hodges University of Wisconsin – Madison
Julia Becker for the IceCube collaboration
Recent Results of Point Source Searches with the IceCube Neutrino Telescope Lake Louise Winter Institute 2009 Erik Strahler University of Wisconsin-Madison.
John Kelley for the IceCube Collaboration
Brennan Hughey for the IceCube Collaboration
Erik Strahler UW-Madison 4/27/2008
Search for point-like source in ANTARES
Point Sources Jacob Feintzeig WIPAC − May 21, 2014
J. Braun, A. Karle, T. Montaruli
J. Braun, A. Karle, T. Montaruli
MC studies of the KM3NeT physics performance Rezo Shanidze
Fluxes and Event rates in KM3NeT detectors
Brennan Hughey for the IceCube Collaboration
2000 Diffuse Analysis Jessica Hodges, Gary Hill, Jodi Cooley
Claudio Bogazzi * - NIKHEF Amsterdam ICRC 2011 – Beijing 13/08/2011
Time-Dependent Searches for Neutrino Point Sources with IceCube
University of Wisconsin-Madison
Presentation transcript:

Examining the balance between optimising an analysis for best limit setting and best discovery potential Gary C. Hill, Jessica Hodges, Brennan Hughey, Mike Stamatikos PHYSTAT 2005, Oxford University of Wisconsin, Madison

The problem? What is an optimal analysis? Do we go for best sensitivity (lowest upper limit) or for best discovery potential (high deviation away from background expectation)? How do we optimise for each? How does choosing one optimisation affect the potential of the other?

Sensitivity Sensitivity – choose cuts so that the average limit on the physics quantity is minimised “Model Rejection Potential” Gary C. Hill and Katherine Rawlins, Astroparticle Physics (2003) Standard optimisation for AMANDA and IceCube

Assessing sensitivity Choose cuts – leaves the signal region, where we will look for an effect above background

Limit on the flux constant A

Limit on a cross section

OPTIMISATION INDEPENDENT OF ASSUMED SOURCE STRENGTH!

Example : sensitivity of IceCube to diffuse extraterrestrial neutrinos

What about discovery potential? Suppose one was interested in discovering something and not just setting best limits? Would the optimisation be different? What would one optimise?

Characterising discovery * A “discovery” is a rejection of the background only hypothesis at a very high confidence level Of course, a small chance probability does not prove the truth of some alternative hypothesis - unlikely data under a background hypothesis doesn’t mean that some alternative hypothesis is true… P(n|H0) small doesn’t mean P(H1|n) is non-zero!

* To give credit where it is due! When I submitted this abstract, Louis Lyons mentioned that there was a paper from Giovanni Punzi, PHYSTAT2003, on this question I had been at that talk and glanced at the paper a long time ago… but obviously things didn’t sink in totally… Upon reading again, I realised that G.P. had come up with the same basic concept of a “least detectable flux” that one chooses cuts to minimise, that I will talk about and use here

Characterising discovery * A “discovery” is a rejection of the background only hypothesis at a very high confidence level Of course, a small chance probability does not prove the truth of some alternative hypothesis - unlikely data under a background hypothesis doesn’t mean that some alternative hypothesis is true… P(n|H0) small doesn’t mean P(H1|n) is non-zero!

Chance probabilities for different sigmas John Kelley, UW Madison, with Mathematica

How likely are we to see such low chance probabilities if there really is a signal?

Least detectable signal

Example

Least detectable signal vs background Solid lines: average upper limit curves, 68, 90, 95, 99%

MRF → MDF Least detectable signal curves replace average upper limit curves Compare minima for limits and discovery

Astrophysical point source search

Radiation Enveloping Black Hole p +  -> n +  + ~ cosmic ray + neutrino -> p +  0 ~ cosmic ray + gamma NEUTRINO BEAMS: HEAVEN & EARTH Neutrino Beams: Heaven & Earth

Atmospheric Muons & Neutrinos “Atmospheric muons” from cosmic ray showers, penetrating to the detector from above “Atmospheric neutrinos” from the same air showers, forming a diffuse background and calibration beam Astrophysical neutrinos: interesting signal 10 2 Hz atm.  atm Hz astrophysical cosmic ray

Amundsen-Scott South Pole station South Pole Dome Summer camp AMANDA road to work 1500 m 2000 m [not to scale]

Astrophysical point source search Know sky position of the source from other wavelength observations. e.g. VHE gamma-ray people look where the x-ray people saw something, neutrino people look where the gamma- ray and/or x-ray people were successful

Reconstruction angular resolution: point spread function Reconstruction response: Gaussian in angle away from true arrival direction differential differential, weighted with solid angle Gaussian signal uniform background

integrated signal and background signal background LDS, 5-σ aul 90%

Curves of MRF, MDF MDF MRF Solid : best dashed : best MRF dotted : best MDF

bg = inside 1 degree

bg = 0.1 inside 1 degree

bg = 1 inside 1 degree

bg = 10 inside 1 degree

bg = 30 inside 1 degree

best MDF/MRF angular cut vs log10 background both MRF and MDF cut converge toward best  b/s cut – 1.58 times resolution – historical “standard” choice of astrophysics community

Comparison of the MDF/MRF performance Difference between best MRF cut and best MDF cut For larger backgrounds, don’t lose much on MRF by choosing to optimise for MDF and vice versa. At very low backgrounds, choose more carefully (discrete Poisson strikes again!)

More to check? Different confidence levels –how do optimisations vary with confidence levels? –should we use consistent levels in limit and discovery optimisation? Different power settings –independent of power in some cases –not true generally –is there a general power rule for MDP/MRP? Punzi method: use power corresponding to limit you would set if no discovery, i.e. 1-β = 1-α

Search for neutrinos from a gamma-ray burst Mike Stamatikos, UW Madison Presentation at 29 th International Cosmic Ray Conference, Pune, India, August 2005

Conclusions Model rejection/discovery* potential –independent of assumed strength of signals! Both discovery and limit optimisations converge to a simple  b/s form for large background expectations For small backgrounds, look at MRF and MDF curves and decide what (mild?) tradeoffs you will accept in your analysis between limit and discovery potential * Also developed by Giovanni Punzi, PHYSTAT 2003