How far away is the moon? ( An adventure in signal detection theory and allusive science) Harold E. Brooks NOAA/National Severe Storms Laboratory Norman,

Slides:

Advertisements

Similar presentations

The results of repeated observations and/or experiments concerning a naturally occurring event (phenomenon) are reasonably the same when performed and.

Advertisements

Chapter 4 Pattern Recognition Concepts: Introduction & ROC Analysis.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

10 / 31 Outline Perception workshop groups Signal detection theory Scheduling meetings.

Ludlum Measurements, Inc. User Group Meeting June 22-23, 2009 San Antonio, TX.

Elicited and Emitted Behavior The Reflex: Elicitation Properties of Elicited Behavior Eliciting Stimuli and Response Probabilities Probabilities or Relative.

ROC Statistics for the Lazy Machine Learner in All of Us Bradley Malin Lecture for COS Lab School of Computer Science Carnegie Mellon University 9/22/2005.

Mick McGuire Christina Henderson Kelly Ranae Giedd-James.

PSYCHOPHYSICS What is Psychophysics? Classical Psychophysics Thresholds Signal Detection Theory Psychophysical Laws.

CAP and ROC curves.

Introduction to Biomedical Statistics. Signal Detection Theory What do we actually “detect” when we say we’ve detected something?

Control Charts for Variables

Tornado Detection Algorithm (TDA) By: Jeffrey Curtis and Jessica McLaughlin.

Chapter 2 Simple Comparative Experiments

Face Processing System Presented by: Harvest Jang Group meeting Fall 2002.

PSY 307 – Statistics for the Behavioral Sciences

CSCI 347 / CS 4206: Data Mining Module 06: Evaluation Topic 07: Cost-Sensitive Measures.

Psychophysics 3 Research Methods Fall 2010 Tamás Bőhm.

4IWVM - Tutorial Session - June 2009 Verification of categorical predictands Anna Ghelli ECMWF.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

Sampling Uncertainty in Verification Measures for Binary Deterministic Forecasts Ian Jolliffe and David Stephenson 1EMS September Sampling uncertainty.

Business Statistics, 4e, by Ken Black. © 2003 John Wiley & Sons. 6-1 Business Statistics, 4e by Ken Black Chapter 6 Continuous Distributions.

A Preliminary Verification of the National Hurricane Center’s Tropical Cyclone Wind Probability Forecast Product Jackie Shafer Scitor Corporation Florida.

CpSc 810: Machine Learning Evaluation of Classifier.

Latest results in verification over Poland Katarzyna Starosta, Joanna Linkowska Institute of Meteorology and Water Management, Warsaw 9th COSMO General.

Signal Detection Theory I. Challenges in Measuring Perception II. Introduction to Signal Detection Theory III. Applications of Signal Detection Theory.

The Benefit of Improved GOES Products in the NWS Forecast Offices Greg Mandt National Weather Service Director of the Office of Climate, Water, and Weather.

Behavioral: Basic Psychology of the Senses of the User IST 331 – Organization and Design of Information Systems: User and System Principles Instructor:

5-1 Objective: Solving proportions. A ratio is the comparison of two numbers written as a fraction. An equation in which two ratios are equal is called.

Lecture 2d: Performance Comparison. Quality of Measurement Characteristics of a measurement tool (timer) Accuracy: Absolute difference of a measured value.

Severe Weather: Tornadoes Harold E. Brooks NOAA/National Severe Storms Laboratory Norman, Oklahoma

Radiation Detection and Measurement, JU, First Semester, (Saed Dababneh). 1 Counting Statistics and Error Prediction Poisson Distribution ( p.

GG 313 Lecture 9 Nonparametric Tests 9/22/05. If we cannot assume that our data are at least approximately normally distributed - because there are a.

Chapter 2: Signal Detection and Absolute Judgement

Bayesian Decision Theory Basic Concepts Discriminant Functions The Normal Density ROC Curves.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 5: Credibility: Evaluating What’s Been Learned.

ISE Recall the HIP model. ISE Beyond sensing & perceiving …  You are sitting at lunch and hear a familiar ring tone. Is that your.

Evidence base of clinical diagnosis 邓贝贝张棣王迎. Category of diagnostic test laboratory examination medical history and information from physical examination.

Spatial Smoothing and Multiple Comparisons Correction for Dummies Alexa Morcom, Matthew Brett Acknowledgements.

Bayesian decision theory: A framework for making decisions when uncertainty exit 1 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e.

USER INTERFACE USER INTERFACE January 12, 2006 Intern 박지현 Performance analysis of filtering software using Signal Detection Theory Ashutosh Deshmukh, Balaji.

Chapter 7: The Distribution of Sample Means. Frequency of Scores Scores Frequency.

Statistics for Political Science Levin and Fox Chapter Seven

Short Range Ensemble Prediction System Verification over Greece Petroula Louka, Flora Gofa Hellenic National Meteorological Service.

Outline of Lecture I.Intro to Signal Detection Theory (words) II.Intro to Signal Detection Theory (pictures) III.Applications of Signal Detection Theory.

Gridded warning verification Harold E. Brooks NOAA/National Severe Storms Laboratory Norman, Oklahoma

Psy Psychology of Hearing Psychophysics and Detection Theory Neal Viemeister

Szalma & Hancock HFES Europe, Fuzzy Signal Detection Theory and Human Performance: A Review of Empirical Evidence for Model Validity J.L. Szalma.

SIGNAL DETECTION THEORY  A situation is described in terms of two states of the world: a signal is present ("Signal") a signal is absent ("Noise")  You.

Describing a Score’s Position within a Distribution Lesson 5.

Chapter 6: Descriptive Statistics. Learning Objectives Describe statistical measures used in descriptive statistics Compute measures of central tendency.

Sensitivity, Specificity, and Receiver- Operator Characteristic Curves 10/10/2013.

Scientific Method Vocabulary Observation Hypothesis Prediction Experiment Variable Experimental group Control group Data Correlation Statistics Mean Distribution.

Receiver Operator Characteristics What is it and where does it come from Statistical aspects Use of ROC.

Construction Engineering 221 Probability and statistics Normal Distribution.

Lecture 1.31 Criteria for optimal reception of radio signals.

Verifying and interpreting ensemble products

Chapter 2 Simple Comparative Experiments

Origins of Signal Detection Theory

Information Units of Measurement

Data Mining Classification: Alternative Techniques

Binary Forecasts and Observations

Project Management Risk and Uncertainty Making decisions using ROC (receiver operating characteristic) curves Graham Collins University College London.

Probabilistic forecasts

Quantitative verification of cloud fraction forecasts

ROC Curves Receiver Operating Characteristic (ROC) curves are used to determine the appropriate operating point of a system, the tradeoff between False.

A Neural Network for Detecting and Diagnosing Tornadic Circulations

How do we make decisions about uncertain events?

Business Statistics, 3e by Ken Black

Roc curves By Vittoria Cozza, matr

Presentation transcript:

How far away is the moon? ( An adventure in signal detection theory and allusive science) Harold E. Brooks NOAA/National Severe Storms Laboratory Norman, Oklahoma, USA

Aside on lead time Three classes of tornadoes –Warned before tornado occurs (predict) –Warned after tornado occurs (detect) –The Unwarned

Signal detection theory Classify event as yes or no with uncertain evidence –Threshold for evidence John Swets 50 th anniversary of first work Mason (1982) Aust. Meteorol. Mag.

Weight of evidence Yes No D'

Weight of evidence Yes No D'

2x2 Table a = Correct fore. of events, d = Correct non-event b = False alarm, c = Missed event Probability of detection=a/(a+c) Probability of false detection=b/(b+d) False alarm ratio = b/(a+b) Ob. YesOb. NoSum Fore.Yesaba+b Fore. Nocdb+d Suma+cb+dN

2x2 Table (D'=1,climo = 0.1) Probability of detection=0.33 Probability of false detection=0.07 False alarm ratio = 0.67 Ob. YesOb. NoSum Fore.Yes Fore. No Sum

2x2 Table (D'=1,climo = 0.1) Probability of detection=0.50 Probability of false detection=0.16 False alarm ratio = 0.74 Ob. YesOb. NoSum Fore.Yes Fore. No Sum

2x2 Table (D'=1,climo = 0.1) Probability of detection=0.90 Probability of false detection=0.61 False alarm ratio = 0.86 Ob. YesOb. NoSum Fore.Yes Fore. No Sum

2x2 Table (D'=1,climo = 0.1) Probability of detection=0.006 Probability of false detection= False alarm ratio = 0.25 Ob. YesOb. NoSum Fore.Yes Fore. No Sum

Relative Operating Characteristics

No skill Hypothetical Forecasts Area under curve is measure of system performance

On-line introduction –Claremont Graduate University –

Relative Operating Characteristics (Normal deviate space) D'

MDA and tornadoes

Model the warning process (statistically) Assume Gaussian model of SDT applies –True for many problems (cancer, code interception, credit card fraud, NWP) –Equal variance for simplicity Vary distance (D'), find curves in POD/FAR space that go through current performance D' ~ “science/technology”

How to do it? Need all 4 elements of 2x2 table Need 3 equations (total events=1) –POD (from ROC) –POFD (from ROC) –Fraction of storms considered that are tornadic (f) Vary until POD/FAR curve goes through current performance

Current status Unless number of events considered possibly tornadic is large, FAR=0.25 is unrealistic Understanding/NEXRAD/training/SPC perhaps moved us from D' ~0.5 to D'>1

NWS Tornado Warning History

Comments (1) For current “science”, FARatio less sensitive than POD Era comparisons –1980s possibly pessimistic –1999, 2001 possibly optimistic

Comments (2) Moved D'~ since late-1980s Need to move D'~1 by 2025 Same or slightly faster rate than last 15 years

Apollo Project Political, not scientific, decision Why haven’t we been to Mars?