Virginia Image and Video Analysis Pavement conditions, fighter jets, and psychophysics! a.k.a. “Did you hear that???” Andrea Vaccari.

Slides:

Advertisements

Similar presentations

...visualizing classifier performance in R Tobias Sing, Ph.D. (joint work with Oliver Sander) Modeling & Simulation Novartis Pharma AG 3 rd BaselR meeting.

Advertisements

Signal Detection Theory. The classical psychophysicists believed in fixed thresholds Ideally, one would obtain a step-like change from no detection to.

Chapter 4 Pattern Recognition Concepts: Introduction & ROC Analysis.

Statistical Significance and Performance Measures

Learning Algorithm Evaluation

THE DISTRIBUTION OF SAMPLE MEANS How samples can tell us about populations.

Evaluation of segmentation. Example Reference standard & segmentation.

10 / 31 Outline Perception workshop groups Signal detection theory Scheduling meetings.

Research methods in sensation and perception (and the princess and the pea)

From the homework: Distribution of DNA fragments generated by Micrococcal nuclease digestion mean(nucs) = bp median(nucs) = 110 bp sd(nucs+ = 17.3.

Bayes rule P[s(t)|x(t)] = P[x(t)|s(t)] P(s(t)] / P[x(t)] s(t) - incoming stimulus that describes the object x(t) – activity in the system resulting from.

The Ethics of Image Analysis Martin Peterson,TU/e.

Thursday, September 12, 2013 Effect Size, Power, and Exam Review.

The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press 1 Computational Statistics with Application to Bioinformatics Prof. William.

Sensation Overview How is perception different from sensation? What is psychophysics? What do sense organs do? How does vision work? How does this compare.

Assessing and Comparing Classification Algorithms Introduction Resampling and Cross Validation Measuring Error Interval Estimation and Hypothesis Testing.

PSYCHOPHYSICS What is Psychophysics? Classical Psychophysics Thresholds Signal Detection Theory Psychophysical Laws.

Sensation Perception = gathering information from the environment 2 stages: –Sensation = simple sensory experiences and translating physical energy from.

Decision making as a model 2. Statistics and decision making.

Statistics for the Social Sciences

Introduction to Biomedical Statistics. Signal Detection Theory What do we actually “detect” when we say we’ve detected something?

Decision Theory Naïve Bayes ROC Curves

Introduction to Signal Detection

Jeremy Wyatt Thanks to Gavin Brown

Chapter 9.2 ROC Curves How does this relate to logistic regression?

Psychophysics 3 Research Methods Fall 2010 Tamás Bőhm.

Sensation & Perception

SPH 247 Statistical Analysis of Laboratory Data May 19, 2015SPH 247 Statistical Analysis of Laboratory Data1.

Evaluation – next steps

Link Reconstruction from Partial Information Gong Xiaofeng, Li Kun & C. H. Lai

Classification II. 2 Numeric Attributes Numeric attributes can take many values –Creating branches for each value is not ideal The value range is usually.

Performance measurement. Must be careful what performance metric we use For example, say we have a NN classifier with 1 output unit, and we code ‘1 =

Data Analysis 1 Mark Stamp. Topics  Experimental design o Training set, test set, n-fold cross validation, thresholding, imbalance, etc.  Accuracy o.

The Method of Constant Stimuli & Signal Detection Theory VISN2211 Sieu Khuu David Lewis.

Lecture notes for Stat 231: Pattern Recognition and Machine Learning 3. Bayes Decision Theory: Part II. Prof. A.L. Yuille Stat 231. Fall 2004.

Signal detection theory Appendix Takashi Yamauchi Texas A&M University.

Research Design & Analysis 2: Class 23 Announcement re. Extra class: April 10th BAC 237 Discrete Trials Designs: Psychophysics & Signal Detection.

Signal Detection Theory I. Challenges in Measuring Perception II. Introduction to Signal Detection Theory III. Applications of Signal Detection Theory.

Fundamentals of Sensation and Perception EXPLORING PERCEPTION BY STUDYING BEHAVIOUR ERIK CHEVRIER SEPTEMBER 16 TH, 2015.

Bayesian Decision Theory (Classification) 主講人：虞台文.

Education 793 Class Notes Decisions, Error and Power Presentation 8.

Overconfidence in judgment: Why experience might not be a good teacher Tom Stewart September 24, 2007.

Sensation Perception = gathering information from the environment 2 stages: –Sensation = simple sensory experiences and translating physical energy from.

Preventing Overfitting Problem: We don’t want to these algorithms to fit to ``noise’’ Reduced-error pruning : –breaks the samples into a training set and.

ISE Recall the HIP model. ISE Beyond sensing & perceiving …  You are sitting at lunch and hear a familiar ring tone. Is that your.

Psych 480: Fundamentals of Perception and Sensation

Machine Learning Tutorial-2. Recall, Precision, F-measure, Accuracy Ch. 5.

Classification Evaluation. Estimating Future Accuracy Given available data, how can we reliably predict accuracy on future, unseen data? Three basic approaches.

1 Performance Measures for Machine Learning. 2 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall –F –Break Even Point.

Outline of Lecture I.Intro to Signal Detection Theory (words) II.Intro to Signal Detection Theory (pictures) III.Applications of Signal Detection Theory.

Psy Psychology of Hearing Psychophysics and Detection Theory Neal Viemeister

Performance Indices for Binary Classification 張智星 (Roger Jang) 多媒體資訊檢索實驗室台灣大學資訊工程系.

Signal Detection Theory.

Power of a Test. Remember: Type I error – Reject the H 0 when you shouldn’t have. Type II error - Fail to reject the H o when you should have.

SIGNAL DETECTION THEORY  A situation is described in terms of two states of the world: a signal is present ("Signal") a signal is absent ("Noise")  You.

Psychophysical theories Signal detection theory: A psychophysical theory that quantifies the response of an observer to the presentation of a signal in.

Receiver Operator Characteristics What is it and where does it come from Statistical aspects Use of ROC.

Lecture 1.31 Criteria for optimal reception of radio signals.

Statistics for the Social Sciences

Measuring Success in Prediction

Information Units of Measurement

LECTURE 05: THRESHOLD DECODING

Chapter 4 Section 2.

Power of a Test.

How do we make decisions about uncertain events?

Inference Concepts Hypothesis Testing.

LECTURE 05: THRESHOLD DECODING

Roc curves By Vittoria Cozza, matr

Given that {image} {image} Evaluate the limit: {image} Choose the correct answer from the following:

Presentation transcript:

Virginia Image and Video Analysis Pavement conditions, fighter jets, and psychophysics! a.k.a. “Did you hear that???” Andrea Vaccari

Virginia Image and Video Analysis Detection theory History: -Introduced in the mid-50s in RADAR field. -It’s a plane! - No, it’s a bird! - No, it’s superman! -Around the same time in psychology. -Mid-60s introduced in psychophysics to deal with sensitivity vs. bias Slide 2

Virginia Image and Video Analysis Detection theory Applications: -Detection (It is there, isn’t it?) -Discrimination (Is VIVA coffee sweeter than BoJo?) -Receiver evaluation (You can’t tell the difference, can you?) -Test evaluation (My algorithm rocks!... NOT!) -Product evaluation (You like my webpage, don’t you?) -… you name it! Slide 3

Virginia Image and Video Analysis Stimulus-response matrix Binary type experiments: -either correct or incorrect (2 classes) Four options: -Hit -Miss (Type II error) -False alarm (Type I error) -Correct rejection Slide 4 Correct rejection False alarm Type I error Miss Type II error Hit Stimulus Response No Yes Yes No

Virginia Image and Video Analysis Can you hear that? Slide 5 Probability Neural activity What’s this noise?? Standard deviation (σ)

Virginia Image and Video Analysis Can you hear that? Slide 6 Probability Neural activity D

Virginia Image and Video Analysis Can you hear that? Slide 7 Probability Neural activity D

Virginia Image and Video Analysis Can you hear that? Slide 8 Probability Neural activity How difficult is it, really? Give me a number! Discriminability, sensitivity (d’)D

Virginia Image and Video Analysis You have to choose! Slide 9 Probability Neural activity No! Yes!

Virginia Image and Video Analysis You have to choose! Slide 10 Probability Neural activity No! Yes! Stimulus Response No Yes Yes No Correct rejection False alarm Type I error

Virginia Image and Video Analysis You have to choose! Slide 11 Probability Neural activity No! Yes! Stimulus Response No Yes Yes No Miss Type II error Hit

Virginia Image and Video Analysis I’m Swiss! Neutral bias! Slide 12 Probability Neural activity No! Yes! Stimulus Response No Yes Yes No Miss Type II error Hit Correct rejection False alarm Type I error Correct rejection False alarm Type I error Miss Type II error Hit

Virginia Image and Video Analysis Are you liberal or conservative? Slide 13 Liberal bias! Probability Neural activity No! Yes! Conservative bias! How biased are you, really? Give me a number! Criterion, bias (β) Correct rejection False alarm Type I error Miss Type II error Hit

Virginia Image and Video Analysis What if? Slide 14 Manipulate biases a.k.a. payoffs! -I give you $10 for each hit! -I give you $5 for each correct rejection! -You pay me $1 for each error! (just because) Now we can actually define an optimum criterion! This also works in real life without artificial payoffs: think RADAR detection of incoming bombers!

Virginia Image and Video Analysis Terminology Slide 15 Sensitivity or Recall: TPR = TP/P = TP/(TP + FN) Specificity: SPC = TN/N = TN/(TN + FP) Fall-out, False Positive Rate: FPR = FP/N = FP/(TN + FP) = 1 - SPC Correct rejection False alarm Type I error Miss Type II error Hit Stimulus Response No Yes Yes No True Negatives False Positives False Negatives True Positives Stimulus Response No Yes Yes No

Virginia Image and Video Analysis So, how good are you? Slide 16 Receiver Operator Curve: TPR = TP/(TP + FN) FPR = FP/(TN + FP) TPR FPR = 1 - SPC

Virginia Image and Video Analysis How do you choose? Slide 17 Probability Neural activity (y) No! Yes! p(y|H 1 ) p(y|H 2 ) p(H 2 |y 0 ) > p(H 1 |y 0 ) Priori: p(H 2 ) = π 2, p(H 1 ) = π 1 p(H i |y 0 ) = p(y 0 |H i )π i / p(y 0 )p(y 0 ) = p(y 0 |H 1 )π 1 + p(y 0 |H 2 )π 2 p(y 0 |H 2 )/p(y 0 |H 1 ) ≥ π 1 /π 2 L(y 0 ) ≥ τ MAP L(y 0 ) => Likelihood ratio τ MAP => Max a posteriori

Virginia Image and Video Analysis How do you choose? Slide 18 If you consider payoffs: Correct rejection False alarm Type I error Miss Type II error Hit Stimulus Response No Yes Yes No U 22 U 21 U 12 U 11 Stimulus Response No Yes Yes No L(y 0 ) ≥ π 1 (U 11 – U 21 )/π 2 (U 22 – U 12 ) L(y 0 ) ≥ τ B