1. Psychophysics 3 Research Methods Fall 2010 Tamás Bőhm

2. Signal detection theory Aka. sensory decision theory (SDT) A model & a data analysis method for decision problems with uncertainty (noise) Originates from World War II: aircraft detection on radar signals Today: widely used in psychophysics, medicine, radiology and machine learning

3. Signal detection theory Experiment setup: –In some trials a stimulus (signal) is presented, in others there is no stimulus; –Observer reports if she/he saw a signal or not –Calculate how many times the observer detected a signal when she/he was presented one (hit rate) Is the hit rate all we want to know? Two observers achieved the same hit rate. Are they certainly behaving the same way? NO, we also need to know how many times the observer said “I see” when there was no signal (false alarm rate)

4. Signal detection theory Confusion matrix: contains all the information about the observer’s performance

5. Signal detection theory Confusion matrix: contains all the information about the observer’s performance As columns add up to 100%, it is enough to know one item from each column 40 trials 20 18 2 6 14 = 100% = 90% = 10%= 70% = 30%

6. Signal detection theory Perfect detection: 100% 0%

7. Signal detection theory No detection at all (1st example): always reporting “Seen” 100% 0% 100%

8. Signal detection theory No detection (2nd example): always reporting “Not seen” 0% 100% 0%

9. Signal detection theory No detection (3rd example): flipping a coin 50%

10. Signal detection theory No detection (4th example): reporting “Seen” in 30% of the trials (no matter what is presented) 30% 70% 30%= = Rows equal  no detection

11. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100%

12. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 90%30% 10%70%

13. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 0% 100% Perfect detection

14. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 0% No detection: always “yes”

15. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 0% 100% No detection: always “no”

16. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 50% No detection: reporting “yes” in 50% of the trials (flipping a coin)

17. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 40% 60% No detection: reporting “yes” in 40% of the trials

18. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 30% 70% No detection: reporting “yes” in 30% of the trials

19. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% 60% 40% No detection: reporting “yes” in 60% of the trials

20. Signal detection theory Receiver operating characteristic (ROC): false alarm rate hit rate100% Diagonal: no detection

21. Signal detection theory SDT model: No way to remove the noise But sensation can be separated from decision by using ROCs Sensation Noise Decision Signal present /absent Sensation level (SL) SL ≥ β Criterion (β) SL < β YES NO

22. Signal detection theory Sensation (Noise) Decision Signal present /absent Sensation level (SL) SL ≥ β Criterion ( β ) SL < β YES NO sensation level probability Without noise: perfect detection is possible criterion signal present signal absent

23. Signal detection theory Sensation (Noise) Decision Signal present /absent Sensation level (SL) SL ≥ β Criterion ( β ) SL < β YES NO sensation level probability criterion signal present signal absent 100%0% 100%

24. Signal detection theory Sensation Noise Decision Signal present /absent Sensation level (SL) SL ≥ β Criterion ( β ) SL < β YES NO sensation level probability Noise: smears the distributions  perfect detection is impossible (if the two distributions overlap) signal absent (noise only) signal present (signal+noise) criterion online demo

25. Signal detection theory Sensation level http://www-psych.stanford.edu/~lera/psych115s/notes/signal/

26. Signal detection theory Sensation level false alarm rate hit rate

27. Signal detection theory false alarm rate hit rate ROC curve β = 8 β = 6 β = 10 β = 6 β = 8 β = 10

28. Signal detection theory false alarm rate hit rate β sensation level probability Criterion (β): specifies where we are on the ROC curve The ROC curve is specified by sensory capacities only (discriminability)

29. Signal detection theory Discriminability: how well the observer can separate the presence of signal from its absence ~ overlap between the two distributions ~ bowing out of the ROC curve Measured by d’ (discriminability index, also called sensitivity) http://www-psych.stanford.edu/~lera/psych115s/notes/signal/

30. Signal detection theory d’: selects the ROC curve β: specifies a point on the selected ROC curve  same information as hit rate & false alarm rate, but: hit rate, false alarm rate: both reflect sensation & decision characteristics; cannot separate the two d’: depends only on sensation β: depends only on decision β The two processes are separated http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.ROC.html

31. Signal detection theory Fechner’s methods: Is a stimulus detectable? Yes or no? Clear-cut threshold value (with some variability) that can be measured –Stimulus intensity > threshold  detectable –Stimulus intensity < threshold  not detectable Dichotic outcome, categorical model Signal detection theory: How well is it detectable? How sensitive the observer is to the stimulus? Measured by d’ –The higher d’ is, the more the stimulus is detectable –d’ = 0  not detectable at all Scalar outcome, dimensional model

32. Signal detection theory Sensation (Noise) Stimulus Sensation level (SL) Different task Correct Incorrect Forced-choice: eliminates the criterion SDT: separates the criterion Decision SL ≥ β Criterion ( β ) SL < β YES NO Problem with Fechner’s methods: criterion

33. Signal detection theory Psychophysical measurements with SDT: 1.Create a stimulus set with a range of intensities (like in the method of constant stimuli) 2.Test each stimulus many times with each observer 3.On each trial, either present a randomly selected stimulus or do not present anything 4.Ask the observer if he/she detected the stimulus 5.Calculate the hit rate and false alarm rate for each observer, for each stimulus intensity 6.Use the formula/table to calculate d’ for each case 7.Examine how d’ changes with intensity: the higher d’ is for a stimulus intensity, the greater the observer’s ability to detect this intensity http://psych.hanover.edu/JavaTest/Media/Chapter2/MedFig.SignalDetection.html

34. Signal detection theory Main results: changes in d’ values Caudek–Rubin Vision Res. 2001

35. Signal detection theory There is also a β value for each d’ value It can be informative about the decision behavior: –Balanced: false alarm and miss rates are equal –Liberal: the observer says “yes” whenever there may be a signal –Conservative: decision is yes only when it is almost certain that there is a signal sensation level probability balanced conservative liberal