1. Sound Localization Strategies in Simple and Complex Environments Norbert Kopčo Department of Cybernetics and AI, TU Košice, Slovakia Hearing Research Center and Center for Computational Neuroscience Boston University

2. 2 Overview Intro – Horizontal sound localization Two studies that examine auditory localization in more complex scenes: Experiment 1: Shifts in localization due to contextual factors during sequential stimulus presentation Experiment 2: A priori information and localization of a talker in a multitalker environment

3. Horizontal Sound localization Is based on binaural cues (ITD, ILD) depends on: stimulus type: spectrum, temporal aspects environment: anechoic, reverberant source movement: static, dynamic But also on: presence of other stimuli a priori knowledge / expectations about the scene

4. Effect of additional stimuli The extra sound can act as a: Masker – localization worse (Gilkey & Good, 1986) Adaptor – localization biased (Attraction/Repulsion) (Carlile et al., 1986) Real sound (of which the target is a reflection) – localization worse/suppressed (Litovsky et al., 1999) Perceptual stream of which the target is or is not a part – localization better or worse (Clifton & Freyman, 1997) Cue – localization better (Rhodes, 1987; Sach et al., 2000) Anchor – change localization strategy (Durlach & Braida, 1969)

5. Current Experiments Understand auditory localization in a more complex scenes: Exp 1: - when target is preceded by another identical sound (distractor) from a known location that the listener should ignore - when the task of single target localization is interleaved with the task of localization of a target preceded by a distractor Exp 2: - when localizing a talker in a multi-talker environment - when we know the location of the distractor talkers Goal: Show that listeners change how they interpret the binaural cues depending on the context, strategy, and a priori information

6. Experiment 1: Perceptual and central effects in a click-sound localization with a preceding distractor

7. Preceding distractor: Intro Several preceding studies indicated that preceding stimulus influences localization at SOAs of several hundreds milliseconds (Kopco et al., 2001, Perrott and Pacheco, 1989) Goal: - Characterize this influence (bias and std.dev. in responses) - Determine its cause. Candidates: - short-term adaptation in brainstem representations - reverberation suppression and acoustics - strategy - perceptual organization - attention: focused away from distractor location

8. Hypotheses Peripheral factors will have short-term effects Central factors will influence results at longer separations Effect of reverberation can be separated by comparing performance in anechoic and echoic rooms Effect of perceptual organization can be addressed by modifying the stimuli

9. 9 Methods We measured azimuthal localization performance for a click target stimulus when preceded by another identical click (coming from a known location): - presented with a short onset asynchrony - from a different azimuthal location - in either an anechoic chamber or an ordinary classroom Performance was compared to that in a control in which there was no preceding distractor click.

10. Stimuli and setup

11. Results: Raw data Complex pattern of biases in responses due to preceding distractor

12. Perceived target location of frontal sources is “attracted” by lateral distractors Interactions for SOAs up to 200 ms Possible mechanisms: - Precedence effect - Short-term adaptation - Grouping of identical target and distractor Results: Bias due to Distractor

13. Replacing the single-click distractor by a 5- Hz click-train (such that the target does not fall into the temporal pattern of the distractor) eliminates the “attractive” bias. Effects not due to acoustics because correct representation is available Exp 1 – Perceptual organization

14. Contextual plasticity There is bias also in the no-distractor responses The bias is always away from the non- present distractor Because the runs were interleaved, this bias had to build up anew during each run

15. Contextual plasticity Difference in no-distractor responses in the frontal and lateral distractor context - Is independent of azimuth - Grows over time - Slightly stronger for the 8-click train context Contextual plasticity on time scale of minutes Similar to effects of long-term exposure Either due to bottom-up factors (distribution of stimuli) or top-down factors (focusing away from distractor) Follow-up experiments and analysis to explore the contextual effect Contextual bias

16. Context Exp 1a – Reverberation? Experimental Design: Compare results from room and anechoic chamber. Result: Contextual plasticity equally strong in anechoic and reverberant spaces

17. Context Exp 1b: Effect of timing Experimental Design: Fix Distractor ahead Add no-distractor run Systematically vary: - SOA (25 – 400 ms) - distr/no-distr trial occurance ratio Result: Contextual plasticity - modulated by SOA - modulated by occurance ratio - persists even at longest SOA & lowest dist-trial occurance Occurance frequency does not matter

18. Context Exp 1c: Target-Distr. Order Experimental Design: Use only longest SOA Fix ratio at 75% distractors Reverse the Distractor- Target order Results: Contextual plasticity - persists even for T-D ordering SOA and D-T T-D order does not matter

19. Context Exp 1c: Target-Distr. Order Experimental Design: Use only longest SOA Fix ratio at 75% distractors Reverse the Distractor- Target order Results: Contextual plasticity - persists even for T-D ordering - affects equally dist and no-dist resps - has sharp on- / off- sets Baseline changes over time Is D used as Anchor?

20. Context Exp 1d: Sp. Arrangement Experimental Design: In separate runs, vary which speakers present targets on distractor-trials No-distractor targets come from all locations Result: Contextual plasticity - strongest where distractor-targets are present (+ D location) - weaker if distr targets on both sides D-T reference required for contextual plasticity

21. Context Exp 1d: Abs and Rel Info Questions: Does distractor provide additional info? Do we combine D-T relative and T-only absolute info? Analysis: Correlation coefficient between actual location and responses on no-distr trials Result: Performance consistency improves when distractor- trials are present Similar results for standard deviations (not shown)

22. Exp 1 - Summary Preceding distractor: - Affects perceived target location at SOAs of up to 200 ms - Is likely due to combination of - short-term plasticity, and - perceptual-organization factors Contextual plasticity: - Is modulated by temporal and spatial characteristics of the inducing task - Persists even when SOA is 400 ms or when target precedes distractor - Is likely due to an automatic process that combines the distractor and target localization information, e.g., to provide the optimum target location estimate based on absolute and relative target location - Results in improvement in localization in terms of correlation coefficient, likely because distractor used as anchor for relative localization (automatic change of strategy?)

23. Experiment 2: Localizing a speech target in a multitalker mixture

24. 24 Introduction Spatial separation of sources enhances speech perception In complex environments (e.g., with multiple talkers), spatial perception also important for “sorting” acoustic scene into objects and focusing attention on sources of interest (Brungart et al 2001; Freyman et al 1999; Kidd et al 2005; Best et al 2007; Shinn-Cunningham 2008) Relatively few studies actually measured localization of speech in a multitalker environment (Yost et al., 1996; Hawley et al.1999; Drullman and Bronkhorst 2000; Brungart et al. 2006)

25. 25 Experiment and Goals Study horizontal localization of speech in a multitalker environment Question 1: How does presence of maskers influence localization performance? Evaluate the effect of maskers on RMS errors in localization responses. Separate effect of detection on localization errors. Question 2: Is performance affected by a priori knowledge / uncertainty about distribution of masker locations? Compare performance when masker distribution fixed vs. varied from trial to trial. Hypotheses: 1. Masker location uncertainty will hurt performance. 2. A priori information will eliminate some of the loss, in particular if a simple strategy can be employed to use it.

26. 26 Setup and masker patterns

27. 27 Methods Stimuli: Target: word “two” spoken by a female talker Maskers: 4 different monosyllabic words, spoken by 4 male talkers (all longer than target) Target-to-Masker energy ratios: 0 dB or -5 dB Task:Subjects pointed head to perceived target location Subjects asked to indicate location only if target heard (5 catch trials with no target per block to monitor obedience) Conditions (separate blocks): - Control: No masker - Fixed: Masker pattern fixed across block of trials - Mixed: Masker pattern randomly chosen for each trial

28. 28 Detection Detection worse at lower TMR, similar in both uncertainty conditions

29. 29 Localization: Control Good performance with no maskers All effects of maskers plotted re. control performance on following slides

30. 30 Average across patterns Detrimental effect of maskers is strong, both for fixed and mixed conditions. Averaged across patterns and target locations, a priori knowledge helps slightly, by approximately 20%.

31. 31 Average across patterns When looking only at off-masker locations, a priori knowledge helps dramatically (by 36%)

32. 32 Average across patterns When looking only at on-masker locations, a priori knowledge has no effect (or hurts performance)

33. 33 Interim Summary Presence of maskers hurts performance (H1 confirmed), even after accounting for lower detectability. A priori knowledge of masker locations influences target talker localizability: - Improving performance at locations from which (the subject knows) no masker can come - Not affecting (or worsening) performance at locations from which (the subject knows) maskers will come (H2 partially confirmed) Possible mechanism: - Redistribution of processing resources - “incorrect” strategy: focusing only on off-masker locations Next, analyze patterns separately to gain more insight into behavior re. H2.

34. 34 Raw Data Complex effect of target location, masking pattern, uncertainty and TMR

35. 35 Analysis of Patterns A priori information helps for off-masker targets - in almost all patterns - at both TMRs (more at -5 dB) A priori information can hurt for on-masker targets, mainly for patterns 1 and 2 Overall, effects large for Patts 1 & 2, small for Patt 5. Complexity of pattern limits use of a priori information.

36. 36 Bias due to Maskers Compression strongest for targets near peripheral maskers Leftward   Rightward

37. 37 Masker Uncertainty and Bias When masker pattern fixed throughout a block, responses biased away from maskers Bias due to Masker Uncertainty Left   Right

38. 38 Exp 2: Summary 1. Mixture has complex effects on localization performance - compression of mean localization responses near peripheral maskers - it increases localization errors, even after detection errors are eliminated - effect depends on masker pattern, location of target re. maskers, and TMR 2. A priori information about the distribution of speech maskers modulates the effect of masking: - reducing it (as expected) - but sometimes increasing it (unexpected) 3. These modulatory effects are - likely to be due to change in strategy / assignment of resources: focusing on off-masker locations in fixed condition - most useful when a priori information can be simply applied (simple patterns) - least useful when a priori information cannot be simply applied (complex patterns) 4. Can models like Feller and Merima (2004) or Dietz, Ewert and Hohmann (2010) predict these results? Kopčo N, Best V, Carlile S (2010). Journal of the Acoustical Society of America, 127, 1450-1457

39. 39 Exp 2a: Hearing Impairment Q: Is the effect of complex masker mixture similar for Hearing-Impaired listeners? Methods: As in Exp. 2, but only the Mixed condition. Result: HI listeners performance only affected in mixture (re. normal hearing listeners) Best, Carlile, Kopčo, van Schaik (2011) J of the Acoust Soc of Am, 129, EL210-EL215

40. 40 Overall Summary Previous studies showed that the effect of attention and a priori information on sound localization is small in simple scenarios. Here we show that localization performance does change - even in simple conditions when the listeners can change their response strategy - In complex scenarios (e.g., in mixture of 5 talker): - People use a priori information to optimize their localization performance. - The result is not always an improvement.

41. Beáta Tomoriová, Ľuboš Hládek, Rudolf Andoga Perception and Cognition Lab, TU Košice, Slovakia Gin Best Hearing Research Center, Boston U & Australian National Acoustics Lab Barbara Shinn-Cunningham Hearing Research Center, Boston University Simon Carlile University of Sydney Financial Support: Human Frontiers Science Program, US National Institutes of Health, US National Science Foundation, US National Academy of Sciences, Slovak Science Grant Agency Collaborators and Support

42. Current Projects TU Košice & Boston University: Contextual Plasticity in Sound Localization (US NIH) Boston University, Harvard Medical School/MGH, UC Riverside, University of Edinburgh: Perceptual and cross-modal learning in auditory distance perception (Marie Curie Project, 7FP EU) More info: http://pcl.tuke.sk, http://cns.bu.edu/~kopco