1. Sensory recruitment during visual Working Memory John Serences Department of Psychology University of California, San Diego

2. Courtney et al., Cereb Cortex, 1996 How are visual details stored in working memory?

3. How many visual objects can we remember at one time? Estimating Storage Capacity Luck and Vogel, 1997

4. How many items can people remember? capacity (K) is about 3… but large individual differences K=(HR-FA)*N, Pashler 1988; Cowan, 2001

5. Memory Capacity (K) Frequency N=170 Mean=2.88 SD=1.04 Individual differences in Visual Working Memory Capacity Reliability: Retest r =.82

6. Why should we care? Working memory capacity predicts: – General measures of intelligence Stanford-Binet IQ (r=.52) Raven’s Progressive Matrices (r=.41-.48) – Scholastic achievement – Efficiency of attentional control Cowan et al., Cognitive Psychology, Mem & Cog Fukuda, Vogel, Mayr & Awh; Psych Bull and Rev (2010) Vogel, E. K., McCollough, A. W., & Machizawa, M. G. (2005)

7. Overview 1) Sensory recruitment during WM 2) Spatially global WM signals 3) Sensory recruitment and WM precision

8. The Sensory Recruitment Hypothesis V1 columns Details are represented in WM via the recruitment of the same neural mechanisms that encode the stored information. This makes a clear prediction: The delay activity observed in these sensory regions should contain stimulus-specific information that represents the stored items.

9. Sustained fMRI amplitude increases are taken as signature of WM Ranganath et al. 2004 Offen et al. 2009 Primary visual cortex (V1)

10. Sample: 1sDelay: 10sTest: 1sITI: 10s Blocked format: remember color or orientation. – Stimulus: 45° or 135° (±5°), Red or Green (variable hue) Performance titrated to 75% on each task Can we predict the remembered feature (45° or 135° / Red or Green) of the sample object based on activity in early visual regions during the delay period?

11. Assessing feature-selective responses in visual cortex Freeman et al., 2011

12. Measuring feature-selective responses in visual cortex <1mm V1 columns ~27mm 3 fMRI voxel Kamitani & Tong, 2005, Nature Neuroscience Haynes & Rees, 2005, Nature Neuroscience Haxby et al., 2001, Science fMRI voxel ~27mm 3

13. Measuring motion selectivity in MT Voxels in MT

14. Voxels in V1 Response

15. Multivoxel Pattern Analysis Test image from scan 8 Mean image for remember 45  scans 1-7 Mean image for remember 135  scans 1-7

16. Functional Localizer Purpose: independently identify most responsive voxels from each sensory region Attend color or orientation on each 10s trial Task: Detect 2-3 oddballs/trial

17. Stimulus-specific patterns of activity represented only the relevant feature during the delay period Classify OrientationClassify Color Remember Orientation Remember Color.5.45.4.35.55.6.65.7 Classification Accuracy 4-10s post-sample Serences, Ester, Vogel, Awh; Psych Science, 2009

18. but not when active discrimination was required without a storage requirement… Classify OrientationClassify Color Remember Orientation Remember Color.5.45.4.35.55.6.65.7 Classification Accuracy 4-10s post-test Serences, Ester, Vogel, Awh; Psych Science, 2009

19. Sensory recruitment also predicts that WM maintains reasonable “copies” of sensory events Can we discriminate remembered attributes using data from a sensory task (the functional localizer)?

20. Post-SamplePost-Test.5.45.4.35.55.6.65.7 Classification Accuracy Yes! Training the classifier using data from a sensory encoding condition (localizer task) enabled reliable classification of stored stimulus values. Serences, Ester, Vogel, Awh; Psych Science, 2009

21. The neural region (V1) that contained these stimulus-specific patterns showed no evidence of increases in the mean amplitude of BOLD responses. Serences, Ester, Vogel, Awh; Psych Science, 2009

22. Sustained amplitude increases Ranganath et al. 2004

23. Summary experiment 1 Feature-selective activation patterns in V1 during delay period predict contents of WM Visual areas that encode sensory information are recruited during maintenance Happens in the absence of sustained amplitude changes – usually taken as the hallmark of WM

24. Overview 1) Sensory recruitment during WM 2) Spatially global WM (and attention) signals 3) Sensory recruitment and WM precision

25. Feature based attention

26. Blank: 6s Experimental design Stimulus: 14s Task: Detect brief slowing of attended dots Eye tracking in subset of observers

27. Visual Pathway Each visual field is initially represented in contralateral visual cortex

28. Functional Localizer 12s 12s fixation Time

29. Contralateral > Ipsilateral Frontal Eye Field (FEF) Intraparietal sulcus (IPS) Occipital visual cortex anterior posterior

30. Feature-selective modulations: 45  vs. 135  hMT+

31. V1V2vV3vV4vV3aMTIPSFEF 0.45 0.5 0.55 0.6 0.65 0.7 0.75 n=10, p<.05 in all areas Visual Area

32. Does feature-based attention spread to stimuli outside the focus of spatial attention? hMT+

33. 0.45 0.5 0.55 0.6 0.65 0.7 0.75 Visual Area *p<.05 * V1V2vV3vV4vV3aMTIPSFEF * * *

34. Does feature-based attention spread to unstimulated regions? hMT+

35. 0.45 0.5 0.55 0.6 0.65 0.7 0.75 Visual Area *p<.05 * * * * * * V1V2vV3vV4vV3aMTIPSFEF

36. Feature-based attention Spatially global modulation: spreads across entire visual field Know what, but not where Serences and Boynton, Neuron, 2007; see Treue and Martinez-Trujillo, 1999; Cohen and Maunsell, Neuron, 2011

37. Are stimulus-specific patterns related to WM also spatially global? Sensory recruitment: Attentional modulations during a perceptual task are global Ester, Serences, Awh, J. Neuroscience

38. Functional Localizer

39. The stored orientation was represented in both contralateral and ipsilateral regions of primary visual cortex, suggesting a spatially global mnemonic representation. Based on data from 6s-16s post-sample Receptive field of cortical ROI

40. Activation patterns during a perceptual task with the same stimuli were similar to those observed during storage in working memory. Receptive field of cortical ROI Location of Localizer Location of WM

41. Sustained increases in response amplitude may not be diagnostic of whether a given region participates in VWM maintenance

42. Conclusions Sensory recruitment during WM is spatially global Global patterns of activation observed during WM maintenance are similar to those observed during perception Ester, Serences, Awh, J. Neuroscience

43. Overview 1) Sensory recruitment during WM 2) Spatially global WM signals 3) Sensory recruitment and WM precision (does any of this actually matter???)

44. Sample: 1sDelay: 12sTest: 3s Experimental Procedure: estimating precision of WM representations using behavior and BOLD Subject rotates probe to match remembered sample orientation

45. Mean response error Orientation Bin %Responses

46. Encoding Model Orientation tuning model Orientation Brouwer & Heeger, 2009, 2011; Freeman and Adelson, 1992 Voxel responses x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x7x7 Using ‘training data’, voxel responses (R) are mapped to responses in the channels (X) by weight matrix (W): R = WX Ŵ = RX T (XX T ) -1 X = (Ŵ T Ŵ) -1 Ŵ T R test Using held out ‘test’ data, channel responses computed:

47. Channel response estimation Training Phase (weight estimation) Given weights and test data, what is the response in each channel? Response pattern from trial in ‘test’ set

48. Shift to common center 0°90°-90° Orientation of the sample stimulus Serences et al., 2009; Brouwer and Heeger, 2011 Constructing orientation selective response profiles in visual cortex

49. Channel offset from sample BOLD Response Orderly channel tuning functions are observed during delay period Fit tuning curve from each subject to estimate bandwidth and amplitude Amplitude BW

50. Mean Recall Error TF Bandwidth (deg) Tuning function bandwidth is tightly coupled with the precision of subjects’ responses Ester, Anderson, Serences, & Awh

51. TF Bandwidth (deg) Recall Bandwidth

52. Mean Recall Error TF Amplitude There was no relationship between tuning function amplitude and task performance

53. Conclusions Precision of responses correlated with precision of neural representation No relationship between behavior and BOLD amplitude – argues against general arousal account

54. General Conclusions Areas that are involved in sensory processing are recruited during WM maintenance Highly selective for relevant information

55. General Conclusions Spatially global spread of activation during WM Spatial invariance of WM for features?

56. General Conclusions Precision of representations in V1 predicts precision of WM This stuff might actually might be important! TF Bandwidth (deg)

57. Thanks Edward Ester, Edward Awh + Edward Vogel (U. Oregon) Geoff Boynton (U. Washington) vogel awh boynton ester Supported by R01-MH087214