Sensory recruitment during visual Working Memory John Serences Department of Psychology University of California, San Diego

Courtney et al., Cereb Cortex, 1996 How are visual details stored in working memory?

How many visual objects can we remember at one time? Estimating Storage Capacity Luck and Vogel, 1997

How many items can people remember? capacity (K) is about 3… but large individual differences K=(HR-FA)*N, Pashler 1988; Cowan, 2001

Memory Capacity (K) Frequency N=170 Mean=2.88 SD=1.04 Individual differences in Visual Working Memory Capacity Reliability: Retest r =.82

Why should we care? Working memory capacity predicts: – General measures of intelligence Stanford-Binet IQ (r=.52) Raven’s Progressive Matrices (r= ) – Scholastic achievement – Efficiency of attentional control Cowan et al., Cognitive Psychology, Mem & Cog Fukuda, Vogel, Mayr & Awh; Psych Bull and Rev (2010) Vogel, E. K., McCollough, A. W., & Machizawa, M. G. (2005)

Overview 1) Sensory recruitment during WM 2) Spatially global WM signals 3) Sensory recruitment and WM precision

The Sensory Recruitment Hypothesis V1 columns Details are represented in WM via the recruitment of the same neural mechanisms that encode the stored information. This makes a clear prediction: The delay activity observed in these sensory regions should contain stimulus-specific information that represents the stored items.

Sustained fMRI amplitude increases are taken as signature of WM Ranganath et al Offen et al Primary visual cortex (V1)

Sample: 1sDelay: 10sTest: 1sITI: 10s Blocked format: remember color or orientation. – Stimulus: 45° or 135° (±5°), Red or Green (variable hue) Performance titrated to 75% on each task Can we predict the remembered feature (45° or 135° / Red or Green) of the sample object based on activity in early visual regions during the delay period?

Assessing feature-selective responses in visual cortex Freeman et al., 2011

Measuring feature-selective responses in visual cortex <1mm V1 columns ~27mm 3 fMRI voxel Kamitani & Tong, 2005, Nature Neuroscience Haynes & Rees, 2005, Nature Neuroscience Haxby et al., 2001, Science fMRI voxel ~27mm 3

Measuring motion selectivity in MT Voxels in MT

Voxels in V1 Response

Multivoxel Pattern Analysis Test image from scan 8 Mean image for remember 45  scans 1-7 Mean image for remember 135  scans 1-7

Functional Localizer Purpose: independently identify most responsive voxels from each sensory region Attend color or orientation on each 10s trial Task: Detect 2-3 oddballs/trial

Stimulus-specific patterns of activity represented only the relevant feature during the delay period Classify OrientationClassify Color Remember Orientation Remember Color Classification Accuracy 4-10s post-sample Serences, Ester, Vogel, Awh; Psych Science, 2009

but not when active discrimination was required without a storage requirement… Classify OrientationClassify Color Remember Orientation Remember Color Classification Accuracy 4-10s post-test Serences, Ester, Vogel, Awh; Psych Science, 2009

Sensory recruitment also predicts that WM maintains reasonable “copies” of sensory events Can we discriminate remembered attributes using data from a sensory task (the functional localizer)?

Post-SamplePost-Test Classification Accuracy Yes! Training the classifier using data from a sensory encoding condition (localizer task) enabled reliable classification of stored stimulus values. Serences, Ester, Vogel, Awh; Psych Science, 2009

The neural region (V1) that contained these stimulus-specific patterns showed no evidence of increases in the mean amplitude of BOLD responses. Serences, Ester, Vogel, Awh; Psych Science, 2009

Sustained amplitude increases Ranganath et al. 2004

Summary experiment 1 Feature-selective activation patterns in V1 during delay period predict contents of WM Visual areas that encode sensory information are recruited during maintenance Happens in the absence of sustained amplitude changes – usually taken as the hallmark of WM

Overview 1) Sensory recruitment during WM 2) Spatially global WM (and attention) signals 3) Sensory recruitment and WM precision

Feature based attention

Blank: 6s Experimental design Stimulus: 14s Task: Detect brief slowing of attended dots Eye tracking in subset of observers

Visual Pathway Each visual field is initially represented in contralateral visual cortex

Functional Localizer 12s 12s fixation Time

Contralateral > Ipsilateral Frontal Eye Field (FEF) Intraparietal sulcus (IPS) Occipital visual cortex anterior posterior

Feature-selective modulations: 45  vs. 135  hMT+

V1V2vV3vV4vV3aMTIPSFEF n=10, p<.05 in all areas Visual Area

Does feature-based attention spread to stimuli outside the focus of spatial attention? hMT+

Visual Area *p<.05 * V1V2vV3vV4vV3aMTIPSFEF * * *

Does feature-based attention spread to unstimulated regions? hMT+

Visual Area *p<.05 * * * * * * V1V2vV3vV4vV3aMTIPSFEF

Feature-based attention Spatially global modulation: spreads across entire visual field Know what, but not where Serences and Boynton, Neuron, 2007; see Treue and Martinez-Trujillo, 1999; Cohen and Maunsell, Neuron, 2011

Are stimulus-specific patterns related to WM also spatially global? Sensory recruitment: Attentional modulations during a perceptual task are global Ester, Serences, Awh, J. Neuroscience

Functional Localizer

The stored orientation was represented in both contralateral and ipsilateral regions of primary visual cortex, suggesting a spatially global mnemonic representation. Based on data from 6s-16s post-sample Receptive field of cortical ROI

Activation patterns during a perceptual task with the same stimuli were similar to those observed during storage in working memory. Receptive field of cortical ROI Location of Localizer Location of WM

Sustained increases in response amplitude may not be diagnostic of whether a given region participates in VWM maintenance

Conclusions Sensory recruitment during WM is spatially global Global patterns of activation observed during WM maintenance are similar to those observed during perception Ester, Serences, Awh, J. Neuroscience

Overview 1) Sensory recruitment during WM 2) Spatially global WM signals 3) Sensory recruitment and WM precision (does any of this actually matter???)

Sample: 1sDelay: 12sTest: 3s Experimental Procedure: estimating precision of WM representations using behavior and BOLD Subject rotates probe to match remembered sample orientation

Mean response error Orientation Bin %Responses

Encoding Model Orientation tuning model Orientation Brouwer & Heeger, 2009, 2011; Freeman and Adelson, 1992 Voxel responses x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 x7x7 Using ‘training data’, voxel responses (R) are mapped to responses in the channels (X) by weight matrix (W): R = WX Ŵ = RX T (XX T ) -1 X = (Ŵ T Ŵ) -1 Ŵ T R test Using held out ‘test’ data, channel responses computed:

Channel response estimation Training Phase (weight estimation) Given weights and test data, what is the response in each channel? Response pattern from trial in ‘test’ set

Shift to common center 0°90°-90° Orientation of the sample stimulus Serences et al., 2009; Brouwer and Heeger, 2011 Constructing orientation selective response profiles in visual cortex

Channel offset from sample BOLD Response Orderly channel tuning functions are observed during delay period Fit tuning curve from each subject to estimate bandwidth and amplitude Amplitude BW

Mean Recall Error TF Bandwidth (deg) Tuning function bandwidth is tightly coupled with the precision of subjects’ responses Ester, Anderson, Serences, & Awh

TF Bandwidth (deg) Recall Bandwidth

Mean Recall Error TF Amplitude There was no relationship between tuning function amplitude and task performance

Conclusions Precision of responses correlated with precision of neural representation No relationship between behavior and BOLD amplitude – argues against general arousal account

General Conclusions Areas that are involved in sensory processing are recruited during WM maintenance Highly selective for relevant information

General Conclusions Spatially global spread of activation during WM Spatial invariance of WM for features?

General Conclusions Precision of representations in V1 predicts precision of WM This stuff might actually might be important! TF Bandwidth (deg)

Thanks Edward Ester, Edward Awh + Edward Vogel (U. Oregon) Geoff Boynton (U. Washington) vogel awh boynton ester Supported by R01-MH087214