Progress in MURI 15 ( ) Mathematical modeling of decision behavior. AFOSR, Alexandria, VA, Nov 17th, 2009 Phil Holmes 1. Optimizing monkeys? Balancing noisy stimulus information and reward expectations (Feng, Holmes, Rorie, Newsome, PLoS Comp. Biol. 5 (2), 2009). 2. Time-dependent perturbations can distinguish integrator dynamics (Zhou, Wong-Lin, Holmes, Neural Comput., 21, , 2009). [3. Mean field reduction of a spiking neuron integrator model with neuromodulation (Eckhoff, Wong-Lin, Holmes, SIAM J. Appl. Dyn. Sys., in review, 2009). Not to be presented today. ]

1. Optimizing Monkeys & Optimistic Math

Leaky competing accumulators reduce to OU (actually, nonlinear DD/OU processes) via strong attraction to a 1-d subspace: (Usher & McClelland, 1995,2001) Subtracting the accumulated evidence yields an OU process (linearized DD case shown here). This generalizes to nonlinear systems: stochastic center manifolds.

Fixed viewing time (cued response) tasks can be modeled by DD/OU processes. We consider the PDF of sample paths of the SDE, which is governed by the forward Fokker-Planck or Kolmogorov PDE: Interrogate solution at cue time T : is ? [Neyman-Pearson] General solutions for time-varying drift (SNR) are available, so we can predict accuracy (psychometric functions):

Integrating bottom-up (stimulus) and top-down (prior expectation) inputs: a motion discrimination task with four reward conditions. Introduces reward priors (expectations). A. Rorie & W.T. Newsome, SfN 2006; in prep., 2008). DD can incorporate priors: reward expectations Reward cue Motion We only model this part HL HH LL LH Why monkeys? Electrophysiology! (Rorie, Reppas, Newsome)

DD/OU models with reward bias priors, 1 Choose a model for reward expectation. Two examples are: 1. Initial condition set at start of motion period: motion period only. 2. Bias applied to drift rate throughout reward cue and motion periods: reward cue period motion period. for predicts shifted PMFs: Feng, H, Rorie, Newsome, PLoS Comp. Biol. 5 (2), Leak-dominatedBalanced Inhibition- dominated (stable OU, recency) (DD, optimal) (unstable OU, primacy) HL LH HH LL

DD/OU models with reward bias priors, 2 These reward expectation models both lead to PMFs that can be written in the simple form where b 1 acuity = ability to use signal (SNR) b 2 shift = estimate of reward bias depending on the model chosen. But unless we have a range of “ priming ” and viewing times*, the models can ’ t be distinguished on behavioral evidence alone : the parameters cannot be separated. [*But see McClelland presentation.] We can nonetheless compute shifts that maximize expected rewards, e.g., in simple case of fixed coherence : (More complex expressions hold for the mix of coherences actually used.) ~ 1/SNR reward ratio

DD/OU models with reward bias priors, 3 Fit the model to the Rorie-Newsome behavioral data from two adult male rhesus monkeys, averaged over all sessions: monkey A monkey T slope shift For fixed the PMFs all reduce to a 2 parameter family:

DD/OU models with reward bias priors, 4 Maximize expected rewards: compute optimal shifts given animals ’ slopes and compare with their actual shifts: Both animals overshift, and T prefers alternative 2 when rewards are equal. But does this cost them much? How steep is the reward hill? reward ratio sums of Gaussians

DD/OU models with reward bias priors, 5 Expected reward functions are rather flat, and overshifting costs less than undershifting. So they don ’ t lose much! monkey A monkey T 99.5% performance bands

DD/OU models with reward bias priors, 6 Overall performance is good, although acuity (= slope b 1 ) and shifts (b 2 ) vary significantly from session to session. In spite of “ visual correlation, ” the animals do not exhibit significant b 1 vs b 2 correlations (performance tuning). Monkey A: Blue is optimal; magenta curves are 99% & 97% of optimal. With few exceptions, A stays inside 97% in every session. Monkey T is not as good, shows bias to T 2.

DD processes extend to include top-down cognitive control: can be used to predict optimal PMFs. Two subjects tested both overshift, but garner 98 − 99% of maximum possible rewards, in spite of significant session-to-session variability. App of rigorous math methods. Brief impulsive perturbations during stimulus presentation can reveal the dynamics of integrators, distinguish pure DD processes from leak- and inhibition-dominated integrators*, and even from nonlinear model. (*stable, unstable OU processes) Conclusions 1

2. Brief perturbations can distinguishing integrators

Drift-diffusion and OU processes with thresholds can model RT tasks. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

Time-dependent perturbations can distinguish integrator types. Decision variable x time threshold for choice 1 threshold for choice 2 Mean RT Variance

Perturbation modifies RT distributions means standard deviations Means and standard deviations of RT distributions are shifted in distinct manners for constant and rising drift rates (CD, TD) and for stable and unstable OU processes (SOU, UOU); positive pulse +, negative pulse _. Zhou, Wong-Lin, H, Neural Comput. 21 (8), Perturbn must be early to have strong effect!

Zero-effect perturbations CD, TD, SOU and UOU are most easily distinguished by tuning a neg-pos pulse pair so that the mean RT is not changed. These employ the recency/primacy effects previously noted by Usher-McClelland. The “ simple ” patterns seen here occur only for low noise and early perturbations (pert time T << mean RT; effects decay as T increases). For high noise the mean and SD patterns are less clear. Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.

Further results and related work The ideas extend to nonlinear systems (subcritical pitchfork case), although for effects are harder to interpret. Zhou, Wong-Lin, H, Neural Comput. 21 (8), Gao-McClelland have replicated a slight decrease in the effect of perturbations delivered later, under the cued response (interrogation) protocol. Also, subjects appear more responsive to coherent motion early in the trial than later. See Gao presentation, and cf. fits of OU and variable drift DD models to data from J. Gold lab (Penn). (Eckhoff, H, Law, Conolly, Gold, New J. Phys. 10, 2008.) Perturbation experiments are planned (Usher, McClelland, Simen, … )

DD processes extend to include top-down cognitive control: it can predict optimal PMFs. Two subjects tested both overshift, but garner 98 − 99% of their maximum possible rewards, in spite of significant session-to-session variability. App of rigorous math methods. Brief impulsive perturbations during stimulus presentation can reveal the dynamics of integrators, distinguish pure DD processes from leak- and inhibition-dominated integrators*, and even from nonlinear model. (*stable, unstable OU processes) Conclusions 2

Preamble: optimal choices The drift-diffusion (DD) process, a cornerstone of 20th century physics, and perhaps the simplest stochastic ODE, is an optimal decision maker for two- alternative forced choice perception tasks with noisy data: drift rate noise strength Here is the accumulated evidence (the log likelihood ratio): when it reaches the threshold, declare R or L the winner. DD is a continuum limit of SPRT (Wald, 1947). The model has only 2 parameters: Behavioral data, neural recordings in primates, fMRI and EEG in humans (MURI 16) and spiking neuron models support the contention that evidence is accumulated in this manner by cortical networks.

Behavioral evidence: RT distributions Human reaction time data in free response mode can be fitted to the first passage threshold crossing times of a DD process. Prior or bias toward one alternative can be implemented by setting starting point. Extended DD: variable drift rates & starting points. Simple expressions for mean RT, accuracy. Ratcliff et al., Psych Rev. 1978, 1999, 2004 Simen et al., J. Exp. Psych.:HPP, in press, thresh. +Z thresh. -Z drift A

Spiking neurons reduce to leaky accumulators Working hypothesis: motion sensitive cells in visual cortex MT pass noisy signals on to LIP, FEF, … where integration occurs. threshold Experimental observations: K.H. Britten, M.N. Shadlen, W.T. Newsome, J.D. Schall & A. Movshon (various papers, ). Simulation, analysis and reduction of spiking neuron models: X.-J. Wang, Neuron, 2002; K-F. Wong & X.-J. Wang, J. Neurosci., Extension to model NE modulation of LIP: P. Eckhoff, K-F. Wong, PH. J. Neurosci, 29 (13), 2009 ; P. Eckhoff, K-F. Wong, PH. (in review, 2009). MT LIP

Neural evidence: firing rates Spike rates of neurons in cortical areas rise during stimulus presentation, monkeys signal their choice after a threshold is crossed (like DD process). J. Schall, V. Stuphorn, J. Brown, Neuron, Frontal eye field (FEF) recordings. J.I Gold, M.N. Shadlen, Neuron, Lateral interparietal area (LIP) recordings. thresholds

We can predict Psychometric Functions (PMFs): Accuracy for fixed viewing time T, sorted by coherences. DD model with variable drift: scale coher. expt. asmpt. decay rate Model fits to data from monkeys during training on the moving dots task. Animals learn to extract signal from noise. Scale factor a : steady increase in SNR. (Eckhoff, H, Law, Conolly, Gold, New J. Phys., 2008) Accuracy

Perturbing a nonlinear integrator A nonlinear model can also be distinguished from CD, TD, SOU and UOU: both mean and SD effects decrease monotonically as T increases. The “ simple ” patterns seen here occur only for low noise and early perturbations (T << mean RT). Experiments are in planning stage (Usher, McClelland, Simen). Zhou, Wong-Lin, H, Neural Comput. 21 (8), 2009.