2. Seeing Patterns in Randomness: Irrational Superstition or Adaptive Behavior? Angela J. Yu University of California, San Diego March 9, 2010

3. “Irrational” Probabilistic Reasoning in Humans 1 2 2 2 2 2 1 1 2 1 2 1 … 1 2 2 2 2 2 Random stimulus sequence: 1 2 “hot hand” 2AFC: sequential effects (rep/alt) (Gillovich, Vallon, & Tversky, 1985) (Soetens, Boer, & Hueting, 1985) (Wilke & Barrett, 2009)

4. “Superstitious” Predictions Subjects are “superstitious” when viewing randomized stimuli O o o o o o O O o O o O O… repetitionsalternations slow fast Trials Subjects slower & more error-prone when local pattern is violated Patterns are by chance, not predictive of next stimulus Such “superstitious” behavior is apparently sub-optimal

5. “Graded” Superstition (Cho et al, 2002) (Soetens et al, 1985) [o o O O O] RARR = or [O O o o o] RT ER Hypothesis: Sequential adjustments may be adaptive for changing environments. t t-1 t-2 t-3

6. Outline “Ideal predictor” in a fixed vs. changing world Exponential forgetting normative and descriptive Optimal Bayes or exponential filter? Neural implementation of prediction/learning

7. I. Fixed Belief Model (FBM) A (0) R (1) hidden bias observed stimuli ? … ?

8. II. Dynamic Belief Model (DBM) A (0)R (1) changing bias observed stimuli ? ?.3.8.3

9. RA bias  What the FBM subject should believe about the bias of the coin, given a sequence of observations: R R A R R R FBM Subject’s Response to Random Inputs

10. What the FBM subject should believe about the bias of the coin, given a long sequence of observations: R R A R A A R A A R A… RA bias 

11. What the DBM subject should believe about the bias of the coin, given a long sequence of observations: R R A R A A R A A R A… RA bias  DBM Subject’s Response to Random Inputs

12. Randomized Stimuli: FBM > DBM Given a sequence of truly random data (  =.5) … FBM: belief distrib. over  Simulated trials Probability DBM: belief distrib. over  Simulated trials Probability Driven by long-term average Driven by transient patterns

13. “Natural Environment”: DBM > FBM In a changing world, where  undergoes un-signaled changes … FBM: posterior over  Simulated trials Probability Adapt poorly to changes Adapt rapidly to changes DBM: posterior over  Simulated trials Probability

14. Persistence of Sequential Effects Sequential effects persist in data DBM produces R/A asymmetry Subjects=DBM (changing world) FBM P(stimulus) DBM P(stimulus) Human Data (data from Cho et al, 2002) RT

15. Outline “Ideal predictor” in a fixed vs. changing world Exponential forgetting normative and descriptive Optimal Bayes or exponential filter? Neural implementation of prediction/learning

16. Bayesian Computations in Neurons? Optimal Prediction What subjects need to compute Too hard to represent, too hard to compute! Generative Model What subjects need to know

17. (Sugrue, Corrado, & Newsome, 2004) Simpler Alternative for Neural Computation? Inspiration: exponential forgetting in tracking true changes

18. Exponential Forgetting in Behavior Exponential discounting is a good descriptive model Linear regression: R/A Human Data Trials into the Past Coefficients (re-analysis of Cho et al)

19. Linear regression: R/A Exponential discounting is a good normative model DBM Prediction Trials into the Past Coefficients Exponential Forgetting Approximates DBM

20. Discount Rate vs. Assumed Rate of Change … DBM  =.95 Simulated trials Probability  =.77 Simulated trials

21. Trials into the Past DBM Simulation Coefficients Human Data Trials into the Past Coefficients  =.57 Reverse-engineering Subjects’ Assumptions  = p(  t =  t-1 )  =.57  =.77  changes once every four trials   2/3 

22. Analytical Approximation Quality of approximation  vs.   .57.77 nonlinear Bayesian computations3-param model 1-param linear model

23. Outline “Ideal predictor” in a fixed vs. changing world Exponential forgetting normative and descriptive Optimal Bayes or exponential filter? Neural implementation of prediction/learning

24. Subjects’ RT vs. Model Stimulus Probability Repetition Trials R A R R R R …

25. Subjects’ RT vs. Model Stimulus Probability Repetition Trials R A R R R R … RT

26. Subjects’ RT vs. Model Stimulus Probability Repetition Trials Alternation Trials R A R R R R … RT

27. Subjects’ RT vs. Model Stimulus Probability Repetition vs. Alternation Trials

28. Multiple-Timescale Interactions Optimal discrimination (Wald, 1947) 2 1 discrete time, SPRT continuous-time, DDM DBM (Yu, NIPS 2007) (Frazier & Yu, NIPS 2008) (Gold & Shadlen, Neuron 2002)

29. SPRT/DDM & Linear Effect of Prior on RT Timesteps RT hist Bias: P(s 1 ) Bias : P(s 1 ) x tanh x 0

30. SPRT/DDM & Linear Effect of Prior on RT Empirical RT vs. Stim Probability Bias: P(s 1 ) Predicted RT vs. Stim Probability

31. Outline “Ideal predictor” in a fixed vs. changing world Exponential forgetting normative and descriptive Optimal Bayes or exponential filter? Neural implementation of prediction/learning

32. Neural Implementation of Prediction Leaky-integrating neuron: Perceptual decision-making (Grice, 1972; Smith, 1995; Cook & Maunsell, 2002; Busmeyer & Townsend, 1993; McClelland, 1993; Bogacz et al, 2006; Yu, 2007; …) Trial-to-trial interactions (Kim & Myung, 1995; Dayan & Yu, 2003; Simen, Cohen & Holmes, 2006; Mozer, Kinoshita, & Shettel, 2007; …) bias input recurrent = 1/2 (1-  )1/3  2/3 

33. Neuromodulation & Dynamic Filters Leaky-integrating neuron: bias input recurrent Norepinephrine (NE) (Hasselmo, Wyble, & Wallenstein 1996; Kobayashi, 2000) Trials NE: Unexpected Uncertainty (Yu & Dayan, Neuron, 2000)

34. Learning the Value of  Humans (Behrens et al, 2007) and rats (Gallistel & Latham, 1999) may encode meta-changes in the rate of change,  Bayesian Learning 001.3.9.3 … … … … Iteratively compute joint posterior Marginal posterior over  Marginal posterior over 

35. Neurons don’t need to represent probabilities explicitly Just need to estimate  Stochastic gradient descent (  -rule) Neural Parameter Learning? learning rate error gradient

36. Learning Results Trials Stochastic Gradient Descent Trials Bayesian Learning

37. Summary H: “ Superstition” reflects adaptation to changing world Exponential “memory” near-optimal & fits behavior; linear RT Neurobiology: leaky integration, stochastic  -rule, neuromodulation Random sequence and changing biases hard to distinguish Questions: multiple outcomes? Explicit versus implicit prediction?

39. Unlearning Temporal Correlation is Slow Marginal posterior over  Marginal posterior over  Trials Probability (see Bialek, 2005)

40. Insight from Brain’s “Mistakes” Ex: visual illusions (Adelson, 1995)

41. lightness depth context Neural computation specialized for natural problems Ex: visual illusions Insight from Brain’s “Mistakes”

42. Discount Rate vs. Assumed Rate of Change Iterative form of linear exponential Exact inference is non-linear Linear approximation Empirical distribution

43. Bayesian Inference Posterior Generative Model (what subject “knows”) 1: repetition 0: alternation Optimal Prediction (Bayes’ Rule)

44. Bayesian Inference Optimal Prediction (Bayes’ Rule) Generative Model (what subject “knows”)

45. Power-Law Decay of Memory Human memory Stationary process! Hierarchical Chinese Restaurant Process 1074  … (Teh, 2006) Natural (language) statistics (Anderson & Schooler, 1991)

46. Ties Across Time, Space, and Modality Sequential effects RT Stroop GREEN SSHSS Eriksen time modality space (Yu, Dayan, Cohen, JEP: HPP 2008) (Liu, Yu, & Holmes, Neur Comp 2008)

48. Sequential Effects  Perceptual Discrimination Optimal discrimination (Wald, 1947) R A discrete time, SPRT continuous-time, DDM DBM PFC (Yu & Dayan, NIPS 2005) (Yu, NIPS 2007) (Frazier & Yu, NIPS 2008) (Gold & Glimcher, Neuron 2002)

49. Monkey G Coefficients Trials into past  =.72 Exponential Discounting for Changing Rewards Monkey F Coefficients Trials into past  =.63 (Sugrue, Corrado, & Newsome, 2004)

50. Monkey G Coefficients Trials into past  =.72 Monkey F Coefficients Trials into past  =.63 Human & Monkey Share Assumptions? MonkeyHuman ≈ !  =.68  =.80

51. Simulation Results Trials Learning via stochastic  -rule

52. Monkeys’ Discount Rates in Choice Task (Sugrue, Corrado, & Newsome, 2004) Monkey F Coefficients Trials into past  =.63.63.68 Monkey G Coefficients Trials into past  =.72.72.80

53. Human & Monkey Share Assumptions?.72.80.63.68 MonkeyHuman ≈ !