Motivation and Cognition: From Regulatory Fit to Reinforcement Learning Darrell A. Worthy University of Texas, Austin

Motivation and Cognition Why study motivation? Why study motivation? Need to understand how goals and rewards influence cognition and behavior. Need to understand how goals and rewards influence cognition and behavior. More complicated than anecdotal notions More complicated than anecdotal notions Approach vs. avoidance distinction Approach vs. avoidance distinction Global incentive vs. local goal pursuit mechanism Global incentive vs. local goal pursuit mechanism Leads to regulatory fit or mismatch Leads to regulatory fit or mismatch Regulatory fit affects cognition and behavior Regulatory fit affects cognition and behavior

Overview of Talk Regulatory Fit Framework Regulatory Fit Framework Regulatory fit affects cognition Regulatory fit affects cognition Tests of the Regulatory Fit Hypothesis Tests of the Regulatory Fit Hypothesis Extend framework to examine effects of social pressure. Extend framework to examine effects of social pressure. Regulatory Fit and Decision-making Regulatory Fit and Decision-making Future Directions Future Directions

Regulatory Fit Framework FitMismatch MismatchFit Promotion FocusPrevention Focus Gains Losses Local Goal Pursuit Mechanism Global Incentive Global incentive focus interacts with local reward structure Global incentive focus interacts with local reward structure Produces a Fit or a Mismatch (e.g. Higgins, 2000). Produces a Fit or a Mismatch (e.g. Higgins, 2000). Almost all cognitive research involves promotion focus with gains reward structure. Almost all cognitive research involves promotion focus with gains reward structure.

Manipulation of Regulatory Focus (Global Task Goal) Promotion Focus (Approach) Achieve Global Task Performance Criterion  Raffle ticket for $50 Prevention Focus (Avoidance) Achieve Global Task Performance Criterion  Keep $50 raffle ticket given initially

Manipulation of Goal-Pursuit Mechanism (Local Trial-by-trial Task Goal) Gains Correct Response = 3 points Incorrect Response = 1 point Losses Correct Response = -1 point Incorrect Response = -3 point

Effects of Regulatory Fit Previous research Previous research Regulatory Fit leads to: Regulatory Fit leads to: Increased sense of ‘feeling right’ (Higgins, 2000) Increased sense of ‘feeling right’ (Higgins, 2000) Increased motivational strength (Spiegel et al., 2004) Increased motivational strength (Spiegel et al., 2004) Increased “cognitive flexibility” (Shah et al., 1998) Increased “cognitive flexibility” (Shah et al., 1998) Flexibility can be defined within tasks Flexibility can be defined within tasks Category-learning -willingness to test various strategies Category-learning -willingness to test various strategies Decision-making -willingness to explore the environment Decision-making -willingness to explore the environment

Perceptual Classification Excellent for testing the effects of regulatory fit Excellent for testing the effects of regulatory fit Stimuli with small number of dimensions Stimuli with small number of dimensions Lines that vary in length, orientation and position Lines that vary in length, orientation and position ‘Gabor’ patches that vary in frequency and orientation ‘Gabor’ patches that vary in frequency and orientation Experimenter control of category structure Experimenter control of category structure Extensive set of tools for modeling performance of individual participants Extensive set of tools for modeling performance of individual participants Can assess the strategies participants use in the task Can assess the strategies participants use in the task

Explicit, Hypothesis-testing system mediates learning of “rule-based” (RB) category structures. -Frontally mediated -Verbalizable rules Implicit, Procedural learning system mediates learning of “information-integration” (II) category structures. -Striatally mediated - Verbalizable rules hurt performance (Maddox and Ashby, 2004; Ashby et al., 1998) Multiple systems mediate different classification tasks

Categorization Tasks Rule-BasedInformation-Integration Learned ExplicitlyLearned Implicitly

Increased cognitive flexibility will increase rule use Increased cognitive flexibility will increase rule use Enhance performance on rule-based tasks Enhance performance on rule-based tasks Will harm performance on information-integration task Will harm performance on information-integration task Rule-use disrupts the procedural system Rule-use disrupts the procedural system Recent tests of this hypothesis (Markman et al., 2005; Maddox et al., 2006; Grimm et al., 2008) Recent tests of this hypothesis (Markman et al., 2005; Maddox et al., 2006; Grimm et al., 2008) Manipulated regulatory focus and reward structure between subjects Manipulated regulatory focus and reward structure between subjects Used rule-based and information-integration tasks Used rule-based and information-integration tasks Influence of Regulatory Fit

Regulatory Fit and Classification Rule-based performance was better in a fit Information-integration performance was better in a mismatch Fit increases rule-use Helps on rule-based, hurts on information- integration

Choking & Excelling Under Pressure Worthy, Markman, & Maddox, 2009a, 2009b; Worthy, Markman, & Maddox, 2008; Markman, Maddox & Worthy 2006

Choking Under Pressure Anecdotal phenomenon (e.g. sports, test-taking, etc.) Anecdotal phenomenon (e.g. sports, test-taking, etc.) People perform worse than normal when under pressure People perform worse than normal when under pressure Some also seem to excel under pressure Some also seem to excel under pressure Might pressure be similar to a prevention focus? Might pressure be similar to a prevention focus?

Motivation and Pressure Working Memory Distraction Hypothesis of choking Working Memory Distraction Hypothesis of choking Pressure reduces WM capacity Pressure reduces WM capacity Should see main effects Should see main effects Pressure decreases rule-use Pressure decreases rule-use Alternative: Pressure affects cognition through its effects on the motivational state Alternative: Pressure affects cognition through its effects on the motivational state Working Hypothesis: Working Hypothesis: Pressure induces an “avoidance” or “prevention” motivational state Pressure induces an “avoidance” or “prevention” motivational state Interacts with goal pursuit mechanism to influence regulatory fit Interacts with goal pursuit mechanism to influence regulatory fit

Pressure and Category-Learning Low pressure – “do your best” Low pressure – “do your best” High pressure: High pressure: -Paired with a ‘partner’ -If both of you reach criterion, both get $6 -If one of you fails neither get $6 bonus -Partner has already reached criterion -Trying to prevent the negative end-state of letting their partner down Run gains and losses Run gains and losses FitMismatch MismatchFit Promotion FocusPrevention Focus Gains Losses Local Goal Pursuit Mechanism Global Incentive Low PressureHigh Pressure

WM Distraction vs. Regulatory Fit Pressure decreases WM Pressure decreases WM Poor rule-based performance Poor rule-based performance Better information- integration Better information- integration Pressure induces a prevention focus Pressure induces a prevention focus Will interact with the reward structure Will interact with the reward structure

Method 2 (Pressure-level) X 2 (Reward Structure) X 2 (Task Type) between-subjects design Performed 8 80-trial blocks Rule-BasedInformation-Integration Worthy, et al., 2009, Psychonomic Bulletin & Review

Results Worthy, et al., 2009, Psychonomic Bulletin and Review

Decision Bound Modeling Used to infer strategy use. Used to infer strategy use. Decision bound models assume stimuli are classified based on which side of the decision bound they fall on Decision bound models assume stimuli are classified based on which side of the decision bound they fall on Several models are fit to the data Several models are fit to the data Best-fitting model gives information about which strategy each participant probably used to classify the stimuli Best-fitting model gives information about which strategy each participant probably used to classify the stimuli

Decision Bound Modeling

Model Fitting Procedure Fit each participant’s data on a block-by-block basis Fit each participant’s data on a block-by-block basis Used AIC to determine best fitting model for that block Used AIC to determine best fitting model for that block Penalizes for free parameters Examined the proportion of data sets best fit by each model over all blocks of the task. Examined the proportion of data sets best fit by each model over all blocks of the task.

Model-Based Analysis Best strategy for rule-based task Best strategy for rule-based task Best strategy for information- integration task Best strategy for information- integration task

Proportion Fit by Best Model Increase in accuracy likely due to improved strategy use. Worthy, et al., 2009, Psychonomic Bulletin & Review

Summary Pressure does appear to operate like a prevention focus during classification learning. Not main effect where WM is decreased Gains mismatches with pressure-induced prevention focus Pressure hurts rule-based performance Pressure helps information-integration performance. Losses fits with pressure-induced prevention focus Pressure helps rule-based performance. Pressure hurts information-integration performance.

Pressure and Experts Examined effects of pressure after extensive training. Examined effects of pressure after extensive training. RB or II task RB or II task trial sessions trial sessions Worthy et al., 2009, Attention, Perception and Psychophysics Supports a different account for effects of pressure on experts

Real World Choking Examined clutch free-throw performance among NBA athletes Examined clutch free-throw performance among NBA athletes Considered point-differential between shooter’s team. Considered point-differential between shooter’s team. Compared percentage to career percentage Compared percentage to career percentage Worthy et al., 2009, International Journal of Creativity and Problem Solving

Regulatory Fit and Decision-Making Worthy, Maddox, & Markman, 2007

Decision-making from experience Basic Design ‘Gambling’ task Participants choose from two or more decks of cards Must either maximize gains or minimize losses GainsLosses

Modeling Task is amenable to reinforcement learning modeling Task is amenable to reinforcement learning modeling Can estimate parameters that describe performance Can estimate parameters that describe performance

Expected Value (EV) EV – How many points one expects to gain or lose from selecting a given deck EV – How many points one expects to gain or lose from selecting a given deck Used to determine which option to choose Used to determine which option to choose Example Example EV red deck = 7 points EV red deck = 7 points EV blue deck = 3 points EV blue deck = 3 points

Exploration/Exploitation Dilemma Exploit the option with the highest EV Exploit the option with the highest EVor Explore other options with lower EVs Explore other options with lower EVs Must balance the need to exploit with the need for new information Must balance the need to exploit with the need for new information Exploration may be more frontally mediated (e.g. Daw et al., 2006). Exploration may be more frontally mediated (e.g. Daw et al., 2006). Working hypothesis: Regulatory fit will increase exploration Working hypothesis: Regulatory fit will increase exploration

Task Design Can design tasks to favor more exploratory or exploitative strategies. Can design tasks to favor more exploratory or exploitative strategies. Experiment 1 – Exploration-optimal Experiment 1 – Exploration-optimal Experiment 2 – Exploitation-optimal (Gains only) Experiment 2 – Exploitation-optimal (Gains only) Use behavioral and model-based analyses to test the regulatory fit hypothesis Use behavioral and model-based analyses to test the regulatory fit hypothesis Worthy et al., 2007

Experiment 1 Designed a task where exploring the deck with lower EV led to better-long-term performance. Designed a task where exploring the deck with lower EV led to better-long-term performance. Had to be willing to explore the Advantageous deck Had to be willing to explore the Advantageous deck Fit should increase exploration; performance Fit should increase exploration; performance

Methods Used raffle-ticket manipulation to manipulate regulatory focus Used raffle-ticket manipulation to manipulate regulatory focus Promotion Focus (Approach) Achieve Global Performance Criterion  Raffle ticket for $50 Prevention Focus (Avoidance) Achieve Global Performance Criterion  Keep $50 raffle ticket given initially

Methods Achieved global criterion by either maximize gains or minimizing losses Achieved global criterion by either maximize gains or minimizing losses Gains Gained between 1 and 10 points on each draw; maximized gains Losses Lost between -10 and -1 points on each draw; minimized losses

Behavioral results Behavioral results Participants in a regulatory fit came significantly closer to the performance criterion than participants in a mismatch Participants in a regulatory fit came significantly closer to the performance criterion than participants in a mismatch

Modeling Choice Behavior EVs of each option are updated via an exponential recency-weighted algorithm EVs of each option are updated via an exponential recency-weighted algorithm Current EVNew EVRewardRecency Parameter Current EV If reward is greater than the current EV the EV increases If reward is less than the current EV the EV decreases

Action Selection Action selection is probabilistically determined via choice rules (e.g. Luce, 1959) Softmax Rule Probability of choosing option “A” EV for option “A” Exploitation parameter Sum of EVs for all options Higher  values indicate greater exploitation Lower  values indicate greater exploration Can directly parameterize degree of exploratory vs. exploitative behavior

Model-based results Fit reinforcement-learning model to estimate the degree of exploratory vs. exploitative behavior. Fit reinforcement-learning model to estimate the degree of exploratory vs. exploitative behavior. Participants in a regulatory fit had significantly lower estimated exploitation-parameter values. Participants in a regulatory fit had significantly lower estimated exploitation-parameter values.

Experiment 2 Designed a task where exploitation of the deck with the best expected value led to the best performance. Designed a task where exploitation of the deck with the best expected value led to the best performance. If fit increases exploration then participants in a fit should do worse. If fit increases exploration then participants in a fit should do worse.

Results Only ran participants with a gains reward structure Only ran participants with a gains reward structure Participants in a regulatory fit were further from the performance criterion Participants in a regulatory fit were further from the performance criterion

Model-Based Results Participants in a fit were less exploitative than those in a mismatch Participants in a fit were less exploitative than those in a mismatch

Summary Regulatory fit influenced the decision-making behavior Fit – greater exploration Mismatch greater exploitation Social pressure induces a prevention focus Influences regulatory fit Differential performance on category-learning tasks Three-way interaction Regulatory focus – Promotion vs. prevention Reward Structure – Maximize gains vs. minimize losses Task Demands – Rule-based vs. information-integration; exploration-optimal vs. exploitation-optimal

Expected Reward Comparison Extended decision-making paradigm to ratio vs. difference comparisons Extended decision-making paradigm to ratio vs. difference comparisons Are EVs compared via ratio or differences? Are EVs compared via ratio or differences? Manipulated whether difference or ratio preserved. Manipulated whether difference or ratio preserved. Changing the ratio between EVs affected performance Changing the ratio between EVs affected performance Worthy et al., 2008, Memory and Cognition

Research Approach Categorization and Decision-making tasks Categorization and Decision-making tasks Behavioral analysis Behavioral analysis Mathematical modeling Mathematical modeling Decision-bound modeling Decision-bound modeling Reinforcement-learning modeling Reinforcement-learning modeling Ground theories in neuroscience Ground theories in neuroscience Leads to novel predictions Leads to novel predictions

Current & Future Directions ‘Why’ does regulatory fit influence behavior and cognition Working memory hypothesis Fit increases WM memory capacity Not yet directly tested Test using regulatory focus and social pressure manipulation in WM tasks. Test by adding WM span as an additional factor on categorization and decision-making tasks. Regulatory fit and short-term vs. long-term decision-making Does fit reduce future discounting? People in a fit may focus more on long-term outcomes

Current & Future Directions Individual Differences Are some less susceptible to situational factors than others? Why do some people tend to choke, while others excel? Aging and decision-making Older adults appear to be more exploratory than younger adult May value long-term over short-term outcomes Positivity bias Neural differences Gender and decision-making Men appear to be more exploitative than women

Current & Future Directions Current & Future Directions Social vs. Monetary rewards Give incrementally happier or angrier faces as feedback in decision-making tasks. Can use same modeling approach Compare to monetary rewards Neural mechanisms

Current & Future Directions Current & Future Directions Category learning Feedback timing Very important for procedural learning system Retention and generalization Desirable difficulties Naturalistic stimuli (x-rays – tumor detection) Interactions between multiple systems – competition vs. cooperation

Thanks! Acknowledgements Todd Maddox, Art Markman, Bo Zhu, MaddoxLab research assistants. Supported by NIMH grant MH to WTM and ABM, and a supplement to DAW.

References Daw, N.D., O’Doherty, J.P., Dayan, P., Seymour, B., & Dolan, R. (2006). Cortical Substrates for exploratory decisions in humans. Nature, 441 (15), Grimm, L. R., Markman, A. B., Maddox, W. T., Baldwin, G. C. (2008) Differential Effects of Regulatory Fit on Category Learning. Journal of Experimental Social Psychology. 44, Higgins, E. T. (1997). Beyond pleasure and pain. American Psychologist, 52, Higgins, E. T. (2000). Making a good decision: Value from fit. American Psychologist, 55, Maddox, W.T., & Ashby, F.G. (1993). Comparing decision bound and exemplar models of categorization. Perception and Psychophysics, 53, Maddox, W. T., & Ashby, F. G. (2004). Dissociating explicit and procedural-learning based systems of perceptual category learning. Behavioural Processes, 66, Maddox, W. T., Markman, A. B., & Baldwin, G. C. (2006). Using classification to understand the motivation-learning interface. Psychology of Learning and Motivation, 47, Markman, A.B., Maddox, W.T., Worthy, D.A. (2006) Choking and excelling under pressure. Psychological Science. 17, Shah, J., Higgins, E. T., & Friedman, R. S. (1998). Performance incentives and means: How regulatory focus influences goal attainment. Journal of Personality and Social Psychology, 74, Spiegel, S., Grant-Pillow, H., & Higgins, E. T. (2004). How regulatory fit enhances motivational strength during goal pursuit. European Journal of Social Psychology, 34, Worthy, D.A., Maddox, W.T., & Markman, A.B. (2007). Regulatory Fit Effects in a Choice Task. Psychonomic Bulleting and Review, 14, Worthy, D.A., Maddox, W.T., & Markman, A.B. (2007). Regulatory Fit Effects in a Choice Task. Psychonomic Bulleting and Review, 14, Worthy, D.A., Maddox, W.T., & Markman, A.B. (2008). Ratio and Difference Comparisons of Expected Reward in Decision Making Tasks. Memory and Cognition, 36, Worthy, D.A., Markman, A.B., & Maddox, W.T. (2009A). What is pressure? Evidence for social pressure as a type of regulatory focus. Psychonomic Bulletin and Review, 16, Worthy, D.A., Markman, A.B., & Maddox, W.T. (2009B). Choking and excelling at the free throw line. The International Journal of Creativity & Problem Solving, 19, Worthy, D.A., Markman, A.B., & Maddox, W.T. (2009C). Choking and Excelling Under Pressure in Experienced Classifiers. Attention, Perception and Psychophysics, 71,

Aging and Decision-Making Aging and Decision-Making Older adults use a more exploratory than younger adults. Task favored an exploitative strategy Worthy et al., in preparation

Aging and Decision-Making Aging and Decision-Making -Task favored an exploratory strategy Worthy et al., in preparation

Aging and Decision-Making Aging and Decision-Making -Looked at “Directed Exploration” – not just more random Worthy et al., in preparation

Aging and Decision-Making Aging and Decision-Making -Older adults explore the decision space -Do not focus only on short-term rewards Worthy et al., in preparation

Gender and Decision-Making Gender and Decision-Making -Males tend to be more exploitative than females

Feedback Delay and Category Learning Feedback Delay and Category Learning -Feedback timing important for II learning only -500ms appears to be the best time for procedural system to receive feedback Worthy et al., in preparation

Feedback Delay and Category Learning Feedback Delay and Category Learning -Separated visual and motor response feedback components -Important for system to receive visual and motor information that a response has been made Worthy et al., in preparation

Desirable Difficulties in II learning Desirable Difficulties in II learning -Discontinuous categories are more difficult to learn but may lead to better transfer performance. Maddox et al., in preparation

Desirable Difficulties in II learning Desirable Difficulties in II learning -Continuous categories are learned easier, but transfer performance is worse. Maddox et al., in preparation