Chapter 8 Instrumental Conditioning: Learning the Consequences of Behavior

8.1 Behavioral Processes

3 The “Discovery” of Instrumental Conditioning Components of the Learned Association Putting It All Together: Building the S–R–C Association Learning and Memory in Everyday Life— The Problem with Punishment Choice Behavior

4 The “Discovery” of Instrumental Conditioning Instrumental conditioning—developing a contingency between response and outcome. Organism learns to make responses to obtain or avoid important consequences. e.g., trained circus animals, waterskiing squirrels AP/Wide World Photos

5 Free-Operant Learning Operant conditioning—a type of instrumental learning. Skinner’s free-operant paradigm: Replaces Thorndike’s discrete trials. Learner can operate apparatus (e.g. Skinner box) at will. Learner’s trial-independent responses are measured with a cumulative recorder.

6 Operant Conditioning

7 Free-Operant Learning Reinforcement—consequences increase behavior probability. Punishment—consequences decrease behavior probability.

8 Components of the Learned Association Three components to instrumental conditioning: Stimulus (S) Response (R) Consequence (C)

9 Stimulus A discriminative stimulus is a cue, not a US or CS. A signal for when response will lead to consequence. Examples: Starting whistle for racing swimmers Potty seat for toilet-training toddler Can increase behavior probability, but does NOT elicit behavior.

10 Response: Shaping Shaping—successive approximations to the desired response are reinforced. Collect baseline data on current behavior (establish operant level). Identify target behavior. Reinforce successive approximations of the target response.

11 Response: Shaping Example: Helping autistic children learn language (Ivar Lovaas, 1987). Say target word (e.g., child’s name). Reinforce with food any sound, then closer imitations. Introduce new words.

12 Response: Chaining Chaining—learning a complicated sequence of responses by adding one discrete “link” (step) at a time. Backward chaining—training steps in reverse order. Examples: Teaching pets unusual tricks. Teaching workers a sequential manufacturing process.

13 Skinner QEyBY

14 Consequence: Primary Reinforcers Reinforcer—behavioral consequence that makes future behavior more likely. Primary reinforcers: Reinforcing events that occur because of their natural characteristics and inherent ability to reinforce behavior (drive reduction theory). Examples: Food or Water Sleep Sex

15 Consequence: Secondary Reinforcers Secondary (conditioned) reinforcers: Reinforcing events that function as reinforcers because they are consistently associated with one or more primary reinforcers. Example: Money No biological imperative. Can be exchanged for primary reinforcers (e.g., food or shelter).

16 Consequence: Punishers Punishers: Behavioral consequence leads to a reduction of future behavior. Strong and enduring aversive stimuli are the most effective suppressors. Aversive stimuli of low intensity may reinforce behavior we intend to suppress! Apply aversive stimuli immediately after the targeted behavior. Delaying punisher decreases contingency.

17 The Negative Contrast Effect Data from Kobre and Lipsitt, 1972.

18 Learning and Memory in Everyday Life— The Problem with Punishment Use of corporal punishment is controversial. Alternatives: Scolding Time-out Grounding Withholding allowance Avoid attention for punished behavior. Reinforce appropriate behavior.

19 Putting It All Together: Building the S–R–C Association Timing affects learning: Immediate consequence = best learning Instrumental conditioning faster if R–C interval is short (temporal contiguity). Timing can also impact: Punishment Immediate punishment more effective than delayed punishment. Self-control Forego immediate reward for greater future reward.

Interim Summary Instrumental conditioning = learning a three- way association (S → R → C) between: Discriminative stimulus (S) Response (R) Consequence (C) C may be reinforcer or punisher. In instrumental conditioning, C occurs only if R is made; whereas, In classical conditioning, the consequence (US) occurs automatically after the stimulus (CS).

Interim Summary Four classes of instrumental conditioning: Positive reinforcement Negative reinforcement Positive punishment Negative punishment. “Negative” and “positive” show if consequence is subtracted or added. “Reinforcement” and “punishment” show response increase or decrease with learning.

Interim Summary Operant conditioning: subclass of instrumental conditioning Organism responds at its own rate. Complex responses may be trained by: Shaping Reinforcement of progressive approximations. Chaining Training a sequence of responses, one step at a time.

8.2 Brain Substrates

Brain Substrates The Basal Ganglia (BG) and Instrumental Conditioning Mechanisms of Reinforcement in the Brain

25 BG and Instrumental Conditioning BG help connect information from the sensory and motor cortices to make a behavioral response. BG may serve as storage for S–R associations (especially those in which R is a movement). With BG lesions (in dorsolateral striatum): Rats learned to lever-press for food. But showed impaired discriminative S training.

26 Basal Ganglia

27 Reinforcement in the Brain: This figure shows that instrumental learning may involve the interaction of several neural systems.

28 Electrical Brain Stimulation One of the “pleasure centers” is the ventral tegmental area (VTA) in the brainstem. The VTA is the center for dopamine neuromodulation. VTA stimulation = powerful reinforcement

29 Consequence C: Electrical Brain Stimulation Motor System (e.g. basal ganglia) Reinforcement System Stimulus S Response R (Sight of lever) (Press lever) Visual System (e.g. visual cortex) Taste System (e.g.brainstem gustatory nuclei) Hungry? Electrical Brain Stimulation: Brain stimulation may directly activate the brain's “reinforcement” system, eliminating the need for natural reinforcers (e.g., food).

30 Dopamine and Reinforcement Some VTA axons extend to the nucleus accumbens in BG. Nucleus accumbens sends dopamine to motor areas in the striatum. Dopamine may be the physiological basis for the “wanting” aspect of reinforcement. “Motivation” or “wanting” in chemical form May contribute to addictive behavior.

31 Reward Prediction by Dopamine Neurons Schultz (2002) trained monkeys to press a lever for food. Electrophysiological recordings indicate that dopamine neurons in a monkey’s midbrain signal reward (or omission of reward).

32 Reward Prediction by Dopamine Neurons In study: Dopamine neurons in a monkey’s midbrain respond strongly after unexpected rewards. If light occurs before food, dopamine neurons increase activation after light, but not after food. Dopamine neurons decrease activity after an expected reward does NOT occur (omission). Illustrates reward prediction hypothesis i.e., dopamine is involved in predicting future reward.

33 (A) Unexpected juice reward: (B) Reward is predicted by light stimulus: (C) Predicted reward is omitted: Adapted from Schultz, 2002.

34 Opioids and Hedonic (Liking) Value Endogenous opioids (endorphins) may mediate “liking.” Opiates (heroin, morphine) bind to the brain’s natural opiate receptors. Opiates may provide information about “liking” that helps stimulate VTA’s “wanting” system.

Interim Summary In the brain, instrumental S-R-C associations may be stored in corticocortical connections and via basal ganglia. Brain’s reinforcement system may include release of dopamine from ventral tegmental area to basal ganglia. Drugs that interfere with the dopamine system disrupt instrumental conditioning.

Interim Summary Several hypotheses on interaction of dopamine and reinforcement. Anhedonia hypothesis: Dopamine gives reinforcers their “goodness.” Incentive salience hypothesis: Dopamine modulates “wanting” rather than “liking” (how hard an organism is willing to work for reinforcement). Reward prediction hypothesis: Dopamine signals whether reinforcement is expected.

Interim Summary Whereas dopamine may be involved in “wanting,” endogenous opioids may be involved in “liking.” Drugs that affect brain opiate receptors affect hedonic (“goodness”) value of primary reinforcers and punishers (e.g., food and pain).

8.3 Clinical Perspectives

Clinical Perspectives Drug Addiction Behavioral Addiction Treatments

40 Drug Addiction Pathological addiction—a strong habit maintained despite harmful consequences. Involves craving a high “euphoria” and avoiding withdrawal. Seeking pleasure involves positive reinforcement. Avoiding pain involves negative reinforcement. As indicated by the incentive salience hypothesis, dopamine is involved in “wanting” a drug.

41 Effects of Drugs on Dopaminergic Neurons

42 Behavioral Addiction Behavioral addiction—addiction to certain behaviors, rather than drugs. Produces euphoria. Understanding drug addiction may help understand/treat behavioral addictions. Examples: Compulsive gambling, eating, sex, Internet use, shopping, exercise, work Everynight Images/Alamy

43 Behavioral Addiction 7Kw Everynight Images/Alamy

44 Treatments Naltrexone (drug) treatment: Indirectly inhibits dopamine production; may help treat heroin addicts and compulsive gamblers. (Cognitive) behavior therapies: e.g., extinction, distancing, reinforcement of alternative behaviors, delayed reinforcement Based on instrumental conditioning principles.

Interim Summary Addictive drugs (e.g., heroin, caffeine) may hijack brain’s reinforcement system. May be psychological as well as physiological addiction. Behavioral addictions may reflect same brain processes as drug addictions.

Interim Summary Treatment for people with addictions may include: Cognitive therapy Medication Behavioral therapy Including principles learned from instrumental conditioning.