1. How do animals “know” when a schedule is on extinction?

2. Remember
Operant conditioning extinction differs from classical conditioning extinction
During extinction, responding decreases to baseline levels for both classical and operant conditioning
BUT: Operantly conditioned behavior shows:
Transient increase
Extinction induced aggression
Question is: Why!?!

3. Neuringer, Kornell & Olufs, 2001
2 groups of rats
Examined response variability
Experimental chamber:
2 response levers: one lever on opposite walls (front and back)
During reinforcement: 3 responses in row to obtain food pellet (in any combination)
One group reinforced for variability of responses
Got food only if sequence was different that what did on an earlier trial
Second group: just any 3 responses
Then both groups shifted to EXT condition:
Predictions on rate of EXT in each group?

4. Results: Examined Rats in variability group
Last 4 sessions of reinforcement phase
First 4 sessions of EXT
Rats in variability group
INCREASED variety of responses during initial extinction,
But reduced responses overall (as predicted by extinction)
Rats in control group showed increase in variability as well! (but never reinforced for variability)

5. So: Why is this study important?
Extinction:
Decreased overall responses
But INCREASED variability of those responses
Animals tried something DIFFERENT
Remember this is for operant conditioning
Animal is in control of contingency
Try something NEW in case the contingency changed
But quit with no reinforcement

6. Emotional effects of Extinction
Frustration: Frustrative non-reward energizes behavior!
Aggression = Frustration aggression
Will aggress towards model if model placed in cage with rat or pigeon during initial extinction
Will bite or attack objects
Will attack a real cohort
Does this occur in humans?

7. What happens to learning during EXT?
Extinction does NOT reverse effects of acquisition
Reduces responding
Emotional outcomes
But does not erase learning!
We aren’t “forgetting” the contingency, but learning a new one!
Four lines of research demonstrate this:
Spontaneous recovery
Renewal
Reinstatement
Reinforcer devaluation

8. Extinction and Spontaneous Recovery
EXT produces decline in conditioned responding, but this decline dissipates with time
That is, the response begins to come back
Because there is no reinstatement of the contingency, this is called spontaneous recovery
Spontaneous recovery = return of previously extinguished response
Critical factor: introducing period of rest between end of extinction training and assessment of responding
Behavior that is suppressed during extinction training recovers after a period of rest
Original learning trumps extinction condition
Better the response is originally learned, and the shorter the extinction training, the more likely spontaneous recovery

9. Renewal of Responding Renewal
Recovery of excitatory responding to an extinguished stimulus
Produced by a shift away from the contextual cues that were present during extinction.
Responding will recover from extinction if extinction trials occur in contextually different situation than original training

10. Renewal of Responding Bouton and King, 1983
Used conditioned suppression to study acquisition and extinction of conditioned fear in rats
Rats first trained to press response lever for food
Then paired CS with foot shock:
Trained in 1 of 2 operant chambers
Shock condition had distinctively different contextual cues

11. Renewal of Responding Bouton and King, 1983
Rats assigned to 1 of 3 groups for extinction phase:
2 groups: 20 EXT trials of CS but no shock
Group A: extinction in SAME context as original fear conditioning
Group B: extinction in different context as original fear conditioning
Group C: no extinction trials = control group

12. Renewal of Responding Results:
Great suppression at start of EXT for both groups
Then, following EXT in either distinct or similar context: series of test trials in ORIGINAL setting
LEAST suppression for group that received training in different context
Training in different context resulted in less generalization

13. Renewal of Responding Why?
Renewal effect occurs because memory of extinction is specific to cues that were present during extinction trials
Shifting away from the context of extinction disrupts memory of extinction
Appears that original learning of response-reinforcement is better remembered than sequence of extinction

14. Renewal of Responding Important applied implications!
Even if a therapeutic intervention successfully results in extinction of a response, the original response may return if the person or animal encounters contexts that are highly similar to the original learning!
Extinction must be well generalized to overcome effects of response-reinforcer contingency!

15. Reinstatement Reinstatement
Recovery of responding to an extinguished stimulus produced by exposure to unconditioned stimulus
Again- problematic for applied situations:
Reduce behavior through EXT
A pairing of the response and reinforcer occurs
You just “reinstated” the behavior

16. Reinstatement Tends to be somewhat context specific
Supports theory that conditioning occurs due to contextual stimuli
S+ for reinforcement; S+ for extinction
Response-reinforcer situation is often fairly context- specific
EXT then, is also context specific and may easily be reinstated
Context of conditioning PLUS any remaining excitement to CS/S+ could summate into a reinstatement of responding
In a way, is a more context specific form of renewal

17. What happens to R-Sr relation during EXTINCTION?
Does the organism retain knowledge of the reinforcer?
Evidence suggests YES!
IF conditioned behavior reflects an S-O responding or R-O association, devaluation of the reinforcer should produce decrement in responding similar to that found in EXT!
What do the data say?
S-O associations are not forgotten or lost during CC procedures
Specificity of reinstatement behavior shows that animal must have knowledge of reinforcer

18. Ostlunk & Balleine, 2007
Rats trained to lever press for 2 reinforcers:
1 lever = food pellet
1 lever = sugar water
Quick extinction procedure for BOTH
Then gave 1 reinforcer for next lever press on either food lever or water lever
If food given: Responding on that lever increased
If water given: Responding on that lever increased
NOT both or random lever: Rats remembered which lever that reinforcer was for!

19. Enhancing Extinction So is extinction just worthless?
No, several things can be done to increase the effectiveness of extinction
Procedures for enhancing extinction
Carefully choose the number and spacing of extinction trials
Reduce spontaneous recovery
Reduce opportunities for renewal
Use compounding extinction stimuli

20. Number and spacing of EXT trials
Increase the number of extinction trials!
Results in further decrease in responding
Conduct extinction trials in massed trials, not spaced out
No opportunity for reinstatement or recovery
No other contextual pairings can occur

21. Reduce Spontaneous Recovery
Repeat periods of extinction, rest and retest
Successive cycles reduce the likelihood of spontaneous recovery
Alternatively (and more often used in appetitive work): Increase the interval between training and extinction
Introduce cues associated with extinction

22. Reducing Renewal
Conduct extinction trials in a variety of contextual settings
Basically, generalization of extinction
Make extinction a generalized response
Present reminder cues that EXT is in effect
Again, providing specific cues NOT to respond

23. Compounding Extinction Stimuli
Present 2 stimuli at same time that both have undergone EXT: Deepens the extinction cues for BOTH cues
That 2 extinction cues increases the level of extinction suggests that extinction operates in an error-correction process (e.g., RW model)
If organism responds and gets what expected- no change
If organism responds and gets MORE than expected- more responding
If organism responds and gets LESS than expected- EXT
2 cues provide better signaling of the outcome!

24. Compounding Extinction Stimuli
VERY interesting prediction here for treatment of fear:
Conditioned inhibitor cue = “don’t do that” cue
If EXT cue is compounded (given) with a conditioned inhibitor cue during extinction training
Conditioned inhibitor = signal for ABSCENCE of a US
Conditioned inhibitor  safety signal indicating that the aversive US won’t happen

25. Compounding Extinction Stimuli
IF safety signal is compounded (given) with the fear stimulus during extinction:
Absence of US (fear) is fully predicted by safety signal
Safety signal = no fear signal
Won’t be any feedback error to encourage learning that the fear stimulus no longer ends in shock
Safety signal should BLOCK extinction of the fear stimulus
Remember blocking?
Can’t learn fear signal = no bad thing because safety signal means no bad thing.
SHOULD be interference rather than facilitation of the EXT process

26. What IS learned in EXT? Inhibitory S-R associations
Remember: learning involves S-O, R-O and S-R associations
Stimulus-Outcome (consequence)
Response- Outcome (consequence)
Stimulus-response
Extinction procedures appear to not change S-O and S-R associations of original learning
Instead: changes in S-R associations occur
Non reinforcement produces inhibitory S-R associations
Nonreinforcement of a response in the presence of a specific stimulus produces an inhibitory S-R association that serves to suppress that response whenever that S is present
What? Learn “don’t respond when presented with that S”

27. What IS learned in EXT? Inhibitory S-R associations
This helps to explain frustration and aggression
Expect that S will have a certain outcome
During extinction, that outcome for S changes
Omission of expected reward creates disappointment and frustration
Nonreinforcement triggers aversive frustrative reaction which in turn discourages responding

28. Pardoxical Reward Effects
Assume that decline in responding during EXT is due to frustative effects of an unexpected loss of reinforcement
Then would expect more rapid extinction following training that establishes GREATER expectations of reward.
This is supported by data showing the
Overtraining extinction effect
Magnitude reinforcement extinction effect
Partial reinforcement extinction effect (PREE)

29. Magnitude Reinforcement Extinction Effect
Less persistence of instrumental behavior in extinction following training with a large reinforcer
More persistence of responding with a small or moderate reinforcer.
Effect is most prominent with continuous reinforcement.
The bigger the reward for responding, the faster you quit when the reward stops coming.

30. Overtraining extinction effect
Less persistence of instrumental behavior in extinction (faster extinction) following extensive training with reinforcement than following only moderate levels of reinforcement training.
Again, effect most prominent with continuous reinforcement
The more you train, the faster you quit when the reinforcement goes away!

31. Partial Reinforcement Extinction Effect: PREE
Extinction occurs at different rates depending on the schedule:
Continuous reinforcement: FAST extinction
Partial reinforcement schedules: SLOWER extinction
Variable schedules show slower extinction than fixed (rate or time) schedules.
Partial Reinforcement Extinction Effect: PREE
Used to describe greater persistence in instrumental responding during extinction after partial (or intermittent) reinforcement training
Faster extinction after continuous reinforcement training.
Partial reinforcement schedules show RESISTANCE TO EXTINCTION

32. Why do we get these effects?
Several theories to suggest PREE and similar effects:
Discrimination hypothesis
Frustration theory
Sequential Theory
Behavioral momentum

33. Discrimination and Frustration
Discrimination hypothesis:
Mowrer and Jones 1945
In order for subjects’ behavior to change during extinction, the subject must be able to discriminate the change in reinforcement contingencies
With CRF: This is immediately noticeable
With PRF: not immediately noticeable
More discriminative on fixed schedules
Less discriminative on variable schedules
Evidence does not completely support this

34. Generalization Decrement Hypothesis
Capaldi, 1966
Responding during extinction will be:
Rapid if the stimuli present during extinction are different from those that occurred during reinforcement
Slow if the stimuli are similar to those that occurred during reinforcement.
Get more resistance to extinction if the stimuli remain the same between reinforcement and extinction
More specific than the discrimination hypothesis, in that states WHEN discrimination is likely to occur

35. Sequential Theory Sequential theory: Memory of reward vs. non reward
Cognitive theory
Fast extinction after CRF
Extinction occurs quickly because the instrumental response has NOT been conditioned to the memory of nonreward
Slow extinction during PREE
Extinction is slowed after partial reinforcement because the instrumental response becomes conditioned to the memory of nonreward.

36. May be an alternative way to think about all of this
Our behavior continues unless we have a cue or a “Reason” to alter it
Keep going unless otherwise signaled to change!
Our behavior has momentum

37. Behavioral Momentum Behavioral momentum:
Degree to which behavior is susceptible to disruption
Based on physics: a rolling object continues until something disrupts it
Momentum of a physical object is the product of its mass and its speed
A fast moving bullet and a slow moving train both have high momentum
But a slow moving train has more mass
Both the bullet and the train are hard to stop because of their momentum times their mass

38. Behavioral Momentum Behavior has both speed and mass
The better learned the response, the harder it is to stop (MASS)
The higher the rate of reinforcement, the harder it is to stop (SPEED)
The hardest behavior to stop is one that is well learned with a high rate of reinforcement!
A behavior will continue unless it meets something that disrupts it!

39. But think carefully about this
Two situations:
Situation 1: CRF with high rate of reinforcement
Situation 2: Partial reinforcement at lower rate of reinforcement
Lots of behavior in both
Now institute EXTINCTION
Which situation invokes a BIGGER disruptor
(think of the size of that bump for that rolling rock)

40. Behavioral Momentum
Suggests that PREE occurs because the animal has a high momentum of responding and it is more difficult to stop this momentum
Two major conclusions about behavioral momentum
Behavioral momentum is directly related to the rate of reinforcement
Behavioral momentum is not tied to response rate alone!

41. Behavioral Momentum What can disrupt behavioral momentum?
Extra food between experimental sessions
Drugs that alter value of reinforcement
Introduce alternative sources of reinforcement
Anything that devalues the S-O relationship!

42. Behavioral Momentum Timberlake and Lucas 1985: Ball bearing studies
Rolled ball bearing across cage; rats had to let it go past to receive reinforcer
Played with the ball bearing, slowing reinforcement
During extinction (ball bearing but no food): played with ball bearings MORE
Does suggest that animals show strong patterns of behavior that may interfere and thus slow the extinction process
Innate behavior (instinctive drift) was a disruptor for learned behavior; disrupted the behavioral momentum

43. Why is behavioral momentum important?
Give easy requests, then a harder request
Foot in the door in social psychology is an example of this
Give a little bit, give a little bit, then ask for a lot!
High-Probability Command Sequence (HPCS)
Give 3 to 5 high probability cues followed by a low probability cue
E.g., 5 easy math problems, then a harder problem
Got your flow going on the easy ones, so enough momentum to get through the hard ones
Walk for 20, run for 10

44. Bottom Line Rate of extinction is affected by: Why frustration?
Discriminability of the situation
Previous experience with non-reinforced responding
Possibly, memory of non-reinforced responding
Why frustration?
Not getting what expected

45. Bottom Line Because of contingency between response and reinforcer:
Subject “controls” responding, and thus reward
During EXT: this contingency is disrupted
But it is the S-O relationship that is disrupted
The S no longer predicts the O: Stimulus no longer predicts reward!
That pisses the organism off!
Subject becomes “frustrated” and
Tries other responses
Increases magnitude of responding
Increases intensity of responding
Pushed into fear/flight/fight sequence: aggression is elicited
Aggression interferes with responding
S-O relation takes on new meaning IN THAT CONTEXT

46. To make EXT effective: Be sure to use:
Lots of EXTINCTION trials
Massed trials
Many training, extinction, retest trials
Alter the cues that predict extinction so they become contextual cues
Then generalize those cues to all the situations that EXT should be in effect
Remember the organism doesn’t forget the old S+:R-O situation, so if that reoccurs, the old response comes back!

47. So what do we know about EXT?
It is far more difficult than “just forgetting”
In fact, animal doesn’t “forget”
Animal learns new contingency
Many things can go wrong when using extinction
Beware and be careful when using EXT!