1. The dynamics of active information maintenance
Kim Uittenhove
Université de Genève

2. Working memory Actively maintain information over time
Cornerstone of human cognition
Severely limited capacity (i.e., 4 chunks, Cowan, 2001; 2015) and duration (in the order of seconds, Baddeley, 2003)  curtails human abilities
Information weighed in during decision making
Planning ahead of chess moves
Reasoning abilities, fluid intelligence
As humans we crucially posess the ability to keep information active over time, in spite of the physical absence of this information. This ability is called working memory and forms the cornerstone of complex cognition. Because of its important role, working memory has been widely studied for the past 40 years, with special attention to the limits of its capacity. As it turns out, working memory has a severely limited capacity, and representations held in this system are prone to loss over a very short timecourse in the order of seconds. These limits will curtail our abilities for complex cognition in a straightforward way. Consider for example the number of moves that can be planned ahead in a chess game, the elements of information that can be simultaneously weighed in while making a decision, and the number of elements that can be considered during the reasoning abilities that comprise fluid intelligence.
It goes without a doubt that the study of the ins and outs of wm and it’s limits is an important endeavour with implications for a large range of abilities in the domain of complex cognition

3. Source of WM limits While distracted from the target information
= Forgetting
Maintaining information in a state appropriate for processing is key to our information processing in the physical absence of this information, however the capacity to do so is limited in quantity of information as well as duration of maintenance.
The source of limits in the capacity and duration of representations in working-memory can be brought back to the problem of forgetting. Forgetting crucially happens during episodes of distraction, when attention is no longer on the information that we are trying to maintain. In this case, representations held in wm are prone to quick forgetting over a matter of seconds.

4. WM capacity = capacity to counteract forgetting
Limited quantity of information (in reality around four, Cowan, 2001)
Limited resource = bottleneck
Mechanisms to counteract forgetting depend on the nature of forgetting
WM capacity can be viewed as the capacity of the system to counteract this forgetting for a certain quantity of information. The fact that it can do so for very few items (i.e., around 4) suggests that these protective processes load high on limited cognitive resources, thus constituting a bottleneck in cognition. In order to understand the nature of these limited processes that protect representations from being lost, we need to understand the nature of forgetting

5. Nature of forgetting
Decay of target information in the absence of attention
Interference between representations
One possibility that naturally comes to mind is that representations naturally decay over time, when nothing is done to maintain them (Brown, 1958; Reitman, 1974). Though, little is known about the exact nature of the mechanisms that would be responsible for temporal decay, rendering this account problematic (McGeoch, 1932). Promising avenues have only recently started to offer solutions to the problem. Let us for example consider that, on the neuronal level, information is represented by sets of neurons that fire in synchrony (e.g., Deiber et al., 2007). Temporal decay could correspond to a gradual de-synchronization of this firing pattern so that representations gradually disintegrate, unless they are rehearsed (Jonides et al., 2008). However, temporal decay is faced with another major problem, as it seems tricky to develop experimental paradigms that permit to observe its action, without the confounding influence of other factors, mainly that of increased interference over time.
This brings us to the second possibility, according to which loss is actually the consequence of increased interference that arises over time. When attention is captured by distracting information, this information will be encoded and its representation will be mixed up in the same neuronal structures as the target information. This will lead to noise that will make it harder to distinguish and retrieve the target information. Interference is a complex phenomenon (e.g., Crowder, 1976) and can occur at many stages including encoding, retrieval, and storage. The interfering representations can be formed before the target representation (proactive) or afterwards (retroactive). The exact quantity of interference generated by each distractor is a complex function of its strength and the amount of overlapping features. In spite of its complexity, the interference-only account for the degradation of target information is nowadays the dominant view.

6. Counteracting forgetting in WM
Restoring decayed target information
Attentional refreshing (Cowan, 1992, 1995; Johnson, 1992)
Domain-specific rehearsal systems (i.e., phonological loop, Baddeley & Hitch, 1974)
Redintegration (Hulme et al., 1999)
Reconstruction (Barrouillet & Camos, 2015)
Reducing interference from distractor representations (Lewandowsky, Oberauer, & Brown, 2009)
To counteract decay or interference, several protective mechanisms have been imagined. Explain the mechanisms

7. Loss & maintenance Loss Time-based decay Interference Maintenance
Attentional refreshing
Rehearsal
Reconstruction
Redintegration
Distractor removal
Whatever the exact nature of the mechanisms involved, recall depends on a temporal dynamic between episodes of loss and episodes of maintenance, with a top-down role for the modulation of attention (Gazzaley & Nobre, 2012).
Recall of information depends on a temporal dynamic between loss & maintenance

8. The dynamic between two opposing forces
Complex span task
Loss of target representations (decay or interference) L C
Recall 3 4 8 5 2 9
Based on whether information is lost through decay, or through interference, competing models have been advanced to formalize the interplay between loss and maintenance. The operation of this interplay can be studied in a complex span paradigm, in which elements of information to be remembered are presented sequentially, and interspersed with distractors that require processing, and episodes of free time.
Maintenance of target representations (refreshing or distractor removal)
How is recall performance determined ?

9. The dynamic between two opposing forces: TBRS
Decay and refreshing (i.e, TBRS, Barrouillet, Bernardin, & Camos, 2004)
 Cognitive load
How do loss and maintenance operate according to TBRS
According to TBRS, the important factor in determining memory performance is the time-based sharing of limited processing resources. This can be more precisely defined by the ratio between the time where attention is diverted by distractors, and the time where it is available for maintenance activities. This proportion is known as the cognitive load (Barrouillet, Bernardin, & Camos, 2004), and will determine the number of items that can be sufficiently refreshed to survive successive distractors.

10. The dynamic between two opposing forces: SOB-CS
Distractor encoding and removal (i.e., SOB-CS, Oberauer et al., 2012)
 accumulation of interference
Instead, the SOB-CS model (Oberauer et al., 2012) proposes that the determining factor is the alternation between distractor encoding and active distractor removal.

11. Signature of the operation of loss & Maintenance
Decay and refreshing
Cognitive load
Effect of processing duration
Effect of free time duration
Distractor encoding and removal
Accumulation of interference
Effects of the number of distractors
We manipulated the number and the duration of distractors interspersed between the target items. Both the TBRS and the SOB-CS model allow to make predictions regarding the effect of the number and the duration of processing episodes. According to TBRS, attention will be unavailable during distractor processing, so that target information (letters or spatial locations) cannot be maintained for a certain duration and will degrade accordingly. This leads to the prediction of an effect of the duration of these distracting episodes on recall performance. The free time following each distractor will be used to refresh a certain number of target items sufficiently in order to survive successive distractor episodes. Hence, the number of such distractors should not have additional effects to the duration. According to TBRS, it does not matter how many distractors intervene between memory items, as long as the total proportion of time spent solving them compared to free time is kept equal. Conversely, according to the SOB-CS model the number (and not the duration) of distractors should be the crucial factor determining recall performance, since every new distractor will be obligatorily encoded.
In our experiments, distractors were arithmetic problems and we manipulated the duration by changing the presentation format, given that solution times are longer for problems presented in number-words than problems presented in numbers.

12. Effect of duration and number of processing episodes
In a complex span procedure
Maintenance of letters or spatial locations

13. Duration of Distractor task
600 ms
Longer processing duration
Constant free time
four three
six two
Give a trial example irl
Is the red number the smallest of the set, yes or no ?

14. Spatial locations

16. Red number smallest
Red number not smallest

17. Red number smallest
Red number not smallest

18. Red number smallest
Red number not smallest

19. Red number smallest
Red number not smallest

21. Red number smallest
Red number not smallest

22. Red number smallest
Red number not smallest

23. Red number smallest
Red number not smallest

24. Red number smallest
Red number not smallest

25. Click on the spatial locations in the correct order

26. Click on the spatial locations in the correct order

27. Click on the spatial locations in the correct order

28. Signature of the operation of loss & Maintenance
Decay and refreshing
Cognitive load
Effect of processing duration
Distractor encoding and removal
Accumulation of interference
Effects of the number of distractors
We manipulated the number and the duration of distractors interspersed between the target items. Both the TBRS and the SOB-CS model allow to make predictions regarding the effect of the number and the duration of processing episodes. According to TBRS, attention will be unavailable during distractor processing, so that target information (letters or spatial locations) cannot be maintained for a certain duration and will degrade accordingly. This leads to the prediction of an effect of the duration of these distracting episodes on recall performance. The free time following each distractor will be used to refresh a certain number of target items sufficiently in order to survive successive distractor episodes. Hence, the number of such distractors should not have additional effects to the duration. According to TBRS, it does not matter how many distractors intervene between memory items, as long as the total proportion of time spent solving them compared to free time is kept equal. Conversely, according to the SOB-CS model the number (and not the duration) of distractors should be the crucial factor determining recall performance, since every new distractor will be obligatorily encoded.
In our experiments, distractors were arithmetic problems and we manipulated the duration by changing the presentation format, given that solution times are longer for problems presented in number-words than problems presented in numbers.

29. In reality: a mixed signature
Overal effect of duration
No effect of the number of distractors for recall of letters
Effect of the number of distractors for recall of spatial locations
 Unexpected role for the nature of the information
In line with a decay hypothesis, the duration of these distractors did seem to crucially impact recall, given a constant free time duration. However, the effect of the number of these distractors was strongly dependent on the type of memoranda maintained. Whereas there was no effect of this parameter for the maintenance of letters, clear effects appeared when spatial locations had to be maintained.
Neither of the models currently take into account that the dynamic between loss and maintenance depends on the nature of the information that is maintained. Theoretically, there is no reason to suppose that a distractor removal process (SOB-CS) would operate differently as a function of the nature of the target information. However, the refreshing mechanisms included in a decay-and-refresh model could theoretically be assumed to be more or less efficient as a function of the information to be maintained.
Barrouillet, Uittenhove, Lucidi, & Langerock (submitted)

30. ????????????
Why does visuospatial information degrade over successive processing episodes ? Why does verbal information resist over successive processing episodes ? Why do none of the proposed models account for the characteristics of the information maintained ?
Why do the models not currently account for this and other potentially important characteristics of information to be maintained, and how can this be amended ?

31. The legacy of the standard model
The multi-component working-memory model (Baddeley & Hitch, 1974)
Specific yet equivalent systems for maintaining verbal and visuospatial information
 An equilibrium between loss and maintenance will be similarly settled for any type of information (TBRS)
Central executive = storage + processing
Phonological loop: verbal storage
Visuospatial sketchpad: visual storage
Episodic buffer: Multimodal integration
Long-term memory
The answer to this question may be found when looking at the standard model of working-memory which has dominated reseach for the past years, the assumptions of which implicitly underlies much of todays working-memory research, even though competing theories have since been advanced, such as the embedded process model (activated part of memory, Cowan). Explain the model. So as we can understand, this model evokes specific yet equivalent storage systems for verbal and visual information.
the implicit assumption is that an equilibrium will necessarily be settled, in a similar way for any type of information. However, depending on the stimuli, maintenance may differ to the extent that it alters this interplay in a fundamental way

32. Against equivalent maintenance
Does visuospatial information benefit from refreshing processes to the same extent as verbal information ?
Phonological loop (Vergauwe et al., 2014) ?
Unrefreshable features (Ricker & Cowan, 2010) ?
Sudden death (Zhang & Luck, 2008) ?
the existence of equivalent systems for the storage of visuospatial and verbal information can be questioned.
One possibility to account for the pattern observed with visual information within TBRS is to suppose that visual information does not benefit from refreshing processes to the same extent as verbal information, and thus progressively suffers from the succession of distracting episodes. Several aspects of visuospatial information could account for such lesser refreshability. Explain the possibilities.
In line with this possibility, Ricker and Cowan (2010) have suggested that some types of visual stimuli contain non-refreshable features that are inevitably lost over time.
Another possibility is that visual information does not have access to similarly efficient maintenance mechanisms as verbal information has access to the phonological loop (Vergauwe et al., 2014). a large body of research only supports such a system for verbal information (i.e., a phonological loop) and not for visual information (Morey & Bieler, 2012; Morey & Mall, 2012; Morey et al., 2013).
The lower refreshability of visuospatial information becomes especially difficult to ignore when considering results from Lilienthal, Hale, and Myerson (2014), who observed that memory for locations was lost over time even when attention was available for maintenance.

33. Against a modular view
Multi-component standard model: processing goes through a verbal stream or a visual stream, with processing dependent on which system is initiated
Diversity in the processes involved in maintaining information:
VSWM = a set of specific processes instead of the operation of a unitary system (Zimmer, 2008)
Verbal WM: distinctions between phonological, semantic and syntactic working memory (Martin et al., 2004)
And many other distinctions (Postle, 2006)
However, there is more to the legacy of the standard model, which is a modular view on the maintenance of information of different modalities. Crucially, an increasing body of results are difficult to accommodate by this view.

34. Subsystems of different efficiency
No equivalent maintenance of different information
Diversity of processes
Proliferation of new subsystems of differing efficiency ?
Or a change in conception ?

35. A change of conception
The mechanisms recruited for information maintenance are flexible
And depend on the processing the stimuli can support
And are not simply those mechanisms packed in predefined subsystems
(Ricker et al. 2016)
Affects the equilibrium between loss & maintenance

36. A novel perspective Taking into account additional stimulus dimensions
Continuous
Verbal
Categorical
Visual
Rich
Taking into account additional stimulus dimensions
Scale (continuous versus categorical)
Richness (LTM representations)
 maintenance mechanisms
 the dynamic between loss & maintenance
However, it should be noted that spatial locations and letters differ not only on the visual-verbal dimension, but in other ways as well. For example, random spatial locations have very little connection to knowledge in LTM, and are of continuous nature,
Poor
A 3D-space of stimulus characteristics

37. Application of the framework to recent results
Uittenhove & Barrouillet (in prep)
Number of distractors
Stimulus dimensions
Balance of loss and maintenance ?

38. Number of distractors C L L C L C L C 4 4 4 4 PE PE PE PE PE PE PE PE

39. Stimulus dimensions
The auditory-visual dimension: Codes in the auditory domain afford additional mechanisms (i.e., phonological loop).
The categorical-continuous dimension: The categorical nature of stimuli may make them more distinguishable and thus easier to refresh.
The richness of the stimulus: Rich stimuli have more elaboration potential, because they can be associated to more representations in LTM.

40. Balance of loss & maintenance
The amount of loss equals amount maintained
LC T
LC T
LC T
LC T
LC T
LC T C LC LC T F S L PE PE PE PE PE PE PE PE PE L PE L
The amount of loss is equal to the amount that can be maintained
The amount of letters that can be sufficiently refreshed to withstand subsequent loss

41. Letters vs spatial locations
Verbal material
No influence of number of PE
Balance of loss and maintenance
Visuospatial material
Influence of number of PE
No balance of loss and maintenance
n.s.
Explain the different materials ?
More decay for visuospatial ? Or less efficient refreshing
The number of processing episodes had no effect on the maintenance of verbal information, but did affect the type of visuospatial information used here, which degraded as the number of intervening processing episodes increased. However, a floor effect was soon reached at a span of around 1.7 items, performance not further decreasing beyond this limit. Our results suggest substantial differences in maintenance processes underlying verbal and visuospatial information, questioning the basic tenets of TBRS and SOB-CS, and of the multi-component model (Baddeley & Hitch, 1974).
The following experiments will further explore the nature of these differences. **
Whatever the processes at play, they seem highly dependent on the type of memoranda

42. Blocking specific maintenance mechanisms
The phonological loop
Articulatory suppression
L… L… L… LC… LC… LC… L C 3 4 8 5 2 9
The aim of this experiment was to test the hypothesis that the differential effect that the increase in the number of processing episodes had on the balance between loss and maintenance of verbal and visuospatial information is due to the lack of a mechanism of maintenance specifically dedicated to visuospatial WM. If this is the case, the same decline should be observed for verbal WM by making the specific verbal maintenance system (i.e., the articulatory rehearsal mechanism) unavailable through concurrent articulation (i.e., repetition of the same syllable). Therefore, we compared a condition where participants were required to maintain letters with or without articulatory suppression (AS), each letter being followed by either 2 or 6 processing episodes.
BA DI BOU BA DI BOU BA DI BOU BA DI BOU BA L C L LC 3 L 4 L 8 5 LC 2 LC 9

43. The phonological loop
n.s.
Role of the phonological loop in the balancing of loss and maintenance of verbal information
***
The results of this experiment are in line with the hypothesis that the balance between loss and maintenance of verbal WM over successive processing episodes, is at least in part due to the availability of a mechanism of maintenance specific to verbal material, and that this mechanism relies on articulatory rehearsal. When the use of this mechanism is hindered by AS, verbal WM declines with the number of processing episodes just like visuospatial WM does. Therefore, articulatory rehearsal seems instrumental in the achieving of a balance between loss and maintenance for verbal material. This suggests in turn that the visuospatial information used until now does not afford such rehearsal mechanisms.

44. Doubling free time L L L LC LC LC
BA DI BOU BA DI BOU BA DI BOU BA DI BOU BA DI BOU L C L LC 3 L 4 L 8 5 LC 2 LC 9
Maintenance mechanisms other than the phonological loop could balance loss and maintenance given sufficient time
BA DI BOU BA DI BOU BA DI BOU BA DI BOU BA DI BOU BA DI BOU BA DI BOU BA DI BOU L L L L C LC LC LC 3 4 8 5 2 9

45. Utilization of free time
Maintenance mechanisms other than the phonological loop can balance loss and maintenance given sufficient time
- For verbal information
- But not for the kind of visuospatial memoranda tested here
***
n.s.
whereas the number of processing episodes did no longer significantly affect verbal recall under AS (mean spans of 3.21 and 3.00 for 2 and 6 processing episodes, respectively), F (1, 25) = 1.41, p = 0.25,
visuospatial recall performance still decreased between 2 and 6 processing episodes (mean spans of 3.63 and 2.77 for 2 and 6 processing episodes respectively), F (1, 25) = 16.35, p < .001.
Note that extra time did not help to reach a stable balance between loss and maintenance for visuospatial material, in spite of the higher initital rates of recall of memoranda. Some features of the visuospatial memoranda that we use here can not be kept stable over successive processing episodes, thus a balance between loss and maintenance is not reached
Now it is time to look at the type of visospatial memoranda that we use
Categorical nature ?
Richness ?

46. making visual information more categoricalx
One peculiarity of the spatial locations in our study is their high level of similarity, differing only by the single feature of their location in the grid, which is continuous in nature. It seems that categorical information may be maintained differently
Articulatory suppression, free time 600 ms

47. Loss & maintenance for categorical visual information
No effect of the number of processing episodes
A balance was reached between loss & maintenance for discrete abstract figures
Categorical nature of visuospatial memoranda plays a role in how they can be maintained to overcome successive distractors
n.s.
Mean spans of 1.50 and 1.52 for 2 and 6 episodes , F < 1

48. Increasing LTM associations for verbal information
Words vs non-words
Pinguin
Rintong R
Tong G

49. Loss & maintenance and semantical richness
Selective strengthening of words over successive PE
Independent from rehearsal mechanisms
LTM representations influence maintenance
600 ms free time
-Number of processing episodes x type of memoranda (p < .01)
-Increasing number of PE had a significant positive effect for words, and a tendential negative effect for non-words
-No interaction with AS

50. factors that balance loss & maintenance
Continuous
Verbal
Categorical
Visual
Rich
Phonological loop
Categorical visuospatial information
Rich LTM representations for verbal material
Further examination is needed to complete the puzzle
For example, do the different dimensions interact or are they orthogonal as presented here ?
Additive effects ?
Poor

51. Conclusion
The interplay between loss and reconstruction does not always reach an equilibrium.
The characteristics of the information to be maintained (its semantic richness and categorical format)determine the equilibrium between loss & maintenance:
reconstruction overcomes loss
contains loss
or fails to counteract loss
Interestingly, we obtained further results suggesting that the benefits from attention increase when the number of distinguishable features of the information is higher, or when more LTM knowledge is available to construct sturdier representations. Overall, our results suggest that the interplay between loss and reconstruction does not always reach an equilibrium. Depending on the nature of the information to be maintained and its semantic richness and discriminability, reconstruction either overcomes loss, just contains it, or fails to counteract it. This highlights the need for an alternative framework that focuses on relevant stimulus dimensions, how these interact with maintenance mechanisms and how they alter the interplay between loss and reconstruction.

52. Further avenues
Complete understanding of what factors influence the maintenance mechanisms, and the balance between loss & maintenance
 revision of models that characterize this interplay (TBRS, …)
An avenue for exploring other questions
Nature of representations maintained
Development and aging

53. Lifespan changes in WM dynamics
Development of the dynamics of loss and maintenance(Camos & Barrouillet, 2014):
Processing speed  the duration of processing steps during which memory traces degrade
Cognitive control  purposefully direct attention in the working-memory task
Other mechanisms that may crucially depend on the characteristics of the stimuli:
Speech rate and the phonological loop
Increasing LTM knowledge and elaborating certain stimuli
The way in which continuous information is maintained
The study of working-memory in the lifespan may benefit from taking into account how the characteristics of the maintained information may alter the interplay between loss and maintenance, and thus determine working-memory capacity
These methods will allow us to determine how stimulus characteristics alter working-memory dynamics throughout development and aging. This will allow us to more precisely pinpoint the mechanisms that undergo evolution in childhood and aging. For example, studying rich information that has great elaborative potential may elucidate the role of LTM knowledge and elaborative rehearsal in the working-memory capacity of older adults and children
The processing speed increase in children and the decline thereof in older adults could then explain part of the variation in working-memory capacity.