Test-Retest Reliability and Convergent Validity of Measures of Executive Function Oliver Sawi & Kenneth Paap Department of Psychology, San Francisco State University, San Francisco, CA The construct of executive function (ef) often viewed as a set of interrelated processes all involving the prefrontal cortex, with each component recruiting other constellations of cortical function. Smith and Kosslyn (2007) partition executive processing into five sub-processes: executive attention, conflict monitoring, switching, sequencing, and response inhibition. Miyake, et al. (2000) have proposed a similar family of interrelated executive processes. This framework, in contrast to the unitary view of the central executive (Baddeley, 1986), allows for the possibility that the sub-processes have some degree of anatomical and functional independence. Consequently, one could be good at interference tasks that require only executive attention (inhibitory control) and poor at tasks that require only switching. Our principle argument is that an investigation of individual differences in executive function should be grounded in a specific conceptual framework that elucidates the nature of executive processes and guides operational definitions for manipulating and measuring them. Furthermore, there needs to be more converging evidence that corresponding indicators from different tasks correlate with one another and, therefore, are valid measures of a specific type of executive function. Participants The 53 participants were psychology majors at SFSU who volunteered to participate as one alternative for satisfying a class requirement. Participants have diverse backgrounds, for example, 26 were monolingual and 27 billingual. Method Introduction No Convergent Validity for Executive Attention All participants completed four computer-controlled tasks commonly assumed to require executive control and a demographic survey. Participants then returned a week later, at the same time, and completed the four computer-controlled tasks again. Antisaccade Task. Task on each trial was to identify the target stimulus (“B”, “P”, “R) by pressing the key with the corresponding label with three fingers of the right hand. (“ ”,”?”) Briefly presented target followed by visually similar mask (“8”). Antisaccade condition: a distractor stimulus is always blinked before on the opposite side from the target stimulus. Flanker task. On each trial either a left or right pointing arrow appeared at fixation and participants were required to press the corresponding key. Half of the trials were incongruent - the central arrow was flanked by a pair of arrows pointing in the opposite direction. No evidence for a Bilingual Advantage in Executive Function Cognitive/Neural Components of Executive Function References Color-shape switching task. On each trial a single target appeared that was either a circle or a triangle and colored either blue or red. Block 1: participants always made color decisions with two fingers of the left hand. Block 2: they always made shape decisions with two fingers of the right hand. These two “pure” blocks were combined to provide a baseline for assessing mixing costs and switching costs. Block 3: the target stimulus was always preceded by a precue Half of the trials were “repeat” trials, half were “switch” trials. Mixing Costs: difference between the mean of the pure trials and the mean of the repeat trials Switching Costs: difference between the mean of the “repeat” and and “switch” trials Baddeley, A. (1986) Working memory. Oxford. Clarendon Press. Fan, J. I., McCandless, B. D., Thomas, K. B., & Posnr, M. I. (2003). Cognitive and brain consequences of Keye, D., Wilhelm, O., Oberauer, K., van Ravenzwaaij, D. (2009). Individual differences in conflict-monitoring: Miyake, A. Friedman, N. P., Emerson, M. Witzki, A. H., Howerter, A., & Wagner, T. D. (200). The unity and Smith, E. E., & Kosslyn, S. M. (2007). Cognitive psychology: mind and brain. Upper Saddle River, NJ: Pearson conflict. NeuroImage, 18, testing means and covariance hypotheses about the Simon and Eriksen flanker task. Psychological Research, 73, diversity of executive functions and their contribution to complex “frontal lobe” tasks. CognitivePsychology, 41, Prentice Hall. Lack of Convergent Validity Across Measures of EF TaskSimonFlankerMixingSwitchingAntisaccade Simon Effect Flanker Effect Mixing Costs Switching Costs Antisaccade Table 5. Correlations between measures of EF Group Main Effect: F(1, 56) = 5.08, p =.028 Group x Congruency Interaction: F(1,56) = 5.56, p =.022 Group Main Effect: F(1, 51) = 3.56, p =.065 Group Main Effect: F(1, 56) = 4.34, p =.042 Group x Congruency Interaction: F(1,56) = 0.811, p =.372 Group Main Effect: F(1, 55) = 8.52, p =.005 Group x Congruency Interaction: F(1,55) = 3.48, p =.068 _nEng. Prof.Eng AOAOther ProfOther AOA% Eng Switching Frequency PED Bilinguals % Monolinguals % Table 6. Language Group Demographics Discussion There were no main effects of group nor were there any GroupxCongruency interactions. In fact, the only significant difference between the groups was in the form of a bilingual disadvantage in the Simon Effect (replicating a finding from a previous study). In addition, the antisaccade and flanker effects also trended towards a bilingual disadvantage. Further, the low test-retest reliability and lack of convergent reliability may contribute to inconsistent findings for the bilingual advantage in EF. Test-Retest Reliability of Measures of Executive Function (EF) TaskrnExact p Antisaccade Effect Flanker Effect Simon Effect * Switching Costs Mixing Costs rdfexact p * Taskrdfexact p Antisaccade Effect Flanker Effect Simon Effect Switching Costs Mixing Costs rdfexact p Table 1. Session 1 vs. Session 2 Table 2. Block 1 to Block 2 (Session 1) Table 3. Block 2 to Block 3 (Session 1) Table 4. Block 1 to Block 3 (Session 1) Discussion The measures of EF produce moderate to low reliability 1. between sessions and 2. between blocks within the same session. For example, while significant, mixing cost in session 1 can only account for 22.3% of the variance in session 2. However, the Simon Effect shows significant block to block reliability, but not session to session. Taken together, this suggests that the Simon Effect is more vulnerable to day to day variation than the other purported measures of EF, more specifically, Executive Attention (antisaccade, flanker) and undermines the idea that these tasks are measuring the same construct. Simon task. On each trial either a Z or / appeared to the left or right of fixation and participants responded by pushing the corresponding key. Half of the trials were incongruent in that the spatial location of the target was opposite the location of the correct response key. Results Discussion The absence of an association between interference effects has important implications for the standard practice of using these tasks as indicators of individual differences in the ability to control executive attention. If there is no association, then only one of the tasks could require executive attention or the inhibitory control exercised in the tasks are completely task dependent. This lack of convergence undermines the confidence that individual or group differences in the magnitude of the flanker, antisaccade or Simon effect signals differences in executive attention. Eng. Prof. Scale level 6 is "as fluent as a typical native speaker", level 7 is "better than a typical native speaker" AOA Age of Acquisition (native language coded as 0) PED Parent's Educational Level t(49)=1.82, p =.075 Executive Attention (Inhibitory Control). The difference between the congruent and incongruent trials in both the Simon and flanker task is used as an indicator of executive attention. The RT for the antisaccade task is also used as an indicator. *All three tasks yield significant interference effects that are typical in magnitude for each task.