1. Symptom Validity Test Performance in Veterans with Mild Traumatic Brain Injury and Post-traumatic Stress Disorder: Embedded versus Free-Standing Measures Mina Dunnam, Ph.D.; Andrea Miele, M.A.; Gary C. Warner, Ph.D.; Kerry Donnelly, Ph.D., ABPP; James P. Donnelly, Ph.D.; Jim Kittleson, Psy.D., ABPP; Charles Bradshaw, Ph.D. & Michelle Alt, M.A. References 1.Boone, K. B. (Ed.). (2007). Assessment of feigned cognitive impairment. A neuropsychological perspective. New York: Guilford Press. 2.Bush, S. S., Ruff, R. M., Troster, A. I., Barth, J. T., Koffler, S. P., Pliskin, N. H., Reynolds, C. R., & Silver, C. H. (2005). Symptom validity assessment: Practice issues and medical necessity NAN policy & planning committee. Archives of Clinical Neuropsychology, 20, 419-426. 3.P. Green, (2004). Green's Medical Symptom Validity Test (MSVT) for Microsoft Windows: User's manual, Green's Publishing, Edmonton, Canada. 4.Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. L., Millis, S. R. & Conference Participants (2009). American academy of clinical neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias and malingering. The Clinical Neuropsychologist, 23, 1093-1129. 5.Larrabee, G. L. (Ed.) (2007). Assessment of malingered neuropsychological deficits. Oxford: Oxford University Press. 6.Tombaugh, T. N. (1996) Test of Memory Malingering (TOMM). New York: Multi Health Systems. INTRODUCTION Evaluation of symptom validity is important in any clinical evaluation.  Both the National Academy of Neuropsychology (NAN) and the American Academy of Clinical Neuropsychology (AACN) have published guidelines for practitioners regarding the importance of effort testing (see Bush et al., 2005; Heilbronner et al., 2009, respectively). Concerns of test security and coaching as well as issues of diagnostic accuracy surrounding frequently used, free-standing symptom validity tests (SVTs) has created impetus for the development of other measures of symptom validity; “Embedded Measures” are one group of tests currently under investigation.  Embedded Measures (EMs) represent a group of tests that hold potential to evaluate symptom validity. Proponents for their use cite shorter evaluation time and more test security, as they exist within standard neuropsychological tests (see Larrabee, 2007 and Boone, 2007).  They are indices derived from specific cutoffs applied to scores within standard neuropsychological tests. Of special importance is the diagnostic validity of EMs compared to free-standing SVTs  Diagnostic validity is commonly investigated using classification accuracy statistics such as sensitivity, specificity and positive and negative predictive power. This study investigated the diagnostic validity of two EMs of effort relative to two free- standing measures in a large veteran cohort RESULTS Classification Accuracy (see Table 2). Suboptimal effort was identified in approximately 13% in this sample of veterans Classification accuracy of the EMs showed lower PPP than NPP and very low SN compared to SP. False positives were much lower than false negatives. The CVLT FCR had higher PPP than RDS, although the two EMs had similar high NPP. The CVLT FCR thus had a lower false positive rate than RDS, but both EMs had similar and high false negative rates. Logistic Regression (see Table 3) Logistic regression revealed that the two EMs significantly predicted suboptimal effort: χ 2 (2, N=439) = 43.44, p<.001. The overall accuracy rate was 88.4%, in which 98.4% of the optimal effort examinees but only 19.6% of the suboptimal effort examinees were correctly classified. In order to determine if other information typically gleaned in an evaluation accounted for any variance in effort over and above these two EMs, number of TBIs sustained, and the presence of symptoms of depression, anxiety and PTSD were also entered in subsequent models (see Table 3). Tolerance tests revealed no significant multicollinearity between predictors.  Percent accurate classification also increased to 88.8%, with 99% of optimal but still only 18.2% of suboptimal individuals correctly classified with the addition of these four predictors into the model  Compared to the 2-EM model, the addition of the PCL-M and number of TBIs accounted for the most variance in effort of the 4 self-report measures (χ 2 (4, N=439) = 60.20, p<.001. Classification was 88.6% (18.2% and 98.7% of individuals correctly classified). DISCUSSION The base rate of suboptimal effort was relatively infrequent (around 13%). The EMs RDS and the CVLT FCR emerged as statistically significant predictors, but they are both likely to misidentify a large portion of individuals. Though few individuals with optimal effort will fail these EMs, many individuals with suboptimal effort will pass these EMs. Are clinically more concerned with false positives (incorrectly labeling a person with optimal effort as having suboptimal effort).  Both EMs are likely to more accurately identify optimal effort (NPP).  Both EMs are more likely to under-identify suboptimal effort (PPP).  Compared to RDS, fewer optimal effort individuals will be over-identified as suboptimal effort using the CVLT FCR (PPP).  RDS had an unacceptably high false negative rate (low SN). Many individuals with suboptimal effort will pass this EM. o Reexamination of both cutoffs in veteran samples is warranted. The self-report measures of number of TBIs sustained as well as presence of clinically significant PTSD symptoms were also significant predictors; however when entered into the model, the PCL-M assumed much of the variance of number of TBIs. Symptoms of clinically significant anxiety and depression did not provide additional information about effort. How might the diagnostic validity of the TOMM and MSVT in veteran samples [used as comparison criteria variables] affect the classification accuracy of RDS and the CVLT FCR? Caution is advised against using either EM alone; however, when considering the combination of PPP, NPP, false positives and false negatives, RDS will inaccurately identify a smaller proportion of individuals than the CVLT FCR. METHOD: Participants and Materials Participants Data was obtained from 440 veterans of the wars in Iraq and Afghanistan participating in a large, multi-site trial. Materials and Cutoff Selection As part of a larger battery, were administered two commonly used free-standing measures of effort: the Test of Memory Malingering (TOMM; Tombaugh, 1996) and the Medical Symptom Validity Test (MSVT; Green, 2004). In addition, all examinees completed two EMs: the California Verbal Learning Test- 2 nd Edition, Forced-Choice Recognition [CVLT- FCR] at a cutoff of <14 and Reliable Digit Span [RDS] at a cutoff of ≤7 Symptoms of post-traumatic stress disorder (PTSD), anxiety and depression were collected using scores on the Posttraumatic Stress Disorder Checklist-Military Version (PCL-M; cutoff raw score of >=50), Beck Anxiety Inventory (BAI; cutoff raw score of >=10) and Beck Depression Inventory-II (BDI-II; cutoff raw score of >=14) Number of traumatic brain injuries (TBIs) sustained was determined using a semi-structured interview developed in coordination with national head injury experts and administered individually by a clinical neuropsychologist The dependent variable was categorical (optimal and suboptimal effort) and was created using pass and fail scores on the TOMM and MSVT:  TOMM cutoff: scores of <45 on either Trial 2 or the Retention  MSVT cutoff: scores of <90% on either IR, DR or the Consistency Index Suboptimal effort was defined as failing either one or both the TOMM and/or the MSVT (optimal effort examinees passed both SVTs) DEMOGRAPHICS Average age was 32.16 (SD=8.9) years. 89.1% of the examinees were male, 84.3% identified themselves as white or Caucasian, and 87% were right handed. Modal years of education was “some college,” (42.3%). 47% of the sample served in the United States Army. Approximately 56% were receiving VA disability Most of the sample sustained no TBIs (57.5%) or 1 TBI (32%). 65.8% and 56.1% reported clinically significant symptoms of depression and anxiety, respectively. 43.6% reported PTSD symptoms The sample base rate (N=439) for suboptimal effort (failing the TOMM and/or MSVT) was 12.8% (n=56). 40 examinees failed only 1 SVT (9.1%) and an additional 16 individuals failed both SVTs (3.6%). See Table 1 for descriptive information about number of TBIs and clinically significant psychiatric symptoms depending on effort. This project was funded by the US Department of Veterans Affairs, Health Services Research & Development: SDR 06-162: Cognitive Assessment of Veterans after Traumatic Brain Injury METHOD: Classification Accuracy Sensitivity (SN), Specificity (SP), Positive Predictive Power (PPP) and Negative Predictive Power (NPP) were calculated  As PPP and NPP take into consideration the sample base rate, PPP and NPP values are more accurate indicators of classification accuracy. Results will discuss these values over SN and SP o False positives were defined as 1 minus SP, (1-SP) o False negatives were defined as 1 minus the SN, (1-SN) The diagnostic validity of the CVLT FCR and RDS to predict suboptimal effort was examined using classification accuracy statistics and logistic regression Table 1. Number of TBIs and Percent Clinically Significant Psychiatric Symptoms Table 2. Classification Accuracy Statistics Table 3. Direct Logistic Regression: Results from Multiple Models