1. Were Prior Mild TBI Meta-Analytic Results Refuted or Replicated? Martin L. Rohling, Ph.D.Laurence Binder, Ph.D. U. of South AlabamaOregon Health Sciences Glenn Larrabee, Ph.D.Danielle Ploetz, M.S. Private PracticeU. of South Alabama

2. This presentation is a rebuttal to the a paper published in Brain Injury. 03/11/2010 2 Rohling et al. Rebuttal to Pertab et al. (2009)

3. 1. Mild traumatic brain injury (mTBI) is one of the most frequently studied topics in clinical neuropsychology and may also be one of the most controversial (p. 428). 2. The first, and most frequently cited, meta- analysis intending to clarify the effects of mTBI was conducted by Binder, Rohling, & Larrabee (1997). They found only small overall effect sizes and concluded that… ‘the size of the overall effect was unimpressive and suggestive of clinical non- significance’. Note:Binder et al. (1997) only analyzed outcomes for patients who were 3-months or longer post mTBI. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 3

4. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 4 1.Are there residual deficits in cognitive functioning that could be attributed to mTBI? 2.If there are not significant overall deficits cognitive deficits caused by mTBI, are there domain specific deficits associated with mTBI (e.g., attention) that might be a “canary in the mine?”

5. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 5 Cognitive DomainsM ES gp Attention-.17.01 Learning-.17.06 Delayed Memory-.14<.01 Manual Dexterity-.07.37 Perceptual Reasoning-.02.44 Abstraction & Flexibility.09.41 Verbal Abilities.09.16 Mean Overall ES.07.10

6.  Reanalyze data reported in 2 prior MAs on mTBI: Binder et al. (1997) & Frencham et al. (2005).  Specifically, focus on previously unanalyzed variables: › (1) mechanism of the injury › (2) diagnostic criteria › (3) assessment instruments utilized › (4) symptomatic vs. non-symptomatic samples  They claimed to have restrict their data analyses to that which was gathered 3-months or greater post mTBI.  Purpose: “…to clarify opposing conclusions in the mTBI literature…” 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 6

7. “Of the 25 original studies, 18 reported neuropsychological variables in a form that enabled effect size calculation on the variables of interest for this investigation (see Table I)” (pp 502). Table I is on the next slide. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 7

8. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 8 Reference Control n mTBI n Total n # of comp Average g [70] Alterman et al. (1985)25 5010.20 [84] Echemendia et al. (2001)[81] 35 706-.51 [86] Macciocchi et al. (1996)[71] 1012226-.24 [71] Bornstein et al. (1993)[90] 24 4818-.07 [87] Mangels et al. (2002)[83] 76501269-.58 [92] Dikmen et al. (1995)12116128221-.02 [88] Mathias & Coats (1999)[84] 20294914-1.20 [67] Goldstein et al. (2001)14183210-.36 [89] McAllister et al. (2001)[85] 10 204-.74 [90] Potter et al. (2001)[86] 4818323115-.20 [91] Voller et al. (1999)[87] 101121 -.16 [92] Dikmen et al. (1995)[88] 27 5412-.39 [93] No such reference # listed[89] 1217298-.62 [68] Ponsford et al. (2000)538413711-.12 [94] No such reference # listed[90] 24 488-.47 [66] Reitan & Wolfson (1999)4118591-.93 [83] Comerford et al. (2002)19254499-.34 [96] No such reference # listed[91] 14122612-.95 Weighted overall effect size5837651348285 -.31 Table I. Studies with analysable neuropsychological data, the number of comparisons and average effect size for each study. red blue References listed in red are incorrectly matched. Corrected reference numbers are listed in blue.

9. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 9 Assessment tool# studiesTotal ngt for g Verbal Paired Memory269-.52-2.17** Coding Tasks4228-.33-2.48** Verbal Fluency269-.33-1.36 Digit Span5257-.31-2.45** List Memory - Delayed2330-.17-1.50 PASAT2185-.13-.89 Story Memory2303-.11-.91 Card Sorting Tasks3123-.11-.59 Trails B4399-.03-.28 Trails A and B6661-.02-.21 List Learning & Memory4496.00-.03 Trails A4399.06.64 Figure Memory3344.111.01 Weighted overall effect size3863 -.08 ?? Table II.Assessment tools — weighted overall effect size 3 months and over, ranked — largest impairment first.

10. 1. Scores on Trails B & List Learning were not affected by mTBI. 2. Scores on Verbal Paired Associates, Digit Span, & Coding were all negatively affected by mTBI. This suggests that the control groups for some of the included studies were not representative. Studies did not use trauma controls like Dikmen et al (1995) 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 10

11.  Acquired Pertab et al.’s (2009) data file  Two coders blindly recoded effect sizes for Binder et al. (1997) & Frencham et al. (2005)  Compared results to Pertab’s calculations  Corrected any questionable procedures › Inappropriate collapsing of ESs across time › Failure to conduct heterogeneity analyses › Failure to include all available cognitive variables › Failure to use inferential statistics to calculate ES 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 11

12. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 12 # Samples Reporting mTBI M mTBI SD Control M Con SD Pd Total Sample482834--2057-- Avg. Sample Size4859.067.242.947.4.007.28 Age (years)4325.17.225.97.8.65.10 Education (yrs)3912.61.312.51.2.63.09 Sex (%male)4373.420.974.319.2.83.05 LOC (min)1710.215.8-- GCS514.6.4-- PTA (hours)1118.954.5--

13. 03/11/2010Rohling et al. Rebuttal to Pertab et al. (2009) 13 High degree of inter-rater reliability on individual effect size calculations when both teams coded a data point. ES represents effect sizes coded by Rohling et al.

14. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 14 ES represents effect sizes coded by Rohling et al. Much lower inter-rater reliability was generated when overall summary ES’s were compared to one another. There is only 1 summary ES per sample. The summary ES is calculated by averaging all of a studies individual ES’s together.

15.  Coding errors were apparent in Pertab et al. › Most critical was the inclusion of data gathered prior to 3-month post mTBI into ESs presented in Table I.  When all errors were corrected, for those data gathered prior to the 3-month post mTBI cutoff, a weighted mean ES was generated: ES = -.33 (n = 32 samples)  By contrast, data gathered at 3-months or greater post mTBI, a weighted mean ES was generated: ES = -.07 (n = 16 samples) 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 15

16. 03/11/2010Rohling et al. Rebuttal to Pertab et al. (2009) 16 Wt M dSE # Studies Total Np for Zp for Q Sum = 0-14 day -.37.04203096 <.0001.04 Sum = 15-92 day -.20.0712829.003.92 Sum > 3 months -.07.0716956.15.98 All Studies-Sum -.27.03484881 <.0001.13 All cognitive measures divided into 3 groups based on Chronicity (i.e., time post injury) and analyzed with a fixed effects model.

17. 03/11/2010Rohling et al. Rebuttal to Pertab et al. (2009) 17 Wt M dSE # Studies Total Np for Zp for Q Sum = 0-14 day -.40.05203096<.0001.35 Sum = 15-92 day -.23.1012829.01.98 Sum > 3 months -.09.1016956.18>.99 All Studies-Sum -.28.04484881<.0001.51 All cognitive measures divided into 3 groups based on Chronicity (i.e., time post injury) and analyzed with a random effects model.

18. 03/11/2010Rohling et al. Rebuttal to Pertab et al. (2009) 18 0-14 days 15-91 days > 3 mo All Time Frames Verbal Memory & Learning-.47-.24-.09-.35 Working Memory-.34-.40-.16-.30 Executive Functioning-.32-.36.07-.27 Visual Memory & Learning-.66-.32.03-.20 Processing Speed-.21-.06-.09-.16 Verbal Comprehension-.42.10-.01 Perceptual Reasoning---.03 Average - ALL DOMAINS-.40-.30-.02 -.18 Cognitive domains for All Time Frames generated from a fixed effects model.

19. 03/11/2010Rohling et al. Rebuttal to Pertab et al. (2009) 19 Wt M dSE # Studies Total Np for Zp for Q Sum 0-1 Day-.41.075 833<.0001.29 Sum 2-4 day-.41.086 666<.0001.06 Sum 5-7 day-.31.065 1189<.0001.08 Sum 8-14 day-.40.114 408.0002.13 Sum 15-31 day-.26.108 388.0100.97 Sum 1-3 mo.-.14.114 441.09.43 Sum 3-6 mo.-.10.106 400.16.84 Sum 7-12 mo..01.105 422.47.74 Summary > 1 yr-.21 *.175 134.11.99 All time frames-.25.105 542--- All cognitive measures divided into 9 groups based on Chronicity and analyzed with a fixed effects model. Two studies should have been excluded, as they did not meet the inclusion initial criteria as coming from an mTBI non-symptomatic patients. Another study in this subsample had effect sizes incorrectly calculated and the corrected ES’s were smaller than previous calculated. *Two studies should have been excluded, as they did not meet the inclusion initial criteria as coming from an mTBI non-symptomatic patients. Another study in this subsample had effect sizes incorrectly calculated and the corrected ES’s were smaller than previous calculated.

20. ES = -.07 to -.09  When analyzed properly, results from Binder et al. (1997) and Frencham et al. (2005) were REPLICATED with an ES = -.07 to -.09  Pertab et al.’s (2009) results were the product of coding errors, mistakes in statistical analyses, and poor written description of results. 1. Including 3-mo. lead to confusion by many readers. 2. Pertab found no residual cognitive effects of mTBI that were different from that which had been found earlier. 3. There was no evidence in the data that a subgroup of mTBI patients remain more significantly impaired. 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 20

21. (1) Mechanism of the injury Could not be analyzed – insufficient data (2) Diagnostic criteria Could not be analyzed – insufficient data (3) Assessment instruments used Questionable stability – outliers, small n (4) Symptomatic patients vs. non-symptomatic Could not be analyzed – insufficient data 03/11/2010 Rohling et al. Rebuttal to Pertab et al. (2009) 21