NOT YOUR GRANDPA'S STATISTICS: NEW MODELING APPROACHES TO STUDENT ACHIEVEMENT & RTI Jill Pentimonti Adrea Truckenmiller Jessica Logan Sara Hart Discussion: Grandpa Schatschneider Presented Feb 6, 2014 Pacific Coast Research Conference, San Diego

Individual differences in response to intervention: An application of Integrative Data Analysis in Project KIDS Sara A. Hart & Grandpa Florida State University

Expanding our search for moderators of intervention A little about me – Behavioral genetics background – PCRC participant Even with modest effect sizes, individual differences in intervention response Bioecological model (Bronfenbrenner & Ceci, 1994) – Provides framework for differentiating students based on non-intervention related traits

Integrative Data Analysis (IDA) Item-level pooled data (Curran & Hussong, 2009) Capitalizes on cumulative knowledge – Longer developmental time span – Increased statistical power – Increased absolute numbers in tails Controls for heterogeneity – Sampling, age/grade, cohort, geographical, design, measurement

Project KIDS Expanded definition of moderators of response to intervention – Cognitive, psychosocial, environmental, genetic risk IDA across 9 completed intervention projects – Approximately 5600 kids Data entry of item level data common across at least 2 projects – ~30 different assessments Questionnaire data collection

Proof of Concept Behavior problems and achievement are associated More behavior problems are typically seen in LD populations Is adequate vs inadequate response status differentiated by behavior problems?

Method Participants – ISI intervention project (Connor et al., 2007) RCTish : 22 treatment, 25 contrast teachers, 3 pilot 821 first graders A2i recommendations vs standard practice – ISI intervention through FL LDRC (Al Otaiba et al., 2011) RCT: 23 treatment, 21 contrast teachers 556 kindergarteners A2i recommendations vs enhanced standard practice

Method Measures – WJ Tests of Achievement Letter-Word Identification (LWID) Pre- and post-intervention testing periods – Social Skills Rating Scale: Behavioral Problems subscale Teacher completed during intervention year 05/06 1 ISI Mean (SD) 07/08 K ISI LDRC Mean (SD) WJ LWID Fall24.28 (7.97)11.96 (5.53) WJ LWID Spring36.54 (7.39)21.64 (7.10) SSRS.53 (.44).48 (.44)

Results: Calibration LWID Randomly selected 1 time point/child/project to form “calibration sample” for LWID IRT with decision to include only items > 5% endorsement rate Reduced item sample from 75  36 – Items 8 to 44

Results: Calibration LWID Generalized linear factor analysis (GLFA) – Combines latent factor analysis and 2-PL IRT model Here, equivalent of 2-PL IRT model with DIF No significant DIF was found

Results: Second data sample LWID Using remaining data, GLFA model run again, setting parameters based on calibration sample Separately by project – If significant, add DIF estimates to parameters

Results: SSRS IRT to GLFA model with Project DIF on full data

Results

Results: Response Proc mixed: covariance adjusted LWID score – 1169 children

Results: Response 648 treatment children

Results: Response 648 treatment children Unresponsive Cutoff < 20% N=110!

Results: Response 648 treatment children Unresponsive Cut off Fall SS = 95 Spring SS= 104 Mean Fall SS = 86 Spring SS = 96

Results: Response 648 treatment children Unresponsive Cut off Fall SS = 95 Spring SS= 104 Mean Fall SS = 86 Spring SS = 96 Responsive Mean Fall SS = 99 Spring SS = 111

Results Logistic regression – SSRS behavior problems significant predictor of response status (OR = 1.45, CI = ) average behavior problems = 19% probability of being “unresponsive” greater than average behavior problems(+ 1SD) = 29% probability of being “unresponsive” Less than average behavior problems (-1SD) = 12% probability of being “unresponsive”

Conclusions Response status is differentiated by behavior problems – Mo’ behavior problems, mo’ (reading) problems! The questionnaire data we will be adding will be real test of bioecological model on response to intervention

Overall IDA conclusions IDA is a “cheap” way to get more power, more n at tails, and show more generalizable effects Given how similar many of our projects are, consider doing item-level data entry – Easy potential to combine data – Can you do factor analysis and IRT? You can do IDA! These data are more useful together than apart – IRT within and between samples? – Treatment effectiveness across samples? – Characteristics of lowest responders?

Acknowledgements Stephanie Al Otaiba Carol Connor Chris Schatschneider Great staff & grad students, and a small army of data enterers NICHD grant HD072286