Data Forensics: A Compare and Contrast Analysis of Multiple Methods Christie Plackner

Outlier Score Applied to most of the methods Statistical probabilities were transformed into a score of 0 to = statistically unusual 2

Erasure Analysis Wrong-to-right (WR) erasure rate higher than expected from random events The baseline for the erasure analysis is the state average One sample t-test 3

Scale Score Changes Scale score changes statistically higher or lower than the previous year Cohort and Non-cohort One sample t-test 4

Performance Level Changes Large changes in proportion in performance levels across years Cohort and Non-cohort Log odds ratio –adjusted to accommodate small sample size –z test 5

Measurement Model Misfit Performed better or worse than expected Rasch residuals summed across operational items Adjusted for unequal school sizes 6

Subject Regression Large deviations from expected scores Within year – reading and mathematics Across year – cohort within a subject One sample t-test 7

Modified Jacob and Levitt Only method not resulting in a school receiving a score Combination of two indicators: –unexpected test score fluctuations across years using a cohort of students, and –unexpected patterns in student answers Modified application of Jacob and Levitt (2003) –2 years of data –Sample size 8

Principal Component Analysis Does each method contribute to the overall explained variance? Can the methods be reduced for a more efficient approach? 9

Multiple Methods 1.Erasure Analysis (mER) 2.Scale score changes using non-cohort groups (mSS) 3.Scale score changes using cohort groups (mSC) 4.Performance level changes using non-cohort groups (mPL) 5.Performance level changes using cohort groups (mPLC) 6.Model misfit using Rasch Residuals (mRR) 7.Across subject regression using reading scores to predict mathematic scores (mRG) 8.Within subject regression using a cohort’s previous year score to predict current score (mCR) 9.Index 1 of the Modified Jacob and Levitt evaluating score changes (mMJL1) 10.Index 2 of the Modified Jacob and Levitt evaluating answer sheet patterns (mMJL2). 10

Principal Component Analysis Grade 4 mathematics exam 10 methods Method MeanStd. DeviationAnalysis N mSS mPL mRG mRR mER mSC mPLC mMJL mMJL mCR

Method Correlations mSSmPLmRGmRRmERmSCmPLCmMJL1mMJL2mCR mSS1.000 mPL mRG mRR mER mSC mPLC mMJL mMJL mCR

Principal Component Statistics Component Initial Eigenvalues Extraction Sums of Squared Loadings Total% of VarianceCumulative %Total% of VarianceCumulative %

Scree Plot 14

Loading Matrix Component mCR mSC mPLC mMJL mMJL mRG mSS mPL E-5 mRR mER

Simplified Loading Matrix +/- greater than 1/2 the maximum value in the component (+)/(-) is between ¼ to ½ the maximum mCR+(-) mSC+(-) mPLC+(-) mMJL2++ mMJL1+.+ mRG(+)++ mSS(+)_+ mPL_+ mRR(-)+ mER(+)+

Principal Component Statistics Component Initial Eigenvalues Extraction Sums of Squared Loadings Total% of VarianceCumulative %Total% of VarianceCumulative %

Scree Plot 18

Reducing Variable Set Determine how many components to retain –Cumulative percentage of total variation –Eigenvalues –The scree plot Method Number of Retained Components 90% Cumulative Variance7 70% Cumulative Variance4 Eigenvalue4 Scree Plot2 19

Reducing Variable Set Select one method to represent a component Selecting methods within components –Positive selection Retain highest loading method with components –Discarded principal components Remove highest loading method with 20

Reducing Variable Set Selection Method Positive Selection Discarded Principal Components Number of Components 4242 mCRXXXX mSC mPLC mMJL2 mMJL1XXXX mRG mSSXX mPL mRRXX mER Cohort regression* Modified J&L, Index 1* Non-cohort scale score change Model misfit 21

Conclusion All methods seem to account for variation in detecting test taking irregularities Accounting for the most –Cohort regression –Cohort scale score change –Cohort performance level change Method reduction results the same 22

Discussion Different component selection methodologies Closer examination of variables –Remove cohort regression or cohort scale score change –Combine the J&L indexes Remove erasures 23