1. Choosing tests for EEF evaluations – reliability and validity and other issues Steve Higgins & Carole Torgerson s.e.higgins@durham.ac.uks.e.higgins@durham.ac.uk & carole.torgerson@durham.ac.ukcarole.torgerson@durham.ac.uk School of Education, Durham University EEF Evaluators Conference, June 2013

2. The perfect test! Highly reliable … Internal consistency Inter-item /Item total correlation Split-Half Reliability Cronbach's Alpha (α) Test/Re-test Parallel forms & Split half Inter-rater Wonderfully valid … Translation validity Face (weak and strong versions) Content (CVI) Criterion validity Concurrent Predictive Construct-related validity Convergent Discriminant.. And eminently practical Short Easy to administer to large groups Cheap Quick and easy to mark/ get the data

3. Other issues Intervention alignment Measures what is taught versus a good measure of school learning (predictive validity) Standardisation Availability of data Sampling (compared with intervention focus) Recency Poor reliability can increase Type II errors (false negatives)

4. Diamond ranking Most important Least important

5. Competing priorities ReliableValidPractical Number of items Time to administer Cost Ease of marking and data entry Aligned to intervention

6. Other issues (Carole T) Choice of test(s) Possible test effects Blinding teachers? intervention designers (developers)? markers Teaching to the test Treatment inherent measures Developers/teachers attitudes to tests

7. Discussion Discuss any of these or other testing issues that have arisen in your evaluation How can EEF help?