1. Validation of the Assessment and Comparability to the PISA Framework Hao Ren and Joanna Tomkowicz McGraw-Hill Education CTB

2. Outline PISA and OECD Test for Schools (OTS) – similarities and differences Linking OTS to PISA Cluster rotation design for OTS Scoring model Selected results Summary Copyright © 2015 CTB/McGraw-Hill LLC2

3. PISA and OTS (Cognitive Test) The same domains covered by PISA and OTS: reading, mathematics and science – PISA focuses on one major domain in each administration (with the other two domains receiving less coverage) – OTS has a balanced design between the three domains. New items were developed for OTS under the same framework as PISA. – PISA 2009 blueprint used in the test development. Similar test question type and format Similar total testing time Copyright © 2015 CTB/McGraw-Hill LLC3

4. PISA and OTS (Student and School Questionnaires) Both PISA and OTS use student questionnaires to collect information from students on various aspects of their home and family life, school environment and attitudes towards learning Both PISA and OTS use school questionnaires to collect information from schools about various aspects of organisation and educational provision in schools. The information collected is the same or very similar to PISA Copyright © 2015 CTB/McGraw-Hill LLC4

5. PISA and OTS Test Scale Comparability OTS was designed to yield school-level scored comparable to PISA scores International field test and equating study conducted during 2011 and early 2012. OTS was equated to PISA scales using a set of common items between PISA 2009 and OTS field test – Linking updated after PISA 2012 administration The three domains of OTS were put on the PISA scales: – Reading is on PISA 2000 – Mathematics is on PISA 2003 – Science is on PISA 2006 Copyright © 2015 CTB/McGraw-Hill LLC5

6. Content design for OTS The design comprises two item clusters for each domain (reading, mathematics and science) and one additional hybrid cluster including items from all three domains The considerations in constructing the test booklets are – There are 7 booklets and 3 clusters in each booklet. – The clusters are organized to appear in 3 separate booklets, in the 1 st, 2 nd and 3 rd block (position) – In order to balance the effect of item position in the booklet, each pair of clusters appears in one (and only one) booklet and each cluster appears with every other clusters – The testing time is 120 minutes, which matches the PISA main design and therefore gives the students a “PISA-like” test experience Copyright © 2015 CTB/McGraw-Hill LLC6

7. Cluster Rotation design for OTS BookBlock 1Block 2Block 3 1R1RSMM1 2RSMM2S2 3M2M1R2 4M1S2S1 5S2R2R1 6R2S1RSM 7S1R1M2 Copyright © 2015 CTB/McGraw-Hill LLC7

8. Scoring Model PISA scoring model was adopted for OTS: – The mixed coefficients multinomial logit model – PISA uses multi-dimensional model – OTS uses uni-dimensional model The generalized form of the Rasch model is used to describe the item response part The population model describes the relationship between the student’s ability and all the background information – Gender, grade, booklet administered, parents’ occupation, and all other information from the student’s questionnaire Copyright © 2015 CTB/McGraw-Hill LLC8

9. Scoring Model All information from the student questionnaire was included in the scoring model for OTS – PISA uses both student questionnaire and school questionnaire in the scoring model – OTS only involve student questionnaire in the scoring model but use school questionnaire for reporting purpose and other analysis – Principal component analysis conducted on all student responses to the questionnaire items and the generated principal scores are used in estimation of student’s latent ability The scores are adjusted by considering the impact of all factors Estimated scores at the student level are plausible values Student level scores are not reported – Main PISA reports country-level scores – OTS reports school-level score The detail description about the method can be found in PISA report Copyright © 2015 CTB/McGraw-Hill LLC9

10. Selected OTS Pilot 2012 Results OTS results are reported at the school level Performance in three domain (scale scores, achievement levels) Extensive comparison with similar schools in the United States and international countries and economies Economic, Social, and Cultural Status (ESCS) and student performance Various analyses related to non-cognitive factors Copyright © 2015 CTB/McGraw-Hill LLC10

11. OTS Pilot School Performance* DomainMeanSDMinMax Reading509.055.3343.9624.9 Mathematics506.967.3320.6679.5 Science515.047.9376.9625.3 Copyright © 2015 CTB/McGraw-Hill LLC11 *The schools with too small number (<42) of students, or too low participation rate (<50%) were excluded from the summary in this table.

12. Mathematics vs. Reading Copyright © 2015 CTB/McGraw-Hill LLC12 US Mean Reading500 US Mean Mathematics487 Correlation 0.98

13. Reading vs. Science Copyright © 2015 CTB/McGraw-Hill LLC13 US Mean Reading500 US Mean Science502 Correlation 0.98

14. How the pilot schools compared with schools in other countries and economies in reading in PISA 2009 Copyright © 2015 CTB/McGraw-Hill LLC14 Schools in The United States Schools in Shanghai-China Schools in Mexico Schools in Pilot

15. Reading vs. ESCS Copyright © 2015 CTB/McGraw-Hill LLC15 US Mean Reading500 ESCS Economic, social, and cultural status Correlation 0.81

16. Reading vs. Disciplinary Climate in English Lessons Copyright © 2015 CTB/McGraw-Hill LLC16 US Mean Reading500 DISCLIMA Disciplinary climate in English Lessons Correlation 0.73

17. Summary (What We Learned from the Pilot) Data analysis methodology implemented for OTS allows for comparing a school’s results to the country’s results and international benchmarks Similar trends of PISA and OTS results observed – Strong correlations between the performance in three domains – Association between ESCS and performance – Association between the non-cognitive factors and academic achievement The OTS Pilot confirmed the administrative conditions and procedures of the assessment, validated the methodology, and explored how results could be reported before the OTS was made publicly available in 2013 and beyond Copyright © 2015 CTB/McGraw-Hill LLC17