John Jerrim UCL Institute of Education PISA 2015 results in England… ….and some things you may not know about PISA John Jerrim UCL Institute of Education

Background 72 countries participated (75 if you separate out the UK)………….. Conducted every three years since 2000 Reading, mathematics, science, collaborative problem solving Computer-based assessment for the first time in 2015…….. Focus in 2015 = science. 30 points = 1 year of schooling. 100 points = One international standard deviation.

1. What are we good at?

1a. Science (average scores) > 20 points above Between 10 and 20 points below Country Mean Singapore 556* Estonia 534* Japan 538* Taiwan 532* Country Mean Belgium 502* United States 496* Denmark Austria 495* Poland 501* France Portugal Sweden 493* Northern Ireland 500* Czech Republic Norway 498* Spain Scotland 497* > 10 points above Country Mean Finland 531* Vietnam 525* Macao 529* Hong Kong 523* Canada 528*   Between 20 and 30 points below Within 10 points Country Mean Latvia 490* Wales 485* Russia 487* Luxembourg 483* Country Mean China 518 Australia 510 South Korea 516 Germany 509 New Zealand 513 Netherlands Slovenia Switzerland 506 England 512 Ireland 503* ….+35 other countries > 30 points below

1a. Science (average scores) Significantly above the OECD average…..

1b. High-achieving pupils in science > 20 points above Between 10 and 20 points below Country 90th percentile Singapore 683* Country 90th percentile Macao 630* Vietnam 624* Belgium 629* France 623* United States 626* Hong Kong 622* Sweden 625* Norway > 10 points above Country 90th percentile Taiwan 655* Japan Between 20 and 30 points below Within 10 points Country 90th percentile Austria 621* Czech Republic 618* Portugal 620* Malta Scotland 619* Ireland Poland Denmark 617* Northern Ireland Luxembourg 615* Country P90 Finland 651* Australia 639 China 649 Netherlands 638 Estonia 648 Slovenia 636 New Zealand 647 Germany Canada 644 South Korea England 642 Switzerland 632 ….+38 other countries > 30 points below

1c. Example question…. ‘Bird migration is a seasonal large-scale movement of birds to and from their breeding grounds. Every year volunteers count migrating birds at specific locations. Scientists capture some of the birds and tag their legs with a combination of coloured rings and flags. The scientists use sightings of tagged birds together with volunteers' counts to determine the migratory routes of birds.’ Identify a factor that might make volunteers’ counts of migrating birds inaccurate, and explain how that factor will affect the count.

1c. Example question

2. What are we not so good at?

2a. Maths (average scores) > 20 points above Between 10 and 20 points below Country Mean score Singapore 564* China 531* Hong Kong 548* South Korea 524* Macao 544* Switzerland 521* Taiwan 542* Estonia 520* Japan 532* Canada 516* Country Mean score Latvia 482* Wales 478* Malta 479* Hungary 477* Lithuania Slovakia 475* > 10 points above Between 20 and 30 points below Country Mean score Netherlands 512* Belgium 507* Denmark 511* Germany 506* Finland Poland 504* Slovenia 510* Ireland Country Mean score Israel 470* Croatia 464* United States Within 10 points Country Mean score Norway 502* Northern Ireland 493 Austria 497 Czech Republic 492 New Zealand 495 Portugal Vietnam Scotland 491 Russia 494 Italy 490 Sweden Iceland 488 Australia Spain 486* England Luxembourg France   ….+27 other countries > 30 points below

2a. ….. Though we are not THAT bad Same as the OECD average…..

2b. Particular issue with the bottom tail in maths…… > 20 points above Within 10 points Country 10th percentile Macao 439* Ireland 400* Singapore 436* Switzerland 394* Hong Kong 426* Slovenia Japan 416* Poland 391* Estonia 415* South Korea Denmark 405* Norway Finland 404* Netherlands 390* Taiwan Germany 389* Canada Country 10th percentile Wales 377 England 369 Sweden 376 Italy 368 New Zealand 375 Iceland 367 Belgium 374 Lithuania 365 Spain Portugal Czech Republic 373 France 364 Australia 371 Luxembourg 363 Austria 370   > 10 points above Between 10 and 20 points below Country 10th percentile Vietnam 388* Russia 387* China Scotland 382* Northern Ireland Latvia Country 10th percentile United States 355* Croatia 351* Hungary Slovakia 349* ….+29 other countries > 30 points below

3. Changes over time

3b. Stable in England, but things have changed in other parts of the UK….. Mean science scores Scotland big decline in 2015…. Wales a consistent downwards trend…

3c. Decline in performance of top 10% in science in NI/ Scotland/ Wales…..

Six things you probably don’t know from PISA….

1. Shanghai did 50 points worse on the computer test in 2012 than in the paper test…..

2. Why does Viet Nam do so well? Country Mean Singapore 556* Estonia 534* Japan 538* Taiwan 532* Country Mean Finland 531* Vietnam 525* Macao 529* Hong Kong 523* Canada 528*   Country Mean China 518 Australia 510 South Korea 516 Germany 509 New Zealand 513 Netherlands Slovenia Switzerland 506 England 512 Ireland 503*

2. Why does Viet Nam do so well 2. Why does Viet Nam do so well? Only 50% of its 15-year-olds are eligible to take part! Figures refer to the 75th percentile in science. ‘Before’ = reported results ‘After’ = including 15-year-olds who are not in school, assuming they would all perform below the national median. http://www.oecd-ilibrary.org/education/pisa-2015-results-volume-i_9789264266490-en Table

3. Around 1 in 3 American schools declines to participate….. See page 71 of http://www.oecd.org/pisa/pisaproducts/PISA%202012%20Technical%20Report_Chapter%204.pdf

4. You can only really compare PISA reading results back to 2009 (not 2000)….. “Changes in design and construct coverage were particularly important in earlier PISA assessments. The change in performance observed between PISA 2000 and later assessments may thus not always reflect genuine changes in what students know and can do……… …..the associated uncertainty associated with comparisons involving PISA 2000, 2003 and 2006 reading results with later results is only imperfectly captured by linking errors. ….some caution is needed when interpreting reading trends before PISA 2009. Footnote 3 page 172 http://www.keepeek.com/Digital-Asset-Management/oecd/education/pisa-2015-results-volume-i/reading-performance-among-15-year-olds_9789264266490-8-en#.WIiw0fmLTIU#page28 Very few link items 3 questions all from 1 unit – which was not technically the best unit……

5. A lot of things changed in PISA 2015 from previous cycles. … 5. A lot of things changed in PISA 2015 from previous cycles...... ….and this had a non-trivial impact upon some of the results “The negative changes between PISA 2012 and PISA 2015 reported for Chinese Taipei (-18 score points) and Viet Nam (-17 score points) are, to a large extent, due to the use of a different scaling approach. Had the PISA 2012 results for mathematics been scaled with the PISA 2015 calibration sample and the PISA 2015 approach to scaling, the differences in results for Chinese Taipei and Viet Nam would have been only -3 points and -4 points, respectively” http://www.oecd-ilibrary.org/education/pisa-2015-results-volume-i/changes-in-science-performance-between-2006-and-2015-based-on-originally-scaled-and-on-rescaled-results_9789264266490-graph117-en

6. Have things really changed that much internationally over the long term? (1981 vs 2011) Has that much changed over the last 25-30 years? East Asian (e.g. Japan / Hong Kong) countries at top of the maths rankings England around the international average Sweden does surprisingly poorly Cross-country correlation All countries = 0.72 Thailand excluded (outlier) = 0.66 Note: TIMSS data from 2011 or closest available year (2007 or 2003)

FIMS (1964) vs TIMSS (2011) Has that much changed over the last 50 years? East Asian (e.g. Japan) countries at top of the maths rankings England around the international average Sweden does surprisingly poorly Cross-country correlation All countries = 0.40 Israel excluded (outlier) = 0.78 Note: TIMSS data from 2011 or closest available year (2007 or 2003)