1. John Jerrim UCL Institute of Education
PISA results in England… ….and some things you may not know about PISA
John Jerrim
UCL Institute of Education

2. Background
72 countries participated (75 if you separate out the UK)…………..
Conducted every three years since 2000
Reading, mathematics, science, collaborative problem solving
Computer-based assessment for the first time in 2015……..
Focus in 2015 = science.
30 points = 1 year of schooling.
100 points = One international standard deviation.

3. 1. What are we good at?

4. 1a. Science (average scores)
> 20 points above
Between 10 and 20 points below
Country
Mean
Singapore
556*
Estonia
534*
Japan
538*
Taiwan
532*
Country
Mean
Belgium
502*
United States
496*
Denmark
Austria
495*
Poland
501*
France
Portugal
Sweden
493*
Northern Ireland
500*
Czech Republic
Norway
498*
Spain
Scotland
497*
> 10 points above
Country
Mean
Finland
531*
Vietnam
525*
Macao
529*
Hong Kong
523*
Canada
528*
Between 20 and 30 points below
Within 10 points
Country
Mean
Latvia
490*
Wales
485*
Russia
487*
Luxembourg
483*
Country
Mean
China
518
Australia
510
South Korea
516
Germany
509
New Zealand
513
Netherlands
Slovenia
Switzerland
506
England
512
Ireland
503*
….+35 other countries > 30 points below

5. 1a. Science (average scores)
Significantly above the OECD average…..

6. 1b. High-achieving pupils in science
> 20 points above
Between 10 and 20 points below
Country
90th percentile
Singapore
683*
Country
90th percentile
Macao
630*
Vietnam
624*
Belgium
629*
France
623*
United States
626*
Hong Kong
622*
Sweden
625*
Norway
> 10 points above
Country
90th percentile
Taiwan
655*
Japan
Between 20 and 30 points below
Within 10 points
Country
90th percentile
Austria
621*
Czech Republic
618*
Portugal
620*
Malta
Scotland
619*
Ireland
Poland
Denmark
617*
Northern Ireland
Luxembourg
615*
Country
P90
Finland
651*
Australia
639
China
649
Netherlands
638
Estonia
648
Slovenia
636
New Zealand
647
Germany
Canada
644
South Korea
England
642
Switzerland
632
….+38 other countries > 30 points below

7. 1c. Example question….
‘Bird migration is a seasonal large-scale movement of birds to and from their breeding grounds. Every year volunteers count migrating birds at specific locations. Scientists capture some of the birds and tag their legs with a combination of coloured rings and flags. The scientists use sightings of tagged birds together with volunteers' counts to determine the migratory routes of birds.’
Identify a factor that might make volunteers’ counts of migrating birds inaccurate, and explain how that factor will affect the count.

8. 1c. Example question

9. 2. What are we not so good at?

10. 2a. Maths (average scores)
> 20 points above
Between 10 and 20 points below
Country
Mean score
Singapore
564*
China
531*
Hong Kong
548*
South Korea
524*
Macao
544*
Switzerland
521*
Taiwan
542*
Estonia
520*
Japan
532*
Canada
516*
Country
Mean score
Latvia
482*
Wales
478*
Malta
479*
Hungary
477*
Lithuania
Slovakia
475*
> 10 points above
Between 20 and 30 points below
Country
Mean score
Netherlands
512*
Belgium
507*
Denmark
511*
Germany
506*
Finland
Poland
504*
Slovenia
510*
Ireland
Country
Mean score
Israel
470*
Croatia
464*
United States
Within 10 points
Country
Mean score
Norway
502*
Northern Ireland
493
Austria
497
Czech Republic
492
New Zealand
495
Portugal
Vietnam
Scotland
491
Russia
494
Italy
490
Sweden
Iceland
488
Australia
Spain
486*
England
Luxembourg
France
….+27 other countries > 30 points below

11. 2a. ….. Though we are not THAT bad
Same as the OECD average…..

12. 2b. Particular issue with the bottom tail in maths……
> 20 points above
Within 10 points
Country
10th percentile
Macao
439*
Ireland
400*
Singapore
436*
Switzerland
394*
Hong Kong
426*
Slovenia
Japan
416*
Poland
391*
Estonia
415*
South Korea
Denmark
405*
Norway
Finland
404*
Netherlands
390*
Taiwan
Germany
389*
Canada
Country
10th percentile
Wales
377
England
369
Sweden
376
Italy
368
New Zealand
375
Iceland
367
Belgium
374
Lithuania
365
Spain
Portugal
Czech Republic
373
France
364
Australia
371
Luxembourg
363
Austria
370
> 10 points above
Between 10 and 20 points below
Country
10th percentile
Vietnam
388*
Russia
387*
China
Scotland
382*
Northern Ireland
Latvia
Country
10th percentile
United States
355*
Croatia
351*
Hungary
Slovakia
349*
….+29 other countries > 30 points below

13. 3. Changes over time

14. 3b. Stable in England, but things have changed in other parts of the UK…..
Mean science scores
Scotland big decline in 2015….
Wales a consistent downwards trend…

15. 3c. Decline in performance of top 10% in science in NI/ Scotland/ Wales…..

16. Six things you probably don’t know from PISA….

17. 1. Shanghai did 50 points worse on the computer test in 2012 than in the paper test…..

18. 2. Why does Viet Nam do so well?
Country
Mean
Singapore
556*
Estonia
534*
Japan
538*
Taiwan
532*
Country
Mean
Finland
531*
Vietnam
525*
Macao
529*
Hong Kong
523*
Canada
528*
Country
Mean
China
518
Australia
510
South Korea
516
Germany
509
New Zealand
513
Netherlands
Slovenia
Switzerland
506
England
512
Ireland
503*

19. 2. Why does Viet Nam do so well
2. Why does Viet Nam do so well? Only 50% of its 15-year-olds are eligible to take part!
Figures refer to the 75th percentile in science.
‘Before’ = reported results
‘After’ = including 15-year-olds who are not in school, assuming they would all perform below the national median.
Table

20. 3. Around 1 in 3 American schools declines to participate…..
See page 71 of

21. 4. You can only really compare PISA reading results back to 2009 (not 2000)…..
“Changes in design and construct coverage were particularly important in earlier PISA assessments. The change in performance observed between PISA 2000 and later assessments may thus not always reflect genuine changes in what students know and can do………
…..the associated uncertainty associated with comparisons involving PISA 2000, 2003 and 2006 reading results with later results is only imperfectly captured by linking errors.
….some caution is needed when interpreting reading trends before PISA 2009.
Footnote 3 page 172
Very few link items
3 questions all from 1 unit – which was not technically the best unit……

22. 5. A lot of things changed in PISA 2015 from previous cycles. …
5. A lot of things changed in PISA 2015 from previous cycles ….and this had a non-trivial impact upon some of the results
“The negative changes between PISA 2012 and PISA 2015 reported for Chinese Taipei (-18 score points) and Viet Nam (-17 score points) are, to a large extent, due to the use of a different scaling approach. Had the PISA 2012 results for mathematics been scaled with the PISA 2015 calibration sample and the PISA 2015 approach to scaling, the differences in results for Chinese Taipei and Viet Nam would have been only -3 points and -4 points, respectively”

23. 6. Have things really changed that much internationally over the long term? (1981 vs 2011)
Has that much changed over the last years?
East Asian (e.g. Japan / Hong Kong) countries at top of the maths rankings
England around the international average
Sweden does surprisingly poorly
Cross-country correlation
All countries = 0.72
Thailand excluded (outlier) = 0.66
Note: TIMSS data from 2011 or closest available year (2007 or 2003)

24. FIMS (1964) vs TIMSS (2011)
Has that much changed over the last 50 years?
East Asian (e.g. Japan) countries at top of the maths rankings
England around the international average
Sweden does surprisingly poorly
Cross-country correlation
All countries = 0.40
Israel excluded (outlier) = 0.78
Note: TIMSS data from 2011 or closest available year (2007 or 2003)