Establishing educational standards and monitoring student performance Directions for methodological improvements in international assessments

Key methodological features 1.Focus on literacy and using knowledge 2.Collection of range of information (attitudes, school-level factors, etc) other than performance data 3.Use of IRT to construct achievement variables 4.Multiple matrix sampling of items 5.Use of a range of item types and authentic assessment tasks 6.Standards-referenced reporting using 6 levels 7.Multilevel modelling to identify student/school/system effects 8.Two-stage approach to reporting results

Areas for improvement What/who is assessed? How it is assessed? What other information is collected? How the data are analyzed? How the results are reported, disseminated and used?

What/who is assessed? What other information to be collected? More open- ended items; Importance of trend and monitoring progress over time; More number of items which requires various type of learning strategies; Classroom environment: “what happen in the classrooms?” (e.g. use Video); Students’ social skill, emotional aspects, creativity, social-cooperation skill; Teachers’ perception of students performance; Teachers’ classroom teaching strategy; Testing 1year younger or older students: Value-added of 1 year of schooling; Longitudinal study; Targeting different age group; Foreign language skill; “How much and what skill/ability retain after few years?”; Conception of teaching: “what is teaching?” (e.g. guiding or facilitating); Conception of learning: “what is learning?”.

How the data are analysed Scaling: need caution for bottom and top ends which are instable; The relationship between combined-scale score and sub-scale scores.

Reporting, dissemination and use of results Feedback to teachers and schools to improve classroom teaching`; Media makes the most use of PISA results. But, often only focus on ranking. Need to avoid over simplified interpretation; Concrete national analysis/reports which goes beyond the international report; Need secondary study and in-depth study (e.g. case study or ethnography); Further involvement t of high level officials; Importance of dissemination seminars to understand other countries reaction to PISA;. Further use of PISA database and technical reports for capacity building of researchers and students; Reaction of students who participated in PISA;. Feedback to students and schools which participated in PISA; PISA reports are often too intensive. More concise results need to be prepared to disseminate to teachers, schools and parents.

Others Link the PISA scale/proficiency level to the national assessments; Link to other international studies such as PIACC.