VISTA 19 August October 2005 Park Hyatt, Melbourne Assessment & Reporting Dr Andreas Schleicher – OECD Key Note Address 1

What effective systems and schools need to do? Is the sky the limit in educational performance? Organisation for Economic Cooperation and Development (OECD) Melbourne, October 2005 Andreas Schleicher Head, Indicators and Analysis Division Directorate for Education

In the dark… …all students, schools and education systems look the same… But with a little light….

But with a little light…. …important differences become apparent….

Growth in university-level qualifications A world of change Approximated by percentage of persons with ISCED 5A/6 qualfications in age groups 55-64, 45-55, und years

Baseline qualifications A significant minority still left out Approx. by % of persons with upper secondary qualfications in age groups 55-64, 45-55, und years r Australia falls behind the OECD average with adult attainment at the upper secondary level With serious consequences for those who have not completed this level –Average earnings are only 75% of those with completed upper secondary schooling, a disadvantage that is larger than at the OECD average level –Among those without upper secondary qualifications 25% earn half or less of the national median earnings, and only 3% are in the group of top earners, whose earnings exceeds twice the OECD median Note that the youngest individuals in this comparison (25-year-olds in 2003) passed the age of 16 in 1994 r Few of those without upper secondary completion catch up later in life with continuing education and training

Overview

The PISA approach Measuring the quality of learning outcomes

Key features of PISA 2003 r Information collected volume of the tests –3½ hours of mathematics assessment –1 hour for each of reading, science and problem solving each student –2 hours on paper-and-pencil tasks (subset of all questions) –½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals –questionnaire (school demography, learning environment quality ) r Coverage PISA covers 15-year-olds in school PISA countries cover roughly nine tens of the world economy

OECD countries participating from PISA 2000 OECD countries participating from PISA from 2003 OECD partner countries participating from PISA 2000 OECD partner countries participating from PISA 2003 OECD partner countries participating from PISA 2006 PISA country participation

Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

OECD framework for PISA National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

Three broad categories of key competencies Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Learning strategies Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively PISA concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society

Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Reading literacy Using, interpreting and reflecting on written material

Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Mathematical literacy Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways

Using “tools” interactively to engage with the world Acting autonomously Interacting in diverse groups e.g. Using language, symbols and texts Interacting with information Capitalising on the potential of technologies e.g. Relating well to others Co-operating, working in teams Managing and resolving conflicts e.g. Acting within the bigger picture Forming and conducting life plans Taking responsibility and understanding rights and limits To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Scientific literacy Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world

Where we are - and where we can be What PISA shows students in Australia can do Examples of the best performing countries

PISA provides five key benchmarks for the quality of education systems

Average performance of 15-year-olds in mathematics High mathematics performance Low mathematics performance

Mathematical literacy in PISA The real world The mathematical World A real situation A model of reality A mathematical model Mathematical results Real results Understanding, structuring and simplifying the situation Making the problem amenable to mathematical treatment Interpreting the mathematical results Using relevant mathematical tools to solve the problem Validating the results

Average performance of 15-year-olds in mathematics Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

Is it all innate ability? Variation in student performance OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p

Variation of performance between schools Variation of performance within schools Is it all innate ability? Variation in student performance in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383. In some countries, parents can rely on high and consistent standards across schools In Canada, Denmark, Finland, Iceland and Sweden average student performance is high… …and largely unrelated to the individual schools in which students are enrolled In other countries, large performance differences among schools persist In Austria, Belgium, Germany, Hungary, Italy, Japan, the Netherlands and Turkey, most of the performance variation among schools lies between schools… …and in some of these countries, most notably those that are highly stratified, a large part of that variation is explained by socio-economic inequalities in learning opportunities

Student performance School performance and schools’ socio- economic background - Australia Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES within schools School performance and school SES OECD

Student performance School performance and schools’ socio- economic background - Germany Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES

Student performance School performance and schools’ socio- economic background - Finland Advantage PISA Index of social background Disadvantage Figure 4.13 Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size

Student performance Universal policies Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES OECD

Student performance Socio-economically targeted policies Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES OECD

Student performance Compensatory policies Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES OECD

Student performance Performance-targeted policies Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES Student performance and student SES within schools School performance and school SES OECD

Gender differences r In reading, girls are far ahead In all countries, girls significantly outperform boys in reading r In mathematics, boys tend to be somewhat ahead In most countries, boys outperform girls …but mostly by modest amounts… …and mainly because boys are overrepresented among top- performers while boys and girls tend to be equally represented in the “at risk” group –Within classrooms and schools, the gender gap is often larger Strong problem-solving performance for girls suggests… …that it is not the cognitive processes underlying mathematics that give boys an advantage… …but the context in which mathematics appears in school Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap –Girls report much lower intrinsic (though not instrumental) motivation in mathematics, more negative attitudes and much greater anxiety with mathematics… …and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

Interest in and enjoyment of mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

Instrumental motivation in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.2a, p.360 and Figure 3.3a, p.122.

Anxiety in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374 and Figure 3.8, p.139.

Other noteworthy student views in Australia r 70% of Australian 15-year-olds aspire to a high-skills white-collar occupation (rank 6 among the 30 OECD countries) r A generally positive view about school (rank 3) r A reasonably strong self-concept and self-efficacy in math but only average control strategies (ranks 8, 10 and 15 respectively) r Anxiety with mathematics an issue, and it has a strong negative net impact on performance (ranks 18).

How can we get there? Levers for policy that emerge from international comparisons

Money matters but other things do too Mexico Greece Portugal Italy Spain Germany Austria Ireland United States Norway Korea Czech republic Slovak republic Poland Hungary Finland Netherlands Canada Switzerland Iceland Denmark France Sweden Belgium Australia Japan R 2 = 0.28 Cumulative expenditure (US$) Performance in mathematics r Spending per student is positively associated with average student performance… …but not a guarantee for high outcomes Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well in terms of “value for money”… …while some of the big spenders perform below-average

Sympathy doesn’t raise standards – aspiration does r In some of the best-performing countries National research teams report a strong “culture of performance” –Which drives students, parents, teachers and the educational administration to high performance standards r PISA suggests… …that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations –Among these aspects, students’ perception of teacher-student relations and classroom disciplinary climate display the strongest relationships

High ambitions and clear standards Access to best practice and quality professional development

Challenge and support Low support High support Low challenge High challenge Strong performance Systemic improvement Poor performance Improvements idiosyncratic Conflict Demoralisation Poor performance Stagnation

Governance of the school system r In many of the best performing countries School-based decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities The provision of standards and curricula at national/subnational levels is combined with advanced evaluation and support systems –That are implemented by professional agencies Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress r Standard setting and equity-related goals Key objectives: –Raise educational aspirations, establish transparency over educational objectives, reference framework for teachers Approaches range from definition of broad educational goals up to formulation of concise performance expectations Some countries go beyond establishing educational standards as mere yardsticks and use performance benchmarks that students at particular age or grade levels should reach Instruments –Minimum standards, targets defining excellence, normative performance benchmarks r Monitoring and equity-related goals Diverging views how evaluation and assessment can and should be used –Some see them primarily as tools to reveal best practices and identify shared problems in order to encourage teachers and schools to improve and develop more supportive and productive learning environments –Others extend their purpose to support contestability of public services or market-mechanisms in the allocation of resources –e.g. by making comparative results of schools publicly available to facilitate parental choice or by having funds following students Differences in type of performance benchmarks being used and reported for the various stakeholders involved, including parents, teachers and schools

Assessment in Australian schools r Percentage of 15-year-olds enrolled in schools where school principals report that the following methods of assessment are used at least three times per year Standardised assessment: 11% (OECD 23%) Student portfolios: 40% (OECD 43%) (performance advantage) Judgmental ratings: 76% (OECD 75%) Teacher developed tests: 96% (OECD 92%) Student assignments/projects/homework: 98% (OECD 86%)

High ambitions Access to best practice and quality professional development Accountability and intervention in inverse proportion to success Devolved responsibility, the school as the centre of action

Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance

Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance School with responsibility for deciding which courses are offered High degree of autonomy Low degree of autonomy

Durchschnittliche Schülerleistungen im Bereich Mathematik Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High mathematics performance Low mathematics performance Early selection and institutional differentiation High degree of stratification Low degree of stratification

Organisation of instruction r In the best performing countries Schools and teachers have explicit strategies and approaches for teaching heterogeneous groups of learners –A high degree of individualised learning processes –Disparities related to socio-economic factors and migration are recognised as major challenges Students are offered a variety of extra-curricular activities Schools offer differentiated support structures for students –E.g. school psychologists or career counsellors Institutional differentiation is introduced, if at all, at later stages –Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

Support systems and professional teacher development r In the best performing countries Effective support systems are located at individual school level or in specialised support institutions Teacher training schemes are selective The training of pre-school personnel is closely integrated with the professional development of teachers Continuing professional development is a constitutive part of the system Special attention is paid to the professional development of school management personnel

Teacher support in mathematics Students’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.1a, p.403 and Figure 5.1, p.213.

Other noteworthy views of school principals and students in Australia r Students perceive a high level of perceived teacher support (rank 6 among the 30 OECD countries) r School principals and student report an average picture concerning student factors affecting disciplinary climate r School principals report a below-average picture concerning teacher-related factors affecting disciplinary climate (rank 19) 31% report that learning is hindered by teachers’ low expectations of students 48% report that learning is hindered by teachers not meeting individual students’ needs 34% report that learning is hindered by staff resisting change … r School principals report a fairly positive picture of teachers morale and commitment and an exceptionally high picture of student morale and commitment (ranks 9 and 1 respectively)

Strong ambitions Access to best practice and quality professional development Accountability Devolved responsibility, the school as the centre of action Individualised learning Integrated educational opportunities

Strong ambitions Access to best practice and quality professional development Accountability and intervention in inverse proportion to success Individualised learning Devolved responsibility, the school as the centre of action Integrated educational opportunities

And the reality today?  

Creating a knowledge-rich profession in which schools and teachers have the authority to act, the necessary knowledge to do so wisely, and access to effective support systems The tradition of education systems has been “knowledge poor” The future of education systems needs to be “knowledge rich” National prescription Professional judgement Informed professional judgement, the teacher as a “knowledge worker” Informed prescription Uninformed professional judgement Uninformed prescription, teachers implement curricula

Further information –All national and international publications –The complete micro-level database …and remember: Without data, you are just another person with an opinion

Space & shape item Answers: Yes, No, Yes, Yes Process skill: connections Context: educational quasi-realistic problem typical in maths classes not genuine occupational problem Form: complex multiple-choice Source: OECD (2004) Learning for tomorrow’s world: First results from PISA 2003, Figure 2.4a, p.52.

Change and relationships item Scores: 1 for n = 140x0.8 = 112 but no further work shown 2 for correct steps/min but not m/min; correct m/min but not km/hr; correct method but error of calculation; correct km/hr but not giving m/sec 3 for correct m/min (89.6) and m/hr (5.4), rounding acceptable. Process skill: score 1=connections score 2=connections score 3=reflection Context: personal Form: open-constructed

Quantity item Form: open constructed response Answer: Yes, with adequate explanation Process skill: reflection Context: public Form: short constructed response Answer: ZAR (unit not required) Process skill: reproduction Context: public Form: short constructed response Answer: 975 SGD (unit not required) Process skill: reproduction Context: public

Uncertainty item Scores: 1 for “No, not reasonable” but explanation lacking detail (e.g. focusing on exact increase in number of robberies without comparison with total) 2 for “No, not reasonable” with argument focusing on only small part of graph shown, ratio or percentage increase, or need for trend data. Process skill: connections Context: personal Form: open- constructed

Development of PISA r Assessments Development of frameworks –Development of a working definition of the domain and a description of the assumptions underlying that definition –Organisation of the domain so as to devise useful reporting scales –Identification of key task variables that will be used in test construction –Validation of the variables and assessment of the contribution each makes to understanding task difficulty –Development of framework weights –In terms of student response time –Preparation of an interpretative scheme for the results Preparation of guidelines for item submissions and item development