Pacific Islands Literacy and Numeracy Assessment (PILNA) 2012 Nina Tu’i OCIES Conference Panel 4-6 November 2015 Port Vila, Vanuatu

Context International Large Scale Assessments (ILSAs) – Education policy informed by evidence Regional or National level LSAs – Capacity building and community engagement Sustainable development goals – Monitoring – Targeted interventions

Research methodology Conducted interviews with all those who were involved at the regional level of implementation Questionnaire for national level coordinators of PILNA 2012

Background 14 PICTs: Fiji, Federated States of Micronesia, Kiribati, Solomon Islands, Tuvalu, Papua New Guinea, Samoa, Cook Islands, Nauru, Tokelau, Palau, Niue, Republic of the Marshall Islands and Vanuatu More than 25,000 Year 4 and 6 students assessed for literacy and numeracy

Purpose Call from Pacific Education Ministers for more data on Literacy and Numeracy in PICTs Establish a baseline of literacy and numeracy levels of year 4 and 6 Pacific Island students Well understood by those implementing at the regional level

Processes Design – Jointly between SPBEA and UNESCO-Apia – Sampling Implementation – Instrumentation – Delivery of Assessment – Processing Reporting

Use of PILNA 2012 results

Dissemination of PILNA 2012 results

Impact of PILNA 2012 results

Lessons learnt from PILNA 2012 Clarity of purpose Capacity development Sampling Strategy for use of results by countries Design of assessment incorporating processing

2015 PILNA Torika Taoi OCIES Conference Vanuatu

2015 DESIGN The design seeks to address the two objectives: i)to develop a tool and processes that are grown out locally from within the region for delivering long term valid and reliable regional assessment data ii)to lift regional capacity in collecting and working with regional assessment data.

Country agreement November 2014: invite sent to countries Conditions for participation: The two key conditions require that: participating countries are committed to sharing the results with other countries for lessons that one can learn from each other. each country is committed to using the findings to carry out policy interventions as well as technical interventions aimed at improving the situation in each country.

PILNA 2015 and FEdMM Results of 2012 were presented to 2014 FEdMM In response to these findings: FEdMM endorsed re-run of PILNA in 2015 for all PICs countries but based on funding availability. FEdMM also called on Development Partners to work with countries to identify courses of poor performance. Focus on identifying specific areas of weaknesses regionally and at country level; using evidence to identify regional and country interventions. Adopting a benchmarking approach using evidence from 2015 PILNA More diagnostic

Purpose of PILNA 2015 Diagnose literacy situation in Pacific to identify main courses of low performance. To trial questionnaires: Require additional data based on factors influencing student performance such as; - school leadership, teacher quality, student assessment, curriculum - parental support, - socio-economic status of families, - policies in place.

PILNA PROCESS Develop Parallel Items: review of the 2012 test items, including its psychometric analysis was carried out in December 2014, to allow for comparability of 2015 results with the 2012 results. Parallel items developed and also new items constructed to fill in gaps identified from the 2012 set od learning outcomes

PILNA Process Components assessed in Literacy Writing Reading and Language features Components assessed in Numeracy Numbers- Recognising Numbers, Place values & Relations, Fractions & % Operations- Addition, Subtraction, Multiplication, Division(Y6 only), Problem Solving Measurement Data (only for Year 6) The cognitive skills were the same for both Year levels, but the skill demand increased in Year 6. These differences in skill demands reflected the learning outcomes that tapped into the different ability levels of the students

PILNA PROCESS Review of items for trial Items panel reviewed by EQAP officers before assembling for trial Trial run As there were new items developed, these had to be trialled. Items were trialled in 5 countries (PaBER countries and 2 non-PaBER countries): Samoa, SI, PNG, Kiribati and Fiji

PILNA PROCESS Data Scoring Different from – Degree of correctness : 0, 1, 2, 3, – Levelling : L0, L1, L2, L3, L Scoring and LevellingScoring: L0 – Inability to demonstrate proficiency in any of the skills that are basic 0 – 0% correct L1 – Not yet working at expected level1 – partially correct; - 100% correct L2 – Working towards at expected level2 – partially correct; - 100% correct L3 - Performing at expected level3 – 100% correct L4 – Performing above expected level9 – no answer

PILNA PROCESS Data processing and Data analysis Samoa, Solomon Islands, PNG, Fiji (Kiribati) – In-country and central scoring – Electronic data entry Cleaning – EQAP + Country (NCs) IRT analysis Assemble items for main test: Items were finally reviewed Spelling and typo errors edited

PILNA PROCESS Translation In consideration of Language Policies or languages of instructions applicable at lower primary in the PILNA participating countries. PILNA instruments are translated into 8 country languages (Cooks, Niue, Tonga, Vanuatu, RMI, Tokelau, Kiribati & Tuvalu Only 4 countries opted for translated versions in 2012

PILNA PROCESS REPORTING Data will be analysed and reports will be developed in the first quarter of As with PILNA 2012, it is anticipated that reports will be developed at the following levels: Country Regional Sub-regional levels

Sampling The sampling design is complex, given the extreme variations that apply across the countries participating in PILNA. Census Approach- applied to relatively smaller countries: o Cook Islands, Niue, Palau, RMI, Tokelau, and Tuvalu (but with certain schools excluded). Sampling approach- applied to FSM, Kiribati,PNG, Samoa, Solomon Islands, Tonga and Vanuatu (also with certain exclusions). The sampling arrangements for Papua New Guinea, Samoa and Solomon Islands reflect decisions made through the PaBER processes. Fiji and Nauru are not participating in the PILNA 2015.

Sampling In the sampling frame, schools are categorised by: country, education district/province/region, location address, locality (urban, non-urban, remote, very remote) school authority (government, private, religious authority) regional authority (government, non-government) school size, Year 4 size, and Year 6 size.

The Pacific Islands Literacy and Numeracy Assessment (PILNA) – Strengthening Regional Educational Assessment ‘Uhila-moe-langi Fasi OCIES Conference Port Vila, Vanuatu 2015

Students performance Students factorss School environment Home environment SURVEY INSTRUMENTS

Student factors language Study habits attitude

Home Environment Socio-economic factors Where student lives House type Home possessions Number of meals

School Environment: TEACHERS Qualifications Experience Professional development Pedagogy Assessment knowledge and skills Community involvement

School Environment: HEAD TEACHER Management Staff development Leadership skills and experience Qualifications and training

Research Questions: How do the socio-economic factors influence student performance in PILNA? How does the student study habit influence his/her achievement in PILNA? What influence does teacher quality have on student achievement in PILNA? How does school management and leadership influence student achievement in PILNA?

Instrument: Questionnaires STUDENTS QUESTIONNAIRETEACHERS QUESTIONNAIREHEAD TEACHERS QUESTIONNAIRE

Instrument development process: Draft – EQAP Review – EQAP Expert review – ACER Trial Implement

The Pacific Islands Literacy and Numeracy Assessment (PILNA) – Strengthening Regional Educational Assessment Elizabeth Cassity OCIES Conference 2015

Feedback / Evaluation ACTION Improved outcomes Improved life consequences Understanding of current situation Knowledge about how to improve Required resources A framework for assessment - Educational decision making loop Masters, 2013

How is the analysis being conducted for PILNA 2015 (in collaboration with EQAP)? Psychometric support – Item Response Theory model Sampling support – strategy that states sampling objectives, accounts for diverse populations and other parameters Questionnaire support – feedback on 2015 pilot questionnaires; support for future cycles Support for trend measures – providing advice on linking PILNA across years

What are plans for reporting and dissemination for PILNA 2015? Plans outlined at PILNA Steering Committee Meeting, August focus on student learning outcomes is paramount Six reporting and dissemination priorities: 1. Interventions 2. Policy 3. Political support 4. Community awareness 5. Monitoring results 6. National validation

What is the long term plan for PILNA? Regional assessment conducted every three years Linking of PILNA cycles to measure trends Questionnaires distributed to all teachers, head teachers and students participating in PILNA (post- 2015) Ongoing development of specification and shared vision for long term assessment program