Assessing Learning Outcomes Polices, Progress and Challenges 1

2 CONTEXT India – the country with the largest education system in the world Worryingly low average levels of learning particularly in the domain of numeracy and literacy

CONTEXT Increasing recognition by the GOI of the challenges presented by poor learning outcomes. As India’s 12 th Five-Year Plan notes: “The four main priorities for education policy have been access, equity, quality and governance. The Twelfth Plan will continue to prioritise these four areas, but will place the greatest emphasis on improving learning outcomes at all levels.” [GoI, 2013: p49]. Driving up learning outcomes is now clearly a key priority for MHRD. 3

4 Federal and State Governments need reliable assessment data to inform evidence-based policy making while educational practitioners need diagnostic information in order to improve teaching and learning in classrooms. Assessments - Health check of education outcomes

Assessment Cycle

Progress towards more meaningful assessments During SSA India made notable progress in developing a multi-faceted approach to the assessment of learning In particular, the capacity building support provided by the DFID-funded Technical Cooperation Fund (TCF) to MHRD and NCERT impacted a transformational change in the way NAS was administered and analysed The TCF encouraged and enabled the adoption of Item Response Theory (IRT) as the basis for analysing test results and reporting outcomes on a meaningful ability scale This now allows results to be compared between groups and over time – even when different test instruments are used 6

Item Response Theory (IRT) IRT assumes there is a mathematical relationship between the probability of a student giving the correct answer - his/her ability and the difficulty of the item. There are two scales: item difficulty and student ability. 7

This girl has a certain level of ability in Maths. We want to measure her ‘Maths ability’. MATHS TEST #1 28/50 56% MATHS TEST #2 (harder) 21/50 42% The girl’s ability in Maths has not changed… but her score has! Classical Test Theory - Percentage correct scores are ‘test dependent

Why Item response theory (IRT) The baselines for learner achievement must be established on scales which are constant over time and independent of the particular tests taken by the students. IRT places results on robust and meaningful scales which are independent of the particular tests used. This is in line with international best practice.

IRT scaled scores are independent of the test taken This girl has a certain level of ability in Maths. We want to measure her ‘Maths ability’. SET OF MATHS ITEMS #2 Pattern of responses   The girl’s ability in Maths has not changed… nor has her score! SET OF MATHS ITEMS #1 Pattern of responses  Apply IRT model Scale score

Why Item response theory (IRT) Through the use of IRT all the test items can be placed on a meaningful scale of difficulty. ‘Items maps' which show what students at different parts of the ability range know and can do can be developed This gives vital information to curriculum designers, textbook writers, teacher trainers and - most importantly - subject teachers 11

GOI – Driving up the agenda for more meaningful assessment Continuous, rolling programme of national achievement surveys made an integral part of the education system A dedicated National Assessment and Evaluation Centre to be established to take the lead in the design and commissioning of such surveys on behalf of the government Continuous Comprehensive Evaluation(CCE) up to Grade VIII introduced. Indicating shift from high-stake external examinations towards school-based assessment for formative and summative purposes. 12

Improvements in reporting of results Homogeneity and heterogeneity of educational outcomes within states can be determined Meaningful comparisons among states Information about performance of students across the ability range. Future trends in educational standards can be judged, with confidence, against these newly established baselines. 13

14 National NAS Class VIII Summary Report, 2013 National NAS Class V Technical Report, 2011 State NAS Class V Summary Reports, 2013 National NAS Class III Technical Report, 2005

GROWING PUBLIC AWARENESS Increased interest in the findings of achievement surveys has been mirrored by increased coverage in the media. Where once attention was only paid to the ASER, other studies, including those conducted by NCERT, are now enjoying significant exposure 15

Challenges and way ahead Need to develop comprehensive and reliable databases for student enrolment across all grades and for all recognized school types in the government, government-aided, and private sectors Develop further the capacity to conduct analysis of student responses using item response theory. Need to develop a framework of criteria- related achievement levels against which to report learning outcomes.