Analysing ANA results Nicky Roberts Wits School of Education Kelello Consulting

Context Focus on Primary Maths project Errors in marking, adding and recording results (2012) Using ANAs as a diagnostic tool for teaching and development (2013)

Research in two schools in Cape Town Focus on Primary Maths – School based maths teams – Professional development for FP teachers in maths Mini seminars Co teaching interventions Mathematical thinking course – Responding to ANA attainment – Sharing lessons

Responding to ANA attainment Tracking ANA results year on year Question-by-question capture and analysis Planning for fourth term based on ANA analysis Rasch analysis comparing the two schools

Background to the two schools 2 schools are equal partners in development project, chose to be named at school level Capricorn Primary School (CPS) – 700 learners, 2011 below provincial mean (ANA and systemics) – Township, backyard dwellers, poverty, gangsterism and taxi violence, drugs (Tic), alcohol (FAS). Mostly households in poverty. – Minority not English main lang, LOLT is English The Grove Primary school – 700 learners, history of excellent attainment (ANA and systemics) – Suburban, model C, mostly affluent households – Majority English main lang, LOLT is English

No anchor questions Predictable test Not year-on-year comparison Assume system is stable, then performance relative to national mean is indicator of overall performance

Indicator of proportion of learners meeting grade level requirements (relative to national pass rate, on that ANA)

WCED systemic (standardized questions and marking, so year- on-year comparison)

Question by question data capture Real power for instructional design Q1 Q2 Q3 Q4... Learner 1 Learner 2 Learner 3 Learner 4 Learner 5 …. Correct Incorrect Facility

ANA analysis template Class capture sheet Summary sheet Question summary sheet Spreadsheet template suburban school Spreadsheet template township school

What we did in 2013 Before ANA marking  Import the learner names from EMIS into classes. One class per sheet.  Check formulas and that the template works as you want it to On the marking day  Laptops/ computers with template in marking venue  Capture marks for each script  Queries – go back to teacher who was marking

What we did in 2013 After marking days  Create grade summaries  Compare to previous years  Look for questions of common difficulty Very poor: colour code in green Poor: colour code code in pink Revise maths teaching and development plan for fourth term

Functions used in Excel template SUM AVERAGE MIN MAX STDEV Some notes on using Excel IF-THEN COUNT IF

What we did in 2014 Repeated the descriptive statistics  Used this to create grade level ‘mini lesson studies’ cycles, eg: Teaching bonds in Grade 1 Teaching views and positional language in Grade 2 Teaching analogue time in Grade 3  Plan – teach- reflect cycle  Teacher presentations at mini conference, AMESA regional and AMESA national

Mini conference (Day 1 internal)

Mini conference (Day 2 district)

Mini conference

AMESA regional

What we did in 2015 Repeated the descriptive statistics Used the data to conduct Rasch analysis:  Misfitting items  Biases between schools (based on DIF values) Used this to target problematic questions and review these qualitatively

Qualitative analysis (from initial qunat data) Total marks1 Mean0.02 Standard deviation0.14 Mean percentage2% Number writing80 Number getting 079 Percentage getting 099% Identified as a major area of weakness

Errors on Q 25.1 Some completely off Most common error was to break up number rather than converting

Conclusion ANA is a major investment in SA education It gives annual standardised data across schools If anchor questions were included, it would also give year-on year comparisons It gives schools rich data for error analysis (if item level analysis is conducted) which can lead to lesson study cycles/ interventions Teachers need support in marking ANAs (again directs attention to the mathematics) and reporting common errors to district/ province