Portfolio Committee on Basic Education STANDARDISATION Dr M Rakometsi CEO: Umalusi Prof M Moodley ASC Member 7 February 2018

Overview Mandate and Regulatory Framework Framework for Quality Assurance of Assessment Standardisation of results Demystifying Myths on standardisation 2017 NSC Standardisation decisions Conclusion

Mandate and Regulatory Framework Functions of the quality council (QC) regarding quality assurance of assessment National Qualification Framework (NQF) Act No. 67 of 2008 The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework. The QC must, in respect of quality assurance within its sub- framework, do the following:

Mandate and Regulatory Framework Functions of the quality council (QC) regarding quality assurance of assessment National Qualification Framework (NQF) Act No. 67 of 2008 develop and implement policy for quality assurance; ensure the integrity and credibility of quality assurance; ensure that quality assurance as is necessary for the sub- framework is undertaken;

Mandate and Regulatory Framework Section17A of the GENFETQA Act (3) The Council must, perform the external moderation of assessment of all assessment bodies and education institutions (4) The Council may adjust raw marks during the standardisation process

Mandate and Regulatory Framework Section17A of the GENFETQA Act (5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has— (i) conducted the assessment free from any irregularity that may jeopardise the integrity of the assessment or its outcomes; (ii) complied with the requirements prescribed by the Council for conducting assessments; (iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and (iv) complied with every other condition determined by the Council.

Mandate and Regulatory Framework- Cont. Section18 of the GENFETQA Act an assessment body—’’. 18(a) must take adequate measures to combat irregularities at assessment and marking centres and must take adequate security measures to ensure the confidentiality and integrity of the assessments; This section read in conjunction with section 42(6) of the Regulations pertaining to the conduct, administration and management of the NSC examination Gazette No 31337, emphasises the importance of keeping confidentiality of all resulting data (written, electronic format and oral) 18(j) may recommend to the Council during the standardisation process that raw marks be adjusted.

Framework for Quality Assurance of Assessment Quality assurance of external examinations through (18 months cycle): - Moderation of question papers; - Moderation and verification of school based assessment (SBA); - Monitoring the state of readiness to conduct the NSC; - Monitoring and audit of the selection and appointment of markers; - Monitoring of the writing of examinations; - Marking guidelines discussion; - Monitoring of marking; - Verification of marking; - Standardisation and resulting; and - Certification

Objectives of Standardisation To ensure that a cohort of learners is not advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude. To achieve comparability and consistency from one year to the next.

Why Umalusi Standardises Results and How International practice – large scale assessment systems; Standardisation – process used to mitigate the effect of factors other than the learners’ knowledge and aptitude on the learners performance. Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions.

Why Umalusi standardises results and how cont. Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably; Process of standardisation: Moderation of question papers; Review of learner performance against historical performance of candidates in each subject; Historical average (norm) constructed using past 3 to 5 years data; Pairs analysis provides further comparisons of raw means; Statistical moderation of internal assessment.

Why Umalusi standardises results and how cont. Qualitative input meetings; Reports (Moderator, Chief Marker and Internal Moderator); Umalusi research (maintaining standards & post exam analysis); Responsibility of Assessment Standards Committee (ASC) - Committee of Council (independent experts drawn from different HE institutions and research institutes): Education, Mathematics and Statistics experts – large data ASC is responsible for setting and maintaining standards; Observers (SAQA, USAf, Teacher Unions – (Portfolio Committee on Basic Education since 2011).

STANDARDISATION WORKSHOP

General Principles Applied in the Standardisation of Examination Marks 1. In general no adjustment should exceed 10% of the historical average. 2. In the case of the individual candidate, the adjustment effected should not exceed half of the raw mark obtained by the candidate. 3. If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to limitations.

Demystifying Myths on Standardisation Standardisation is a manipulation of figures to increase learners’ pass rate Fact 1: Standardisation is not a process aimed at increasing learners’ pass rate but it is a process aimed at ensuring the equivalence of marks from year to year, regardless of variability of examinations, in the pursuit of a high level of fairness. Standardisation is an international process which is not unique to South Africa. The overall pass rate is never known at this stage.

Demystifying Myths on Standardisation Fact 1: In other words an A,B or C symbol should be equivalent to those of previous years and future years regardless of variability in exams. Candidates’ marks are only adjusted if there is compelling evidence from both qualitative and quantitative reports that the standard of the examination does not compare fairly with the previous examinations. Candidates examination marks are then adjusted to ensure the candidates are not ,advantaged or disadvantaged due to factors that are beyond their control.

Demystifying Myths on Standardisation Mainly upward or downward adjustment of marks involves the increase or decrease of learners’ marks by the same mark from 0-300 Fact1: Standardisation is a process guided by principles and guidelines of importance as follows: No adjustment should exceed half of the raw mark Adjustment should not exceed 10 percentage points above or below the raw mark For example, a raw mark of 10 will never exceed 15 and a raw mark of 130 will never be adjusted to more than 160 or less than 100

Demystifying Myths on Standardisation Fact 2: Mainly upward/downward adjustment does not necessarily mean that all marks from 0-300 increase/decrease by the same mark. A mainly upward adjustment can have some marks adjusted downwards while a mainly downward adjusted can have some marks adjusted upwards depending on the candidates performance , in relation to the norm(see Annexure A) The adjustment is identified as mainly upward/downward ,when most marks are adjusted upward or downward respectively (see Annexure B)

Demystifying Myths on Standardisation The decision to adjust is only determined by the mean of the current cohort A maximum adjustment is measured from the mean Fact: A maximum adjustment is not measured by the mean alone but by the spread of adjustments (see Annexure A) Decision to adjust is not determined by the mean only as the following factors are also taken into consideration: The qualitative input reports The previous five years and the norm (see Annexure C & D) pass rate distinction rate failure rate As well as the median.

Demystifying Myths on Standardisation Candidates’ final marks are finalised during standardisation Fact : Candidates final marks are not finalised at standardisation meetings NSC resulting consist of two components: the external component (examination) and the internal component (SBA). The final result/mark of the candidates consists of 75% ,exam and 25% SBA During the standardisation meeting adjustment are made only on the external component/examination raw mark (These adjustments are then effected on the examination…sense??) del The SBA marks are subjected to a process of statistical moderation before being added to the adjusted exam mark in the above ratio. The underlying assumption is that the learners’ performance in SBA and examination bears a positive relationship

2017 NSC Standardisation Decisions SACAI Description Total 2015 2016 2017 Number of subjects presented 27 28 Raw marks 24 16 18 Adjusted (mainly upwards) 2 9 7 Adjusted (mainly downwards) 1 3 Number of subjects standardised: Description Total Number of instructional offerings presented 59 Raw marks accepted 28 Computer adjustment 17 Half computer adjustment 10 Block Adjustment (mainly upwards) None Block Adjustment (mainly downwards) 1

2017 NSC Standardisation Decisions IEB Description Total 2015 2016 2017 Number of subjects presented 61 Raw marks 49 46 Adjusted (mainly Upwards) 12 7 5 Adjusted (mainly downwards) none 10 Number of subjects standardised:

2017 NSC Standardisation Decisions DBE Description Total 2015 2016 2017 Number of subjects presented 59 58 Raw marks 29 26 38 Adjusted (mainly upwards) 30 28 16 Adjusted (mainly downwards) none 4 Number of subjects standardised:

In conclusion The ASC, the committee of Council that standardises the results, is an independent body made up of experts from different universities and research institutes in the country. The ASC standardizes the results of the NSC, NC(V), Nated Courses (N2-N3) and the GETC - offered by the DBE, DHET, IEB, SACAI and Benchmark Assessment Agency. Equal attention must also be given to other qualifications - not just the NSC.

Conclusion The standardisation process is cumulative - it starts with the appointment of examiners, setting of question papers, moderation of question papers, verification of marking and the so-called standardisation meetings whose processes mark the end of the actual standardisation process. The full framework of standardisation is an 18 month process encompassing the exam cycle from the setting of the paper to the final standardisation meeting.

Conclusion The final pre-standardisation meeting starts on 15 December and the climax is the open meeting like the one attended by members of the Portfolio Committee on 23 December 2016. During pre-standardisation meetings all evidence - both quantitative and qualitative - is considered. Why is the standardisation data confidential? The standardisation marks are not final because they only include 75% written component and not the 25% of the School-Based Assessment (SBA). Standardisation marks also do not include rules of combination, condonation and language compensation marks.

Conclusion That is why it is dangerous to speculate the outcome of the results based on standardisation outcomes. Any predictions are misleading as the final outcome may be different after taking everything into account. The marks attained after standardisation do not form the final mark on the subject and the overall pass rate is never known at this stage

12th Conference of the Southern Africa Association for Educational Assessment (SAAEA) Host: Umalusi Date: 14 – 16 May 2018 Venue: The Capital Hotel, Menlyn, Pretoria Theme: Local context in Global context: encouraging diversity in assessment See you there!!!

Thank You!!