1. Umalusi Moderation & Release of the 2010 Standardisation Decisions National Consultative Forum Meeting 07 May 2011

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Context, principles, approaches and processes

3. Reasons for Disclosure Unprecedented interest in the role of Umalusi Request: Access to Information Uninformed comments Credibility of the NSC Credibility of Umalusi

4. Legal Mandate Chapter 2A of GENFETQA Act 58 of 2001 (as amended in 2008)

5. Section17 of the GENFETQA Act (1) The Council must assure the quality of assessment at exit points (2)(a) The Council must develop policy for the accreditation of assessment bodies other than departments of education and must submit it to the Minister for approval (b) The Minister must make regulations in which the policy for accreditation is set out (c) The Council must accredit an assessment body in accordance with the regulations contemplated in paragraph (b) (3) The Council must perform the external moderation of assessment of all assessment bodies and education institutions (4) The Council may adjust raw marks during the standardisation process.

6. Section17 of the GENFETQA Act (5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has— (i) conducted the assessment free from any irregularity that may jeopardise the integrity of the assessment or its outcomes; (ii) complied with the requirements prescribed by the Council for conducting assessments; (iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and (iv) complied with every other condition determined by the Council. (6) The Council must issue certificates to learners who have achieved qualifications or part qualifications

7. Framework for QA of Learner Achievement Based on established and existing practices in assessment for certification Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

8. Framework for Quality Assurance of Assessment Evaluation and /or accreditation of assessment bodies Periodic inspection of assessment systems Ongoing monitoring of assessment systems Quality assurance of external examinations through: Moderation of examination question papers Monitoring and moderation of SBA Monitoring the “State of readiness Monitoring the conduct of examinations Moderation of marking Standardization of assessment outcomes

9. Approval of Results Compliance with minimum requirements Shift from Moderation to Verification (Moderation for verification) Emphasis on Assessment Body reports The monitoring of the conduct of examinations and the moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process. Broadened definition of irregularities (including assessment system processes)

10. Why Umalusi standardises results, and how Provision of GENFETQA – Council may adjust raw marks. International practice – large scale assessment systems Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance. Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

11. Objectives for Standardisation 1)To ensure that a cohort of learners is not advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude. 2)To achieve comparability and consistency from one year to the next.

12. Why Umalusi standardises results, and how Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably Process of standardisation  Moderation of question papers  Review of learner performance against historical performance of candidates in each subject.  Historical average (norm) constructed using past 3 to 5 years data.  Pairs analyses provides further comparisons of raw means  Statistical moderation of Internal assessment

13. Why Umalusi standardises results, and how Qualitative input meetings  Reports (moderator, Chief marker and Internal Moderator)  Umalusi research (maintaining standards &post exam analysis) Responsibility of Assessment Standards Committee  Committee of Council  Responsible for setting and maintaining standards Observers (SAQA, HESA, Teacher Unions)

14. Process for Standardisation Continuation of JMB and SAFCERT model Assessment Standards Committee Qualitative Reports Pre-standardisation and Standardisation meetings. Standardisation booklets (data) – subject raw mark distributions (external written component only) of entire cohort. Subjects are standardised individually, in a linear and non- iterative manner

15. General principles applied in the standardisation of examination marks 1. In general no adjustment should exceed 10% or the historical average. 2. Adjustments in excess of 10% could be considered at the upper end to increase the number of distinctions in a subject. 3. In the case of the individual candidate, the adjustment effected should not exceed 50%of the raw mark obtained by the candidate. 4. If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to the limitations.

16. General principles applied in the standardisation of examination marks (cont.) 5. If the distribution of the raw marks is above the historical average, the marks may be adjusted downwards subject to the limitations. 6. The computer adjusted mark is calculated based on these principles. 7. For those subjects with a practical component of 50%, raw marks could be accepted. 8. Umalusi however retains the right to amend these principles where and when deemed to be necessary based on sound educational principles.

17. Main instruments of standardisation 1.Historical average (HA) of raw scores over 3-5 years 2. Distribution of raw and adjusted marks in 10% intervals 3. Graph plot showing cumulative percentage of candidates at each mark 4. Raw mark distribution including the number and cumulative percentage of candidates at each raw mark

18. Main instruments of standardisation 5. Pairs analyses 6. Qualitative inputs: examiners’, moderators’ and examining bodies’ reports. 7. Qualitative and quantitative inputs: Umalusi’s ‘research group’.

19. Guiding Principles of Standardisation 1. Apply both positive and negative adjustments 2. No adjustments of more than 10% in either direction 3. Positive adjustments should generally not push the results above the HA (norm) 4. Marks that are well beyond the HA/norm should be adjusted downwards, but not necessarily as far as the norm

20. Guiding Principles of Standardisation 5. Raw marks which are close to the norm should not be adjusted 6. Take account of reports by examiners, moderators and research group 7. Any adjustment – no change in rank order 8. Small subjects generally not adjusted - raw

21. Specific issues / debates Does Umalusi meet with government and look at the pass rates on raw exam marks? Does Umalusi adjust marks to achieve a particular pass rate? Is Umalusi subjected to political pressure to achieve a particular pass rate?

22. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS ERCO NSC 2010

23. SUBJECTADJUSTMENT Accounting 0 at 0 scale to +18 at 72, block 18 at 222 and scale to 0 at 300 Afrikaans First Additional LanguageRaw Afrikaans Home LanguageRaw Agricultural Management PracticeRaw Agricultural ScienceRaw Business Studies Raw Civil Technology (DBE) 0 at 0; scale to -15 at 30, scale to -8 at 248, scale to 0 at 300

24. SUBJECTADJUSTMENT Computer Application Tech.Raw Consumer Studies (DBE) 0 at 0, scale to +4 at 8, block +4 from 8 to 296, scale to 0 at 300 Dramatic Arts(DBE)Raw EconomicsRaw Electrical TechnologyRaw Engineering Graphics and DesignRaw English First Additional Lang. 0 at 0, scale to +6 at 234 and scale back to 0 to 300 English Home Language 0 at 0, scale up to +9 at 111, scale down to 0 at 300

25. SUBJECTADJUSTMENT French 2nd Additional Language (IEB) 0 at 0, +20 at 70 scale back to 0 at 300 Geography (DBE) 0 at 0, scale to +8 at 41, block +8 at 86 ; scale to 0 at 154, and then raw. History (DBE) Raw Hospitality StudiesRaw Information Technology (DBE)Raw IsiZulu First Additional (DBE)Raw Life OrientationRaw

26. SUBJECTADJUSTMENT Life SciencesRaw Mathematics 0 at 0, scale to +27 at 63, scale to +12 at 228 and scale to 0 at 300 Mathematical Literacy Raw Mathematics P3Raw Mechanical Technology (DBE)Raw Physical Science 0 at 0, scale to +12 at 78 and scale to +2 at 238 and 0 at 300 Portuguese 1st Additional Language (IEB)Raw

27. SUBJECTADJUSTMENT Religion StudiesRaw Sesotho 1st Additional Language (DBE)Raw TourismRaw Visual Arts (DBE)Raw

28. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS IEB NSC 2010

29. SUBJECTADJUSTMENT Accounting 0 at 90 scale to at -7 to 247; Scale back to 0 at 300 Afrikaans First Additional Language 0 at 0, scale to +6 at 84, block +6 from 84 to 234, scale down to 0 at 300 Afrikaans Home LanguageRaw Business Studies 0 at 0, scale up +6 at 84, scale to +9 at 231, scale back to 0 at 300 Computer Application TechnologyRaw Consumer StudiesRaw DesignRaw

30. SUBJECTADJUSTMENT Dramatic Arts 0 at 200 scale -4 at 244 and scale back to 0 at 300 EconomicsRaw Engineering Graphics and Design 0 at 0 + 12 at 78 scale to 0 at 200, scale to -10 at 250, scale back to 0 at 300 English First Additional LangRaw English Home LanguageRaw GeographyRaw HistoryRaw Information TechnologyRaw

31. SUBJECTADJUSTMENT IsiZulu First Additional LanguageRaw Life OrientationRaw Life SciencesRaw Mathematical Literacy 0 at 0 to -30 at 63, block -30 from 63 to 281, scale back to 0 at 300 Mathematics 0 at 0 to +6 at 105, block +6 from 105 to 144, scale back to 0 at 300 Mathematics P3Raw Physical Science 0 at 90 scale to +6 at 144, block of +6 from 144 to 234, scale back to 0 at 300

32. SUBJECTADJUSTMENT TourismRaw Visual Arts 0 at 200 + 2 at 238, scale back to 0 at 300 Dance StudiesRaw Hospitality StudiesRaw MusicRaw Sepedi First Additional LanguageRaw Sesotho First Additional LanguageRaw Setswana First Additional LanguageRaw

33. SUBJECTADJUSTMENT Siswati First Additional LanguageRaw Siswati Home LanguageRaw IsiXhosa First Additional languageRaw IsiZulu Home LanguageRaw Arabic 2nd Additional Lang 0 at 0, scale to +20 at 70 scale back to 0 at 300 French 2nd Additional Lang 0 at 0, scale to +20 at 70 scale back to 0 at 300 German Home LanguageRaw German 2nd Additional Language 0 at 200,scale to -7 at 247, scale back to 0 at 300

34. SUBJECTADJUSTMENT Gujurati 2nd Additional LangRaw Hebrew 2nd Additional Lang 0 at 200, scale to -7 at 247, scale to 0 at 300 Hindi 2nd Additional LangRaw Italian 2nd Additional LangRaw Latin 2nd Additional LangRaw Modern Greek 2nd Additional LangRaw Portuguese 1st Additional LangRaw Portuguese 2nd Additional LangRaw

35. SUBJECTADJUSTMENT Spanish 2nd Additional LangRaw Tamil 2nd Additional LangRaw Telegu 2nd AdditionalRaw Urdu Home LanguageRaw Advanced Programme Mathematics 0 at 0, scale to +15 at 150, block +15 to 240, scale to 0 at 300 Equine StudiesRaw Sport and Exercise Science 0 at 0, scale to +15 at 30, block +15 at 200, scale to 0 at 300

36. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2010

37. SUBJECTADJUSTMENT Accounting 0 at 0, scale to +23 at 67, scale to +6 at 234, scale to 0 at 300 Afrikaans FAL Raw Afrikaans HL Raw Afrikaans SAL Raw Agricultural Management Practice Raw Agricultural Science 0 at 0, scale to -9 at 18, block -9 from 18 to 220, scale down to 0 at 240 and raw to 300

38. SUBJECTADJUSTMENT Agricultural TechnologyRaw Business StudiesRaw Civil Technology 0 at 0; scale to -15 at 30, scale to -8 at 248, scale to 0 at 300 Computer Application Tech Raw Consumer Studies 0 at 0, scale to +4 at 8, block +4 from 8 to 296, scale to 0 at 300 Dance Studies Raw Design Raw Dramatic ArtsRaw

39. SUBJECTADJUSTMENT EconomicsRaw Electrical Technology 0 at 0, scale -6 at 46, block -6 to 226, scale to 0 at 240 and raw up to 300 Engineering Graphics and Design Raw English HL 0 at 0, scale to +3 at 117, scale back to 0 at 171, thereafter raw to 300 English First Additional Lang FAL 0 at 0, scale to +3 at 6, block +3 from 6 to 297, scale to 0 at 300

40. SUBJECTADJUSTMENT English Second Additional Lang Raw Geography 0 at 0, scale to +8 at 41, block +8 at 86 ; scale to 0 at 154, and then raw to 300. HistoryRaw Hospitality StudiesRaw Information TechnologyRaw IsiNdebele HL 0 at 0, scale to -6 at 12, block of -6 from 12 to 300 IsiZulu HL 0 at 0, scale to -6 at 12, block of -6 from 12 to 300 IsiZulu FALRaw

41. SUBJECTADJUSTMENT IsiZulu SALRaw IsiXhosa HL Raw IsiXhosa FAL 0 at 0, scale to -6 at 12, block of -6 from 12 to 300 IsiXhosa SAL Raw Life OrientationRaw Life Sciences 0 at 0 ; scale to -9 at 23; block -9 from 23 to 286 and scale down to 0 to 300 Mathematics 0 at 0, scale to +22 at 68, scale down to +15 at 225, scale to 0 at 300

42. SUBJECTADJUSTMENT Mathematical Literacy Raw Mathematics P3Raw Mechanical TechnologyRaw MusicRaw Physical ScienceRaw Religion Studies 0 at 0 scale to -12 at 24; block -12 to 291 and scale down to 0 at 300 Sepedi HLRaw Sepedi FALRaw

43. SUBJECTADJUSTMENT Sesotho HL 0 at 210, scale to +4 at 236, scale down to 0 at 300 Sesotho FALRaw Sesotho SALRaw Setswana HL Raw up to 210; scale to +7 at 233; scale to 0 at 300 Setswana FALRaw Setswana SALRaw Siswati HL 0 at 0, scale to -6 at 12, block of -6 from 12 to 300 Siswati FALRaw

44. SUBJECTADJUSTMENT Tshivenda HL 0 at 210 scale to +7 at 233 and scale to 0 at 300 Tshivenda FALRaw TourismRaw Visual ArtsRaw Xitsonga HL 0 at 120 scale to -6 at 148 and block of -6 to 272 and scale to 0 at 300 Xitsonga FALRaw Nautical ScienceRaw Maritime EconomicsRaw