1. 7 February 2012 Dr Mafu S Rakometsi
Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC
7 February 2012
Dr Mafu S Rakometsi

2. Introduction to context, principles, approaches and processes
WHAT CAN WE LEARN FROM THE NSC RESULTS?
Introduction to context, principles, approaches and processes
Dr Mafu Rakometsi - CEO of Umalusi

3. Quality Assurance of Assessment
Regulatory Framework
Quality Assurance of Assessment
NQF Act Section 27 (h)
The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

4. Section17 of the GENFETQA Act
(5) The Council must, with the concurrence of the Director-General and
after consultation with the relevant assessment body or education
institution, approve the publication of the results of learners if the
Council is satisfied that the assessment body or education institution
has—
(i) conducted the assessment free from any irregularity that may
jeopardise the integrity of the assessment or its outcomes;
(ii) complied with the requirements prescribed by the Council for
conducting assessments;
(iii) applied the standards prescribed by the Council which a learner is
required to comply with in order to obtain a certificate; and
(iv) complied with every other condition determined by the Council.

5. Framework for QA of Learner Achievement
Based on established and existing practices in assessment for certification
Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment
Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

6. Framework for Quality Assurance of Assessment
Evaluation and /or accreditation of assessment bodies
Periodic inspection of assessment systems
Ongoing monitoring of assessment systems
Quality assurance of external examinations through:
Moderation of examination question papers
Monitoring and moderation of SBA
Monitoring the conduct of examinations
Moderation of marking
Standardization of assessment outcomes

7. Approval of Results Compliance with minimum requirements
Shift from Moderation to Verification (Moderation for verification)
Emphasis on Assessment Body reports
The monitoring of the conduct of examinations and the moderation of marking as well as the monitoring of the implementation of internal assessment and the moderation of internal assessment is undertaken by Umalusi to verify the veracity of the assessment body reports in respect of these process.
Broadened definition of irregularities (including assessment system processes)

8. Approval for the release of Results
Approval is based on the following requirements:
The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations
At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.

9. Quality Assurance of the 2011 National Senior Certificate Examination
Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments

10. Moderation of Question papers
Number of NSC 2011 question papers moderate
Number of subjects
Number of papers
Approved/ Conditionally approved at 1st moderation
Approved/ Conditionally approved at 2nd moderation
Approved/ Conditionally approved at 3rd moderation
4th moderation and beyond
38 (excluding the non-official languages and LO)
Nov 41 73 18 3
Mar 38 72 19 1

11. Moderation of question papers
PURPOSE
To ensure that the question papers are of the required standard
(standard captured in the NCS and SAG’s)
To ensure that the question papers are relatively:
- fair
- reliable
- representative of an adequate sample of the curriculum
- representative of relevant conceptual domains
- representative of relevant levels of cognitive challenge
External moderators

12. Moderation of the question papers
Approach
Question papers set by panel of examiners – DBE
Internally moderated by DBE
Externally moderated by Umalusi
Subsequent moderations and approval.

13. Moderation of the question papers
Criteria
Technical Criteria
Internal moderation – quality of internal moderator reports.
Content coverage
Cognitive Demand
Marking Guidelines
Language and Bias
Predictability

14. Moderation of the question papers
Findings
Areas of Good Practice
Additional back-up papers for CAT and IT
Simultaneous moderation of final and supplementary question papers.
Decrease in the number of question papers going beyond second moderation.
Standard of Internal moderation

15. Moderation of the question papers
Findings
Areas of Concern
Adherence to timeframes
Simultaneous moderation of final and supplementary question papers.
Predictability in Afrikaans SAL – prescribed short stories.

16. Moderation of internal /continuous assessment
Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification
panels of moderators / subject specialists

17. Moderation of SBA
Previous model - moderation of learner and educator portfolios at the end of the year.
Umalusi implemented a new model in 2010 – focus on tasks rather than on portfolios
Ongoing – 3 moderations across 9 PDE’s
Feed back to educators
Improvement in SBA implementation
DBE implemented a similar model at a national level in 2011.

18. Moderation of internal assessment (cont.)
Purpose of Umalusi’s verification:
To verify the rigour and appropriateness of the DBE moderation process.
Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across
Ascertain the standard and quality of the tasks
Determine the extent and quality of internal moderation and feedback.
Determine the reliability and validity of the assessment outcomes

19. Moderation of SBA - Scope
Subjects
Mathematics, Life Sciences,
Physical Sciences
Accounting, English FAL, History, Geography, Life Orientation,

20. Moderation of SBA Findings Adherence to policy guidelines
Quality of Assessment tasks
Content Coverage
Cognitive demand and difficulty levels
Marking (tools, application and mark allocation)
Internal Moderation
Moderation instruments
Moderation at School level & district level
Feedback and Support

21. Moderation of SBA Findings Areas of Good Practice
DBE National Moderation
Provincially set preliminary examinations.
Increased use of common tasks

22. Moderation of SBA
Findings
Areas of Concern
Standard of tasks
Marking tools – use of rubrics
Life Orientation- Physical Education Tasks

23. Monitoring of Examinations State of readiness Conduct of examinations
Marking
State of Readiness
Monitored the following provinces with the DBE
Eastern Cape
Free State
Mpumalanga
North West
Limpopo
KZN

24. Monitoring of Examinations
State of readiness
Findings
All provinces have examination systems in place
Vacant posts and use of contract staff is a major concern.

25. Monitoring of the writing phase.
Findings
General Management of the examinations
Generally examinations conducted in line with policy
Isolated instances of non-compliance (suitability of venue, noise, security)
Shortage of question papers and answer books
No of exam centres
No of exam centers monitored by Umalusi
No of Umalusi monitors per province
No of exam centers monitored by Umalusi staff
6593
186 32 17

26. Monitoring of the marking phase
Findings
In general marking processes in line with policy
Concern with the appointment of markers
Concern with the quality and standard of marking
Large number of marking centres – security risk

27. Monitoring of the marking phase
Province
No of marking centers
(2010)
(2011)
Eastern Cape 14 15
Free State 16 18
Gauteng 08 07
KwaZulu-Natal 33 29
Limpopo 20 19
Mpumalanga 17
Northern Cape 03
North West
Western Cape 02 01
Total
127
123

28. Verification of marking
PURPOSE
Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices
Umalusi engages the following during the moderation of marking
1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres
2. Off-site moderation of marking

29. Verification of marking
Memoranda discussion meetings
Approved and signed off finalized memoranda/ marking guidelines for all the NSC subjects
Concern with non attendance and pre- memo discussion preparations

30. Centralized verification of marking
Findings
General improvement in standard of marking
Competency of markers to mark interpretation type questions.
Internal moderation
Unmarked questions
Unauthorised changes to marking guidelines.

31. Statistical moderation
Provision of GENFETQA – Council may adjust raw marks.
International practice – large scale assessment systems
Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance.
Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

32. Statistical moderation
Standardisation
56 subjects standardised
Raw marks accepted: 45 subjects
Moderated Upward : 3 subjects
Moderated Downward : 8 subjects

33. Verification of the Resulting Process
Planned
Status
Subject structures
Subject structures verified.
Candidate registration
System verified during state of readiness visits.
Generation of mark Sheets
Monitored
Capturing of Marks
Standardisation data & Booklets
Data sets received and verified
Capturing of adjustments
Verified
Statistical moderation & resulting

34. STANDARDISATION DECISIONS
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION DECISIONS
ERCO NSC 2011

35. SUBJECT
ADJUSTMENT
Accounting
Raw
Afrikaans First Additional Language
Afrikaans Home Language
Agricultural Management Practice
Agricultural Technology (DBE)
Agricultural Science
Business Studies

36. SUBJECT
ADJUSTMENT
Computer Application Tech.
Raw
Consumer Studies (DBE)
Civil Technology (DBE)
Dramatic Arts(DBE)
Design (DBE)
Economics
Engineering Graphics and Design
Electrical Technology

37. SUBJECT
ADJUSTMENT
English First Additional Lang.
CA (0 at 0, scale to +10 at 168, scale to +15 at 233, scale to 0 at 300)
English Home Language
Raw
Geography
History (DBE)
½ CA( 0 at 0, scale to -10 at 185, scale back to 0 at 300)
Hospitality Studies
Information Technology (DBE)
Life Orientation
Life Sciences
CA (0 at 0, scale to +30 at 120, scale to 0 at 300)

38. SUBJECT
ADJUSTMENT
Mathematics
CA (0 at 0, scale to +30 at 67, block +30 from 72 to 199, scale to 0 at 300
Mathematical Literacy (DBE)
Raw
Mathematics P3
Mechanical Technology (DBE)
Physical Science

39. SUBJECT
ADJUSTMENT
Religion Studies
Raw
Tourism
Visual Arts (DBE)
IsiZulu FAL (DBE)

40. STANDARDISATION DECISIONS
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION DECISIONS
IEB NSC 2011

41. SUBJECT
ADJUSTMENT
Accounting
CA (0 at 0, scale to +13 at 210; scale back to 0 at 300)
Business Studies
0 at 0, scale to -9 at 99, block of -9 from 99 to 135, scale to
-16 at 181, scale back to 0 at 300
Computer Application Technology
Raw
Consumer Studies
½ CA (0 at 0, scale to +8 at 139; scale back to 0 at 300)
Design
Dance Studies

42. SUBJECT
ADJUSTMENT
Dramatic Arts
Raw
Economics
CA(0 at 0, scale to -30 at 100, block -30 from 100 to 163, scale down to 0 at 300
Engineering Graphics and Design
CA (0 at 0, scale to -12 at 233, scale down to 0 at 300

43. SUBJECT
ADJUSTMENT
Geography
Raw
History
½ CA (0 at 0, scale to +13 at 77, scale down to 0 at 300)
Information Technology
Hospitality Studies
Tourism
Visual Arts
½ CA (0 at 0, scale to -15 at 142, block -15 from 142 to 157, scale to 0 at 300)
Music

44. SUBJECT
ADJUSTMENT
Life Orientation
Raw
Life Sciences
0 at 0 to -11at 222, scale to -5 at 245, scale back to 0 at 300
Mathematical Literacy
CA(0 at 0 scale to +10 at 72 scale back to 0 at 110, scale to -14 at 214, scale back to 0 at 300)
Mathematics
0 at 0 to -5 at 95, scale to -8 at 158, scale to -4 at 244, scale back to 0 at 300
Mathematics P3
Physical Science

45. SUBJECT
ADJUSTMENT
English FAL
Raw
English HL
Afrikaans FAL
0 at 0, raw to 90, scale to -14 at 225, scale to -7 at 247, scale to 0 at 270, raw to 300.
Afrikaans HL
Sepedi FAL
Sesotho FAL
Setswana FAL

46. SUBJECT
ADJUSTMENT
Siswati FAL*
Raw
Siswati HL*
IsiXhosa FAL*
IsiZulu HL*
IsiZulu FAL*
Arabic SAL*
0 at 0, raw to 180, scale to -6 at 225, block –6 from 225 to 270, scale back to 0 at 300
French SAL*
German HL*
German SAL*

47. SUBJECT
ADJUSTMENT
Gujurati SAL*
Raw
Hebrew SAL*
CA (0 at 0, scale to at 200, scale to -30 at 166, block -6 from 166 to 231, scale to 0 at 300
Hindi FAL*
Hindi SAL*
Italian SAL*
Latin SAL*
Portuguese SAL*

48. SUBJECT
ADJUSTMENT
Modern Greek SAL*
Raw
Portuguese HL*
Portuguese FAL*
Spanish SAL*
Tamil SAL*
Urdu HL*
Urdu FAL*
Urdu SAL*

49. SUBJECT
ADJUSTMENT
Equine Studies
Raw
Sport and Exercise Science
0 at 0, scale to +20 at 70, scale to +17 at 193, scale to 0 at 300
Maritime Economics
Nautical Science
Advanced Programme Mathematics
CA (0 at 0, scale to +13 at 109, scale to 0 at 300)
Advanced Programme English

50. STANDARDISATION DECISIONS
WHAT CAN WE LEARN FROM THE NSC RESULTS?
STANDARDISATION DECISIONS
DBE NSC 2011

51. SUBJECT
ADJUSTMENT
Life Sciences
0 at 0, scale to +10 at 106, block + 10 from 106 to 235, scale to 0 at 300
Physical Sciences
Raw
Mathematics
Mathematical Literacy
Mathematics: Probability; Data Handling
Geography

52. SUBJECT
ADJUSTMENT
Life Orientation
Raw
Hospitality Studies
Tourism
Agricultural Science
0 at 0, scale to -12 at 24, block
-12 from 24 to 281, scale 0 at 300
Consumer Studies
Agricultural Management Practices
½ CA (0 at 0, scale to -15 at 65, block -15 up to 300
Agricultural Technology
(Raw)

53. SUBJECT
ADJUSTMENT
Music
Raw
Dance Studies
Design
Dramatic Arts
Visual Arts
Religion Studies
Information Technology
Computer Applications Technology

54. SUBJECT
ADJUSTMENT
Civil Technology
Raw
Electrical Technology
½ CA(0 at 0, scale to -15 at 241, scale to -6 at 300
Mechanical Technology
Engineering Graphics and Design
Accounting
0 at 0, scale to +5 at 85, scale to 0 at 180, raw to 300
Business Studies
½ CA(0 at 0, scale to -15 at 145, block -15 to 239, scale to -3 at 300

55. SUBJECT
ADJUSTMENT
Economics
0 at 0, raw to 19, scale to +12 at 78, scale to 0 at 179, and then raw to 300.
History
½ CA( 0 at 0, scale to -10 at 185, scale back to 0 at 300)
Afrikaans HL
Raw
Afrikaans FAL
Afrikaans SAL
English HL
English FAL
0 at 0, scale to -4 at 94, block of -4 from 94 to 280, scale to 0 at 300.

56. SUBJECT
ADJUSTMENT
English SAL
Raw
IsiNdebele HL
IsiZulu HL
0 at 0, scale to -6 at 48, block of -6 from 48 to 247, scale to 0 at 300.
IsiZulu FAL
IsiZulu SAL
Setswana HL
Setswana FAL
Setswana SAL

57. SUBJECT
ADJUSTMENT
Siswati HL
Raw
Siswati FAL
IsiXhosa HL
IsiXhosa FAL
IsiXhosa SAL
Xitsonga HL
Xitsonga FAL

58. SUBJECT
ADJUSTMENT
Tshivenda HL
0 at 0, scale to -6 at 71, block of -6 from 71 to 207, scale to 0 at 300.
Tshivenda FAL
Raw
Sepedi HL
Sepedi FAL
Sesotho HL
Sesotho FAL
Sesotho SAL

59. Areas of good practice
Generally papers catered for both the weaker & high performing candidates,
In many cases the Nov 2011 & March 2012 were presented at the same time for moderation, this ensured comparability of standards
External moderators and Umalusi staff signed the final papers off before dispatch to provinces for printing,
Great improvement observed in the monitoring of exams by provinces,
Security measures were intensified in the majority of centers across provinces,

60. Areas of concern Late submission of question papers for moderation.
SBA tasks developed by educators focus mainly on assessing the lower cognitive skills,
Educators battle with the development & use of rubrics used for scoring learners,

61. Areas of concern…
Implementation & Assessment of Life Orientation (LO) requires serious attention. LO is 100% internally assessed and standards vary from one province to the other, even within provinces.
Incidents of improper registration of candidates resulted in duplications and some candidates not registered or registered for the incorrect subjects,
The appointment of suitable Chief Invigilators is encouraged. This will address problems relating to the flouting of even basic examination regulations.

62. Areas of concern…
Criteria for the appointment of markers should be strictly adhered to so as not to disadvantage candidates.
Additions were made to the final memoranda without the consent of the DBE and Umalusi.
Some markers do not have the ability and experience to handle higher-order cognitive level questions that required insight and logical reasoning.
There were instances of inaccurate totaling of marks and incorrect transferring of marks to the cover page during marking.
Large number of marking centres – security risk .

63. Recommendations Moderation of SBA:
Introduction of a common (standardised task)
Moderation of the quality and standard of SBA must be implemented at the school and cluster levels. Checks for compliance can still be part of the moderation process to ensure adherence to policy.
All the assessment bodies should ensure that all educators are familiar with the policy requirements to ensure the meeting of national standards. .

64. Recommendations … Verification of marking:
Criteria for the appointment of markers should be strictly adhered to so as not to disadvantage candidates.
All stakeholders need to be consulted when changes are made relating to the appointment of markers.
Standardization of marks:
Common framework for the assessment of Languages.
Teaching and learning of Mathematics and Science needs to be strengthened.

65. Examination Irregularities
The majority of irregularities were of a technical nature and these were reported to Umalusi according to the established channels.
Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects.
Umalusi represented on NEIC
Major irregularities:
No major irregularities reported

66. Conclusion
The findings of the quality assurance processes are a clear indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed.
The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.

67. Conclusion…
In general Umalusi is pleased with the manner in which the NSC examination was administered.
Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner.
Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination.
Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives.

68. Thank you!