1. Select Committee on Education and Recreation
24 February 2010
Dr Mafu S Rakometsi

2. WHAT CAN WE LEARN FROM THE NSC RESULTS?
Prof. John Volmink - Chairperson of Umalusi

3. The Class of 2009
Second cohort of candidates to write the new National Senior Certificate (NSC)
Completely new qualification based on the National Curriculum Statement
The NSC replaces the Senior Certificate which was written at the end of 2007 and which will finally be phased out in 2011
Some sat for the NSC ( enrolled including repeater candidates)
Approximately 60,25% of all those who have written have met the requirements of a pass

4. Comparing the NSC with that of previous years
The NSC differs from the previous Senior Certificate both in structure as well as in grading
A critical feature of the Senior Certificate was that what determined a pass or fail was based on the aggregate score
The NSC determines a pass or fail by 40% in at least three subjects and 30% in three others
Not completely appropriate to compare overall results between the two qualifications

5. Some criteria for comparison
The number of candidates passing
The quality of the results
The standardisation of the exam
The predicative quality of the exam for performance at HE level

6. The number of candidates passing
Massification is a natural outcome of an education system that is non-racist, non-sexist and democratic
It is the clear that there has been a steady increase over the last decade in the number of learners who enrolled for and wrote the Senior Certificate
It is also significant that almost of the Class of did not pass because they narrowly missed the pass mark in one or two subjects
They qualify to write supplementary examinations and if half of these candidates pass, the final pass mark may go beyond the 67% mark
The underlying philosophy of the NCS had been designed to ensure that most people achieve the minimum requirement for a pass

7. The number of candidates passing : Trends in senior certificate enrolment and passes
Senior Certificate examination results, all schools,
Year
No. of
Candidates
Passes
Number
Change in number
(%)
Change in passes (%)
1995 7 -1
1996 -3 -2
1997 -6
1998 4
1999 -7 -8
2000 -4 13
2001
2002 5 17
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

8. The number of candidates passing : Trends in senior certificate enrolment and passes
Senior Certificate examination results, all schools,
Year
No. of
Candidates
Passes
Number
Change in number
(%)
Change in passes (%)
2002 5 17
2003 -7 -1
2004 6 3
2005 9
2006 4 1
2007 7
2008 -6 -9
2009
3.5
334718
0.33
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

9. The quality of the results
Concerns about quality has become a thorny issue and the subject of much debate albeit uninformed at most times.
SA public uses the matric results as the main indicators of quality of the education system and so there is an understandable concern about what these new results of a new qualification is saying about the current state of education
It is important however that we interrogate some of our assumptions that inform our beliefs about standards and quality

10. Assumptions that affect our perceptions Assumption 1
The first assumption is that “more means less” or “more means worse”
Stellenbosch University Website:
“In the light of phasing out of matriculation exemption and of the distinction between subjects on the higher and standard grade, a significant number of learners will probably qualify for basic admission to university on the basis of their subject combination…..The implication is that many more learners may be able to apply for admission to Stellenbosch University. ……. With this background, it becomes clear why it is necessary for the University to have its own measure to interpret the meaning and value of the new NSC and also to generate additional information in order to make finer distinctions.”

11. Assumption 2: Trustworthiness The standardisation of the exam
Another assumption is that we now use vastly different methods of standardising the results
Responsibility for matriculation results has changed hands:
JMB ( )
SAFCERT (1992 – 2001)
Umalusi (2002 – to date)
The one constant has been the standardisation process used for the examinations.
This is crucial to:
Obtain equivalence of the standard of the SCE
Across years, subjects and examination authorities
It is also necessary in order to deliver a relatively constant product to the HE sector and to the workplace

12. There is an assumption that we are at a much worse place today than in 1995

13. Senior Certificate examination results
Senior Certificate examination results, all schools,
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

14. Umalusi’s research on comparisons of NSC with NATED 550
Another assumption is that the NCS is of a lower standard than the previous curriculum.
Research completed by Umalusi into the standard of the NCS curriculum confirms that in most cases the NCS presents a greater cognitive challenge
The NCS also represents modern, updated and more demanding versions of previous subjects

15. The NCS philosophy
It needs to be said however that the National Curriculum Statement represents different sets of standards for all
Not only does it reflect the new values embedded in the Constitution but also aims to develop learners who can respond to the growth and development of knowledge and technology and the demands of the 21st century

16. What is the NCS? Nationally set curriculum Internationally benchmarked
Modern and relevant: 21st Century skills
Reflect the progressive values of the constitution
Focus on Africa and South Africa

17. The design of the NCS ASs = leaves of the tree
LOs = branches of the tree
COs and DOs = trunk of the tree
10 FUNDAMENTAL VALUES = roots of the tree
CONSTITUTION = soil in which the Fundamental Values are grounded

18. Nine principles of the NCS
Social transformation
Outcomes-based education
High knowledge and skills
Integration and applied competence
Progression
Articulation and portability
Human rights, inclusivity, environmental and social justice
Valuing indigenous knowledge systems
Credibility, quality and efficiency
PRINCIPLES = nutrients that feed the tree

19. Five critical and seven developmental outcomes
Solve problems
Work with others
Manage self
Communicate early
Use science and technology
Understand world as a set of related systems
Strategies to learn
Citizenship
Cultural and aesthetic sensitivity
Education and career opportunities
Entrepreneurial opportunities

20. Can HE Trust the NSC Results?

21. Senior Certificate examination results and HE
Senior Certificate examination results, all schools,
Year
No. of
Candidates
Passes
University
Exemption
Number
Rate (%)
1994 58
88 497 18
1995 53
78 821 15
1996 54
79 768
1997 47
69 007 12
1998 49
69 856 13
1999
63 725
2000
68 626 14
2001 62
67 707
2002 69
75 048 16
2003 73
82 010 19
2004 71
85 117
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

22. Senior Certificate examination results and HE
Senior Certificate examination results, all schools,
Year
No. of
Candidates
Passes
University
Exemption
Number
Rate (%)
2004 71
85 117 18
2005 68
86 531 17
2006 66
85 830 16
2007 65
85 454 15
2008 63 20
2009 60
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

23. Senior Certificate examination results and HE
Senior Certificate examination results, all schools,
Year
No. of
Candidates
University
Exemption
Number
Change in number
(%)
Change in exemptions (%)
1995 7
78 821
-11
1996 -3
79 768 1
1997
69 007
-13
1998 -1
69 856
1999 -7
63 725 -9
2000 -4
68 626 8
2001 -8
67 707
2002 5
75 048 11
2003
82 010 9
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

24. Senior Certificate examination results and HE
Senior Certificate examination results, all schools,
Year
No. of
Candidates
University
Exemption
Number
Change in number
(%)
Change in exemptions (%)
2003 -7
82 010 9
2004 6
85 117 4
2005
86 531 2
2006
85 830 -1
2007 7
85 454
2008 -6 26
2009
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

25. Senior Certificate examination results
Senior Certificate examination results, all schools,
Sources: DoE, 2004, 2005a, 2005b, 2005c, 2007, 2008, 2009

26. Maintaining Standards Research Project
Purpose of the Research
To provide Umalusi’s Assessment and Statistics Committee with information on the comparability of the old NATED 550 and new National Curriculum Statement curricula, and on the comparative difficulty of the exams associated with each.

27. Methodology Comparing curricula Benchmarking NCS exams
In-depth policy analysis of content, skills and assessment
Design features i.e. outcomes, assessment standards, sequencing, progression, etc.
Benchmarking NCS exams
Comparison: NATED 550 HG & SG (2005-7) and 2008 & 2009 NSC exam papers
Analysis of NSC 2009 examination in correlation with SAG

28. Subjects Considered in 2008
Curriculum Evaluation
Exam Analysis
English FAL √
Geography
Life Sciences
Mathematics
Mathematics Literacy
Physical Sciences
The available budget, time, and capacity meant that six subjects could be researched: the main (high enrolment) ‘gateway’ subjects used to assess suitability for entrance to tertiary institutions were selected.
Reports are available

29. Subjects Considered in 2009
Curriculum Evaluation
Exam Analysis
English FAL √
Geography
Life Sciences
Mathematics
Mathematics Literacy
Physical Sciences
Accounting
Business Studies
Economics
History
The available budget, time, and capacity meant that six subjects could be researched: the main (high enrolment) ‘gateway’ subjects used to assess suitability for entrance to tertiary institutions were selected.

30. The case of Mathematics
This special role of mathematics has seen it continuously singled out in discussions on:
Curriculum reforms
Examination results
Historically 30% to 40% of secondary schools in the country simply did not offer any mathematics beyond grade nine
In 2009 all 620,000 learners took some form of mathematics

31. Mathematics and the Class of 2009
The Class of 2009 had learners who passed mathematics at the 50% level or higher
The Class of 2008 had learners who passed mathematics at the 50% level or higher
This stands in contrast to the 25,000 who passed Higher Grade mathematics in 2007
A further learners passed mathematical literacy, most of whom would not have done any mathematics in the previous system.
A total of passed ML at the level of 80% or higher

32. Math pass rate Wrote Passed at 30% Passed at 40% Passed at 50%
Number passing Math at different thresholds
Wrote
Passed at 30%
Passed at 40%
Passed at 50%
Passed at 60%
Total did not pass
2008
89 788
63 038
42 323
2009
85 491
52 866
31 786
Sources: DoE, Abdridged Report on the 2008 NSC Examniation Results, December 2008; Doe, Technical Report – NSC, December 2008

33. Trends in HG Math pass rate
Number passing HG Math
Year
No. Wrote Math in 000’s
No. Wrote Math HG in 000s
No. Passed Math HG in 000s
1995
200,000
60,000
29,000
1996
215,000
65,000
22,000
1997
231,000
68,000
20,000
1998
280,000
1999
281,000
50,000
2000
284,000
39,000
19,000
2001
264,000
35,000
2002
261,000
21,000
2003
258,000
36,000
23,000
2004
276,000
40,000
24,000
* Pass at 50% or higher
Sources: Vithal, 2008, Table constructed from data in CDE Research Report No 13; Kahn 2007, DoE Senior Certificate Technical Report 2006; DoE Senior Certificate Report Figures rounded off

34. Trends in HG Math pass rate
Number passing HG Math
Year
No. Wrote Math in 000’s
No. Wrote Math HG in 000s
No. Passed Math HG in 000s
2004
276,000
40,000
24,000
2005
303,000
44,000
26,000
2006
318,000
47,000
25,000
2007
348,000
46,000
2008
298,821
n.a.
63,038*
2009
290,630
52,866*
* Pass at 50% or higher
Sources: Vithal, 2008, Table constructed from data in CDE Research Report No 13; Kahn 2007, DoE Senior Certificate Technical Report 2006; DoE Senior Certificate Report Report on the National senior Certificate Examination results 2009 Figures rounded off

35. Some remarks about Mathematics and the Class of 2008
The cognitive challenge for mathematics for the class of 2008 was closer to the old SG level as judged by the Umalusi research panel
Furthermore there was a mismatch between the suggested cognitive demand as provided by the SAGs and the cognitive challenge in the exemplars and the final mathematics examination
Only about 46% of learners met the minimum requirements for a pass in mathematics

36. Some remarks about mathematics and the Class of 2009
2009 exam was more challenging than the exam
2009 aligned to the exam guideline
Provided a challenge at the top end
Only about 44% of learners met the minimum requirements for a pass in mathematics

37. Cognitive demand for HG and SG Mathematics
% of marks at each level of cognitive demand on HG and SG Math papers SG HG
2008 NSC
2009 NSC
Knowledge (%) 21 10 9 18
Routine procedures (%) 53 42 62 36
Complex procedure (%) 20 40 24 35
Problem solving (%) 7 8 4 11
Sources: Draft Composite Exam Analysis Report: Maths, Umalusi Research, 2008

38. The case of Physical Science
Curriculum needs urgent attention
Content overload
Selection of examinable content (especially Chemistry)
2009 exam was of a similar standard to 2008
Difficult for both top and lower achieving learners

39. Physical Science and the Class of 2009
The Class of 2009 had learners who passed Physical Science at the 40% level or higher
The Class of 2008 had learners who passed Physical Science at the 40% level or higher
This stands in contrast to the who passed Higher Grade Physical Science in 2007

40. Physical Science pass rate
Number passing Physical Science at different thresholds
Wrote
Passed at 30%
Passed at 40%
Passed at 50%
Passed at 60%
Total did not pass
2008
61 480
32 524
16 620
98 042
2009
81 507
45 531
22 329
10 308
Sources: DoE, Abdridged Report on the 2008 NSC Examniation Results, December 2008; Doe, Technical Report – NSC, December 2008

41. Trends in HG Physical Science pass rate
Number passing HG Physical Science
Year
No. Wrote Phys. Sc in 000’s
No. Wrote Phys. Sc in 000s
No. Passed Phys.Sc HG in 000s
2006
69,302
29,781
2007
71,172
28,122
2008
217,300
n.a.
32,524 *
2009
220,957
22,329*
* Pass at 50% or higher
Sources: Vithal, 2008, Table constructed from data in CDE Research Report No 13; Kahn 2007, DoE Senior Certificate Technical Report 2006; DoE Senior Certificate Report Report on the National senior Certificate Examination results 2009 Figures rounded off

42. Cognitive demand for HG and SG Physical Science
% of marks at each level of cognitive demand on HG and SG Physical Science papers SG HG
2008 NSC
2009 NSC
Factual (%) 21 12 15
Conceptual (%) 29 30 38 39
Problem Solving (%) 50 58 46
Sources: Draft Composite Exam Analysis Report: Maths, Umalusi Research, 2008

43. Results of other Critical Subjects
Life Sciences
2009 exam was more difficult for below-average candidates, but not so challenging for above-average candidates
Accounting
curriculum
NCS content more than Nated 550 content
Examination guidelines cover % of new disciplines
2009 exam was of a similar standard to 2008
difficult for both top and lower achieving learners
The available budget, time, and capacity meant that six subjects could be researched: the main (high enrolment) ‘gateway’ subjects used to assess suitability for entrance to tertiary institutions were selected.

44. Quality Assurance of the 2009 National Senior Certificate Examination
Dr Mafu S Rakometsi – CEO of Umalusi

45. 2009 Quality assurance processes
Moderation of the question papers;
Moderation of Site-Based Assessment (SBA);
Monitoring of the conduct of the NSC examination;
Verification of marking; and
Standardization of marks.

46. Scope of the quality assurance processes
Number of NSC 2009 question papers moderated
Number of subjects
Number of papers
Approved/ Conditionally approved at 1st moderation
Approved/ Conditionally approved at 2nd moderation
Approved/ Conditionally approved at 3rd moderation 38
130 44 73 13

47. Scope of the quality assurance processes…
Moderation of SBA
Subjects
Provinces EC FS GP
KZN
Limp MP NW NC WC
Accounting √
Business Studies
Economics
English First Additional Lang
Geography
History
Life Sciences
Life Orientation
Mathematics
Mathematical Literacy
Physical Science

48. Scope of the quality assurance processes…
Monitoring of the NSC examination: writing phase
Province
No of exam centres
No of candidates enrolled
No of exam centers monitored by Umalusi
No of Umalusi monitors per province
No of exam centers monitored by Umalusi staff EC
908
76 242 17 04 01 FS
314
31 412 GP
787 14 02
KZN
1 670 29
Limp
134 30 MP
528
58 759 34 NC
135
11 461 NW
382
32 420 22 03 WC
416
48 198 12
Total
5 272
169 09

49. Scope of the quality assurance processes…
Monitoring of the NSC examination: marking
Province
No of marking centres
No of marking centers visited by Umalusi monitors
No of marking centers visited by Umalusi staff EC 14 05 01 FS 15 06 GP 07 02
KZN 31 08
Limp 19 MP 16 NC 03 NW - WC
Total
121 45

50. Scope of the quality assurance processes…
Moderation of marking
Memoranda discussion meetings
Approved and signed off finalized memoranda/ marking guidelines for all the NSC subjects
Centralised moderation (see next slide)

51. Scope of the quality assurance processes…
Centralized moderation of marking
Non language subjects
Language subjects
Accounting
Afrikaans First Additional Language P1, P2, P3
Agricultural Science P1, P2, P3
English First Additional Language P1, P2, P3
Business Studies
IsiXhosa Home Language P1, P2, P3
Economics
Sesotho Home Language P1, P2, P3
Geography P1, P2
Setswana Home Language P1, P2, P3
History P1, P2
Life Sciences P1, P2
Mathematics P1, P2, P3
Mathematical Literacy P1, P2
Physical Science P1, P2

52. Areas of good practice
Generally papers catered for both the weaker & high performing candidates,
In many cases the Nov 2009 & March 2010 were presented at the same time for moderation, this ensured comparability of standard,
External moderators signed the final papers off before dispatch to provinces for printing,
Great improvement observed in the monitoring of exams by provinces,
Security measures were intensified in the majority of centers across provinces,

53. Areas of concern Moderation of SBA
Majority of provinces do not conduct actual moderation of SBA tasks & learner portfolios – moderation exercise is limited to a compliance audit.
Majority of the SBA tasks reflect over reliance on past examination papers, educators lack creativity in developing suitable tasks,
SBA tasks developed by educators focus mainly on assessing the lower cognitive skills,
Educators battle with the development & use of rubrics used for scoring learners,

54. Areas of concern…
Implementation & Assessment of Life Orientation (LO) requires serious attention. LO is 100% internally assessed and standards vary from one province to the other, and even within provinces.
Monitoring of exams:
Incidents of improper registration of candidates resulted in duplications and some candidates not registered or registered for the incorrect subjects,
The appointment of suitable trained Invigilators and Chief Invigilators is essential. This will address problems relating to the flouting of even basic examination regulations.
The security systems in Mpumalanga during the printing, distribution, and storage of question papers require urgent attention.

55. Areas of concern…
Criteria for the appointment of markers should be strictly adhered to so as not to disadvantage candidates.
Moderation of marking:
Memoranda discussion meetings for small subjects did not receive the same attention as that given to the gateway subjects. These memorandum discussions were poorly attended. Added to this was the absence of input from the provinces.
The problems with the improper translation of question papers into Afrikaans is still of great concern.

56. Areas of concern…
Additions were made to the final memoranda without the consent of the DoBE and Umalusi. A few incidents of non- adherence to the final memoranda were observed, (pg 37, 38 of the NSC report).
Some markers do not have the ability and experience to handle higher-order cognitive level questions that required insight and logical reasoning.
There were instances of inaccurate totaling of marks and incorrect transferring of marks to the cover page during marking.

57. Areas of concern… Standardization of marks:
Data on the raw mark distribution for 2008 did not correspond with data contained in the 2009 standardization booklets. The number of candidates reflected in the pairs analysis was also not accurate.
The data sets submitted for verification of the resulting process was incomplete and sometimes not in accordance with the Umalusi specifications. This resulted in a delay of the verification process.
Poor learner performance especially in Mathematics, Physical Science & Accounting point to problems with the curriculum as well as teaching and learning.

58. Recommendations Moderation of SBA:
Moderation of the standard of SBA must be implemented at the school and cluster levels. Checks for compliance can still be part of the moderation process to ensure adherence to policy.
All the assessment bodies should ensure that all educators are familiar with the policy requirements to ensure the meeting of national standards.
Monitoring of exams:
The appointment of suitably trained Chief Invigilators is essential. This will address problems relating to the flouting of even basic examination regulations.

59. Recommendations … Verification of marking:
Criteria for the appointment of markers should be strictly adhered to so as not to disadvantage candidates.
Standardization of marks:
Data submitted for standardisation purposes must be accurate and in the specified format.

60. Recommendations …
There is a need to establish a basis for the standardization of Life Orientation (LO). Umalusi recommends that an externally- written component be introduced for LO with a view to set standards for the assessment of this subject.
There is an urgent need for very intensive teacher training focusing on the content of the new Accounting curriculum. This must then translate into the effective teaching, learning, and assessment of the subject.
The Physical Science curriculum appears to be challenging in terms of breadth and needs to be reviewed.
Teaching and Learning of Mathematics needs to be strengthened

61. Examination Irregularities
The majority of irregularities were of a technical nature and these were reported to Umalusi in terms of established processes.
Some irregularities were as a result of registration-related problems, eg candidates not appearing on mark sheets, some registered for incorrect subjects.
Major irregularities:
Leakage of the 5 papers in Mpumalanga: Mathematics P1 & 2, Physical Science P1 & 2 and Accounting. These were reported to Umalusi in October Back-up papers were moderated under tremendous time pressure.

62. Examination Irregularities…
On 20 December 2009, Umalusi was informed that the DoE investigations revealed that other papers may have been compromised in Mpumalanga.
An official report from the DoE was sent to Umalusi on 29 Dec.
Umalusi EXCO constituted on 30 Dec and agreed that the matter be investigated further.
Based on the information in the DoE report, Umalusi decided not to approve the release of Mpumalanga results at its approval meeting of 04 Jan 2010.

63. Examination irregularities…
The Assessment Standards Committee of Umalusi was tasked to do a thorough verification of the Mpumalanga results between 04 and 06 Jan 2010.
Method used:
comparison of the average performance of schools in 2009 with their average performance in 2008 per subject and based on their raw marks scored in both years.
the Standard deviations of these raw marks were also compared as well as the difference between the raw examination mean & the raw SBA mean between 2008 & Small positive differences in these means from 2008 to 2009 would mean that there were positive interventions. Significant (high) differences would indicate that there were possible irregularities.

64. Examination irregularities…
The investigation showed that most of the differences were negative, indicating that there was a downward trend in the performance of candidates in Mpumalanga.
Based on the investigation, the Assessment Standards Committee concluded that their was no indication of any patterns to suggest systemic inflation of results.
Umalusi EXCO presented the Umalusi findings to the Minister on the 6th January 2010.
Based on this Umalusi Council took a decision to approve release of Mpumalanga results.

65. Conclusion
The findings of the quality assurance processes are a clear indication of a maturing system that has, on the one hand, made positive strides towards improvement in certain areas of assessment and examination, but, on the other hand, still has a few challenges that need to be addressed.
The quality assurance of each of these processes presented above was conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.

66. Conclusion…
As a final quality assurance measure, Umalusi moderated the final marks awarded to the candidates. This is done through a rigorous standardization process carried out in line with established principles and procedures. Through this process, Umalusi ensures the consistency of the NSC examination over time and across the provinces.
Apart from the statistical inputs presented, Umalusi considered qualitative inputs, as well as very sound educational reasoning to arrive at the 2009 standardization decisions. Umalusi is proud to indicate that in the main, the candidates’ raw marks were accepted. There were very few instances where it was necessary to perform minor adjustments to the candidates’ raw marks.

67. Conclusion…
In general Umalusi is pleased with the manner in which the 2009 NSC examination was administered.
Umalusi acknowledges that a number of technical irregularities were reported, but these were addressed in a fitting manner.
The few serious irregularities reported were also afforded proper investigation and resolution to ensure that the credibility of the 2009 NSC examination is maintained.
Umalusi takes this opportunity to express appreciation to the national & provincial departments of education for their concerted effort in ensuring a credible examination.
Umalusi expresses appreciation also to all the relevant stakeholders for the necessary support given in line with Umalusi quality assurance initiatives.

68. Thank you!