1. Wednesday, 3 June 2015 Dr Mafu S Rakometsi
Portfolio Committee on Higher Education Report on the Quality Assurance of NC(V) Level 2-4; NATED N1-3 and GETC: ABET L4
Wednesday, 3 June 2015
Dr Mafu S Rakometsi

2. Introduction to context, principles, approaches and processes
WHAT CAN WE LEARN FROM THE NSC RESULTS?
Introduction to context, principles, approaches and processes
Dr Mafu S Rakometsi
Chief Executive Officer of Umalusi

3. Regulatory Framework
Established through the promulgation of the General and Further Education and Training Quality Assurance (GENFETQA) Act, no. 58, 2001, as amended 2008;
Two predecessors, namely the Joint Matriculation Board - JMB (1918) and the South African Certification Council – (SAFCERT) (1986); and
Umalusi started work in 2001 having taken over from SAFCERT.

4. Regulatory Framework Section 17A. GENFETQA ACT, 2001 amended 2008
(1) The Council must assure the quality of assessment at exit points.
(3) The Council must perform the external moderation of assessment of all assessment bodies and education institutions.
(4) The Council may adjust raw marks during the standardisation process.

5. Section17 of the GENFETQA Act
(5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has—
(i) conducted the assessment free from any irregularity that may
jeopardise the integrity of the assessment or its outcomes;
(ii) complied with the requirements prescribed by the Council for
conducting assessments;
(iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and
(iv) complied with every other condition determined by the Council.

6. Regulatory Framework
National Qualifications Framework (NQF) Act, 2008 (Act No. 67 of 2008):
The (3) Quality Councils were established.
Umalusi - Quality Council for the General and Further Education and Training.
Development, management and promotion of the GENFETQS-f.
The capacity of private providers to offer qualifications that are registered on the GENFETQS-f.
NQF Act Section 27 (h)
The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework

7. Framework for Quality Assurance of Assessment
Evaluation and /or accreditation of assessment bodies;
Periodic inspection of assessment systems;
Ongoing monitoring of assessment systems;
Quality assurance of external examinations through:
Moderation of examination question papers;
Monitoring and moderation of SBA;
Monitoring the conduct of examinations;
Moderation of marking; and
Standardization of assessment outcomes.

8. Quality Assurance of the
DHET, IEB and BENCHMARK November 2014 examination
Ms Faith Ramotlhale
Senior Manager: Quality Assurance of Assessment

9. Moderation of question papers
PURPOSE
To ensure that the question papers are:-
of the required standard; and
are relatively:
fair
reliable
representative of an adequate sample of the curriculum
representative of relevant conceptual domains
representative of relevant levels of cognitive challenge

10. Approach Moderation Process
DHET:NATED/NC(V)
DHET (GETC)
IEB (GETC)
BENCHMARK (GETC)
Offsite
Centralized
Moderation Process
Question paper development - examiners (assessment body)
Internal moderation - Internal moderators (assessment body)
External moderation - External moderators (Umalusi)
Subsequent moderations and approval

11. Moderation of the question papers
Criteria
Technical criteria
Internal moderation
Content coverage
Text selection, types and quality of questions
Cognitive demands
Marking memorandum/ guidelines
Language and bias
Predictability

12. Question Paper Approval
Assessment body
No of papers
Moderated
Approval of Papers
DHET: GETC
26/26
54% first moderation
46% second and third moderation
IEB:GETC
8/8
75% first moderation
25% second moderation
BM:GETC
4/4
DHET: NATED N2-3
N2: 15/15
N3: 23/23
Different model used
DHET: NC(V) L2-4
L2: 25/25
L3: 24/25
L4: 80/80

13. Moderation of the question papers
Areas of Good Practice
No question paper leakages( all GETC- BM,IEB & DHET)
Questions were not predictable- Creativity and innovation (GETC- BM,DHET & IEB)
The setting of an extra set of papers, first step towards eighteen month cycle, was a step in the right direction (DHET-NC(V))
To ensure equivalence of standards, the November, March and back-up examination question papers were moderated simultaneously, (DHET-NC(V) and NATED)

14. Moderation of Question Papers
Areas of concern
Improve the quality of internal moderation, (DHET, IEB, BM).
Training of examiners and internal moderators to enhance understanding of Subject and Assessment Guidelines, (GETC- DHET & Benchmark)
Focus areas of training: depth and breadth (ECD4, INCT4, LCZU4, LCSO4, LCXI4, TECH4, TRVT4 and WHRT4), (GETC- DHET)
Poor quality memoranda - over reliance on Memorandum Discussion process to improve quality of marking guidelines, (GETC-DHET)

15. Moderation of Question Papers
Areas of concern
The question papers for A4TECH & A4HSSC did not meet the cognitive demands at first moderation (GETC-IEB)
Correct versions question papers, marking guidelines, internal moderator reports and assessment frameworks to be made available for external moderation, (DHET -NC(V))
Resubmission of rejected question papers to Umalusi – turn around time to be improved, (DHET - NC(V)& NATED)
Standard of back-up papers to be the same as November and supplementary papers, (DHET-NC(V)
Quality and standard of internal moderation – a rigorous process should be evident, (DHET-NATED and NC(V))

16. Moderation of Question Papers
Areas of concern
Outdated and/or underspecified curricula requires urgent revision (DHET-NATED);
Predictability of papers due to repetition of questions from previous question papers should be avoided; (DHET- NATED), and
Quality of assessment grids and marking guidelines should be improved (DHET-NATED, NC(V)).

17. Internal Assessment (SBA)
Definition and Purpose
Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications
GETC: 50%; NC(V) fundamental subjects 25%; NC(V) vocational subjects 50%; and NATED subjects 40%
Ascertain the degree to which the assessment body is attempting to ensure standardisation across providers
Ascertain the standard and quality of the tasks
Establish the extent and quality of internal moderation and feedback, and
Verify the reliability and validity of the assessment outcomes

18. Site-Based Assessment: GETC
Areas of Good Practice
Year-on-year improvement regarding the administration compliance requirements, (DHET)
The SBA user guides developed by IEB were helpful in structuring the content of Common Assessment Tasks

19. Moderation of Internal Assessment - GETC
Assessment Body
Common Assessment Tasks (CATs) moderated and approved
CASS
DHET
5 sets x 29 learning areas
492 portfolios moderated for 29 LA
IEB -
120 portfolios moderated for 6 LA
Benchmark
2 x 2 learning areas
107 portfolios moderated for 2 LA

20. Moderation of Internal Assessment - GETC
Areas of concern
No system to monitor the implementation of the CATs, (IEB)
The CATs did not adequately cover the specific assessment criteria as specified in the unit standards at first moderation, (BM)
The weightings of the CATs also did not comply with the SAG requirements, (BM)
Centres do not submit educator portfolios with sample requested – no evidence of assessment plans (IEB, Benchmark, DHET)
Assessors (educators) do not understand the use of rubrics, and marking guidelines (Benchmark, DHET)

21. Moderation of Internal Assessment - GETC
Areas of concern
Poor compilation of SBA portfolios and insufficient supporting documents provided, (Benchmark)
Very poor internal moderation system and processes (DHET)
Authentication is ad-hoc and sporadic (considering 50% weighting) (DHET)
Substantial differences between SBA and examination marks (DHET).

22. Moderation of Internal Assessment NC(V)
Areas of concern: ISAT
The quality of some tasks remains a cause for concern
Enrolments must be in line with available resources to effectively assess candidates’ acquisition of the practical skills required in the workplace
ISATs should be distributed to colleges in time for effective procurement and implementation
Care should be taken to prevent subjective and overly lenient marking
Moderation should be improved
Absenteeism among students affected the completion of ISATs,

23. Internal Assessment : ICASS
Directives for Improvement (NC(V)
Improving and deepening the understanding of the latest ICASS Guidelines (NCV)
Development of assignments and tasks
Implementation of practical assessments is still problematic,
Implementation of analysis grids and Bloom’s taxonomy
Development of marking tools and rubrics
Assessment and moderation practices need improvement

24. Moderation of Internal Assessment ICASS
Areas of good practice: (NATED)
There were pockets of excellence at some of the sites moderated. dedication of innovative lecturers  
Areas of concern
To ensure the availability and implementation of the latest version of ICASS guideline
Training of lecturers - limited experience in the industry
Teaching aids e.g. models are needed to improve learners’ understanding of the subject content
To address class attendance and retention rates at some colleges

25. Monitoring of Examinations: GETC Areas of Concern (DHET)
Lack of storage rooms and inadequate security for question papers
Transportation of exam material using public transport (EC)
Candidates allowed to write without proper identification
Some centres keep answer scripts overnight - poor security
Assessment body
DHET
IEB
Benchmark
No exam centres
125 7 2
No of monitors 68 4

26. Monitoring of Examinations: GETC
Areas of concern
Low number of candidates who wrote the exam, (IEB, BM)
Poor compliance with exam regulations (BM, IEB)
Inadequate storage facilities for QPs with poor security arrangements
Poor administration regarding exam regulations
Chief invigilators lack expertise to manage exams
Private exam centres often apply to the IEB to conduct the exam on alternative dates (IEB)

27. Monitoring of Examinations: TVET
Areas of good practice
General management of the examinations was good in terms of security of papers, appointment and training of invigilators
Areas of Concern
Irregularities during the writing phase of the examination, candidates were found with cell phones
Others entered the examination without identity documents or examination
Some Colleges monitored, papers were not sent to the marking centre within 24 hours, nor securely sealed

28. Monitoring Of Marking Phase: GETC
Areas of good practice
Marking centres were generally well organised and suitable for the task, (DHET,IEB,BM)
Adequate and experienced security personnel at the gates and entrances to the venues, (DHET,IEB, BM)
Training of marking personnel was generally thorough and effective, (DHET,IEB,BM)
DHET
IEB
Benchmark
No. of marking centres 8 1
No. centres monitored

29. Monitoring of Marking Phase: GETC
Areas of Concern
Shortage of markers in NC and KZN due to appointing markers for more than one learning area, (DHET)
Internal moderation model different and inconsistent across marking centres,(DHET)
Markers in LP disturbed the training process – apparently because of outstanding remuneration issues, (DHET)
The IEB marking centre management plan was not well documented – its based on experience and practice, (IEB)
There was no security in the building where the actual marking was done to control the flow of scripts, (BM)

30. Monitoring of Marking Phase: TVET
Areas of good practice
Safety and security of marking personnel and material was in general very good
Markers were aware of what constituted an irregularity and how it should be dealt with.

31. Monitoring of Marking Phase: TVET
Areas of concern
Distribution incorrect versions of marking guidelines - English and Afrikaans
Quality of marking and internal moderation of low enrolment subjects
Inappropriate allocation of marking venues according to the nature of subject, and number of markers - drawing subjects

32. Verification of Marking
PURPOSE
Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices
Umalusi engages with the following processes during the moderation of marking:
Centralised memorandum discussions
Verification of marking (centralised and on-site)

33. Memorandum Discussions
Areas of good practice
Good procedure observed: Panel discussions followed by marking of exemplar scripts – marking commences after markers have demonstrate an acceptable level of competence, (IEB,BM)
Analysis of evidence bound in educator portfolios were meticulous and contained all relevant documents as per policy requirements
Areas of concern
Poor practice of pre-marking in 15/26 LAs – discussions very difficult and time-consuming, (DHET)
No pre-marking or preparing alternative responses in advance, (IEB)

34. Memorandum Discussion TVET
Areas of good practice
Sample marking improved
Areas of concern
Appointments should be finalised well in advance
Marking personnel must attend marking guideline discussions,
Preparation of marking personnel for the marking process must be improved
Correct version of marking guidelines must be made available

35. Verification of Marking: GETC
Areas of Good Practice
Good internal moderation practices (DHET, IEB)
Good quality and standard of marking, (DHET, IEB)
DHET
IEB
Benchmark
Centralised
9 LA
702 Scripts -
On-site
5 LA
542 scripts
4 LA
215 Scripts
2 LA
70 scripts

36. Standardisation and Resulting
Provision of GENFETQA – Council may adjust raw marks
International practice – large scale assessment systems
Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance
Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

37. Verification of the Resulting Process
Planned
Status
Subject structures
Subject structures verified
Candidate registration
System verified during state of readiness visits
Generation of mark Sheets
Monitored
Capturing of Marks
Standardisation data & Booklets
Data sets received and verified
Capturing of adjustments
Verified
Statistical moderation & resulting

38. Statistical Moderation
Scope of standardisation 2014: GETC
Decision
Number of subjects
DHET
IEB BM
Subjects standardised 26 8 2
Raw marks accepted 11
Moderated Upwards 10 -
Moderated Downwards 05

39. Statistical Moderation
Areas of good practice
Booklets and data sets were submitted according to the prescribed timelines (IEB, BM)
Standardisation and approval meetings went smoothly

40. Statistical Moderation
Areas for Concern
Late submission and inaccurate standardisation data delays the approval of results, (DHET)
Late submission of qualitative input reports, (DHET)
Poor performance of learners,30-40% candidates do not write the exam, (DHET)
Non-adherence to the prescribed colours to draw the graphs (IEB)
Very low enrolment numbers (BM)

41. Statistical Moderation
Scope of standardisation 2014: NATED / NCV
Decision
Number of subjects
NATED
NC(V)
Subjects standardised 83
241
Raw marks accepted 46
125
Moderated Upwards 15 38
Moderated Downwards 22 78

42. Statistical Moderation: TVET
Areas of good practice
Standardisation booklets complied to the prescribed colours
Areas of concern
Late submission of standardisation booklets delayed standardisation
Inaccurate data sets and low capture rate of data delayed the approval of results
Non submission of consolidated irregularity report to Umalusi for standardisation and approval of results

43. Conclusion
The quality assurance processes presented above were conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations
From the findings of the quality assurance processes many good practices were evident but on the other hand, there are still many challenges that need to be addressed
Significant interventions are required to improve the curriculum, teaching and learning, exposure of learners to a compulsory practical component and assessment for the NATED programmes to make it a meaningful qualification

44. Conclusion
The outdated and underspecified curriculum and lack of subject assessment guidelines for the NATED programmes creates problems for appropriate quality assurance to be conducted
Umalusi takes this opportunity to express appreciation to the Department of Higher Education and Training for their concerted efforts in delivering credible examinations in

45. Thank you!