19 February 2014 Dr Mafu S Rakometsi

19 February 2014 Dr Mafu S Rakometsi
Select Committee on Education and Recreation Report on the Quality Assurance of the NSC 19 February 2014 Dr Mafu S Rakometsi

Role of Umalusi, principles, approaches and processes of QA
WHAT CAN WE LEARN FROM THE NSC RESULTS? Role of Umalusi, principles, approaches and processes of QA Dr Mafu Rakometsi - CEO of Umalusi

Role of Umalusi Umalusi is the quality assurer in general and further education and training of the National Qualifications Framework (NQF) The Council ensures that the qualifications and curricula within GFET are of quality, that providers of education and training have the capacity to deliver and assess qualifications and are doing so to expected standards of quality, and that assessments are of the required standard

Establishment of Umalusi
Established through the promulgation of the General and Further Education and Training Quality Assurance Act, 2001 (GENFETQA Act number 58 of 2001, as amended 2008) Two predecessors, namely the Joint Matriculation Board - JMB (1918) and the South African Certification Council - SAFCERT (1986) Umalusi started work in 2002 having taken over from SAFCERT

Establishment of Umalusi
Umalusi was established as a band education and training quality assurance body under the GENFETQA Act in 2001 (NQF levels 1-4): Quality Assuring exit point assessments for qualifications in schools (National Senior Certificate), FET Colleges (N3, NCV) and for Adult (GETC) Accrediting Independent schools, private FET Colleges and Adult Learning Centres as well as private assessment bodies

Umalusi’s brief In the review of the Implementation of the NQF was completed and in 2008 the GENFETQA Act was amended creating Umalusi as one of three Quality Councils with extended mandates, the other two being Council on Higher Education and the Quality Council for Trades and Occupations The object of the amended Act is to enhance the quality of general and further education and training through: Development and management of a sub- framework of qualifications for GFET

Umalusi’s brief Quality assurance of: Qualifications and curricula
Provision through the accreditation of private providers of education and assessment, to provide and assess these qualifications Exit point assessments of the qualifications

Umalusi’s brief Certifying learner attainments for these qualifications Conducting research on matters pertaining to the GFET sub-framework of qualifications Advising the Minister on matters related to the GFET sub-framework of qualifications

Quality Assurance of the DBE 2013 National Senior Certificate Examination
Emmanuel Sibanda – Acting Sen. Manager : Quality Assurance of Assessments

Moderation of question papers
PURPOSE: To ensure that the question papers are of the required standard (standard captured in the NCS and SAG’s) To ensure that the question papers are relatively: - fair - reliable - representative of an adequate sample of the curriculum - representative of relevant conceptual domains - representative of relevant levels of cognitive challenge

Moderation of the question papers
Approach: Question papers set by panel of examiners – DBE Internally moderated by DBE Externally moderated by Umalusi Subsequent moderations and approval

Criteria : Technical criteria Internal moderation Content coverage Text selection, types and quality of questions Predictability Cognitive skills Marking memorandum or guidelines Language and bias

Findings: Areas of Good Practice Percentage of question papers and memoranda approved after first and second moderation ( Nov % ; Mar 14 – 78%) (out of 130 papers) Simultaneous moderation of final and supplementary question papers.

Findings: Areas of Concern Adherence to timeframes and impact on quality of setting and moderation. Question papers requiring more than four moderations. 2 papers for November 2013 (Isizulu HP P1, Isixhosa HL P1) and 2 for March 2014 (Business Studies, Isizulu HL P1)

Moderation of internal assessment
Definition: Internal assessment refers to any assessment conducted by the provider , the outcome of which count towards the achievement of the qualifications Umalusi appoints panels of moderators / subject specialist to carry out this mandate

Moderation of internal assessment
Purpose of Umalusi’s verification: To verify the rigour and appropriateness of the DBE moderation process – linked to DBE plans Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across Ascertain the standard and quality of the tasks Determine the extent and quality of internal moderation and feedback. Determine the reliability and validity of the assessment outcomes

Moderation of Internal Assessment
Approach 1 (June/July 2013)- verifying the DBE SBA moderation PROVINCIAL EDUCATION DEPARTMENT Subjects All Mathematics, Life Sciences, Physical Sciences Accounting, English FAL, History, Geography, Life Orientation,

Approach 2 (June/July 2013)- Umalusi independent moderation (own sample) PROVINCIAL EDUCATION DEPARTMENT Subjects All Economics, Business Studies, Mathematical Literacy, Music, (Practical subjects) Mechanical Technology, Electrical Technology

Areas of good practice: DBE conducted very rigorous moderation and provided useful verbal feedback to PDE’s at the end of each moderation session General adherence to policy in terms of number of tasks done, presentation of learner evidence/portfolios.

Areas of concern: While internal moderation is being done in most schools, much of the focus is on compliance (monitoring) and not on qualitative issues (actual moderation) Persistent problem of lack of constructive feedback given back to learners after moderation Teachers are still unable to develop tasks pitched at appropriate cognitive levels: focus is more on lower cognitive level.

Monitoring of Examinations
“State of readiness” Conduct of examinations Marking State of Readiness Monitored the DBE state of readiness visits in 6 provinces: Eastern Cape, Free State, Kwazulu-Natal, Limpopo, Northern Cape, Gauteng Comprehensive approach that monitors exam systems (including SBA)

Monitoring of Examinations
“State of readiness” Findings: All provinces have working examination systems in place Of concern: Many vacant posts and use of contract staff In some PDE’s closer monitoring printing of question papers needs attention Inefficient or lack of coordination with districts wrt exam related processes

Monitoring of the writing phase
Findings: Generally examinations conducted in line with policy Isolated instances of non-compliance (suitability of venue, identification of learners) No of exam centres No of exam centers monitored by Umalusi Total number of Umalusi monitors 6699 159 36

Monitoring of the marking process
Number of marking centres No of marking centers monitored by Umalusi Total 118 77

Monitoring of marking process
Findings: Marking centres were generally well organised and suitable for the task. Inadequate and inexperienced security and in some provinces, Absence of communication facilities reported at one centre in one province

Verification of marking
PURPOSE: Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices Umalusi engages the following during the moderation of marking Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres Moderation of marking (centralised and on-site)

Marking verification Memo discussion meetings: Areas of good practice:
The memo discussions for the approval of final memoranda went relatively well in Provision of an extra day of training for marking was very welcome.

Memo discussion meetings
Areas of concern: The time between the examination dates and the memo discussions was generally far too short to allow pre-marking to take place. This was reported in several subjects, and seriously compromised the validity of the process, as meaningful discussion depends on the pre-marking of scripts. Some provinces sent only one representative or none at all to the memo discussions.

Centralised & on-site marking verification
Areas of good practice: Many external moderators expressed the opinion that the accuracy of marking had improved slightly. External moderators unanimous in appreciation of the impact that thorough training at the memo discussion meetings had on the quality of marking.

Centralised & on-site marking verification
Areas of concern: Markers still experience problems with regard to interpreting answers to open-ended and higher order questions Use of rubrics continues to be a serious concern e.g. the inappropriateness of the rubrics used for P3 of HL and FAL: the descriptors do not facilitate good marking. There are still markers marking literature questions, who do not have a thorough knowledge of the stories/dramas/novels/poems they are marking.

Areas of good practice: The DBE exam was generally administered in line with policy. No major concerns were reported. One can see growth in the administration and conduct of the exams by PDEs. No of exam centres No of exam centers monitored by Umalusi No of Umalusi monitors per province 6699 159 36

Examination Irregularities
The majority of irregularities were of a technical nature and these were reported to Umalusi according to the established channels. Some irregularities were as a result of registration- related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects. Umalusi represented on NEIC

WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION PROCESS

Why Umalusi standardises results, and how
Provision of GENFETQA – Council may adjust raw marks. International practice – large scale assessment systems Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance. Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

Objectives for Standardisation
To ensure that a cohort of learners is not advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude. To achieve comparability and consistency from one year to the next.

Why Umalusi standardises results & how
Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably Process of standardisation Moderation of question papers Review of learner performance against historical performance of candidates in each subject. Historical average (norm) constructed using past 3 to 5 years data. Pairs analyses provides further comparisons of raw means Statistical moderation of Internal assessment

Why Umalusi standardises results & how
Qualitative input meetings Reports (Moderator, Chief Marker and Internal Moderator) Umalusi research (maintaining standards & post exam analysis) Responsibility of Assessment Standards Committee Committee of Council Responsible for setting and maintaining assessment standards Observers (SAQA, HESA, Teacher Unions)

Process for Standardisation
Continuation of JMB and SAFCERT model Assessment Standards Committee Qualitative Reports Pre-standardisation and Standardisation meetings Standardisation booklets (data) – subject raw mark distributions (external written component only) of entire cohort. Subjects are standardised individually, in a linear and non-iterative manner

Principles applied in the standardisation of examination marks
In general no adjustment should exceed 10% or the historical average In the case of the individual candidate, the adjustment effected should not exceed 50%of the raw mark obtained by the candidate If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to the limitations

Statistical moderation
Scope of standardisation 2013 59 standardised Raw marks accepted: 38 subjects Moderated upward : 5 subjects Moderated downward : 16 subjects

STANDARDISATION DECISIONS
WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2013

SUBJECT ADJUSTMENT Life Sciences ½ CA = 0 at 0, scaled to -6 at 87, block -6 up to 162, scaled to 0 at 281, scaled to -3 at 300 Physical Sciences Raw Mathematics 0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300 Mathematical Literacy 0 at 0, raw to 180, scale 0 to +6 from 180 to 234, scale to 0 at 300 Mathematics: Probability; Data Handling ½ CA = 0 at 0 scaled to -15 at 57, block -15 up to 260, scaled to 0 at 300

SUBJECT ADJUSTMENT Geography 0 at 0, scale to -12 at 102, block -12 from 102 to 268, scale to 0 at 300 History Raw Religion Studies Life Orientation Hospitality Studies 0 at 0, scale to -6 at 96, block -6 from 96 to 257, scale to 0 at 300 Tourism ½ CA = 0 at 0 scaled to -6 at 47 scaled to 0 at 119, scaled to +11 at 241, scaled to 0 at 300 Consumer Studies

SUBJECT ADJUSTMENT Agricultural Science Raw Agricultural Management Practices Agricultural Technology Music Dance Studies Design Dramatic Arts

SUBJECT ADJUSTMENT Visual Arts Raw Information Technology Computer Applications Technology ½ CA = 0 at 0 scaled to -12 at 50, scaled to 0 at 269, raw up to 300 Civil Technology ½ CA = 0 at 0 scaled to -14 at 129, scaled to -7 at 300 Electrical Technology

SUBJECT ADJUSTMENT Mechanical Technology 0 at 0, scale to -6 at 126, block -6 from 126 to 246, scale to 0 at 300 Engineering Graphics and Design Raw Accounting Business Studies 0 at 0, scale to -6 at 96, block -6 from 96 to 246, scale to 0 at 300 Economics

SUBJECT ADJUSTMENT Afrikaans HL block -7 English HL Raw IsiNdebele HL CA = 0 at 0 scaled to -13 at 90, scaled to up to 163, scaled to +16 at 218, scaled to 0 at 300 IsiZulu HL Setswana HL Block -6

SUBJECT ADJUSTMENT Siswati HL ½ CA = 0 at 0 scaled to -10 at 93, scaled to 0 at 154,, scaled to +9 at 258 , scaled to 0 at 300 IsiXhosa HL Raw Xitsonga HL 0 at 0, scale to -12 at 73, block of -12 from 73 to 232, scale to 0 at 300. Tshivenda HL 0 at 0, scale to -10 at 30, block of -10 from 30 to 245, scale to 0 at 300. Sepedi HL ½ CA = 0 at 0 scaled to -13 at 98, block -13 up to 159, scaled to +5 at 276, scaled to 0 at 300

SUBJECT ADJUSTMENT Sesotho HL 0 at 0, scale to -6 at 126, block of -6 from 126 to 233, scale to 0 at 300. Afrikaans FAL Raw English FAL IsiNdebele FAL IsiZulu FAL Setswana FAL CA = 0 at 0 scaled to -30 at 60, block -30 up to 222. scaled to -10 at 259, scaled to -30 at 300 Siswati FAL IsiXhosa FAL Xitsonga FAL

SUBJECT ADJUSTMENT Tshivenda FAL Raw Sepedi FAL ½ CA = 0 at 0 scaled to -15 at 99, scaled to -5 at 142, block -5 up to 269, scaled to -15 at 300 Sesotho FAL Afrikaans SAL English SAL IsiNdebele SAL IsiZulu SAL

SUBJECT ADJUSTMENT Siswati SAL Raw IsiXhosa SAL Sepedi SAL Sesotho SAL

Quality Assurance of the IEB 2013 National Senior Certificate Examination
Emmanuel Sibanda – Acting Sen. Manager : Quality Assurance of Assessments

Areas of Good Practice : Percentage of question papers and memoranda approved after first and second moderation ( Nov % Mar 2014 – 100%)

Term 4 moderation- focus on learner evidence and teacher files. IEB Subjects Mathematics, Maths Lit, Life Sciences, Geography, Physical Sciences, Accounting, English HL, History, Life Orientation, Business Studies, Economics, Afrikaans FAL,

Findings: Areas of Good Practice IEB has very good systems and processes in place for SBA implementation and support of their educators. This is borne out by the generally good quality of tasks set by the IEB educators. Findings: Areas of Concern The main challenge is how rubrics are developed: there is some element of vagueness and subjectivity – this leads to differences in interpretation by educators, resulting in differences in allocation of marks Moderation at school level can also be improved

Monitoring of the writing phase Scope:
No of exam centres No of exam centers monitored by Umalusi No of Umalusi deployed for IEB monitoring 187 22 09

Findings: Management of the examinations: Generally examinations conducted in line with policy There is good and ongoing communication between the IEB and their chief invigilators who control the writing phase competently and professionally Isolated instances of non-compliance (candidate identification) Monitoring of centres occurs but it should be done in a more visible onsite and regular basis

Monitoring of the marking phase Scope:
No of marking centres No of marking centers monitored by Umalusi No of Umalusi deployed for IEB monitoring 03 01

Focus: Memo discussion and on-site marking verification Scope: Accounting, Business Studies, Economics, English HL, English FAL, Afrikaans FAL, History, Geography, Life Sciences, Mathematical Literacy, Mathematics, and Physical Sciences

Findings: The IEB conducts memo discussions in the presence of sub-examiners/Senior markers who are responsible for groups of markers during marking. The sub- examiners/Senior markers then train markers thoroughly on day 2 Additions made to the final memo are ratified by the internal moderator and chief examiner. Commendable – IEB is a small system

Verification of marking (cont)
Findings: Marking was found to be fair, valid and reliable: attributed to the continuous feedback sessions between the chief examiner, senior sub-examiners, and sub-examiners The practice of double marking and therefore verification is highly commendable Attention should be paid to instances where it was indicated that moderation doesn’t always involve full moderation of a script but sampled questions

Statistical moderation
Scope of standardisation 2013: 62 subjects standardised Raw marks accepted: 47 subjects Moderated upward : 2 subjects Moderated downward : 13 subjects