Using the RUMM2030 outputs as feedback on learner performance in Communication in English for Adult learners Nthabeleng Lepota 13th SAAEA Conference 19-22 May 2019

Outline Background Aim of the study Methodology Findings Samples and variables Analysis Findings Recommendations 2

Background In terms of the General Further Education and Training Quality Assurance(GENFETQA) Act of 2008 (as amended), Umalusi is mandated amongst others to: Quality assure exit point assessments for qualifications in schools, Technical Vocational Education and Training (TVET) Colleges and for Adult centers. Although, there are intervention programs in place, they are not focused, targeted or evidence based. -UMALUSI Has a legislation mandate, laid out in the GENFETQA Act -One of the responsibilities is to quality assure exit point assessments -The focus will be on Adult sector -General Further Education and Training Quality Assurance Act

Background… There are challenges observed in performance in the Adult sector. To acquire the NQF Level 1 qualification, adult learners have to complete internal assessment (School Based Assessment) and the written component (Examination). Each of these components contributes 50% towards the final mark. Umalusi conducts quality assurance of assessment processes for the qualification. -QA processes are conducted in this qualification: That includes -Moderation of Question papers -Moderation of SBA -The conduct of the examination is monitored -Verification of marking

Aim of this study To analyze performance in Communication in English for Adult learners. To identify items where learners performed poorly. To identify struggling learners in the subject. -This study intends to achieve the following:

Methodology: Sample and variables The sample of 270 Adult learners scripts sourced from the assessment body. The June 2017 examination consisted of 37 items recorded at item level. The examination made up of open-ended questions. 6

Methodology: Sample and variables Industry/Occupation Females Males Total % Cohort Agriculture 9 5 14 5.19% Construction 1 4 1.85% Education Training & Development 31 24 55 20.37% Fibre, Process & Manufacturing 0.37% Local Government 3 7 2.59% Mining 57 82 139 51.48% Not Indicated 2 17 19 7.04% Wholesale & Retail 13 30 11.11% 117 153 270   -Profile of candidates -Mostly males from the mining sector -Age varied from 21-70+ 7

Methodology: Analysis The RUMM2030 software was used to produce all statistical outputs. Provide estimates of item and person parameters, and fit statistics. Data fitted to the Partial Credit Model. Rasch measurement model for polytomous items with ordered scores. Assumptions: Responses with more than one possible categories i.e. Partially correct responses. -Including dichotomous, rating scale and partial credit models. -Unlike in Multiple Choice questions where a candidate can either score zero or 1 (completely wrong or completely right). -The model is chosen because of responses with more than two possible categories. -The are more possibilities than scoring correct or wrong eg. Since they are open ended questions, a candidate may receive partial credits. 8

Methodology: Analysis Metric Parameter June 2017 Location index δ< -0.5 -0.5<=δ <=0.5 δ> 0.5 E (10), M (14), D (13) Goodness of fit -2.5 to 2.5 Item 20, item 31 and 37 -Criteria used -Location index used to measure the learner ability in a question and item difficulty -Also the fitness of the data to the Partial Credit model

Findings: Item-Person Map -The distribution of learners performance on the left, relative to item difficulty on the right side. -Location scale reads from bottom to top.. -Item 27 the easiest item at the bottom. (93% of the candidates scored correctly) -Item 29, 05 and 23 most difficult item. More than 80% scored incorrectly in the item.(they were accessible) 10

Findings: Item-Person Map All items were attempted by learners. Item 27 the easiest in the examination. No item was noted as not accessible to learners. Even the most difficult items in the paper: Item 29, 5 and 23 were accessible to some candidates. The majority of the items and learners were sitting in the middle of the location scale, between -2 and 2. 11

Findings: Mark distribution Item No 1 2 3 4 I0023 233 37   I0024 160 61 49 I0025 86 184 I0026 33 237 I0027 18 252 I0028 163 107 I0029 227 43 I0030 114 119 I0031 34 11 218 I0032 35 235 I0033 I0034 20 71 179 I0035 51 62 76 30 I0037 140 130 -This distribution shows how the candidates performed in each category per item -Majority of shell items in the paper worth one mark -Item 23 (86%) What does this say about this item? -Item 29 (84%) -Item 27 (93%) -Item 31 (80%) -With this it can be identified the items that were tough or demanding to the candidates. 12

Findings: Mark distribution Four items (23,5,29 and 22) identified with more than 80% of the cohort scoring zero in the items. Such items can be investigated to identify reasons for poor performance. Topics aligned to such items may need to be strengthened and additional support provided to learners. Excellent performance noted in five items (32,31,26,33 and 27). More than 80% scoring full marks in these items. 13

Findings: Person Fit Statistics Max Score ID Location Fit Residual 50 3 118 -2.985 -0.441 4 309 -2.641 0.284 5 263 -2.354 1.343 15 154 -0.888 1.316 20 119 -0.511 -1.875 28 205 0.171 -1.211 33 114 0.646 0.44 43 248 1.903 -0.282 280 -0.758 46 340 2.543 -0.429 Metric Parameter Location index δ< -0.5 -0.5<=δ <=0.5 δ> 0.5 Fit Residual -2.5 to 2.5 -all candidates labelled with unique ID (column 3) -First column, maximum score. -Second column, expected score/score obtained by the learner -Location: describes the difficulty parameter on the latent trait scale -High negative location indicates a person of lower ability and higher positive location indicates persons of a higher ability. -Sorts the candidate Locations in order of their performance and therefore their average ability from lowest to highest. -The mean location parameter is at zero. -Fit residual or goodness of fit measure tells us how well the data fits to the model. Acceptable range between -2.5 and 2.5. 14

Findings: Person Fit Statistics Lowest performing learners with locations < -0.5 is 25%. Average performing learners with locations between -0.5 and 0.5 is 53%. Learners performing above average with locations above 0.5 is 22%. Therefore majority of learners sitting in the middle of the location scale in this examination. Use the learner unique ID to identify candidates with performance below average. -Support should be given to these candidates where they struggle 15

Conclusion The Partial Credit Model is valuable in identifying challenging areas and struggling candidates in the examination. Therefore, the Assessment Bodies can look into using this approach to improve the results in the Adult sector. The candidates unique ID field should be included in the mark sheets and registration database in order to link the two datasets. -Identify where support can be offered and teaching and learning strengthened -Link the datasets in order to identify the sector candidates are from. 16

THANK YOU!