1. Using the RUMM2030 outputs as feedback on learner performance in Communication in English for Adult learners
Nthabeleng Lepota
13th SAAEA Conference
19-22 May 2019

2. Outline Background Aim of the study Methodology Findings
Samples and variables
Analysis
Findings
Recommendations 2

3. Background
In terms of the General Further Education and Training Quality Assurance(GENFETQA) Act of (as amended), Umalusi is mandated amongst others to:
Quality assure exit point assessments for qualifications in schools, Technical Vocational Education and Training (TVET) Colleges and for Adult centers.
Although, there are intervention programs in place, they are not focused, targeted or evidence based.
-UMALUSI Has a legislation mandate, laid out in the GENFETQA Act
-One of the responsibilities is to quality assure exit point assessments
-The focus will be on Adult sector
-General Further Education and Training Quality Assurance Act

4. Background…
There are challenges observed in performance in the Adult sector.
To acquire the NQF Level 1 qualification, adult learners have to complete internal assessment (School Based Assessment) and the written component (Examination).
Each of these components contributes 50% towards the final mark.
Umalusi conducts quality assurance of assessment processes for the qualification.
-QA processes are conducted in this qualification: That includes
-Moderation of Question papers
-Moderation of SBA
-The conduct of the examination is monitored
-Verification of marking

5. Aim of this study
To analyze performance in Communication in English for Adult learners.
To identify items where learners performed poorly.
To identify struggling learners in the subject.
-This study intends to achieve the following:

6. Methodology: Sample and variables
The sample of 270 Adult learners scripts sourced from the assessment body.
The June 2017 examination consisted of 37 items recorded at item level.
The examination made up of open-ended questions. 6

7. Methodology: Sample and variables
Industry/Occupation
Females
Males
Total
% Cohort
Agriculture 9 5 14
5.19%
Construction 1 4
1.85%
Education Training & Development 31 24 55
20.37%
Fibre, Process & Manufacturing
0.37%
Local Government 3 7
2.59%
Mining 57 82
139
51.48%
Not Indicated 2 17 19
7.04%
Wholesale & Retail 13 30
11.11%
117
153
270
-Profile of candidates
-Mostly males from the mining sector
-Age varied from 7

8. Methodology: Analysis
The RUMM2030 software was used to produce all statistical outputs.
Provide estimates of item and person parameters, and fit statistics.
Data fitted to the Partial Credit Model.
Rasch measurement model for polytomous items with ordered scores.
Assumptions:
Responses with more than one possible categories i.e. Partially correct responses.
-Including dichotomous, rating scale and partial credit models.
-Unlike in Multiple Choice questions where a candidate can either score zero or 1 (completely wrong or completely right).
-The model is chosen because of responses with more than two possible categories.
-The are more possibilities than scoring correct or wrong eg. Since they are open ended questions, a candidate may receive partial credits. 8

9. Methodology: Analysis
Metric
Parameter
June 2017
Location index
δ< -0.5
-0.5<=δ <=0.5
δ> 0.5
E (10), M (14), D (13)
Goodness of fit
-2.5 to 2.5
Item 20, item 31 and 37
-Criteria used
-Location index used to measure the learner ability in a question and item difficulty
-Also the fitness of the data to the Partial Credit model

10. Findings: Item-Person Map
-The distribution of learners performance on the left, relative to item difficulty on the right side.
-Location scale reads from bottom to top..
-Item 27 the easiest item at the bottom. (93% of the candidates scored correctly)
-Item 29, 05 and 23 most difficult item. More than 80% scored incorrectly in the item.(they were accessible) 10

11. Findings: Item-Person Map
All items were attempted by learners.
Item 27 the easiest in the examination.
No item was noted as not accessible to learners.
Even the most difficult items in the paper: Item 29, 5 and 23 were accessible to some candidates.
The majority of the items and learners were sitting in the middle of the location scale, between -2 and 2. 11

12. Findings: Mark distribution
Item No 1 2 3 4
I0023
233 37
I0024
160 61 49
I0025 86
184
I0026 33
237
I0027 18
252
I0028
163
107
I0029
227 43
I0030
114
119
I0031 34 11
218
I0032 35
235
I0033
I0034 20 71
179
I0035 51 62 76 30
I0037
140
130
-This distribution shows how the candidates performed in each category per item
-Majority of shell items in the paper worth one mark
-Item 23 (86%) What does this say about this item?
-Item 29 (84%)
-Item 27 (93%)
-Item 31 (80%)
-With this it can be identified the items that were tough or demanding to the candidates. 12

13. Findings: Mark distribution
Four items (23,5,29 and 22) identified with more than 80% of the cohort scoring zero in the items.
Such items can be investigated to identify reasons for poor performance.
Topics aligned to such items may need to be strengthened and additional support provided to learners.
Excellent performance noted in five items (32,31,26,33 and 27). More than 80% scoring full marks in these items. 13

14. Findings: Person Fit Statistics
Max
Score ID
Location
Fit Residual 50 3
118
-2.985
-0.441 4
309
-2.641
0.284 5
263
-2.354
1.343 15
154
-0.888
1.316 20
119
-0.511
-1.875 28
205
0.171
-1.211 33
114
0.646
0.44 43
248
1.903
-0.282
280
-0.758 46
340
2.543
-0.429
Metric
Parameter
Location index
δ< -0.5
-0.5<=δ <=0.5
δ> 0.5
Fit Residual
-2.5 to 2.5
-all candidates labelled with unique ID (column 3)
-First column, maximum score.
-Second column, expected score/score obtained by the learner
-Location: describes the difficulty parameter on the latent trait scale
-High negative location indicates a person of lower ability and higher positive location indicates persons of a higher ability.
-Sorts the candidate Locations in order of their performance and therefore their average ability from lowest to highest.
-The mean location parameter is at zero.
-Fit residual or goodness of fit measure tells us how well the data fits to the model. Acceptable range between -2.5 and 2.5. 14

15. Findings: Person Fit Statistics
Lowest performing learners with locations < is 25%.
Average performing learners with locations between and 0.5 is 53%.
Learners performing above average with locations above 0.5 is 22%.
Therefore majority of learners sitting in the middle of the location scale in this examination.
Use the learner unique ID to identify candidates with performance below average.
-Support should be given to these candidates where they struggle 15

16. Conclusion
The Partial Credit Model is valuable in identifying challenging areas and struggling candidates in the examination.
Therefore, the Assessment Bodies can look into using this approach to improve the results in the Adult sector.
The candidates unique ID field should be included in the mark sheets and registration database in order to link the two datasets.
-Identify where support can be offered and teaching and learning strengthened
-Link the datasets in order to identify the sector candidates are from. 16

17. THANK YOU!