1. THE FEASIBILITY OF APPLYING STATISTICAL MODERATION IN PRACTICAL COMPONENTS USING EXTERNAL EXAMINATIONS AS THE ANCHOR

2. Background Moderation only applied to practical subjects
Their syllabuses contained practical objectives
Practical subjects differ in nature, e.g. HE, Agriculture, Music, PE, D&T, Art, hence different modes of assessment
Different format of moderation depending on the subject
Advent of Outcome-Based Assessment requires SBA in all subjects
A construct is some postulated attribute of people, assumed to be reflected in test performance. In test validation the arguments
Meehl and Cronbach 1953 construct validity subsumed all the other form of validity
Validity can only be measured through test scores
Classical an individual’s observed test score X is a chance variable with some unknown frequency distribution
the errors of measurement are unbiased and hence the expectation of E is O.
true score and error are uncorrelated in any group, ρET=0
Assumptions are very abstract and cannot be proven practically.
Tautologies
Test parameters are relative –variant
IRT ICC, unidiemnsionality, local indipendnece speededness

3. Background Cont’d
e.g BGCSE D&T, employs statistical moderation techniques in its moderation, without necessarily linking it to External exams (Kempa, 1986)
(y = *Sch Mark).
In some subjects visit schools (JC Agric), in some artifacts centrally marked (BGCSE Agric)
Visiting moderation is expensive, and resources are limited.
Stats moderation is cheap
A construct is some postulated attribute of people, assumed to be reflected in test performance. In test validation the arguments
Meehl and Cronbach 1953 construct validity subsumed all the other form of validity
Validity can only be measured through test scores
Classical an individual’s observed test score X is a chance variable with some unknown frequency distribution
the errors of measurement are unbiased and hence the expectation of E is O.
true score and error are uncorrelated in any group, ρET=0
Assumptions are very abstract and cannot be proven practically.
Tautologies
Test parameters are relative –variant
IRT ICC, unidiemnsionality, local indipendnece speededness

4. STATISTICAL MODERATION
Statistical moderation tries to adjust the level and spread of each school’s assessments of its students in a particular study, to match the level and spread of the same students’ scores on a common external examination.
Thus statistical moderation is applied to two different marks measuring the Same Learning Outcomes.

5. Stats Mod Cont’d
Thus students’ External Score should be a valid predictor of coursework achievement.
The Coursework score must itself be valid
Hence strong correlation between the two!!!!.
A construct is some postulated attribute of people, assumed to be reflected in test performance. In test validation the arguments
Meehl and Cronbach 1953 construct validity subsumed all the other form of validity
Validity can only be measured through test scores
Classical an individual’s observed test score X is a chance variable with some unknown frequency distribution
the errors of measurement are unbiased and hence the expectation of E is O.
true score and error are uncorrelated in any group, ρET=0
Assumptions are very abstract and cannot be proven practically.
Tautologies
Test parameters are relative –variant
IRT ICC, unidiemnsionality, local indipendnece speededness

6. Subjects Components JC Subject 1 PP1: MC – 35% PP2: SASI – 40%
PP3: SBA – 25%
JC Subject 2
PP1: MC – 30%
PP2: Prac Exam – 45%
JC Subject 3
PP1: Listening – 25%
PP2: SASI – 25%
PP3: Prac Exam – 30%
PP4: SBA – 20%
JC Subject 4
PP1: MC – 20%
PP3: SBA – 40%
BGCSE Subject 5
PP1: Pract Exam – 50%
PP2: Pract Exam – 50%
PP4 or 5: SBA– 50%
PP3: Pract Exam – 50%
BGCSE Subject 6
PP1: Written Exam – 50%
PP3: Indiv Study – 30%
PP2: SBA – 20%

7. Methodology
A mixed methodology employing Correlational study design as baseline and Design-Based Research design as developmental
Purposive sampling of Subjects (3 from JC and 2 from BGCSE) and schools.
Teacher coursework scores correlated with all other scores

8. Methodology Cont’d
Agreement (Concordance)/Inter-Rater Reliability between moderator and Teacher scores for coursework scores
National descriptive statistics and Schools descriptive statistics were computed

9. RESULTS Teacher Coursework score and Moderator Coursework score Means.
Level
Subject
Score type
Mean SD
JC Subjects 1
Teacher Mark
40.17
6.40
Sample Moderation Mark
40.22
6.53 2
51.88
19.49
53.59
18.23 3
42.68
13.89
36.52
14.34 4
45.74
17.26
12.91
6.37
BGCSE 5
59.01
17.22
47.55
14.52 6
64.22
14.06
61.57
15.17

10. The relationship between Teacher Coursework score and Moderator Coursework score.
Level
Subject r r2
ICC JC 1
.93
86%
.96 2
.98
96% 3
.97
94%
.95 4
.65
42%
.20
BGCSE 5
.85
72%
.80 6
.91
83%

11. Correlation coefficients between Teacher Coursework score and other components.
Component
JCE
BGCSE 1 2 3 4 6 5
Paper 1
.25
.27
.54
.44
.50
.41
Paper 2
.26
.43
.53
.51 -
.49
Paper 3
.63
.85
Scaled score
.45
.66
JC Subject 2, JC Subject 3, and BGCSE Subject 5 Teacher’s Coursework scores didn’t have the highest correlation with Practical Exam!

12. Shared Variance between SBA and External Score with highest r
Coefficient JC
BGCSE
Subject 1 2 3 4 5 6 r
.27
.45
.66
.52
.41
.49
.85 r2 7%
20%
44%
27%
17%
24%
72%
Paper with highest r
Scaled Score
External Practical Exam
Indiv. study
NB: JC Subjects 2, 3, and BGCSE Subjects 5, and 6 all have a Practical Exam Component. BGCSE Subject 5 has 3 papers to choose 1 from.
Need to investigate such low r2 for All subjects except BGCSE Subject 6!!

13. The Statistical Moderation Process: School X JSS – Subject 1 Descriptive Stats.
Component
N/n
Range
Min
Max
Mean
S D
Sample Moderated Mark 46 31 18 49
36.89
8.93
Teacher C\work mark
154 37 12
36.07
7.84
Scaled Score 6 55
30.69
11.79
Paper 1 25 5
18.21
5.41
Final Moderated Mark 19
18.04
3.92
Paper 2 28
12.50
6.88

14. Standardise the coursework scores using the Scaled Score – JSS School X.
Scaled Score
Scaled Score’s Z-Scores
Teacher Coursework Score
Teacher CourseWork Score Z-Score 27
-0.31 38
0.25 16
-1.25 13
-2.94
-1.50 42
0.76 52
1.81 43
0.88 55
2.06 34
-0.26 32
0.11 33
-0.39 12
-1.58 40
0.50 20
-0.91
-3.07 28
-0.23 41
0.63 49
1.55 51
1.72 47
1.40 45
1.14 53
1.89
1.65
0.79 39
0.70 35
0.37
-0.14 6
-2.09 30
-0.77

15. Standardizing the Teacher Coursework score.
Scaled Score
Scaled Score Z-Scores
Teacher C/W Mark
Teacher C/W Mark Z-Score
Stsd Teacher C/W Score 27
-0.31 38
0.25 16
-1.25 13
-2.94
-1.50 42
0.76 52
1.81 43
0.88 55
2.06 34
-0.26 32
0.11 33
-0.39 12
-1.58 40
0.50 20
-0.91
-3.07 28
-0.23 41
0.63 49
1.55 51
1.72 47
1.40 45
1.14 53
1.89
1.65
0.79 39
0.70 35
0.37
-0.14 6
-2.09 30
-0.77
-0.06

16. Applying the VCE correction factor
Then Subjecting the Teacher coursework score to the model:

17. Final Moderated Ext. Score Ext. Score Z-Scores Teacher Mark
Teacher Mark Z-Score
Final Moderated Mark (orig)
Stsd Teacher C/Work Score y
t_moderated 27
-0.31 38
0.25 19 16
-1.25 13
-2.94
6.5 55
2.06 34
-0.26 17 32
0.11 33
-0.39
16.5 12
-1.58 40
0.50 20
-0.91
-3.07 6 3 28
-0.23 41
0.63
20.5 45
1.14
22.5 53
1.89 49
1.65
24.5
0.79 42
0.76 21 35
0.37
-0.14
17.5
-2.09 30
-0.77 15
-0.06

18. Agric stats moderation outcome.
Domain
Statistics
Schools and Pass Rate 1 2 3 4 5 6 7 8 9 10
Low
High
Average
Raw Teacher Coursework
Min Mark 29 12 36 34 32 30 26 21
Max Mark 50 22 49 47 48
Mean
43.51
13.94
36.07
46.99
44.18
43.57
42.18
43.39
44.81
41.09 SD
3.81
4.76
7.84
1.89
3.32
2.63
3.652
4.074
2.33
5.142
External Score
min 20
max 53 55 65 60 59 62 54
mean
24.19
25.01
30.69
42.29
35.18
34.03
35.96
30.20
30.57
27.54
11.06
10.23
11.79
8.96
11.14
10.27
12.40
12.81
10.80
11.40
Standardised Teacher Coursework Score
-18.00
-4.95
-6.35
-9.83
-11.12
-107
-24.5
-10.20
43.06
42.33
50.36
56.56 52
47.41
69.83
50.98
45.33
25.02
30.54
42.28
35.32
34.02
12.01
11.19
12.33
Stats moderated score
1.63
6.00
16.21
9.59
3.82
7.00
4.15
2.89
53.12
49.01
55.00
65.01
54.32
60.01
54.00
59.00
62.00
24.12
25.00
30.68
34.16
35.95
30.19
30.50
11.71
10.55
11.60
11.08
11.45
15.03
11.93.
13.11
17.14

19. Development of a Statistical Model for BEC
Employed Design-Based Research (DBR) methodology.
DBR develops product with practitioners interactively and iteractively
To ensures practicability and utility

20. Development of the model
Field test
Baseline (RQ 1)
Intervention
Baseline survey
Intervention (R Q 2)
Baseline survey findings
Expert Review
Pilot
Prototype 2
Final Prototyp e 4
Define
Measur e
Analys e
Design
Develop
Implement
Design Specifications
Prototyp e 1
Prototype 3
Try-out

21. System-wide Approach (pathways/OBE)
Developmental work .
System-wide Approach (pathways/OBE)
Moderation Observation and interviews
Need to understand why some subjects teacher coursework and moderator coursework are high and others’ moderate
What is it that the teachers or moderators or students or schools or the system does/doesn’t do

22. Conclusion
Subjects showed high correlation and agreement between Teacher Score and Moderator Score suggesting that Teachers and Moderators applied the scoring guide consistently the same. Thus the teachers scoring could be relied upon.
Correlations between the Teacher score and the External scores was low to moderate for all the subjects
Correlations between Teacher score and Practical test scores was found to be low to moderate

23. Conclusion
Development of the model is a system-wide endeavour hence takes long Interventionist: developing an intervention in a real world setting.
Interactive: Involves active participation of practitioners & professionals in the various stages and activities
Iterative: incoporates cycles of analysis, design & development, evaluation & revision.

24. THANK YOU!!

25. JC Subject 1 Corr Matrix Components.
PP 1
PP 2
Ext. Score
Teacher C/W Mark
Sample Mod Mark
Final ModMark
PP1 1
PP2
.82**
Ext. Score
.94**
.97**
Teacher C/W Mark
.25**
.26**
.27**
Sample Mod Mark
.23**
.24**
.93**
Final Mod Mark
.99**

26. Corr matrix for JC Subject 2 Components
PP 1
PP 2
ExtScore
Teacher C/work Mark
Sample Mod Mark
Final Mod Mark 1
.35**
.73**
.27**
.26**
.28**
.89**
.44**
.42**
.45**
.98**
1.0**
.99**

27. Corr matrix for JC Subject 3 Components
PP 1
PP 2
PP 3
External Score
Teacher Mark
Sample Mod Mark
Final Mod Mark
PP 1 -Listening 1
PP 2 (SASI)
.77**
PP 3 (PRACT Exam)
.57**
.60**
.83**
.87**
.90**
Teacher Mark (SBA)
.54**
.53**
.63**
.66**
.62**
.73**
.75**
.96**
.56**
.67**
.70**
.94**
1.0**

28. Correlation matrix for JC Subject 4
Component
Paper 1
Paper 2
External Score
Teacher C/Work Mark
Sample Mod Mark
Final Mod Mark 1
Paper 2
.769**
.872**
.982**
Teacher C/work Mark
.443 **
.511**
.516**
.447**
.509**
.512**
.647**
.417**
.476**
.482**
.711**
.658**

29. Corr matrix for BGCSE Subject 6 Components
PP1
PP 2
PP3
External Score
Teacher Mark
Sample Mod Mark
Final Mod Mark
PP 1 1
PP2
.46**
.48**
.39**
1**
.50**
.34**
.85**
.35**
.98**

30. Corr matrix for BGCSE Subject 5
Component
PP 1
PP2
PP 3
Final Scaled Mark
Teacher Mark
Sample Mod Mark
Final Mod Mark 1 .a
1.000**
.414**
.495**
.512**
PP 2
.487**
.570**
.537**
.999**
.492**
.573**
.482**
.553**
.559**
.906**
.959**
Sampled Mod Mark
.907**