1. BiH Test Piloting
Mary Jo DI BIASE

2. In the beginning……

3. In the beginning…… Workshops on: item development validation
practice in oral interviews
rating
In the beginning……

4. Bosnia’s Items ready
Italy’s guinea pigs ready

6. Summary Test population background Trialling phases
Statistical procedures
Results
…. and more

7. Test population background
Staff officer’s career
Course specifications
JFLT –correlation

8. Staff officer’s career
ISSMI: selected Army officers
one year duration including language course with final SLP 3333

10. Test population background
Three-month course
Staff officer’s career
34 hours a week
Course specifications
Additional activities
JFLT –concurrent validity
Periodic diagnostic tests

11. Test population background
Staff officer’s career
Piloting held two weeks before JFLT
Both are proficiency tests
Course specifications
Both based on STANAG 6001 ed. 2
JFLT –correlation
Both have similar test types

12. Summary Test population background Trialling phases
Statistical procedures
Results
…. and more

13. Listening & Reading Items in booklet form: total 60
level two and three items
Instructions and proctoring
Questionnaires

14. Writing Speaking 15 minute interview between interviewer and candidate
(observer in background with rating scale)
Writing
Writing scripts for inter rater reliability
Trialling prompts to check for level appropriateness, etc

15. Summary Test population background Trialling phases
Statistical procedures
Results
…. and more

16. Score Distribution Cluster: mean, mode, median
Dispersion: standard deviation, range

17. Item Behavior Classical Item analyses
(Facility Value &Discrimination Index)
Distracter Analysis

18. Reliability Scale if item is deleted
Inter-rater reliability coefficient
(Speaking and Writing)

19. Interpretation of Results
Items in relation to:
Facility Value and Discrimination Index
Distracter behavior (ambiguity, implausible options, etc)
Overall mean of test
Reliability of test if item is deleted

20. Example According to the report…
From an economic report (common knowledge-discarded)
According to the report…
approvals of bank loans have gone up recently.
the UK economy is severely going downhill.
recession started earlier than it was thought.
positive figures are predicted for the next year.

21. C
lassical I
tem A
nalysis
Key B
94% A 0% C 5% D 1% FV
94% DI
0,05

22. Summary Test population background Trialling phases
Statistical procedures
Results
…. and more

23. READING LISTENING Mean 60,87% Median 60,40% Mode 67,79% Range 62
Standard Deviation
13,35%
Skewness
-0,486
Kurtosis
0,043
READING
Mean
77,05%
Median
80,00%
Mode
75,00%
Standard Deviation
13,33%
Range 60
Skewness
-1,150
Kurtosis
1,477

24. Listening
Reading

25. Reliability

26. Speaking Inter-rater reliability between Interviewer (holistic rating)
observer (analytic rating)
trainer in background
Language generated by prompts (wording, level alignment)
Tester conduct and elicitation techniques during interview

27. Correlation Coefficient:
Writing
Inter-rater Reliability
Correlation Coefficient:
0,617
0,517
0,458
0,250

28. WRAPPING UP
Overall beneficial:
positive statistical results
(30% items discarded)
- additional ‘live’ training
- more piloting needed

29. Summary Test population background Trialling phases
Statistical procedures
Results
…. and more

30. First Official Administration
of BiH test:

31. Fotografie
Say…..

33. The BiH Testing Team in Rome

34. Thank you!!!