1. Republic of Bulgaria Ministry of Defense
Piloting STANAG 6001 CAT Reading and Listening
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

2. Content Plan for STANAG 6001 CAT implementation;
STANAG 6001 CAT software;
Evaluation algorithms – computer-assisted and computer-adaptive;
Piloting STANAG 6001 CAT for reading;
Piloting STANAG 6001 CAT for listening;
CAT – pros and cons.
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

3. Plan for STANAG 6001 CAT Implementation
since exam session Autumn 2013 – STANAG 6001 CAT and pen-and-paper tests;
since exam session Spring 2015 only STANAG 6001 CAT.
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

4. Administrative Module main menus :
STANAG 6001 CAT software
Administrative Module main menus :
Lists
Protocols
Tasks
Tests
Evaluation
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

5. 6-9 May 2013
Tbilisi, Georgia
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva
Emilija Nesheva 5

6. 6-9 May 2013
Tbilisi, Georgia
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva
Emilija Nesheva 6

7. 6-9 May 2013
Tbilisi, Georgia
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva
Emilija Nesheva 7

8. STANAG 6001 CAT software (cont.)
Candidates’ module:
Reading Test
Listening Test
Writing Test
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

9. Piloting STANAG 6001 CAT reading
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

10. Piloting STANAG 6001 CAT listening
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

11. Evaluation algorithm computer-assisted test
follows the current Specifications for STANAG 6001 exam for the Bulgarian Armed Forces;
starts with items for L1, then L2, then L3 ;
level is given for min. 70% correct answers;
50%-69% correct answers for the next level → (+) level.
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

12. Evaluation algorithm computer-adaptive test
Each candidate is given 7 items for L2 and depending on the correctness of the answers, the test continues moving further to L3 or L1 and again to L2 if appropriate. The pass rate is 70% per level (50-69% for + levels).
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

13. STANAG 6001 CAT Piloting READING and LISTENING
Methods of validation:
- statistical analyses of the performance of the participants comparing their performance at a recent pen-and-paper test and at the CAT;
- retrospective written feedback by the participants in the piloting.
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

14. Piloting STANAG 6001 CAT Reading Results
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

15. STANAG 6001 CAT Piloting Reading Results
1→1 1 1→
1+→1 2
1+→2 1
2→1 3
2→1+ 4
2→2 5
2→2+ 1
2→3 2
2+→1+ 1
2+→2 2
2+→2+ 2
2+→
3→3 1
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

16. Piloting STANAG 6001 CAT Listening Results
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

17. STANAG 6001 CAT PILOTING LISTENING RESULTS
0→0 9
0→1 1
0→1+ 1
0→2 2
1→0 6
1→1 10
1→1+ 5
1→2 4
1+→0 5
1+→1 6
1+→1+ 13
1+→2 4
1+→2+ 2
2→1 2
2→1+ 5
2→2 3
2→2+ 3
2→3 7
2+→1 2
2+→1+ 3
2+→2 2
2+→2+ 5
2+→3 10
3→3 1
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

18. Participants’ Feedback
Nearly 90% of the participants in the piloting approve the implementation of STANAG 6001 CAT for Reading and Listening.
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

19. CAT – pros and cons Pros:
candidates can manage their time and this makes them calm and confident;
decreased candidates’ frustration
-immediate feedback;
-highly objective feedback;
-less items are compromised;
-testing-time is saved;
-new items can be piloted and the statistical calculations can be made automatically.
Cons:
-due to test method and conditions (e.g.computer screen clarity, motor skills needed to respond to test items, and ability of test administrators to troubleshoot);
-due to test-taker attributes (e.g. computer familiarity, ability to concentrate in front of the computer, danger of bad time-management).
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

20. NB! Lessons Learnt!
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

21. Questions?
6-9 May 2013
Tbilisi, Georgia
Emilija Nesheva

22. May 6th – St. George’s Day Valor Day – Day of Bulgarian Armed Forces
Tbilisi, Georgia
Emilija Nesheva