2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA COMPUTER-ASSISTED TESTING PETYA GEORGIEVA BULGARIAN STANAG 6001 TESTING TEAM HEAD 2 – 6 September 2013 SHAPE (Mons) Belgium

 till 1 March, 2012 – approbation of the CAT “writing” module;  till 1 July, 2012 – analysis of the results;  since exam session Autumn 2012 – optional CAT “writing” module;  March, 2013 – approbation of CAT modules for “reading” and “listening”;  till 1 July, 2013 – analysis of the results;  since exam session Autumn 2013 – optional CAT “writing”, “reading”, and “listening” modules;  since exam session Autumn 2014 – all candidates are to be tested on computers for the modules “writing”, “reading”, and “listening”(Action Plan 1 iaw Ministerial Order № 857/ ).  since exam session Autumn 2013 – parallel pen-and-paper and CAT in the three skills.  As of Spring End of pen-and-paper STANAG 6001 tests (Action Plan 2 iaw Ministerial Order № 219/ ). 2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva * * BG CAT Background GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA Administrative Module main menus:  Lists  Protocols  Tasks  Tests  Evaluation and Analysis

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva Candidates’ module:  Reading Test  Listening Test  Writing Test GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva  follows the current Specifications for STANAG 6001 exam for the Bulgarian Armed Forces;  starts with items for L1, then L2, then L3 ;  a minimum of 12 and a maximum of 18 questions per level;  level is given for a minimum of 70% correct answers;  a candidate can terminate the exam when a level is achieved or decide to go up;  50%-69% correct answers of the next level → (+) level. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva Paper-based withdrawals % Failed – Reading (1.5%) Listening (2.5%) Writing (1%) CAT withdrawals – 17.16% Failed – Reading (2.56%) Listening (16.21%) Writing (0%) GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva CAT – 33% of candidates improved their results (R, L, or both). Reading – 30.76% showed lower results; 33.33% confirmed the results; 35.91% scored higher. Listening – 27.92% showed lower results; 39.63% confirmed the results; 32.45% scored higher. Data refers to March 2013 CAT piloting session (R and L). GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva PROS:  candidates can manage their time;  decreased candidates’ frustration;  time saving (for both candidates and testers);  anonymity criteria unconditionally complied with;  immediate feedback;  highly objective feedback;  fewer items are compromised;  new items can be piloted and statistics is facilitated. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva Cons:  interface problems for the test-takers (computer screen clarity, font, size, color of the letters)  due to test-takers’ characteristics (computer literacy, motor skills needed to respond to test items, danger of bad time-management);  no option for test administrators to troubleshoot during the test process. GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva RECOMMENDATIONS FUTURE ACTION PLANS (based on candidates retrospective written and oral feedback and test administrators’ impressions); Several groups:  technical (software perfection, interface layout);  methodological (revision of test components and test administration)  facility use and administrative support GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva  Instructions – reviewed;  Layout of the reading material – black font; screen allowing for parallel viewing of the text and answer sheet;  Increase time allotted for looking through the options for L1 and L2 candidates at the Listening;  Reducing number of items if more than 3 per text (R and L). GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA

2 – 6 September 2013 SHAPE (Mons) Belgium Petya Georgieva ? ? ? ? ? ? GEORGI RAKOVSKI NATIONAL DEFENSE ACADEMY - SOFIA BULGARIA