1. Francesco Gratton 2013 Testing in the time of crisis BILC PROFESSIONAL SEMINAR Stockholm, October 14 - 17, 2013 INNOVATIVE TEST DESIGNS AND FORMATS Lt.Col. F. Gratton

2. Francesco Gratton 2013 Summary: Past situation (up to Sept 2013) New Course Of Actions adopted A do-it-all software Proposals course of action course of action

3. Francesco Gratton 2013 1:Multilevel test (level 1 to 4) 2:Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4:Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correct answers multiplied for a coefficient 7: Potential use of “F” factor RECEPTIVE SKILLS (Listening & Reading) Stanag Proficiency Test 1.0

4. Francesco Gratton 2013 Stanag Proficiency Test How STANAG levels were awarded (Stanag Proficiency Test 1.0) # of correct answers fixed coefficient (1,66) Multiplied by RECEPTIVE SKILLS

5. Francesco Gratton 2013

6. Francesco Gratton 2013 1:Functional language assessed in a global manner 2: Structured interview 3:Tailored to the candidate 4: Checks & probes 5: 1 to 2 role plays SPEAKING (holistically assessed ) PRODUCTIVE SKILLS Proficiency Test 1.0

7. Francesco Gratton 2013 Three tasks (one for each level) PRODUCTIVE SKILLS JFLT 1.0 WRITING (holistically assessed )

8. Francesco Gratton 2013 Summary: Past situation Past situation New COAs adopted New COAs adopted A do-it-all software A do-it-all software Proposals Proposals

9. Francesco Gratton 2013 Specifications Specifications Cut-off score Cut-off score Joint Database Joint Database New COAs adopted

10. Francesco Gratton 2013

11. Francesco Gratton 2013 PURPOSE PURPOSE ADMINISTRATION PROCEDURES ADMINISTRATION PROCEDURES VALIDATION PROCEDURES VALIDATION PROCEDURES TEST FORMAT TEST FORMAT LEVELS OF LINGUISTIC KNOWLEDGE LEVELS OF LINGUISTIC KNOWLEDGE TEST CONTENT TEST CONTENT

12. Francesco Gratton 2013 1:Multilevel test (level 1 to 4) 2:Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4:Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correct answers multiplied by a coefficient 7: Potential use of “F” factor RECEPTIVE SKILLS (Listening & Reading) Stanag Proficiency Test 1.0 TWAS HIGH TIME!!!

13. Francesco Gratton 2013 Stanag Proficiency Test 2.0 New Key-factors Each section is a mini-test (L & R) Plus levels Percentages Each section is a mini-test (L & R) Plus levels Percentages

14. Francesco Gratton 2013 Specifications Specifications Cut-off score Cut-off score Joint Database Joint Database New COAs adopted

15. Francesco Gratton 2013 Section 1 –Stanag level 1 (questions from 1 to 15) No.Correct answ. Level awarded 123456 789 101112131415 0 0+1 Section 2 –Stanag level 2 (questions from 16 to 30) No.Correct answ. Level awarded 123456 789101112131415 1 1+2 Section 3 –Stanag level 3 (questions from 31 to 45) No.Correct answ. Level awarded 123456 789 101112131415 2 2+3 Section 4 –Stanag level 4 (questions from 46 to 60) No.Correct answ. Level awarded 123456789 101112131415 3 3+4 JFLT 2.0: RDS (Listening & Reading)

16. Francesco Gratton 2013 EXAMPLE Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions) Max score: 45 Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions) Max score: 45 All candidates answer correctly to 30 questions Old test Old test WithWith they would all get the same score same score

17. Francesco Gratton 2013 CANDIDATO CORRECT ANSWERS LEVEL 1 (15 Questions ) LEVEL 2 (15 Questions ) LEVEL 3 (15 Questions ) FINAL LEVEL BIANCHI 30 10 3 ROSSI 30 11109 2+ VERDI 30 13125 2 GIALLI 30 1587 1+ ARANCIONI 30 1569 1

18. Francesco Gratton 2013 Specifications Specifications Cut-off score Cut-off score Joint Database Joint Database New COAs adopted

19. Francesco Gratton 2013 NEW LISTENING & READING ITEMS JDB (JOINT DATA BASE) JOINT EFFORT NOT TO HOW NOT TO MAKE JOINT EFFORTS

20. Francesco Gratton 2013 Test-writers involved Accustomed to military environment Accustomed to military environment Language Testing Seminar Language Testing Seminar Qualified Qualified Norming Sessions Norming Sessions

21. Francesco Gratton 2013 THE JDB FLOW CARABINIERI ARMY AIR FORCE NAVY

22. Francesco Gratton 2013 SEPOCTNOVDECJANFEB MAR APR MAJ JUNJUL 20122013 1st Group PREPARATION OF FIRST BATCH OF ITEMS SENT TO OTHER GROUP Phase 1 TESTERS MEET FOR 1° REVISION Phase 2 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) Phase 3 FINAL MODIFICATION APPROVAZIONE Phase 4 AUG WGGdL 2nd Group FEBRUARY 2013 20 NEW ITEMS. INTO JDB PREPARATION OF FIRST BATCH OF ITEMS SENT TO OTHER GROUP Phase 1 TESTERS MEET FOR 1° REVISION Phase 2 Phase 3 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) Phase 4 FINAL MODIFICATION APPROVAZIONE PREPARATION OF FIRST BATCH OF ITEMS SENT TO OTHER GROUP Phase 1 TESTERS MEET FOR 1° REVISION Phase 2 Phase 3 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) 3rd Group 4th Group REPERIMENTO PREPARAZIONE # 120 TTIVAZIONI INVIO AD ALTRO GRUPPO Phase 1 MAJ 2013 40 NEW ITEMS. INTO JDB NOVEMBER 2013 80 NEW ITEMS. INTO JDB TIMINGS AUGUST 2013 60 NEW ITEMS. INTO JDB

23. Francesco Gratton 2013 Summary: Past situation Past situation COA (specs, JDB, cut-off score) COA (specs, JDB, cut-off score) A do-it-all software A do-it-all software Proposals Proposals

24. Francesco Gratton 2013 WHAT’S THE DIFFERENCE ?

25. Francesco Gratton 2013 PC-assessment-related terminology TermDefinition Stakes Assessment Any systematic method of obtaining evidence (through questions) for a purpose. Quiz … measures for the purpose of providing feedback to the student. Low Survey … to determine needs required to fulfill a defined purpose. Low Test … measures knowledge for the purpose of informing the student on their current level Medium Exam … measures knowledge for the purpose of documenting the current level of knowledge High

26. Francesco Gratton 2013 the software is used for: Needs analysis:(surveys ) Placement test Any training activity Assessment: – First level survey – Post-course – Pre-certification – Certification

27. Francesco Gratton 2013 Create questions And organize them in tests using a windows based PC

28. Francesco Gratton 2013 Assessment …… via Browser Assessment Definitions Questions … allows to choose: Time limits Feedback to test-taker Styles (Template) Jumps Question shuffling Instructions to test- takers … allows to choose: Time limits Feedback to test-taker Styles (Template) Jumps Question shuffling Instructions to test- takers On Windows PC

29. Francesco Gratton 2013 Assessments … also created with authoring manager by selecting Qs previously created Any question can be chosen from the database … also created with authoring manager by selecting Qs previously created Any question can be chosen from the database Assessment Definitions Questions su PC Windows  via Browser

30. Francesco Gratton 2013 Create questions And organize them in tests using a windows based PC Set security parameters, schedule assessment and link to other (Learning Management Systems assessment published using any browser, secure browsers, or a PC/MAC Result reports, CIA, graphs, gimmicks, you name it …..

31. Francesco Gratton 2013 Types of Questions: Multiple Choice

32. Francesco Gratton 2013 Likert Scale (for questionnaires)

33. Francesco Gratton 2013 Essay Question 1.Candidate can write free text in the space provided 2.Testers will evaluate later 1.Candidate can write free text in the space provided 2.Testers will evaluate later

34. Francesco Gratton 2013

35. Francesco Gratton 2013 Summary:sing Past situation Past situation COA (specs, JDB, cut-off score) COA (specs, JDB, cut-off score) A do-it-all software A do-it-all software Proposals Proposals

36. Francesco Gratton 2013 Wide project Wide project Sharing experience & capabilities Sharing experience & capabilities Optimization of resources Optimization of resources No alternative to B.A.T. No alternative to B.A.T. Testing in the time of crisis

37. Francesco Gratton 2013

38. Francesco Gratton 2013 A Bilateral-based CDB A multilateral-based CDB COMBINED DATABASE or …

39. Francesco Gratton 2013

40. Francesco Gratton 2013 FLOW TIME SCHEDULE SPECS 12 3

41. Francesco Gratton 2013

42. Francesco Gratton 2013 Thank you francesco.gratton@gmail.comcanutesting@sclingue.esercito.difesa.it