Francesco Gratton 2013 Testing in the time of crisis BILC PROFESSIONAL SEMINAR Stockholm, October , 2013 INNOVATIVE TEST DESIGNS AND FORMATS Lt.Col. F. Gratton

Francesco Gratton 2013 Summary: Past situation (up to Sept 2013) New Course Of Actions adopted A do-it-all software Proposals course of action course of action

Francesco Gratton :Multilevel test (level 1 to 4) 2:Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4:Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correct answers multiplied for a coefficient 7: Potential use of “F” factor RECEPTIVE SKILLS (Listening & Reading) Stanag Proficiency Test 1.0

Francesco Gratton 2013 Stanag Proficiency Test How STANAG levels were awarded (Stanag Proficiency Test 1.0) # of correct answers fixed coefficient (1,66) Multiplied by RECEPTIVE SKILLS

Francesco Gratton 2013

Francesco Gratton :Functional language assessed in a global manner 2: Structured interview 3:Tailored to the candidate 4: Checks & probes 5: 1 to 2 role plays SPEAKING (holistically assessed ) PRODUCTIVE SKILLS Proficiency Test 1.0

Francesco Gratton 2013 Three tasks (one for each level) PRODUCTIVE SKILLS JFLT 1.0 WRITING (holistically assessed )

Francesco Gratton 2013 Summary: Past situation Past situation New COAs adopted New COAs adopted A do-it-all software A do-it-all software Proposals Proposals

Francesco Gratton 2013 Specifications Specifications Cut-off score Cut-off score Joint Database Joint Database New COAs adopted

Francesco Gratton 2013

Francesco Gratton 2013 PURPOSE PURPOSE ADMINISTRATION PROCEDURES ADMINISTRATION PROCEDURES VALIDATION PROCEDURES VALIDATION PROCEDURES TEST FORMAT TEST FORMAT LEVELS OF LINGUISTIC KNOWLEDGE LEVELS OF LINGUISTIC KNOWLEDGE TEST CONTENT TEST CONTENT

Francesco Gratton :Multilevel test (level 1 to 4) 2:Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4:Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correct answers multiplied by a coefficient 7: Potential use of “F” factor RECEPTIVE SKILLS (Listening & Reading) Stanag Proficiency Test 1.0 TWAS HIGH TIME!!!

Francesco Gratton 2013 Stanag Proficiency Test 2.0 New Key-factors Each section is a mini-test (L & R) Plus levels Percentages Each section is a mini-test (L & R) Plus levels Percentages

Francesco Gratton 2013 Specifications Specifications Cut-off score Cut-off score Joint Database Joint Database New COAs adopted

Francesco Gratton 2013 Section 1 –Stanag level 1 (questions from 1 to 15) No.Correct answ. Level awarded Section 2 –Stanag level 2 (questions from 16 to 30) No.Correct answ. Level awarded Section 3 –Stanag level 3 (questions from 31 to 45) No.Correct answ. Level awarded Section 4 –Stanag level 4 (questions from 46 to 60) No.Correct answ. Level awarded JFLT 2.0: RDS (Listening & Reading)

Francesco Gratton 2013 EXAMPLE Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions) Max score: 45 Level 1 (15 questions) Level 2 (15 questions) Level 3 (15 questions) Max score: 45 All candidates answer correctly to 30 questions Old test Old test WithWith they would all get the same score same score

Francesco Gratton 2013 CANDIDATO CORRECT ANSWERS LEVEL 1 (15 Questions ) LEVEL 2 (15 Questions ) LEVEL 3 (15 Questions ) FINAL LEVEL BIANCHI ROSSI VERDI GIALLI ARANCIONI

Francesco Gratton 2013 Specifications Specifications Cut-off score Cut-off score Joint Database Joint Database New COAs adopted

Francesco Gratton 2013 NEW LISTENING & READING ITEMS JDB (JOINT DATA BASE) JOINT EFFORT NOT TO HOW NOT TO MAKE JOINT EFFORTS

Francesco Gratton 2013 Test-writers involved Accustomed to military environment Accustomed to military environment Language Testing Seminar Language Testing Seminar Qualified Qualified Norming Sessions Norming Sessions

Francesco Gratton 2013 THE JDB FLOW CARABINIERI ARMY AIR FORCE NAVY

Francesco Gratton 2013 SEPOCTNOVDECJANFEB MAR APR MAJ JUNJUL st Group PREPARATION OF FIRST BATCH OF ITEMS SENT TO OTHER GROUP Phase 1 TESTERS MEET FOR 1° REVISION Phase 2 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) Phase 3 FINAL MODIFICATION APPROVAZIONE Phase 4 AUG WGGdL 2nd Group FEBRUARY NEW ITEMS. INTO JDB PREPARATION OF FIRST BATCH OF ITEMS SENT TO OTHER GROUP Phase 1 TESTERS MEET FOR 1° REVISION Phase 2 Phase 3 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) Phase 4 FINAL MODIFICATION APPROVAZIONE PREPARATION OF FIRST BATCH OF ITEMS SENT TO OTHER GROUP Phase 1 TESTERS MEET FOR 1° REVISION Phase 2 Phase 3 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) 3rd Group 4th Group REPERIMENTO PREPARAZIONE # 120 TTIVAZIONI INVIO AD ALTRO GRUPPO Phase 1 MAJ NEW ITEMS. INTO JDB NOVEMBER NEW ITEMS. INTO JDB TIMINGS AUGUST NEW ITEMS. INTO JDB

Francesco Gratton 2013 Summary: Past situation Past situation COA (specs, JDB, cut-off score) COA (specs, JDB, cut-off score) A do-it-all software A do-it-all software Proposals Proposals

Francesco Gratton 2013 WHAT’S THE DIFFERENCE ?

Francesco Gratton 2013 PC-assessment-related terminology TermDefinition Stakes Assessment Any systematic method of obtaining evidence (through questions) for a purpose. Quiz … measures for the purpose of providing feedback to the student. Low Survey … to determine needs required to fulfill a defined purpose. Low Test … measures knowledge for the purpose of informing the student on their current level Medium Exam … measures knowledge for the purpose of documenting the current level of knowledge High

Francesco Gratton 2013 the software is used for: Needs analysis:(surveys ) Placement test Any training activity Assessment: – First level survey – Post-course – Pre-certification – Certification

Francesco Gratton 2013 Create questions And organize them in tests using a windows based PC

Francesco Gratton 2013 Assessment …… via Browser Assessment Definitions Questions … allows to choose: Time limits Feedback to test-taker Styles (Template) Jumps Question shuffling Instructions to test- takers … allows to choose: Time limits Feedback to test-taker Styles (Template) Jumps Question shuffling Instructions to test- takers On Windows PC

Francesco Gratton 2013 Assessments … also created with authoring manager by selecting Qs previously created Any question can be chosen from the database … also created with authoring manager by selecting Qs previously created Any question can be chosen from the database Assessment Definitions Questions su PC Windows  via Browser

Francesco Gratton 2013 Create questions And organize them in tests using a windows based PC Set security parameters, schedule assessment and link to other (Learning Management Systems assessment published using any browser, secure browsers, or a PC/MAC Result reports, CIA, graphs, gimmicks, you name it …..

Francesco Gratton 2013 Types of Questions: Multiple Choice

Francesco Gratton 2013 Likert Scale (for questionnaires)

Francesco Gratton 2013 Essay Question 1.Candidate can write free text in the space provided 2.Testers will evaluate later 1.Candidate can write free text in the space provided 2.Testers will evaluate later

Francesco Gratton 2013

Francesco Gratton 2013 Summary:sing Past situation Past situation COA (specs, JDB, cut-off score) COA (specs, JDB, cut-off score) A do-it-all software A do-it-all software Proposals Proposals

Francesco Gratton 2013 Wide project Wide project Sharing experience & capabilities Sharing experience & capabilities Optimization of resources Optimization of resources No alternative to B.A.T. No alternative to B.A.T. Testing in the time of crisis

Francesco Gratton 2013

Francesco Gratton 2013 A Bilateral-based CDB A multilateral-based CDB COMBINED DATABASE or …

Francesco Gratton 2013

Francesco Gratton 2013 FLOW TIME SCHEDULE SPECS 12 3

Francesco Gratton 2013

Francesco Gratton 2013 Thank you