1. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. ILR Government Skill Level Descriptors for Foreign Language Education & Faculty Training Curriculum Design Assessment Development Assessment Development James Dirgin Washington DC January 2014

2. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. The Goal of the Presentation To familiarize with the ILR (Interagency Language Roundtable) scale and the skill descriptors in general To familiarize with the ILR (Interagency Language Roundtable) scale and the skill descriptors in general To understand the tasks, content, and accuracy (TCA) statements in each level To understand the tasks, content, and accuracy (TCA) statements in each level To understand the transfer of TCA to test items To understand the transfer of TCA to test items To distinguish item difficulty based on the ILR To distinguish item difficulty based on the ILR Have fun… Have fun…

3. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. The ILR (Interagency Language Roundtable) Created in the 60s to serve the USG for defining Created in the 60s to serve the USG for defining foreign language proficiency First used by the diplomatic and intelligence agencies First used by the diplomatic and intelligence agencies currently over 100 members Focuses on what a person can do Focuses on what a person can do by using a “foreign language” regardless of how it is “learned” when interacting with “foreigners” in a “foreign setting” Used for measuring non-native (*) user abilities Used for measuring non-native (*) user abilities

4. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. The ILR (Interagency Language Roundtable) Six Base Levels (Levels 0 through 5) Six Base Levels (Levels 0 through 5) Five plus levels (Levels 0+ through 4+) Five plus levels (Levels 0+ through 4+) Increased accuracy within a level leads to plus levels Increased accuracy within a level leads to plus levels Scale: Courtesy of the FBI Testing Division Scale: Courtesy of the FBI Testing Division

5. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. UNDERLYING PRINCIPLES OF THE ILR SYSTEM It is a criterion-referenced proficiency scale It is a criterion-referenced proficiency scale Describes functional performances Describes functional performances Focuses on both the abilities and deficiencies Focuses on both the abilities and deficiencies Approaches the functionality from three overlapping planes: global tasks, content and accuracy Approaches the functionality from three overlapping planes: global tasks, content and accuracy It is a range* system It is a range* system It is a threshold* system It is a threshold* system Each level consists of a plus level range Each level consists of a plus level range Each level subsumes every level below it Each level subsumes every level below it It is presented in ever-expanding intervals It is presented in ever-expanding intervals

6. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. THE ILR SYSTEM The scale is an “output oriented” scale – The scale is an “output oriented” scale – founded upon global proficiency functionality (knowledge, cognitive skills, experience, education???) It approaches functionality from three overlapping and interconnected planes: It approaches functionality from three overlapping and interconnected planes: global tasks contentaccuracy ALL THREE CRITERIA MUST APPLIED TO EVERYTHING IN EACH AND EVERY INSTANCE Functionality is defined through “can do” and “cannot do” statements in the ILR skill descriptors Functionality is defined through “can do” and “cannot do” statements in the ILR skill descriptors

7. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. THE ILR SYSTEM (Cont’d) Threshold (minimum requirement) system Level definitions describe the lowest required performance (threshold)* for meeting the criteria… Any user at a given level level must sustain criteria for that level according to the ILR accuracy requirements… Sustainment is defined as “showing evidence of performing most of the time at that level…” also known as the mastery principle.

8. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. The ILR (Interagency Language Roundtable) Thresholds for each level

9. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. THE ILR SYSTEM (Cont’d) RANGE (within the level) There may be varying degrees of performance within the same range… However…They all have two things in common:

10. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. THE ILR SYSTEM (Cont’d) 1- They all meet the criteria for that level… However, some sustain “better” than others… However, some sustain “better” than others… 2- They all fail to sustain at the next level However, some fail “better” than the others However, some fail “better” than the others

11. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. THE ILR SYSTEM (Cont’d) ACCURACY Degree of accuracy in between “all or nothing” is defined through modifiers and qualifiers also known as the “weasel words”… Degree of accuracy in between “all or nothing” is defined through modifiers and qualifiers also known as the “weasel words”… most of the time / frequently / mostly / usually /can… indicate sustainment, strength, “above the threshold” some of the time / rarely / infrequent / may… indicate failed sustainment, weakness, deficiency There are very few definitions that are “none” or “all”

12. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. Authentic Texts ORIENTATION MODE ORIENTATION MODE (Basic and simple information – L1) INSTRUCTIVE MODE INSTRUCTIVE MODE (Detailed concrete/factual information-L2) EVALUATIVE MODE EVALUATIVE MODE (Analysis and evaluation of events – L3) PROJECTIVE MODE PROJECTIVE MODE (Highly abstract and complex texts – L4)

13. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. The Framework ILR Proficiency Scale and Authentic Texts

14. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. ILR AND TEXT MODES ILR READING DESCRIPTORS FOCUS ON…ILR READING DESCRIPTORS FOCUS ON… - NON-NATIVE READER’S FUNCTIONALITY - THE CONTENT AREAS OF TEXTS - HOW A NON-NATIVE READS THE TEXT TEXT MODES FOCUS ON …TEXT MODES FOCUS ON … - WRITER’S COMMUNICATIVE PURPOSE - TEXT RATED FROM IN-LANGUAGE PERSPECTIVE - HOW A NATIVE READS THE TEXT

15. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. PLUS LEVELS Not in the original ILR document Supplementary (!) to the base levels Indicate performance that is “substantially exceeding the base level and (but?) not consistently meeting (sustaining?) the criteria for the next level” Includes properties of both the base and the next level There are NO plus level tasks!

16. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. CHARACTERISTICS OF PLUS LEVELS 1+ MAINLY L1 WITH L2 ELEMENTS ADDED (VOCABULARY AND GRAMMAR ) 2+ a) L2 TEXT RAISED BY GRAMMAR/LEXICON b) L3 TEXT WITH EASY ATTRIBUTES (TOO MUCH FACTUAL INFORMATION) L3+ MAINLY L4, BUT L4 PROPERTIES NOT SUSTAINED THROUGHOUT (NOT ENOUGH ABSTRACT FORMULATIONS)

17. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. ORIENTATION MODE L1 Short, discrete, simple, and compound sentences Short, discrete, simple, and compound sentences Simple and generic vocabulary Simple and generic vocabulary Simple structure Simple structure Loosely ordered information Loosely ordered information

18. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. CHARACTERISTICS OF LEVEL 1 TEXTS ORIENTATIONAL IN NATURE ORIENTATIONAL IN NATURE PROVIDE SIMPLE GENERAL INFO PROVIDE SIMPLE GENERAL INFO MOSTLY ADS AND ANNOUNCEMENTS MOSTLY ADS AND ANNOUNCEMENTS AUTHOR ANONYMOUS/IRRELEVANT AUTHOR ANONYMOUS/IRRELEVANT SIMPLE DISCOURSE SIMPLE DISCOURSE ONE MAIN IDEA ONE MAIN IDEA GENERIC VOCABULARY GENERIC VOCABULARY PRESENT TIME FRAME PRESENT TIME FRAME

19. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. LEVEL 1 GLOBAL TASKS AT L1, READER CAN UNDERSTAND: THE GENERAL MAIN IDEA THE GENERAL MAIN IDEA PROMINENT DETAILS (WHO/WHEN/WHERE) PROMINENT DETAILS (WHO/WHEN/WHERE)

20. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. LEVEL 2 TEXTS INSTRUCTIVE MODE INSTRUCTIVE/CONVEYS FACTUAL INFO NO AUTHOR PRESENT INSTRUCTIVE/CONVEYS FACTUAL INFO NO AUTHOR PRESENT MOSTLY NEWS-MEDIA CONTENT MOSTLY NEWS-MEDIA CONTENT ANONYMOUS AUTHOR ANONYMOUS AUTHOR COHESIVE DISCOURSE COHESIVE DISCOURSE TOPIC-SPECIFIC VOCABULARY TOPIC-SPECIFIC VOCABULARY MULTIPLE TIMEFRAMES MULTIPLE TIMEFRAMES

21. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. LEVEL 2 GLOBAL TASKS AT L2, READER CAN UNDERSTAND: THE MAIN IDEAS, MAIN PLAYERS (WHO/WHAT, WHERE, WHEN) THE MAIN IDEAS, MAIN PLAYERS (WHO/WHAT, WHERE, WHEN) SEQUENCING OF EVENTS (MULTI-TIME FRAMES) SEQUENCING OF EVENTS (MULTI-TIME FRAMES) INTERACTION AMONG PLAYERS (DIRECTION) INTERACTION AMONG PLAYERS (DIRECTION) CAUSE/EFFECT RELATIONSHIPS (WHY) CAUSE/EFFECT RELATIONSHIPS (WHY) CONDITIONALS & NEGATIONS (IF/HOW) CONDITIONALS & NEGATIONS (IF/HOW) STATED FACTUAL OUTCOMES & RESULTS STATED FACTUAL OUTCOMES & RESULTS

22. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. CHARACTERISTICS OF LEVEL 3 TEXTS EVALUATIVE MODE  EVALUATIVE MODE  OPINION OR EDITORIAL PIECES  SOCIAL/CULTURAL/POLITICAL ISSUES  AUTHOR IS PRESENT BY PERSONAL VIEWS  ABSTRACT AND CONCRETE LEXICON  ABSTRACT LINGUISTIC FORMULATIONS IN COMPLEX SENTENCES  EXTENDED DISCOURSE  DOMAIN SPECIFIC VOCABULARY  SCIENTIFIC OR TECHNICAL DOCUMENTS

23. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. MAIN LEVEL 3 GLOBAL TASKS AT L3, READER CAN UNDERSTAND: ALL L2 ELEMENTS, PLUS HIERARCHY OF ABSTRACT IDEAS HIERARCHY OF ABSTRACT IDEAS SUPPORTING FACTS AND IDEAS SUPPORTING FACTS AND IDEAS ABSTRACT HYPOTHETICAL CONDITIONS ABSTRACT HYPOTHETICAL CONDITIONS (ASSUMPTIONS, SUPPOSITIONS) PROS AND CONS OF AN ISSUE PROS AND CONS OF AN ISSUE INFERENCES & CONNOTATIONS INFERENCES & CONNOTATIONS COMPARISON AND CONTRAST OF IDEAS COMPARISON AND CONTRAST OF IDEAS CONCLUSIONS, SUGGESTIONS, SOLUTIONS CONCLUSIONS, SUGGESTIONS, SOLUTIONS TECHNICAL & DOMAIN SPECIFIC LEXICON TECHNICAL & DOMAIN SPECIFIC LEXICON

24. Dirgin, James. January 2014. Please seek the author’s permission for distribution or commercial use. PROJECTIVE MODE Unpredictable turns of thought Unpredictable turns of thought Mixture of register Mixture of register Subtlety and nuance Subtlety and nuance Highly organized, abstract, sophisticated Highly organized, abstract, sophisticated Tone, inferences; irony & humor Tone, inferences; irony & humor

25. Dirgin, James. December 2012. Please seek the author’s permission for distribution or commercial use. Components of an Item (1) Orientation (Providing a brief context) (2) Text (Passage) XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXX (3) Stem (Question) MULTIPLE CHOICE(4) KEY (Correct answer) (5) Distracter 1 (6) Distracter 2 (7) Distracter 3

26. Dirgin, James. December 2012. Please seek the author’s permission for distribution or commercial use. Components of an Item Any single ONE of these 7 factors may and will change the level of difficulty Of an item!