1. Frameworks, Standards, and Scales in the USG
On behalf of the Interagency Language Roundtable Testing Committee, I thank you for attending our presentation today. The Interagency Language Roundtable, or ILR, is an unfunded Federal interagency organization established for the coordination and sharing of information about language-related activities at the Federal level. Each year at ECOLT, we highlight issues that we believe will be of interest to the broader language testing community, such as testing language aptitude, or testing less commonly taught languages.
ECOLT 2009
November 6, 2009
ILR Testing Committee

2. Role of the ILR Testing Committee
Provide forum for discussion
Build network among government, academe and industry
Develop understanding of Skill Level Descriptions
Who we are: Attendance has averaged around 17 participants in 2009 from all three sectors.
Presentations this past year by not-for-profit and testing companies
Major participation in OPI and Listening Summits
Development of Can-Do Self Assessments

3. Characteristics of the ILR Scale
Non-negotiable frame
Mastery principle
Proficiency not performance or achievement
Global rating
Apex is well-educated native speaker

4. Overview of Presentation
Use of the ILR Self Assessments
OPI Summits
First Listening Summit
ASTM Standard Practice
Today we are going to discuss several projects we’ve been involved in related to the ECOLT theme of standards: In this panel presentation, government members on the Interagency Language Roundtable will discuss the impact of several ongoing projects on our understanding of the Interagency Language Roundtable Skill Level Descriptions. The ILR Testing Committee has undertaken projects that speak to the need for transparency and collaboration among the testing community. With those themes in mind, this ECOLT panel will review the outcomes of projects as they relate to the ILR scale. The first project is the use of the ILR-scale based Self Assessments. The second and third are the Summits on Speaking and Listening: the ACTFL-DLI Summit on Testing Speaking at the Higher Levels, whose goal is to develop a clearer understanding of Levels 3 through 5 in the context of the Oral Proficiency Interview. And the fourth project is the ASTM-sponsored effort to develop a Standard Practice on Language Proficiency Assessment. An outgrowth of testing requirements levied by the National Language Service Corps, the ASTM Practice will, for the first time, document best practices for US Government language testing projects. Key to these projects is the overall goal of improving our understanding of the ILR scale.

5. Frameworks, Standards and Scales
Impact of Internet-based Speaking Self-Assessment Tool on
Department of State New Hires

6. Setting
Until September 2007 – one pencil and paper Foreign Service Examination/year resulting in:
Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service

7. Setting: applicant testing
Since September 2007 – 4 online Foreign Service examinations/year resulting in:
Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service
plus
Approximately 1000 additional tests in Super Critical Need Languages: Arabic, Chinese, Hindi, Urdu, Dari, Farsi

8. Setting: expectations
Expected threshold level for testing in:
World languages:
Speaking Proficiency 3
All other languages:
Speaking Proficiency 2

9. Challenges High testing numbers with strict deadlines
Low passing rates
Play audio skit: straight A student in college who failed the Arabic language test after having taken 2 semesters of Arabic]

10. Solution
Internet-available Speaking Self-Assessment Tool on the ILR Homepage

11. Results Before Speaking Self-Assessment Tool: Applicants tested: 85%
Passing rate: 42%
After Speaking Self-Assessment Tool:
Applicants tested: 63%
Passing rate: 63%

12. Additional Results More realistic expectations Happier examinees
Test administration resource savings.

13. Frameworks, Standards and Scales
Oral Proficiency Interview Summits
Listening Summit

14. Impetus for Speaking and Listening Summits
Started in June 2008 as a result of ACTFL OPI work for DLI at Mid & Higher levels
Questions about comparability at the higher levels in Speaking
Success of Speaking Summits led naturally to Listening
Opened up to the wider ILR community and non-USG
SPEAKING I June 2008
SPEAKING II February
SPEAKING III July 2009
LISTENING I September 2009
11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

15. Summit Overall Goals Create common understandings
Achieve comparable scores
Supplement the ILR Skill Level Descriptions
Long-term goal to update the SLDs within the ILR Framework
11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

16. OPI Summit Highlights
Highlighted similarities and differences in approaches to testing Speaking
Recognize that agencies’ tests are different, but all aim to measure speaking in the ILR context
Uncovered differences in interpretations of the SLDs
Discussion on keeping the Educated Native Speaker at the Apex of the Scale
Agreement to continue to tighten wording and standards
11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

17. OPI Summit Outcomes
Work underway to write explanatory appendix of terms
TAEG-Sponsored study of OPI comparability has been designed and is waiting approval

18. Listening Summit Goals
Surface areas in the SLD that need our attention
Build a common understanding on the challenge of listening proficiency
11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

19. Listening Summit Outcomes
Identified participative and non-participative listening as a major issue
Generated a list of terms that need to be further defined and/or clarified
Recognized need to bring in new research in the field

20. Next Steps
OPI Summit work is underway to draft explanatory notes for Speaking Skill Level Descriptions
Next Listening Summit in the winter 2010

21. Frameworks, Standards and Scales
Development of the
ASTM Standard Practice

22. What is ASTM? ASTM: American Society for Testing and Materials
industry standards
Develops but does not enforce standard Independent, private sector, not-for-profit
Mission is to provide a system for experts to directly participate in developing market relevant, high quality, international standards
Standards developed in an open process involving all interested parties (government & industry)
ASTM does not enforce standards
Standards may be referenced in contracts

23. The ASTM Standard
Many ILR users are competent test developers and can design their own test specifications
BUT
Some ILR users need a test but do not have the expertise to develop it or to determine what existing tests might suit their need
ILR users might differ in their interpretations of the ILR for testing; these differences might not be apparent in test specifications

24. The ASTM Standard
Having an overarching standard allows multiple test developers all referencing the ILR to have a common reference point
A formal standard is transparent and allows contractors and government agencies to know what is required
Standard is a basis for quality assurance
Standard guide vs. standard practice vs. standard specification

25. Challenges
The ASTM Standard Practice is NOT an interpretation of the ILR
As of now, there is no enforcement mechanism for the standard
The standard is modular, since different testing needs require different types of tests and test development processes—not lockstep standardization

26. Looking forward
The process of articulating the standard has brought about unprecedented dialogue among different agencies and contractors about what is expected and needed
The requirements for needs analysis and frameworks that are stated in the standard practice raise awareness among stakeholders that will lead to better-fitting tests

27. Conclusions Broaden access to the ILR framework
Improve common understanding of the ILR scale
Update Skill level Descriptions
. Key to these projects is the overall goal of improving our understanding of the ILR scale.