Frameworks, Standards, and Scales in the USG On behalf of the Interagency Language Roundtable Testing Committee, I thank you for attending our presentation today. The Interagency Language Roundtable, or ILR, is an unfunded Federal interagency organization established for the coordination and sharing of information about language-related activities at the Federal level. Each year at ECOLT, we highlight issues that we believe will be of interest to the broader language testing community, such as testing language aptitude, or testing less commonly taught languages. ECOLT 2009 November 6, 2009 ILR Testing Committee

Role of the ILR Testing Committee Provide forum for discussion Build network among government, academe and industry Develop understanding of Skill Level Descriptions Who we are: Attendance has averaged around 17 participants in 2009 from all three sectors. Presentations this past year by not-for-profit and testing companies Major participation in OPI and Listening Summits Development of Can-Do Self Assessments

Characteristics of the ILR Scale Non-negotiable frame Mastery principle Proficiency not performance or achievement Global rating Apex is well-educated native speaker

Overview of Presentation Use of the ILR Self Assessments OPI Summits First Listening Summit ASTM Standard Practice Today we are going to discuss several projects we’ve been involved in related to the ECOLT theme of standards: In this panel presentation, government members on the Interagency Language Roundtable will discuss the impact of several ongoing projects on our understanding of the Interagency Language Roundtable Skill Level Descriptions. The ILR Testing Committee has undertaken projects that speak to the need for transparency and collaboration among the testing community. With those themes in mind, this ECOLT panel will review the outcomes of projects as they relate to the ILR scale. The first project is the use of the ILR-scale based Self Assessments. The second and third are the Summits on Speaking and Listening: the ACTFL-DLI Summit on Testing Speaking at the Higher Levels, whose goal is to develop a clearer understanding of Levels 3 through 5 in the context of the Oral Proficiency Interview. And the fourth project is the ASTM-sponsored effort to develop a Standard Practice on Language Proficiency Assessment. An outgrowth of testing requirements levied by the National Language Service Corps, the ASTM Practice will, for the first time, document best practices for US Government language testing projects. Key to these projects is the overall goal of improving our understanding of the ILR scale.

Frameworks, Standards and Scales Impact of Internet-based Speaking Self-Assessment Tool on Department of State New Hires

Setting Until September 2007 – one pencil and paper Foreign Service Examination/year resulting in: Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service

Setting: applicant testing Since September 2007 – 4 online Foreign Service examinations/year resulting in: Approximately 400 applicants/year tested for language proficiency in languages considered useful to the Foreign Service plus Approximately 1000 additional tests in Super Critical Need Languages: Arabic, Chinese, Hindi, Urdu, Dari, Farsi

Setting: expectations Expected threshold level for testing in: World languages: Speaking Proficiency 3 All other languages: Speaking Proficiency 2

Challenges High testing numbers with strict deadlines Low passing rates Play audio skit: straight A student in college who failed the Arabic language test after having taken 2 semesters of Arabic]

Solution Internet-available Speaking Self-Assessment Tool on the ILR Homepage http://www.govtilr.org/

Results Before Speaking Self-Assessment Tool: Applicants tested: 85% Passing rate: 42% After Speaking Self-Assessment Tool: Applicants tested: 63% Passing rate: 63%

Additional Results More realistic expectations Happier examinees Test administration resource savings.

Frameworks, Standards and Scales Oral Proficiency Interview Summits Listening Summit

Impetus for Speaking and Listening Summits Started in June 2008 as a result of ACTFL OPI work for DLI at Mid & Higher levels Questions about comparability at the higher levels in Speaking Success of Speaking Summits led naturally to Listening Opened up to the wider ILR community and non-USG SPEAKING I 12-13 June 2008 SPEAKING II February 9-10 2009 SPEAKING III 22-23 July 2009 LISTENING I 16-17 September 2009 11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

Summit Overall Goals Create common understandings Achieve comparable scores Supplement the ILR Skill Level Descriptions Long-term goal to update the SLDs within the ILR Framework 11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

OPI Summit Highlights Highlighted similarities and differences in approaches to testing Speaking Recognize that agencies’ tests are different, but all aim to measure speaking in the ILR context Uncovered differences in interpretations of the SLDs Discussion on keeping the Educated Native Speaker at the Apex of the Scale Agreement to continue to tighten wording and standards 11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

OPI Summit Outcomes Work underway to write explanatory appendix of terms TAEG-Sponsored study of OPI comparability has been designed and is waiting approval

Listening Summit Goals Surface areas in the SLD that need our attention Build a common understanding on the challenge of listening proficiency 11/10/2018 3:41:09 PM11/10/2018 3:41:09 PM

Listening Summit Outcomes Identified participative and non-participative listening as a major issue Generated a list of terms that need to be further defined and/or clarified Recognized need to bring in new research in the field

Next Steps OPI Summit work is underway to draft explanatory notes for Speaking Skill Level Descriptions Next Listening Summit in the winter 2010

Frameworks, Standards and Scales Development of the ASTM Standard Practice

What is ASTM? ASTM: American Society for Testing and Materials 30000+ industry standards Develops but does not enforce standard Independent, private sector, not-for-profit Mission is to provide a system for experts to directly participate in developing market relevant, high quality, international standards Standards developed in an open process involving all interested parties (government & industry) ASTM does not enforce standards Standards may be referenced in contracts

The ASTM Standard Many ILR users are competent test developers and can design their own test specifications BUT Some ILR users need a test but do not have the expertise to develop it or to determine what existing tests might suit their need ILR users might differ in their interpretations of the ILR for testing; these differences might not be apparent in test specifications

The ASTM Standard Having an overarching standard allows multiple test developers all referencing the ILR to have a common reference point A formal standard is transparent and allows contractors and government agencies to know what is required Standard is a basis for quality assurance Standard guide vs. standard practice vs. standard specification

Challenges The ASTM Standard Practice is NOT an interpretation of the ILR As of now, there is no enforcement mechanism for the standard The standard is modular, since different testing needs require different types of tests and test development processes—not lockstep standardization

Looking forward The process of articulating the standard has brought about unprecedented dialogue among different agencies and contractors about what is expected and needed The requirements for needs analysis and frameworks that are stated in the standard practice raise awareness among stakeholders that will lead to better-fitting tests

Conclusions Broaden access to the ILR framework Improve common understanding of the ILR scale Update Skill level Descriptions . Key to these projects is the overall goal of improving our understanding of the ILR scale.