Assessing proficiency in languages Curricular implications

Through an innovative partnership between the federal government and higher education, The Language Flagship seeks to graduate students who will take their place among the next generation of global professionals, commanding a superior level of proficiency in one of many languages critical to U.S. competitiveness and security.

Arabic – Chinese – Korean Portuguese - Russian عربى,中文, 한국어, Português, русский 2,000+ tests administered Background questionnaire in addition. Collecting lots of data and engaged in a number of research projects. Will talk here about two of them. Assessment in other languages was added using internal funding

Testing plan Semester before graduation End of 3rd year 2020 1020 Students were tested at the end of spring and fall semesters

Measure student proficiency in speaking, reading and listening in Arabic, Korean, Portuguese and Russian at all levels of instruction Determine baseline and target proficiency goals for each language at each level of instruction Determine and develop curricular and pedagogical shifts Develop a system to integrate proficiency assessment into student placement, which will also facilitate transfer Devise a model to sustain proficiency assessment practices RECURRING CYCLES

Data collected Data includes: Proficiency test scores: Speaking, Reading and Listening (standardized tests) Responses to background questionnaire, e.g., age, gender, other languages spoken, study abroad experience, language use outside the classroom, etc. Classroom observation data I’ll talk now about the correlation between our stated LO and what the data show

Proficiency and learning outcomes

Median assessment scores For Chinese, French, Russian, and Spanish (by year) I am going to show you some 2017 results from a limited number of languages Median assessment scores

Median assessment scores by year: Chinese IH Chinese Year 1 Year 2 Year 3 Year 4 Speaking NM NH IL IM Reading NL Listening IM IL NH Year 1 Year 2 Year 3 Year 4 Speaking n=56 n=70 n=45 n=27 Reading n=49 n=67 n=41 n=12 Listening n=53 n=64 n=33 NM Explain levels (NL, NM, etc.) NL NL

Median assessment scores by year: French AM AL French Year 1 Year 2 Year 3 Year 4 Speaking NH IL IM IH Reading AM Listening NM IH IM IL NH Year 1 Year 2 Year 3 Year 4 Speaking n=243 n=286 n=251 n=85 Reading n=268 n=222 n=80 Listening n=235 n=266 n=217 n=78 NM Mention that these are all languages in which a majority of students are RMs NL

Median assessment scores by year: Russian AL IH Russian Year 1 Year 2 Year 3 Year 4 Speaking NH IL IM IH Reading NL Listening IM IL NH Year 1 Year 2 Year 3 Year 4 Speaking n=87 n=101 n=31 n=4 Reading n=89 n=105 n=5 Listening n=86 NM NL

Median assessment scores by year: Spanish AM AL Spanish Year 1 Year 2 Year 3 Year 4 Speaking NH IL IM IH Reading AM Listening NM IH IM IL NH Year 1 Year 2 Year 3 Year 4 Speaking n=353 n=456 n=524 n=264 Reading n=346 n=428 n=475 n=169 Listening n=344 n=418 n=467 n=165 NM NL

Implications Data show that most students are not attaining the intended proficiency LO by graduation. Expectations need to be adjusted to reflect what is realistic. Other LOs may need to be adjusted too. LOs need to reflect the proficiency profiles of learners of different languages (e.g. reading and listening in French and Spanish vs. Russian and Chinese)

Implications (cont.) More emphasis needs to be placed on the development of certain skills. Classroom observation data uncover some potential explanations for proficiency findings.

Proficiency outcomes and grading: Walk the walk? All language programs list proficiency outcomes. All language courses have explicit or implicit proficiency outcomes. BUT… Current formative and summative assessments may not accurately reflect (or encourage) development of proficiency. Do grades in a course reflect proficiency level?

UofU Grade-score correlations (AY 15-16) Arabic, Chinese, Korean, Portuguese, Russian, all levels Correlations Speaking Reading Listening Grade Pearson Correlation .119* .031 .009 N 196 207 *. Correlation significant at the 0.05 level

UofU Grade-score correlations (AY 16-17) Arabic, Chinese, Korean, Portuguese, Russian, all levels Correlations Speaking Reading Listening Grade Pearson Correlation .229** .194** .159** N 227 224 **. Correlation significant at the 0.01 level

Sustainability? Adding a fee? Doing sample assessments Doing regular self-assessment  BOSSA Sustainability?

Basic Outcomes Student Self Assessment (BOSSA) How does BOSSA work? 50-minute lab session Lab session: 6 components 1. They record themselves speaking in response to prompts in the language they study. 2. They listen to their recordings afterward. Now students have a concrete point of reference and can start to notice how they use language. 3. They use criteria on a worksheet to practice measuring how well they speak. 4. Next, students talk with classmates and discover what they have in common: where they are on their language-learning path and where they are going. 5. Students lead a class discussion about their strengths and challenges, and come up with suggestions and short-term goals for improvement while instructors take notes on the board. 6. Students are now ready for the last step — they assess their own language skills using the online Build on Language Track (BoLT) survey, and get immediate feedback estimating their proficiency level based on how they have assessed themselves.