1. Inés Alonso-García Hervé Didiot-Cook Astrid Kuellmann-Lee Peter Wren Language Level Assessments using WebCT Quizzes

2. In French, German & Spanish, involving 3 constituencies Language teaching teams: Test Writers, WebCT Designers, Project Coordinators. Language teaching teams: Test Writers, WebCT Designers, Project Coordinators. Learning technology: Online pedagogy and technical administration. Learning technology: Online pedagogy and technical administration. Language Centre facilitation: Online access to assessments. Language Centre facilitation: Online access to assessments.

3. The Key element: Team Work Liaison and coordination between test writers, designers and learning technologists, test writers, designers and learning technologists, the Language Centre administration (website and class / level data) the Language Centre administration (website and class / level data) and at the same time, the Centre for Learning Technology technical administration (WebCT enrolment). and at the same time, the Centre for Learning Technology technical administration (WebCT enrolment).

4. Why ? Needs analysis based on written test AND an oral assessment. Needs analysis based on written test AND an oral assessment. Previous assessment was a paper version only, PDF to download and print. Previous assessment was a paper version only, PDF to download and print. 4 questions in English to answer in the target language. 4 questions in English to answer in the target language.

5. Previous paper version Paper-based free-text data is lost to further analysis.

6. Weaknesses No time to fully mark according to criteria as the student comes in for a 10 min needs analysis during which his/her spoken skills are also assessed: Therefore, no score to use as a level reference i.e. no numerical score as basis of decision. No time to fully mark according to criteria as the student comes in for a 10 min needs analysis during which his/her spoken skills are also assessed: Therefore, no score to use as a level reference i.e. no numerical score as basis of decision. Time consuming process as time is spent justifying the placement to the student. Time consuming process as time is spent justifying the placement to the student. Time spent printing, photocopying paper test. Time spent printing, photocopying paper test.

7. “Que faire?” Unsatisfactory to use Dialang as there is no teacher’s access to student score, no opportunity to retrieve, collate & analyse data. Unsatisfactory to use Dialang as there is no teacher’s access to student score, no opportunity to retrieve, collate & analyse data. Design an online level assessment using the quiz tool in WebCT with MCQ, short answer + paragraph questions. Design an online level assessment using the quiz tool in WebCT with MCQ, short answer + paragraph questions.

8. What did we test ? Language proficiency over 4 levels of ability from level 2 (CofE A1/2) to level 5 (CofE C1/2)… by testing literacy: grammar, tenses, lexis, reading comprehension… through MCQ, short answer, and paragraph questions.

9. How ? The overall process (1) Accessibility to WebCT LN 00: All LSE students were registered and given access (12,000 plus) Self registration for LSE staff External students contact language centre to gain access

10. To access the above test and complete their needs analysis All applicants are expected to go to the LSE language centre website (Link on LSE homepage), 1. To find the link for the test, and 2. To book an interview (needs analysis) during which a Level Assignment Form is signed by a teacher / assessor.

11. How ? The different components and stages ONLINE MCQs generating a numerical score MCQs generating a numerical score Paragraph answers in Target Language Paragraph answers in Target Language Linguistic Profile (Short paragraph questions in English on WebCT) Linguistic Profile (Short paragraph questions in English on WebCT) FACE TO FACE + Oral interview (during needs analysis: oral fluency is key component (most important for Spanish))

12. Much easier to evaluate! … and quantify – choice C is an error of tense, choice A is a spelling error. Each choice of distractor tells a story, every one can be counted.

13. One of the paragraph questions

14. The first few questions

15. The Language Assessment WebCT Course

16. The Quizzes: Access is very easily controlled.

17. Instructions and Question 1 Questions are presented to the student one at a time; each can be revisited and revised before final quiz submission.

18. No scores released to student In WebCT the designer has total control of the process: feedback was deliberately limited.

19. But we had instant access to each student’s grade and indicators of overall ability

20. How were the test results used for level placement? The scoring scheme, i.e. graded MCQ complemented by short answer and paragraph questions… follow the levels of proficiency, i.e. from level 2 to level 5, allowing the assessor to see the range and gaps in the student’s ability.

21. Full access to all student responses, which were later downloaded into Access, then Excel, for detailed analysis.

22. Is our assessment process fair and accurate ? Yes, because level transfer is down from last year (especially for French). … but it may be not 100% accurate as data shows discrepancies between test results and level allocation. … but it may be not 100% accurate as data shows discrepancies between test results and level allocation.

23. Numerical scores are not the whole story

24. Discrepancies tracked

25. Numerically High Score but in Level 2

26. Numerically Low Score but in Level 3

27. Positive outcomes Over 1000 students took the test. Over 1000 students took the test. Paper version used in exceptional circumstances. Paper version used in exceptional circumstances. Instant access to student’s score and test. Instant access to student’s score and test. Fast-tracking of student’s weaknesses in grammar. Fast-tracking of student’s weaknesses in grammar. Potential for data collection and analysis from a different perspective. Potential for data collection and analysis from a different perspective.

28. ISSUES We need to look into discrepancies between test results and allocation of students to levels, We need to look into discrepancies between test results and allocation of students to levels, We need to compare the accuracy of LN00 level allocation with Dialang assessment, We need to compare the accuracy of LN00 level allocation with Dialang assessment, Different for German to evaluate this option; short-answer questions used instead of MCQs. Different for German to evaluate this option; short-answer questions used instead of MCQs. We need to develop staff training. We need to develop staff training.

29. Future developments Plan / build in time for data analysis and to track discrepancies between test result and level allocation. Plan / build in time for data analysis and to track discrepancies between test result and level allocation. Get students to take Dialang as a benchmark for our test, at the beginning and end of the programme. Get students to take Dialang as a benchmark for our test, at the beginning and end of the programme. Get students to take the same test again and measure the extent of the progress according to their numerical score; compare this with other means of evaluation. Get students to take the same test again and measure the extent of the progress according to their numerical score; compare this with other means of evaluation.

30. To contact us Inés Alonso-García Inés Alonso-García i.alonso-garcia@lse.ac.uk Hervé Didiot-Cook Hervé Didiot-Cook h.didiot-cook@lse.ac.uk Astrid Kuellmann-Lee Astrid Kuellmann-Lee a.t.kuellmann-lee@lse.ac.uk Peter Wren, LSE (CED, Imperial College from January) Peter Wren, LSE (CED, Imperial College from January) p.wren@lse.ac.ukp.wren@lse.ac.uk then p.wren@imperial.ac.uk p.wren@imperial.ac.uk p.wren@lse.ac.ukp.wren@imperial.ac.uk