1. Users reactions to innovative computerised feedback – the case of DIALANG Ari Huhta Centre for Applied Language Studies University of Jyväskylä Finland

2. What is DIALANG? computerized diagnostic language assessment system in 14 languages (www.dialang.org) feedback (& activities) that help learners to diagnose their strengths and weaknesses plan further language learning raise awareness unusually wide range of feedback for a test; partly unique / innovative, e.g. self-assessment

3. What is feedback? Why feedback? In educational contexts FB can be: … any communication or procedure to inform a learner of the accuracy of a response, usually to an instructional question (Knowledge of the Correct Response) but also … other information such as precision, timeliness, learning guidance, motivational messages,... critical comparisons, and learning focus … any message or display that a computer presents to the learner after a response (Mory 2004) improve performance & learning (directly) but also to motivate, make learning more interesting, encourage different learning, raise awareness, …

4. Why study (DIALANG) feedback? feedback is an integral part of all behaviour, performance and learning social psychology, organizational theory, education... feedback from tests is not much researched (cf. teachers as assessors) feedback from computerized (language) tests is particularly little researched

5. DIALANG FEEDBACK

6. Research topics / questions This study: What are users reactions to different types of feedback in DIALANG? How do new types of feedback compare with more traditional feedback? Are users reactions associated with their background characteristics (e.g. sex, age, language use, language proficiency)? How might possible associations be explained? -------------------------------------------------------- related, more in-depth studies users reactions to self- assessment and Vocabulary Size Placement Test

7. Research methods & data Quantitative & qualitative (cross- sectional) Questionnaire 553 respondents in two countries mostly non-language majors at universities (typical users) Interviews of users and their teachers Written reports from users & teachers

8. Users reactions? Defined in terms of 1) Reading / using particular feedback? 2) Interest in particular feedback? 3) Perceived usefulness of particular feedback? 4) Other reactions (e.g. ease of use) Likert scale + open-ended questions cf. Technology Acceptance Model

9. Who were the respondents?

10. Age of respondents

11. # of languages & tests taken

12. Most common test languages

13. Results: Overall reaction to FB - use & interest & usefulness

14. Reactions to Level (test result)

15. the most popular part of feedback the fact that the meaning of A1 etc was defined was appreciated associations with user background: correlated with proficiency: higher proficiency – more positive reactions men were less interested in this than women

16. Reactions to Vocabulary Size Placement Test feedback / result

17. Reactions to Vocabular Size Placement Test & its feedback VSPT and its result was the most controversial part of DIALANG although, overall, the reactions were positive strong affective reactions, opinions rather divided Interesting, I had never thought of a way like this to test language proficiency. Fun to do! Absolute nonsense! This cant be my level! background: in general, those who got low VSPT results regarded it very negatively, whereas those who got good results regarded it very positively in particular, women at level C1/C2 liked VSPT feedback explanation: VSPT scoring is affected by test-taking strategy and produces very low scores too easily

19. Reactions to immediate item review almost 60% preferred immediate review, 30% post- test review, 10% preference Why like: want to know immediately, remember better if see the result right away, want to learn (during test) Why dont like: disturbs test-taking, can be depressing background: beginners liked it more than advanced learners in particular, C-level (C2) learners considered it less useful and interesting C-level men used it particularly infrequently possible explanation: many beginners & intermediate learners want to learn from tests?? advanced learners make so few mistakes that they may feel immediate feedback is waste of time??

20. Reactions to post-test item review less popular than immediate review but a fair number of users absolutely want to have it Background: more experienced users (# of languages used & length of language studies) considered this more interesting & useful possible explanation (very tentative): more experienced language learners may find information about subskills interesting??

21. Reactions to self-assessment feedback (comparison of SA and test result)

22. Reactions to self-assessment feedback according to users, SA is important part of (a system like) DIALANG, as popular as e.g. item review (joint second) background: men had more often very negative opinions compared with women the youngest group (under 18) found it less interesting than the older groups possible explanation: younger learners may not (yet) be so interested in the kind of reflection on ones skills involved in self-assessment??

23. Reactions to self-assm. information (why SA and test result may not match)

24. Reactions to SA information the least used / interesting / useful part of feedback Background: negative reactions were associated with higher proficiency, especially C1/C2 women more experience in lg learning (more lgs studied) male respondents (more very negative opinions) young age (under 18) positive reactions were associated with : lower proficiency, especially A1/A2 men less experience in lg learning (fewer lgs studied) men were divided in their opinions of SA information possible explanations: if SA and test agreed, no need to read this beginners have less experience in what is described in SA information and may find it new & potentially useful; advanced learners may feel they already know all that

25. Reactions to extended descriptions

26. fairly interesting, useful and used but less so than e.g. item review background: men made more often very negative evaluations of this feedback than women

27. Reactions to advice

28. quite popular but not among all users – surprise? Background: high proficiency (C1/C2) & more experience more negative views of advice men had more very negative views than women A1 / A2 learners were divided: they often chose it as one of the 3 most useful types of FB but many also chose it as one of the least useful types of FB possible explanation: C-level learners no longer need advice, whereas A-level learners do, but some of them may feel the particular advice in DIALANG is not useful??

29. Implications / questions implications for DIALANG and/or other such computerized systems users clearly like to have feedback about their self-assessment younger learners may need to be motivated first good to have two types of item review: immediate and post-test, based on choice preferences differ; level of proficiency

30. Implications / questions VSPT divides opinions: love – hate scoring needs revising advice appears useful for beginners in particular challenges I dont bother to read the advice because I know Ill do things my way anyway... you dont remember them any more when you study longitudinal studies how best integrate systems like DIALANG into teaching (long-term) effectiveness of feedback?