2. Gaming the System in the CALL Classroom Peter Gobel Kyoto Sangyo University pgobel@cc.kyoto-su.ac.jp

3. Introduction J. Carrol (1963)- time on task hypothesis: the learner will succeed in learning a given task to the extent that he spends the amount of time that he needs to learn the task. Baker et. al (2004a)- off-task behavior associated with reduced learning

4. Cont. Baker et. al (2004b)- gaming the system (misuse of tutoring systems) associated with substantially lower learning. 1/3 less than those who do not engage in this behavior. Learned helplessness and ‘performance orientation’ related to gaming.

5. Basic Question Is motivation related to misuse of the system or inappropriate help-seeking?

6. Research Questions In Japanese university students: Is frequency of off-task behavior a predictor of test performance? Does the kind of off-task behavior make a difference? Are motivational factors predictors of performance behavior and test performance?

7. Kyoto Sangyo CALL Curriculum Students lower proficiency non-English majors System meeting 90 minutes/week DynEd ALC Course requirements grade linked to completion of level tests and time spent on software system

8. Participants Three intact classes - 105 students All non-English majors All streamed at low level

9. Observation Off-task behavior (physical and virtual) Non-system related activity Inactivity Misuse of system (gaming) On-task behavior (physical and virtual) Appropriate system-related activity Seeking help from peers, software, and teacher

10. Observation (continued) Total amount of study time Recorded automatically by software system Study score Based on total amount of study time and ‘proper’ use of the software.

11. Observation (continued) Test performance Gain scores on general proficiency test Listening section Reading section Administered in April and July

12. Motivation Questionnaire 35seven-point Likert scale items administered in Japanese 30 items were written referring mainly to motivational theory 5 items referring to CALL activity

13. Questionnaire Items Expectancy for success Attainment Value Intrinsic value Extrinsic utility value Cost Attitudes toward target group Effort CALL related items

14. Results Observation Off-task behavior On-task behavior Total study time Study score Questionnaire Structure Motivational predictors of performance Motivational predictors of behavior

15. Classroom Observation

16. Off-task behavior Off-task non software Off-task inactive Off-task software Help Study record Off-task gaming

17. On-task behavior On-task software Microphones and speech analyzers seldom used Certain activities overused and recycled On-task help Rarely used Translation rarely used

18. Behavior Regression Results Whether students were generally off task or on task was a significant predictor of test performance. Gaming was not a significant predictor of test performance.

19. Behavior Regression Results Listening Gain Study Score a significant predictor of performance Reading Gain Study Score &Total Study Time significant predictors of performance.

20. Questionnaire results Four factors Attitudes toward the target group Perceived usefulness of studying English Expectancy for success Attitude towards CALL

21. Motivation regression analysis Factor 1 (Attitudes towards the target group) was a predictor of gain scores in both listening and reading, but not for behavior.

22. Research Questions Revisited Task behavior was a predictor of test performance. The kind of off-task behavior did not make a difference. Attitude toward the TG was a predictor of test performance, but not off/on-task behavior.

23. Conclusion Off-task behavior Evidence of off-task behavior related mostly to inactivity rather than gaming On-task behavior All aspects of the software not fully used Motivation Attitude towards TG most important for effective CALL use

24. References Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the cognitive tutor classroom: When students ‘game the system’. Proceedings of ACM CHI 2004: Computer Human Interaction, 383-390. Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004). Detecting student misuse of intelligent tutoring systems. Proceedings of the Seventh International Conference on Intelligent Tutoring Systems, 531-540. Carrol, J. (1963). A model of school learning. Teachers College Record, 64(8), 723-733.