1. StoryStation Peter Wiemer-Hastings Judy Robertson Kath Glasswell DePaul University University of Edinburgh University of Illinois at Chicago

2. Motivation Learning to write (well) is hard Multiple constraints / voices (Flower, 1994) –Reader: What do I want R to know? What do I know about what R knows? What do I want from R? –Situation: What am I supposed to write about? What do I know about my topic? What do I want to say about it? –Socio-linguistic context: How can I say this in a way that my teacher will accept?

3. Metacognitive Double Whammy Students must learn how to think about meeting the demands of multiple, possibly conflicting constraints. Students ordinarily don’t know what those constraints are.

4. AI to the rescue!? Appropriate feedback on compositions can point students in the right direction. Teachers don’t have time to give the best feedback to every student all the time. NLP techniques can analyze compositions, generate feedback. Associating different types of feedback with different animated agents might help the students learn the various constraints.

5. Research goals To investigate the effect of animated pedagogical agents on 10-11 year old pupils’… –motivation, –satisfaction, –learning (eventually). To develop an intelligent tutoring system to help children write stories

6. Agenda Previous work Design process StoryStation features Evaluation – preliminary results Future directions

7. Previous research on agents Mayer and Wittrock, 1996: –With agents, students have more social and emotional investment, more motivated, try harder, problem solve better. Herman the Bug (Lester et. al, 1997) –Students more motivated when using agents, and learn more. Johnson, Rickel, and Lester, 2000 –increased communication bandwidth and social interaction with agents Review of agent research (Dehn and van Mulken, 2000) –Some serious methodological problems. No control in Lester et al. Moreno, Mayer, and Lester, 2001: –Included control. Agents are social actors, increase motivation, learning, transfer. Voice is important, face not.

8. Previous writing research Previous writing environments were designed to support cognitive processes of writing (e.g. Sharples 1988), based on Flower and Hayes (1980) theory. Writing apprehension (Madigan, Linton and Johnson, 1996) More recently, interest in providing emotional support for writers (e.g. T’riffic Tales project) StoryStation basic strategy: Provide positive feedback to encourage pupils to take responsibility for their own learning

9. Design process Requirements gathering with 2 teachers and 37 pupils Integration of National English Language Curriculum goals Design team of 8 pupils and 2 teachers, working on the project regularly for 18 months Observation of classroom practice 3 pilot studies

10. StoryStation features Spelling Word banks Dictionary Thesaurus Vocabulary Characterization Plot (in progress)

11. Story re-telling Task Students hear a story-teller present a story (or a video) Students are asked to create their own version with StoryStation. Scaffolds student’s writing. Facilitates text analysis.

17. Evaluation – 18 teachers Positive feedback is a good feature, contrasts with programs which simply point out mistakes. Useful for independent learning because it teaches pupils how to self correct. Good for pupils to get instant feedback rather than waiting till the next day. Would like StoryStation to help them in assessing pupils’ progress Generally, teachers are surprisingly trusting of technology

18. Evaluation - pupils 60 pupils (10 –12 year olds) Half used agents version, half text-only Pupils: –Listened to 10 minute recording of “The Screaming Skulls of Calgarth” –Watched demo of StoryStation –Wrote their own version of “Screaming Skulls” using StoryStation (40 minutes to 75 minutes, depending on timetabling constraints) –Filled in questionnaire –Answered structured interview questions

19. Children’s comments on StoryStation “I’d probably buy it for my home computer” –Roseanna, age 12 “If I had StoryStation on my computer I’d definitely use it more than Microsoft Word” –Rosemary age 12 “It’s really good and it helps you to write better” Shenel, age 12 “It makes writing fun. You enjoy it more when you’ve got the helpers there” –Joanna, age 11

20. Questionnaire results 6. StoryStation makes writing stories easier 7. I think I need someone to help me use StoryStation 8. I think the StoryStation advice was useful 9. A teacher is more helpful than StoryStation 10. StoryStation is boring 1. I enjoyed using StoryStation 2. I think StoryStation made my writing worse 3. I think I would like to use StoryStation again 4. Using StoryStation help me to write better 5. I found StoryStation confusing

21. Non-agentAgent GirlsSDBoysSDGirlsSDBoysSD 1. I enjoyed using StoryStation 0.880.200.900.120.970.060.900.13 2. I think StoryStation made my writing worse 0.06 0.090.120.02 0.070.11 3. I think I would like to use StoryStation again 0.850.220.860.180.980.060.930.13 4. Using StoryStation help me to write better 0.670.320.780.240.710.080.730.29 5. I found StoryStation confusing 0.220.290.150.240.040.060.100.13 6. StoryStation makes writing stories easier 0.850.200.780.320.950.160.760.26 7. I think I need someone to help me use StoryStation 0.080.090.110.140.150.210.180.20 8. I think the StoryStation advice was useful 0.780.290.850.180.890.030.860.16 9. A teacher is more helpful than StoryStation 0.280.220.360.290.240.280.330.32 10. StoryStation is boring 0.120.160.090.130.110.280.090.17

22. Nice pictures

26. Some issues Children’s thoughts about agents –Mixed ideas about whether agents are “real”. Children’s comparisons of teachers to StoryStation “A teacher can tell from your eyes what you’re thinking, but StoryStation can’t” Saima, age 11 Social relationships: The teacher can shout at you, but you can shout at StoryStation Knowledge: Take StoryStation’s advice more because it is a computer

27. Current Proposal NSF-ROLE grant proposal with Elliott and Glasswell Clean up the metacognitive hypotheses and evaluations Address some of the questions about agents.

28. Conclusions Children think StoryStation is useful for writing stories StoryStation is motivating to children Agent users more likely to find StoryStation advice helpful, and want to use it again Agent users less likely to find StoryStation confusing Interesting social / gender interactions

29. Challenges How to foster metacognition –“Appropriate” feedback depends on genre, context, audience, etc –Dialog about student choices? How to measure learning How/where to get funding to continue

30. Questions or Comments?