1. ICCHP 20061 Adaptive Systems Research Group University of Hertfordshire Megan Davis Kerstin Dautenhahn Chrystopher Nehaniv Stuart Powell TouchStory: Towards an Interactive Learning Environment for Helping Children with Autism to Understand Narrative

2. ICCHP 20062 Project Aims Children with autism have a deficit in narrative ability which adversely effects their social comprehension. Our long term goal is to develop interactive systems which improve social comprehension in individual children with autism by enhancing their narrative abilities. This study is concerned with the design of an interactive picture-narrative completion game for children with autism.

3. ICCHP 20063 Autism Children with autism have  impaired social interaction  impaired communication (verbal and non-verbal)  impaired imagination They form a diverse group but share a difficulty in making sense of the world, particularly the social world. Children with autism  generally like repetitive games  may be very routine oriented  may have particular sensory sensitivities  may focus on seemingly irrelevant details (local cohesion)

4. ICCHP 20064 Squirrel Rubbish bin Ref: Story told by Rita Jordan “Oh, look…”

5. ICCHP 20065 Squirrel Rubbish bin “….…. someone has thrown away a perfectly good squirrel.” Ref: Story told by Rita Jordan

6. ICCHP 20066 Narrative and Social Comprehension ► Narrative is fundamental to the perception, creation and communication of meaning in social interaction ► By fitting events into a narrative pattern we construct and inhabit a meaningful social world ► Narrative comprehension involves building an internal representation  Gaps in the narrative exposition must be filled by experiential or semantic knowledge

7. ICCHP 20067 The longitudinal study ► Aim: To investigate primitive aspects of narrative as a pre-cursor to narrative comprehension  Can an adaptive interactive system be used to identify aspects of narrative which individual children find difficult?  Can an adaptive interactive system be used to enhance children’s understanding of narrative?

8. ICCHP 20068 Why use computers? ► Most children with autism enjoy using computers  highly predictable and controllable  naturally monotropic  may be highly visual ► Easy logging, record keeping; timely analysis ► Computers don’t get bored or irritable ► Quicker turn round, children are exposed to more good examples

9. ICCHP 20069 Challenges of designing software systems for children with autism ► Usual methods (interviews, focus groups, collaborative design, etc.) not possible  children may have limited language  they typically find what? why? questions difficult and so answers are not helpful  they are not socially oriented -- collaborative or group based work not appropriate ► Local cohesion and the variability of autism both mean it is difficult to predict what a child will find difficult ► Repetitive behaviour issues

10. ICCHP 200610 Requirements elicitation ► explore the children’s reaction using laminated cards and a replica of the game using a touchscreen ► 18 children (14 ASD), ages 4-9 years, from two schools ► 5 stories from published picture story books Focussed: no extraneous sound or animation, rewards re-enforce the task Designed to minimise meaningless repetition (autonomy/control balance) Expect a desire for repetition so no need to design to keep attention Builds on previous autism and technology research

11. ICCHP 200611 Why a touch screen? ► Children with autism typically have difficulty with remote object references. A touch screen allows direct manipulation – a perceptually direct link between the object and its indication. Why pictures? ► We are concerned with narrative comprehension not ‘literacy’ or reading ability. Why a game? ► Enjoyment and a playful context is central to our approach.

12. ICCHP 200612 Creating proto-narratives ► Primitives identified as  Character  Background or setting  Sequence ► Reversible ► Temporal (may have causality, or be episodic) ► Narrative (involves motivations of characters) ► t-stories (picture narratives and proto- narratives collectively) were prepared

13. ICCHP 200613 TouchStory displaying a t-story from the category ‘reversible sequence’ The game is to select the correct answer from the lower row and drag the picture to the gap in the upper row

14. ICCHP 200614 More example t-stories character sequence type c reversible sequence type rs

15. ICCHP 200615 temporal sequence type ts simple narrative sequence type ns

16. ICCHP 200616 L ongitudinal study ► 12 children (10 ASD) ages 5 - 11 years ► series of 12 visits ► 50 sequences used; simple graphics, photos, picture story books ► about 12 t-stories / child / visit, replacement rate about 20% ► later visits, the sequences seen were tailored to each individual child using an adaptive formula Children given help or autonomy as appropriate to the child.

17. ICCHP 200617 Study Phases Phase 1: information gathering phase  visit 1 or 2 picture-story book narrative comprehension task  visit 1 and 3 -7 using TouchStory, no individual adaptation Phase 2: adaptive phase  visits 8 -12 using TouchStory with the t-story set adapted to the individual child  (visit 11 or 12 final story-book task)

18. ICCHP 200618 Adaptation ► Uses a simple adaptive formula  the issue is to identify what this particular child finds difficult  a predefined ordering of difficulty cannot be assumed ► Aim of adaptation is to provide more exposure to ‘difficult’ t-stories, providing a challenging but still enjoyable game

19. ICCHP 200619 Expectations ► Levelling out of performance during phase 1 ► Initial decline in overall performance during phase 2 as higher proportion of difficult t-stories are introduced ► Followed by improvement to above previous level if learning occurs

20. ICCHP 200620 Initial observations (1) Child ch1 e.g Children in this group  have no trouble with character and background sequences  do least well with TEMPORAL sequences  have narrative comprehension task profiles in the range 3 – 6.25 out of a possible 20

21. ICCHP 200621 Initial observations (2) Child ch3 e.g Children in this group have a less differentiated profile are most successful with character and background do least well with NARRATIVE sequences have narrative comprehension task profiles in the range 0.5 – 2.5 out of a possible 20 children ch2 and ch3 do not have autism

22. ICCHP 200622 Initial observations (3) Child ch5 e.g Children in this group do very well with TouchStory have narrative comprehension task profiles in the range 8.5 – 18.5 out of a possible 20

23. ICCHP 200623 Conclusions ► TouchStory did distinguish among aspects of narrative; some aspects of narrative were found easier than others  In general the children were more successful choosing characters, backgrounds, and size of character than with reversible, temporal or narrative sequences  The relative difficulty did vary from child to child

24. ICCHP 200624 Relevance to real world ► The children still enjoyed TouchStory on the last visit; some became increasingly engaged and confident. ► There is a significant correlation between the average number of t-stories answered correctly per visit and the picture-story narrative comprehension task scores ► There was a great deal of difference among the children in their narrative ability, productive language, and understanding of the task ► No evidence of enhanced understanding of narrative: a longer study needed?

25. ICCHP 200625 What Next? ► New version of TouchStory with automatic adaptation ► Another longitudinal study with longer adaptive phase; does learning take place ► Issues include  the focus and latency of the adaptive formula,  the iconic range and variety of transition types of the proto ‑ narratives ► Methods of enquiry must remain indirect e.g. observation

26. ICCHP 200626 Thank you ► Our goal is ambitious, we do not expect to provide any quick ‘cure’ for the narrative deficit found in autism ► Even small steps are worthwhile.