ICCHP Adaptive Systems Research Group University of Hertfordshire Megan Davis Kerstin Dautenhahn Chrystopher Nehaniv Stuart Powell TouchStory: Towards an Interactive Learning Environment for Helping Children with Autism to Understand Narrative

ICCHP Project Aims Children with autism have a deficit in narrative ability which adversely effects their social comprehension. Our long term goal is to develop interactive systems which improve social comprehension in individual children with autism by enhancing their narrative abilities. This study is concerned with the design of an interactive picture-narrative completion game for children with autism.

ICCHP Autism Children with autism have  impaired social interaction  impaired communication (verbal and non-verbal)  impaired imagination They form a diverse group but share a difficulty in making sense of the world, particularly the social world. Children with autism  generally like repetitive games  may be very routine oriented  may have particular sensory sensitivities  may focus on seemingly irrelevant details (local cohesion)

ICCHP Squirrel Rubbish bin Ref: Story told by Rita Jordan “Oh, look…”

ICCHP Squirrel Rubbish bin “….…. someone has thrown away a perfectly good squirrel.” Ref: Story told by Rita Jordan

ICCHP Narrative and Social Comprehension ► Narrative is fundamental to the perception, creation and communication of meaning in social interaction ► By fitting events into a narrative pattern we construct and inhabit a meaningful social world ► Narrative comprehension involves building an internal representation  Gaps in the narrative exposition must be filled by experiential or semantic knowledge

ICCHP The longitudinal study ► Aim: To investigate primitive aspects of narrative as a pre-cursor to narrative comprehension  Can an adaptive interactive system be used to identify aspects of narrative which individual children find difficult?  Can an adaptive interactive system be used to enhance children’s understanding of narrative?

ICCHP Why use computers? ► Most children with autism enjoy using computers  highly predictable and controllable  naturally monotropic  may be highly visual ► Easy logging, record keeping; timely analysis ► Computers don’t get bored or irritable ► Quicker turn round, children are exposed to more good examples

ICCHP Challenges of designing software systems for children with autism ► Usual methods (interviews, focus groups, collaborative design, etc.) not possible  children may have limited language  they typically find what? why? questions difficult and so answers are not helpful  they are not socially oriented -- collaborative or group based work not appropriate ► Local cohesion and the variability of autism both mean it is difficult to predict what a child will find difficult ► Repetitive behaviour issues

ICCHP Requirements elicitation ► explore the children’s reaction using laminated cards and a replica of the game using a touchscreen ► 18 children (14 ASD), ages 4-9 years, from two schools ► 5 stories from published picture story books Focussed: no extraneous sound or animation, rewards re-enforce the task Designed to minimise meaningless repetition (autonomy/control balance) Expect a desire for repetition so no need to design to keep attention Builds on previous autism and technology research

ICCHP Why a touch screen? ► Children with autism typically have difficulty with remote object references. A touch screen allows direct manipulation – a perceptually direct link between the object and its indication. Why pictures? ► We are concerned with narrative comprehension not ‘literacy’ or reading ability. Why a game? ► Enjoyment and a playful context is central to our approach.

ICCHP Creating proto-narratives ► Primitives identified as  Character  Background or setting  Sequence ► Reversible ► Temporal (may have causality, or be episodic) ► Narrative (involves motivations of characters) ► t-stories (picture narratives and proto- narratives collectively) were prepared

ICCHP TouchStory displaying a t-story from the category ‘reversible sequence’ The game is to select the correct answer from the lower row and drag the picture to the gap in the upper row

ICCHP More example t-stories character sequence type c reversible sequence type rs

ICCHP temporal sequence type ts simple narrative sequence type ns

ICCHP L ongitudinal study ► 12 children (10 ASD) ages years ► series of 12 visits ► 50 sequences used; simple graphics, photos, picture story books ► about 12 t-stories / child / visit, replacement rate about 20% ► later visits, the sequences seen were tailored to each individual child using an adaptive formula Children given help or autonomy as appropriate to the child.

ICCHP Study Phases Phase 1: information gathering phase  visit 1 or 2 picture-story book narrative comprehension task  visit 1 and 3 -7 using TouchStory, no individual adaptation Phase 2: adaptive phase  visits using TouchStory with the t-story set adapted to the individual child  (visit 11 or 12 final story-book task)

ICCHP Adaptation ► Uses a simple adaptive formula  the issue is to identify what this particular child finds difficult  a predefined ordering of difficulty cannot be assumed ► Aim of adaptation is to provide more exposure to ‘difficult’ t-stories, providing a challenging but still enjoyable game

ICCHP Expectations ► Levelling out of performance during phase 1 ► Initial decline in overall performance during phase 2 as higher proportion of difficult t-stories are introduced ► Followed by improvement to above previous level if learning occurs

ICCHP Initial observations (1) Child ch1 e.g Children in this group  have no trouble with character and background sequences  do least well with TEMPORAL sequences  have narrative comprehension task profiles in the range 3 – 6.25 out of a possible 20

ICCHP Initial observations (2) Child ch3 e.g Children in this group have a less differentiated profile are most successful with character and background do least well with NARRATIVE sequences have narrative comprehension task profiles in the range 0.5 – 2.5 out of a possible 20 children ch2 and ch3 do not have autism

ICCHP Initial observations (3) Child ch5 e.g Children in this group do very well with TouchStory have narrative comprehension task profiles in the range 8.5 – 18.5 out of a possible 20

ICCHP Conclusions ► TouchStory did distinguish among aspects of narrative; some aspects of narrative were found easier than others  In general the children were more successful choosing characters, backgrounds, and size of character than with reversible, temporal or narrative sequences  The relative difficulty did vary from child to child

ICCHP Relevance to real world ► The children still enjoyed TouchStory on the last visit; some became increasingly engaged and confident. ► There is a significant correlation between the average number of t-stories answered correctly per visit and the picture-story narrative comprehension task scores ► There was a great deal of difference among the children in their narrative ability, productive language, and understanding of the task ► No evidence of enhanced understanding of narrative: a longer study needed?

ICCHP What Next? ► New version of TouchStory with automatic adaptation ► Another longitudinal study with longer adaptive phase; does learning take place ► Issues include  the focus and latency of the adaptive formula,  the iconic range and variety of transition types of the proto ‑ narratives ► Methods of enquiry must remain indirect e.g. observation

ICCHP Thank you ► Our goal is ambitious, we do not expect to provide any quick ‘cure’ for the narrative deficit found in autism ► Even small steps are worthwhile.