A Multisensory and Multimodal Game for Children with Cochlear Implants Maria Carmela Sogono Supervisor: Dr. Deborah Richards ITEC810 Information Technology Project Department of Computing, Macquarie University

Agenda Problem Related work Methodology Outcomes – Design principles – Instructional design – Game evaluation Conclusion 2

Problem Even with hearing aids, quality is still not the same as with normal hearing Can affect brain development in children Problems in areas such as sound localisation Can a multimodal, multisensory game be used to test, train, and improve sound localisation acuity? Photo source: 3

Why focus on children? 35:10000 children Normal hearing is important for brain development 35:10000 children Normal hearing is important for brain development MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMM MMMMMMMMMM 4

Why focus on SSD? There is less help available for children with SSD. 5 Photo source:

What is sound localisation? The ability to identify where sound has come from Photo source: 6

Baha Processes sound from the impaired side and sends it to the hearing ear Bone anchored hearing aid 7 Photo source:

Would it be possible to improve sound localisation acuity for children with SSD and Bahas ? 8

How? 9 Games Multisensory Multimodal To appeal to the target audience To make it easier to track, collect, then analyse data Supported by perceptual motor training theory Supported by technological advancements

Can we design games to improve sound localisation acuity for children with SSD and Bahas ? 10 Would it be possible to improve sound localisation acuity for children with SSD and Bahas ?

Relevant work Research studies testing sound localisation acuity Existing sound localisation games developed at Macquarie University Other games developed for children with SSD Research studies testing sound localisation acuity Existing sound localisation games developed at Macquarie University Other games developed for children with SSD 11 Photo source:

Methodology 1. Review of related literature 12

Methodology 2. Analysis of existing work 13

Methodology 3. Development of design principles 14

Methodology 4. Development of instructional design template 15

Methodology 5. Development of game evaluation guidelines 16

Design principles 1.The learning environment shall be realistic. 2.The stimulus shall be meaningful. 3.The game shall start at an easy level. 4.When the game level of difficulty increases, the increase shall be incremental. 5.The game shall incorporate motivational elements. 6.The game shall provide feedback after each trial. 17

7.The game shall propose a challenge. 8.The game shall make use of fantasy. 9.The game shall incorporate sensory and cognitive curiosity. 10.The game shall adapt to the user’s progress. 11.The game shall allow for the repetition of tasks. 12.The game shall track user data. 13.The game shall use bright colours. 18 Design principles

14.The game shall be visually appealing to the target audience. 15.The game shall make use of high-quality audio. 16.The game shall allow for sound to originate from at least 2 distinct sources. 17.A non-accented, trained voice shall be used. 18.The game shall make use of multiple modalities. 19 Design principles

Interaction modalities ModalityDescriptionTechnology VisionVisual perceptionMonitor Touchscreen Projector & screen 3D goggles AudioHearingSpeakers HapticDeals with the sense of touch; currently only used to provide pressure Joystick Wand Action, manipulation Directed by limbs and bodyKeyboard Mouse Gamepad Joystick Wii controller Kinect Touchscreen 20

Instructional design template 21 Title Description Objectives  To improve skills in localization acuity through training  To track user progress whilst training Game Style Game genres:  Puzzle (Hidden object, reveal the picture)  Action (Maze game, rhythm action game, shooter games)  Strategy (Time management)  Adventure Background Initial Setup Game Model Four general models of game flow:  Sound identification  Sound evasion  Sound tracking  Sound training

Instructional design template 22 Success Condition Fail Condition Challenges There are several ways to increase difficulty:  Include increasing instances of the stimulus  Decrease angular distance between the instances  Include background noises  Include additional elements onscreen to increase distraction  Decrease stimulus visibility (for example, it is hiding behind another object)  Include a time limit  Include a “number of times” limit Additional Interaction

Sample game modules 23 Bug Killer Treasure Hunt Wild Animal Evasion Dance Beat Shooter, sound identificatio n Adventure, sound tracking Maze, sound evasion Rhythm action, sound training

Game evaluation Game design Usability Content 24 “Does the game follow design principles and best practices ?” “Is the game usable ?” “Does the game teach what it’s supposed to?”

Game design evaluation 25 Module Name: Does the module have…Rating …a realistic learning environment? Explanation: …meaningful stimulus? Explanation: …motivational elements? Explanation: …feedback? Explanation:

Game design evaluation 26 …a challenge? Explanation: …elements of fantasy? Explanation: …adaptation to the user’s progress? Explanation: …repetition of tasks? Explanation:...multiple modalities? Explanation:

Game modules evaluated 27 Bug Killer Treasure Hunt Wild Animal Evasion Dance Beat

Conclusion 28

Augmented reality Multiplayer games Haptic, large scale mats Games for adults 29 Recommendations

Thank you! For questions, contact 30