A MBI L EARN Ambient Intelligent Multimodal Learning Environment for Children 100 day review December 2008 Jennifer Hyndman Supervisors: Dr. Tom Lunney, Prof. Paul Mc Kevitt School of Computing and Intelligent systems Faculty of Computing and Engineering University of Ulster, Magee. 1

O VERVIEW Research Objectives Multimodal Interfaces HCI ~ Child Computer Interaction Games in Education Project Proposal Software Analysis & Prospective Tools Relation to other work Project Plan Conclusion 2

R ESEARCH O BJECTIVES Design and Develop AmbiLearn Means to merge multiple modalities Mobile Devices e.g. Smart Phones (Nokia) /Personal Digital Assistant (PDA) Interface Design /Usability AmbiLearn Educational game e.g. Treasure hunt / puzzels Demonstrate and Test AmbiLearn 3

M ULTIMODAL I NTERFACES Natural communication methods Bolt’s Media Room (Bolt, 1980) SmartKom (Wahlster et al. 2001) Spoken dialogue systems Speech Embodied Agents GandalfSteveREA (Thórisson, 1997) (Johnson et al. 1997) (Cassell et al. 1999) 4

HCI ~ C HILD C OMPUTER I NTERACTION Child centred design Dexterity/speech/interaction style/background/reading (Bruckman et al. 2007) Adaptive and appropriate feedback / cognitive strategies (Rosas et al. 2003) Augmented Reality Story Telling Collaboration / Educational Networking e.g. Web 2.0 G AMES IN EDUCATION 5

P ROJECT P ROPOSAL AmbiLearn PDA based Multimodal input e.g. Speech / 2D gesture (touch screen) Multimodal Output e.g. Graphics / Embodied Agent / speech Educational Game e.g. Treasure hunt / puzzels Designed for children Collaborative learning / problem solving 6 Speech 2D Gesture Speech Graphics Embodied Agent

S OFTWARE A NALYSIS & P ROSPECTIVE T OOLS Platforms Psyclone COLLAGEN Spoken dialogue systems CSLU Toolkit Microsoft Speech Embodied Agents X-Face Microsoft Agent Languages C++ Java Visual Basic Web Tops Glide Nivido eyeOS PDA e.g. Compaq iPAQ 7

R ELATION TO OTHER WORK SystemYearDeviceInput ModalityOutput modalityEmbodied Agent Support collaboration Educational Domain Desktop PCPDASpeechTouchvisualSpeechGraphicsText Gandalf1997  Solar system Steve1997  Virtual Procedural Training KidsRoom1998  Imaginative Story Telling Storyteller1999  Story Telling TICLE2002  Tangram Puzzle Read-It2004  Reading and Language NICE2005  Life and Works of Hans Christian Anderson Virtuoso2006  Art History Augmented Knights Castle 2007  History (Middle Ages) AmbiLearn2011  Treasure Hunt / Puzzles 8

R ESEARCH P LAN Research Activities Oct-DecJan-MarApr-JunJul-SepOct-DecJan-MarApr-JunJul-SepOct-DecJan-MarApr-JunJul-Sep Perform literature review 100 Day Review and VIVA Confirmation report 2 nd Year Poster 3 rd Year Presentation ISEA 2009 conference AICS 2009 conference Develop AmbiLearn Test and Evaluate Thesis write up Submit Thesis Activities Submissions Deliverables 9

C ONCLUSION Advance knowledge Multimodal mobile games in education Multimodal interfaces on mobile devices Collaborative learning with mobile devices AmbiLearn Ambient intelligent Multimodal Learning Environment for children Multimodal communication supporting collaborative learning using mobile devices 10

T HANK YOU ! Q UESTIONS ? 11