Irek Defée Signal Processing for Multimodal Web Irek Defée Department of Signal Processing Tampere University of Technology W3C Web Technology Day

Irek Defée Current status Web is developed for traditional data and computer I/O: text, keyboard, mouse This is simple and effective but not a natural way of human interaction with the world Humans interact via perceptual system

Irek Defée Human Perceptual System Human perceptual system has multiple senses: visual, acoustical, haptic (touch, body position, temperature) and actuators (vocal tract, muscles, motoric system) The perceptual system is intrinsically MULTIMODAL: multiple senses and actuators operate in perfectly coordinated way

Irek Defée Perceptual Information Technology Information technology is evolving towards natural MULTIMODAL human interaction:  Touch gestures revolutionized mobile devices  Intelligent speech input is available  There is more to come: new sensors, cameras and intelligence

Irek Defée Signal Processing Role Perceptual Information Technology requires sophisticated signal processing and it is hard due to: - Complex input signals - Complex information encoding - Complex databases of knowledge Highly sophisticated algorithms and huge processing power are required

Irek Defée Multimodal Web The trend towards perceptual information is noted at the W3C: Extending the Web to allow multiple modes of interaction: GUI, Speech, Vision, Pen, Gestures, Haptic interfaces,... Multimodal Interaction Activity: - Multimodal Architecture and Interfaces - EMMA - InkML - EmotionML

Irek Defée Multimodal Architecture

Irek Defée EMMA Extensible Multimodal Markup Language for Annotations - containing and annotating the interpretation of user input - transcription into words of a raw signal, for instance derived from speech, pen - interpretation is to be generated by signal interpretation processes, such as speech and ink recognition, semantic interpreters

Irek Defée Ink Markup Language data format for representing ink input and processing of handwriting, gestures, sketches, music using traces of pen Trace attributes

Irek Defée Emotion Markup Language Annotation of material involving emotionality Automatic recognition of emotions from sensors Generation of emotion-related system responses: speech, music, colors, gestures, synthetic faces Emotion vocabularies and representations: <emotion category- set=" voc/xml#big6"> <category name="surprise" confidence="0.9

Irek Defée Department of Signal Processing Signal processing has a key role as a front-end for the Multimodal Web Department is on the forefront of research in the natural information processing: - Multimedia information analysis, retrieval and databases - Audio information analysis : speech and music - Media information handling: representation and compression

Irek Defée for your attention!