Vocabularies for Description of Accessibility Issues in MMUI Željko Obrenović, Raphaël Troncy, Lynda Hardman Semantic Media Interfaces, CWI, Amsterdam

Introduction What (output) modalities are most suitable in which situation? How should different (output) modalities be combined? Using rich accessibility description of MM UIs to answer these two questions

Motivation Expressing human functionality and anatomical structures required by modalities –Capturing existing knowledge –Reasoning over modalities Limited scope, small non-standard vocabularies –Card, Mackinlay, Robertson: Morphological analysis of input device; –Modality theory, Niels Ole Bernsen: Based on taxonomy of output modalities; Linguistic, analog, arbitrary, static, media…

Describing Accessibility Issues in Multimodal User Interfaces MM user interfaces - systems that communicate a message, an effect, –Stimulating a particular human functionality or anatomical structures Interaction constraints –Influence of various factors on human anatomical structures and functionalities.

Describing Accessibility Issues in Multimodal User Interfaces Modality User constraints Social constraints Environment constraints Device constraints Accessibility issues (interaction constraints) Multimodal issues Computer Human Sensing Perception Cognition Motor skills Linguistic skills

Interaction Modalities Interaction Constraints Interaction Context described in terms of Reasoning Framework define queryresult Applications described in terms of Vocabularies

WHO Resources Bioinformatics Additional concepts Reasoning Framework queryresult Applications Interaction Modalities Interaction Constraints Interaction Context described in terms of define described in terms of

Vocabularies: ICF 1/2 WHO International Classification of Functioning, Disability and Health (ICF): –2001, 9 years revision, widely used in health community Describing “the person in his/her world“ –Applicable to all people, whatever their health condition –Carefully designed to be relevant across cultures as well as age groups and genders –Uses neutral terms (function vs. disease) –Allows for an assessment of the degree of disability

Vocabularies: ICF 2/2 Around 1500 concepts: –Body functions and structure –Activities (tasks and actions by individual) and participation (involvement in a life situation) –Environmental factors Problems with formalization –Not defined by knowledge engineering experts Formalized ICF Checklist: 180 core concepts Possibility to reuse millions of profiles and statistical data expressed by ICF!

Vocabularies: Anatomy Foundational Model of Anatomy (FMA) –Detailed description of human anatomy –Open source and available for general use –Used in bioinformatic community –~ concepts –Available in OWL format –Problems: size

Vocabularies: Interaction Effects Some examples: –Gestalt visual grouping By similarity, motion, texture, symmetry, proximity, parallelism, closure, good continuation –Gestalt visual highlighting By color, polarity, brightness, orientation, size, motion, flicker, depth, shape –3D cues Visual: stereo vision, motion parallax, linear perspective, rel. size, shadow, familiar size, interposition, horizon Audio: inter-aural time/intensity difference, HRTF, head movement, echo, attenuation of high frequencies

Vocabularies WHO Resources Bioinformatics Additional concepts Reasoning Framework queryresult Applications define described in terms of Interaction Context described in terms of Interaction Modalities Interaction Constraints

Example Modality: Speech

Example Constraint: Noise

Interaction Constraints Interaction Context Device profile User profile Environment profile User interface described in terms of Vocabularies WHO Resources Bioinformatics Additional concepts Reasoning Framework queryresult Interaction Modalities define described in terms of Applications

Reasoning: Using Descriptions Possible with rich and explicit descriptions of (implicit) accessibility requirements What modalities are suitable in which situation? –Combining descriptions of constraints and modalities (speech in noisy environment?) How should different modalities be combined? –Combining modality descriptions to identify conflicting requirements (speech & short term memory);

Implementation In early stage Using Semantic Web technologies: –OWL, RDF standards –Tools and database to support reasoning Sesame, Jena…; –Reusing some of our and existing tools for data visualization and exploration Facet browsing

Conclusions Expressing human functionality and anatomical structures required by modalities Relations with other projects: –W3C Multimodal Interaction Framework - content –W3C Accessibility Initiative (WAI) - guidelines Future work: –Full implementation of reasoning framework –Solving problems with vocabularies: No relations among concepts, overlapping –New vocabularies are coming: The Human Brain Project (with FMA) DARPA Digital Human – unifying medical ontologies