1. The Concept of Connecting Ontology and its Exploitation for Knowledge and Information Presentation for People with Special Needs Muhammad Shuaib Karim 1,2 1 Institute of Software Technology and Interactive Systems Vienna University of Technology (http://www.ifs.tuwien.ac.at/)http://www.ifs.tuwien.ac.at/ 2 Computer Science Department Quaid-i-Azam University, Islamabad (http://www.qau.edu.pk/)http://www.qau.edu.pk/ October 09 / 2007 Technisches Universität Wien

2. 2 Motivation Providing a generic accessibility solution for SemanticLIFE using the Semantic Web Technology

3. 3 Semantic Web – Promises ●Ability to integrate heterogeneous data sources ●Ability to formally describe the information  Formal data description makes it understandable, sharable and thus processable by software agents  Automatic reasoning becomes possible ●Abundance of OS tools from modeling, storage, annotation, reasoning, & query to user interfaces

4. 4 Semantic Web - Architecture

5. 5 Semantic Web – Resource Description (1/5) ●Every Thing can be thought of as an Entity ●The Entity is described as a Resource ●The Resource may range from literal to concept map, and has formal Description ●The Resource is identified by a URI ●The Description has the consensus within CoP

6. 6 Semantic Web – Resource Description (2/5) ●The Description is asserted in terms of ( Subject, Predicate, Object ) triples where each of them is a Resource ●For example, 

7. 7 Semantic Web – Resource Description (3/5) In principle, every “piece of information” ●can be conceptualized in terms of inter-relation of entities – Ontological (Philosophical) Level, AND ●has schema (ontology) described in terms of chains of triples as well as its instances (individuals) – Developer or Content Author Level, AND ●has implementation for those chains of triples based upon strong theoretical formalism of Description Logic – Implementation Level  Allows Reasoners to make inference  Deduce relationships such as containment, symmetrical, transitive, inverse relations

8. 8 Semantic Web – Resource Description (4/5) ●So, Semantic Web provides a platform from Conceptual Modeling to Implementation with added inference capabilities ●Not necessary to implement all of it for getting started. “Even a little semantics go a long way – James Hendler”

9. 9 Semantic Web – Resource Description Example (5/5) John is working on tasks “Ontology_Templates´´ and “Message_Bus´´. The task “Ontology_Templates´´ is for project “DynamOnt´´ and the task “Message_Bus´´ is for project “SemanticLIFE´´ Person John Shuaib Task Ontology_Templates Message_Bus Project DynamOnt SemanticLIFE workingOnTask partOf workingOnProject

10. 10 SemanticLIFE - Architecture (Ahmed et al., 2004). SemanticLIFE - A Framework for Managing Information of A Human Lifetime. In Austrian Computer Society Book Series: iiWAS'04 Proc., (vol.183), pages 687-696.

11. 11 Concept of SemanticLIFE UI

12. 12 Accessibility – Key Issues ●Not only an issue for people with popular disabilities and specific age group, rather it is Acc4All ●Not only related with user interface issues, rather applicable to whole system

13. 13 Accessibility – Web Accessibility Components

14. 14 Research Questions ●Can Semantic Web Technology be used for providing a Generic Accessibility solution? ●How far is it true that the investment on UI alone could provide maximum possible Accessibility? ●Is the sought-after approach exploitable towards diversity in general, and integration of Information Management Systems in particular?

15. 15 Sample Components Requiring Connections User Profile ↔ Assistive Technology Assistive Technology ↔ User Agents User Agents ↔ Contents …. User Interfaces ↔ User Profile Info. Semantics of Domain1 ↔ Info. Semantics of Domain2 Encoded in software or in data → based upon some rules → ? Connection of heterogeneous domains =>

16. 16 Significance of Connections (Ex. Food & Wine) Suggests different foods and wine combinations → Domain expert´s knowledge encoded as OWL DL, such as; taste appetite or stimulation to eat digestion availability

17. 17 Ex. Food & Wine Another Approach (“Connecting Ontology´´)  Ontology of Food – Taxonomy with concepts such as; the calories with respect to the quantity the tendency to get digested on ist own (Quick, Medium duration, Long duration) …  Ontology of Wine - Taxonomy with concepts such as; the appetizing effects (Nil, Low, Medium, High) the digestive effects (Nil, Low, Medium, High) … Above knowledge normally accessible from the available literature

18. 18 Food & Wine (new CQ) New CQ possible to fulfill:  Food requiring no drinks  Food requiring drinks of certain quality & brand  Food & wine combination requiring least digestion time

19. 19 Connecting Ontology - Characteristic Features of O C between O 1 and O 2 ●O 1, O 2 not related with same domains of discourse ●O 1, O 2 developed using their own CODeP ●Similarity b / w CODeP indicate inter-connection ●Creation of new knowledge

20. 20 Connecting Ontology - Benefits ●Useful for top-down evolution of ontologies / applications ●Incompatibilities between the two ontologies are solved at the ontological level ●Helpful in code automation ●Possibility to determine the Cause - Effect relationship between the two ontologies ●Reliance on domain experts reduced due to encoding of domain knowledge The key is “How to capture and represent the domain knowledge ??´´

21. 21 Connecting Ontology - Domain Knowledge Available as;  Structured documents  Unstructured documents  Tacit knowledge with domain experts

22. 22 Connecting Ontology – Capturing Domain Knowledge ●Text processing of knowledge about domains of CO and the two participating ontologies ●Refinement of the above by aligning participating ontologies with global standard ontologies ●Exploiting valid queries on the two ontologies ●Exploiting queries result set and data mining ●Using Semantic Web Rules

23. 23 Connecting Ontology - Workflow

24. 24 Accessibility Plug-in (A scalable framework using Semantic Web Technology) Karim, S., Latif, K., and Tjoa, A. M. (2007). Providing Universal Accessibility using Connecting Ontologies: A Holistic Approach. Universal Access to Applications and Services, In Proc. of HCII’07., LNCS 4556 (vol. 7).

25. 25 CODeP for Accessibility - Generic Pattern with Spatio-Temporal Dimensions

26. 26 CODeP for Accessibility – Simplified Generic Pattern

27. 27 CODeP for Accessibility - Memory Recall Pattern

28. 28 CODeP for Accessibility - Perception Effect Pattern

29. 29 CODeP for Accessibility – Mobility Enhancement Pattern

30. 30 Persistence of Patterns Formal description of semantics for each component => Ontology O i for each component Formal description of consequences and effects of potentially interacting component on each other => Connecting Ontology for O 1 and O 2

31. 31 Test case Connecting Ontology for User’s Impairments & UI Characteristics

32. 32 Role of User Impairments Assistance to be provided = User's Needs - User's Capabilities...(1) Assistance to be provided = User's Needs + User's Impairments needs - User's Capabilities of (1) Quality of Life Technology (QoLT) Ref: Kanade, T. (2007). Digital Human Modeling and Quality of Life Technology. In Keynote address in 12th International Conference on Human-Computer Interaction, Beijing, China.

33. 33 Motivation - Ontology of Visualization Techniques Ref: Chi, E. (2000). A Taxonomy of Visualization Techniques using the Data State Reference Model. In INFOVIS, pages 69-75.

34. 34 Overview of Ontology Interaction for On- Demand Visualization

35. 35 Human Disease Ontology Ref: Hadzic, M. and Chang, E. (2005). Ontology-Based Support for Human Disease Study. In Proc. of HICSS'05, IEEE Computer Society.

36. 36 Extension of Human Disease Ontology Ref: Hadzic, M. and Chang, E. (2005). Ontology-Based Support for Human Disease Study. In Proc. of HICSS'05, IEEE Computer Society.

37. 37 Impairment-User interface Connecting Ontology - Sample user scenarios ●Avoiding the confusing colors on an interface for a user with particular type of color blindness ●Font adjustments according to user‘s visual acuity ●Information presentation on the better part of the screen for a user suffering from Hemianopsia (absence of vision in half of visual field)

38. 38 Impairment-User Interface Ontology Karim, S. and Tjoa, A. M. (2006). Towards the Use of Ontologies for Improving User Interaction for People with Special Needs. In Proc. of ICCHP’06, LNCS (vol. 4061), pages 77-84.

39. 39 Exploring Relationship between Motor Control Impairments and UI Components ImpairmentRelated WithUI Component NormalMotorControlsuggestsComboBox NormalMotorControlsuggestsRadioButton NormalMotorControlsuggestsCheckBox NormalMotorControlsuggestsScrollBar NormalMotorControlsuggestsSpinner NormalMotorControlsuggestsToggleButton VeryLowMotorControlsuggestsAudioFeedback VeryLowMotorControlsuggestsToggleButton VeryLowMotorControlprohibitsSpinner VeryLowMotorControlprohibitsScrollBar

40. 40 Representing Relationship between Motor Control Impairments and UI Components

41. 41 Representing Relationship in DL ●MotorControlImpairment with SeverityMeasure “VeryHigh” suggests UiComponent with MotoricConvenienceMeasure “VeryHigh” ●MotorControlImpairment with SeverityMeasure “Normal” suggests UiComponent with MotoricConvenienceMeasure “Normal” and also;

42. 42 Impairment ontology - competency questions For a given impairment name; ●What is / are the related body parts ? ●What is the impaired side (right, left,...) ? ●What is its severity (on a predefined scale) ? ●What are the perception cues which are affected, and up to what degree (on a predefined scale) ? ●What is the effect of one impairment on another w.r.t. affected perception ?

43. 43 Impairment Ontology

44. 44 Some derived concepts (1 / 2) ●LeftSidedImpairment ●RightSidedImpairment ●BothSidedImpairment

45. 45 Derived concepts (2 / 2) ●AnySidedImpairment

46. 46 User interface Ontology - competency questions ●Find the part-whole relationship of UI components ●Find the attributes of a component and their values (according to predefined usability scale for a normal user in normal conditions) ●For a given attribute, find the related UI components

47. 47 User Interface Ontology

48. 48 Some derived concepts (1 / 2) ●GoodUsabilityComponent ●FairUsabilityComponent ●FairUserControlComponent

49. 49 Rules to Connect Impairments and UI Low perception implies suggesting high usability components (VisualAcuityLow  UI LegibilityGood) Karim, S. and Tjoa, A. M. (2007). Connecting User Interfaces and User Impairments for Semantically Optimized Information Flow in Hospital Information Systems. Journal of Universal Computer Science: In Proc. of I-MEDIA'07 and I-SEMANTICS'07, pages 372-379. (?x rdf:type imp:VisualAcuity) (?x imp:perceptionMeasure imp:Low) (?y rdf:type ui:UiComponent) (?y ui:hasLegibility ui:Good)  (?x eg:suggests ?y).

50. 50 Rules to Connect Impairments and UI High perception implies suggesting fair usability components (VisualAcuityHigh  UI LegibilityFair) (?x rdf:type imp:VisualAcuity) (?x imp:perceptionMeasure imp:High) (?y rdf:type ui:UiComponent) (?y ui:hasLegibility ui:Fair)  (?x eg:suggests ?y).

51. 51 Rules to Connect Impairments and UI High rheumatism implies suggesting easily operatable components (RheumatismHigh  UI UserControlGood) (?x rdf:type imp:Rheumatism) (?x imp:impairmentMeasure imp:High) (?y rdf:type ui:UiComponent) (?y ui:userControl ui:Good)  (?x eg:suggests ?y). CO Workflow

52. 52 Generated RDF triples

53. 53

54. 54

55. 55

56. 56 Visualization and Accessibility Plug-ins

57. 57 Grouping and Aggregation of Items Arranged on a Timeline

58. 58 Consequences of Impairments-user Interface Connecting Ontology Helpful in automatically adapting UI for the user Helpful in deducing the best match of UI characteristics for a user with multiple impairments Possibility to use the ontology for diversity Historical data for studying the cause-effect relationship b/w the impairments and the computer interfaces Useful for rehabilitation purposes Possibility to extract impairment related semantics from user´s information stored in SemanticLIFE repository, and modify the impairments ontology accordingly

59. 59 Concluding Remarks – Contributions (1/5) ●Ontology for Impairments and Usability introduced ●A number of Accessibility CODeP introduced ●The concept of Connecting Ontology introduced ●Demonstration of Semantic Web Rule Layer for developing higher level ontologies ●The Connecting Ontology concept using the ontological rule-based approach paves the way for a generic solution ●Introduced the integration of heterogeneous domains by persisting the tacit knowledge using Connecting Ontology ●Application of the approach in the motivating scenarios

60. 60 Concluding Remarks - Future directions (2/5) ●User testing ●Semantic Web Service for the Accessibility and the Info-Viz Bridge modules ●Instantiating the Impairments and UI ontologies ●Integrating capability measuring tools (MMSE) ●Realization of the remaining CODeP for Accessibility ●Integration for ontology of visualization techniques ●Elevation / Lifting of Connecting Ontology

61. 61 Concluding Remarks – Goals Evaluation (3/5) ●Can Semantic Web Technology be used for providing a Generic Accessibility solution? Generic foundation, not the complete generic solution UI adaptation possible  Visualization toolkits still not ready

62. 62 Concluding Remarks – Goals Evaluation (4/5) ●How far is it true that the investment on UI alone could provide maximum possible Accessibility? Annotation of resources, ontologies of life events, tasks UI & Applications Accessibility

63. 63 Concluding Remarks – Goals Evaluation (5/5) ●Is the sought-after approach exploitable towards diversity 1 in general, and integration of Information Management Systems 2 in particular? 1 Modeling of user specific attributes (impairments) 2 Domain Knowledge, Business rules 2 Motivating scenarios

64. 64 Thanks! http://www.ifs.tuwien.ac.at/~skarim SemanticLIFE Project http://storm.ifs.tuwien.ac.at/http://storm.ifs.tuwien.ac.at/ ASEA-UNINET (Asean European University Network) http://www.uibk.ac.at/asea-uninet/ HEC (Higher Education Commission of Pakistan) http://www.hec.gov.pk/ Quaid-i-Azam University, Islamabad http://www.qau.edu.pk/ Technisches Universität Wien