1. Semantic Web Services for Smart Devices in a “Global Understanding Environment” () Semantic Web Services for Smart Devices in a “Global Understanding Environment” (SmartResource) Vagan Terziyan Industrial Ontologies Group Agora Center, University of Jyväskylä HCISWWA, November 7, 2003, Catania (Sicily), Italy http://www.cs.jyu.fi/ai/OntoGroup/index.html

2. SmartResource: HCISWWA -2003 Presentation 2 of 40 ContentContent Resources in Semantic Web and Beyond Global Understanding Environment Resource Adaptation Remote Diagnostics of Resources Resource Maintenance and Networking

3. MAIN RESEARCH OBJECTIVE Our intention is to make “resources” (Web documents and services, industrial devices, human experts, etc.) active in a sense that they can analyze their state independently from other systems and applications, initiate and control own maintenance proactively. Resource state can provide knowledge about resource condition, whereas both resource condition and goal of the resource will result in certain behavior of active resource towards effective and predictive maintenance.

4. SmartResource: HCISWWA -2003 Presentation 4 of 40 Self-maintenanceSelf-maintenance Do not expect that someone cares about you, take care yourself even if you are just an industrial device ! proactiveYou should be proactive enough to “realize” that you exist and want to be in a “good shape”; sensitiveYou should be sensitive enough to “feel” your own state and condition; smartYou should be smart enough to “understand” that you need some maintenance.

5. SmartResource: HCISWWA -2003 Presentation 5 of 40 Resource Agents 2. “Yeah, your condition is not good. You need urgent help” 1. “I feel bad, temperature 40, pain in stomach, … Who can advise what to do ? “ 3. “Hey, I have some pills for you” Resource agents Resource agents are intelligent “supplements” of various resources. They represent these resources in Semantic Web-enabled environment and interoperate, realizing resource’s (pro-)active behavior

6. SmartResource: HCISWWA -2003 Presentation 6 of 40 Industrial Resources Classes of resources in maintenance systems: Device - machines, equipment, etc. Processing Unit – embedded, local and remote systems, for monitoring, diagnostics and control over devices Human (Expert) – users of the system, operators, maintenance experts

7. SmartResource: HCISWWA -2003 Presentation 7 of 40 Research Challenges Resource Adaptation and Interoperability (Semantic Web )  Unify data representation for heterogeneous environment  Provide basis for communication Resource Proactivity (Agent Technology)  Design of framework for delivering self-maintained resources to industrial systems Resource Interaction (Peer-to-Peer, Web Services technologies)  Design of goal-driven co-operating resources  Resource-to-Resource communication models in distributed environment (in the context of industrial maintenance)  Design of communication infrastructure

8. SmartResource: HCISWWA -2003 Presentation 8 of 40 GUN Concept Global Understanding eNvironment

9. First Slice of Gun Architecture RESOURCE ADAPTATION

10. SmartResource: HCISWWA -2003 Presentation 10 of 40 TargetsTargets A generic resource-access mechanism (semantic adapter) for devices, diagnostic services and humans An environment for remote access and resource browsing via semantic-based communication interface

11. SmartResource: HCISWWA -2003 Presentation 11 of 40 Diversity of Resources GUN (Global Understanding eNvironment) concept considers notion of resource in a very general sense. Types of resources that can be integrated into GUN are not limited only to digital documents and database content. Real-world objects can be also represented as resources capable, for example, to accept and respond to queries, interact with other resources in order to achieve own goals. Generic GUN- resource

12. SmartResource: HCISWWA -2003 Presentation 12 of 40 Generic Resource Adapter The integration requires development of the Generic Resource Adapter, which will provide basic tools for adaptation of the resource to Semantic Environment. It should have open modular architecture, extendable for support of variety low- and high-level protocols of the resources and semantic translation modules specific for every resource (e.g. human, device, database). Generic Resource Adapter must be configurable for individual resource. Configuration includes setting up of communication specific parameters, choosing messaging mechanism, establishing messaging rules for the resource and providing a semantic description of the resource interface. GUN- resource Communication-specific connector of a resource Resource-specific messaging Semantic “wrapping” of resource actions; translation of external messages into resource-native formats Connectivity Layer Semantic Layer GUN environment Generic Adapter configuration Messaging Layer

13. SmartResource: HCISWWA -2003 Presentation 13 of 40 Semantic adapter for Devices API Semantic environment If to consider field devices as data sources, then information to be annotated is data from sensors, control parameters and other data that presents relevant state of the device for the maintenance process. Special piece of device-specific software (Semantic Adapter) is used for translation of raw diagnostic data into standardized maintenance data based on shared ontology. Shared ontology Adapter Semantic message Device-specific calls

14. SmartResource: HCISWWA -2003 Presentation 14 of 40 Semantic adapters for Services Semantic environment The purpose of Service Semantic Adapter is to make service component semantic web enabled, allowing communication with service on semantic level regardless of the incompatibility on protocol levels, both low-level (data communication protocol) and high-level (messaging rules, message syntax, data encoding, etc.). Shared ontology Adapter Semantic message Service- specific calls

15. SmartResource: HCISWWA -2003 Presentation 15 of 40 Semantic Adapters for Human-experts Human in the system is an initiator and coordinator of the resource maintenance process. The significant challenge is development of effective and handy tools for human interaction with Semantic Web-based environment. Human will interact with the environment via special communication and semantic adapter. User interface Human GUN- resource Action translated into semantic message Semantic message that will be visualized Shared ontology

16. Second Slice of Gun Architecture REMOTE DIAGNOSTICS

17. SmartResource: HCISWWA -2003 Presentation 17 of 40 GoalsGoals Development of remote diagnostic model with  semantic-based communication  expert (human) and diagnostic (Web) service  with learning capabilities

18. SmartResource: HCISWWA -2003 Presentation 18 of 40 Device: local platform Device is a sample of a device, which state is to be automatically annotated with “diagnosis”. It is supplied with Local Platform, which contains Local Alarm Service and History Data Storage. “History Data Storage” “Device” “Local Alarm Service” Local Platform Device state data Remote Diagnostic  Local Alarm Service is a local device-specific algorithm capable to detect alarm states of the Device  History Data is collected by Device via the maintenance ontology for history data representation

19. SmartResource: HCISWWA -2003 Presentation 19 of 40 Services (are able to learn) Learning sample Diagnostic model Labelled history data “Service” Service is a standalone diagnostic algorithm capable to learn Diagnostic (Classification, Prediction) Model of an expert based on labelled history data about the device state.

20. SmartResource: HCISWWA -2003 Presentation 20 of 40 Device – Expert : interactions “Expert” “Device” Querying diagnostic results Labelled data Watching and querying diagnostic data Labelled data History data  Accepts semantic description of device state and can respond with classification label (semantic description of diagnosis)  Can make semantic query to request device-state data (also labeled history data), get response from Device and provide own label for observed device state Expert :

21. SmartResource: HCISWWA -2003 Presentation 21 of 40 Device – Service : interactions  Service presents to a Device possibility to use it as a tool for self-diagnostics.  If classification model has to be built first (no model yet) than perform learning: Service accepts semantic description of device state from a Device and responds with classification label obtained using existing learned classification model  Request data required for learning using semantic query  Build (via a machine learning technique) a classification model  Notify Device about readiness to perform diagnostics

22. SmartResource: HCISWWA -2003 Presentation 22 of 40 Device – Service, learning “Service” “Device” Querying data for learning Diagnostic model Learning sample Labelled data History data Learning process: creation of the Diagnostic Model

23. SmartResource: HCISWWA -2003 Presentation 23 of 40 Device – Service, servicing “Device” Querying diagnostic results Labelled data “Service” Diagnostic model History data Labelled data

24. SmartResource: HCISWWA -2003 Presentation 24 of 40 System structure “Expert” “Service” Labelled data Diagnostic model Querying diagnostic results Labelled data Watching and querying diagnostic data Labelled data History data “Device” Querying data for learning Learning sample and Querying diagnostic results Simple remote diagnostic model with semantic-based communication, expert and diagnostic service with learning capabilities.

25. Third Slice of Gun Architecture MAINTENANCE NETWORKING

26. SmartResource: HCISWWA -2003 Presentation 26 of 40 NetworkingNetworking

27. SmartResource: HCISWWA -2003 Presentation 27 of 40 GoalsGoals Develop network infrastructure for resource maintenance system; Support global experience reuse; Support automated search of potential partners for services and resources (devices); Support collaborative resource diagnostics by multiple services and servicing multiple resources by one service.

28. SmartResource: HCISWWA -2003 Presentation 28 of 40 P2P networking - highly scalable - fault-tolerable - supports dynamic changes of network structure - does not need administration Why to interact? resource summarizes opinions from multiple services service learns from multiple ”teachers” one service for multiple similar clients resources exchange lists of services services exchange lists of clients

29. SmartResource: HCISWWA -2003 Presentation 29 of 40 Notice boards Service 1 Service 2 Service 3 Client 1 Client 2 Client 3 Component advertisement solution Allows search for new partners Source of new entry points into P2P network Allows automated search based on semantic profiles

30. SmartResource: HCISWWA -2003 Presentation 30 of 40 P2P semantic resource discovery P2P network formation through Notice Boards; Search for necessary partners in P2P network according to their semantic descriptions; Establishment of additional P2P links via exchanging addresses between partners;

31. SmartResource: HCISWWA -2003 Presentation 31 of 40 Discovery: sample scenario Number of queried peers is restricted due to: superhub based structure; query forwarding mechanism based on analysis of semantic profile; Resource Service Matched service Wrong service Response Query propagation

32. SmartResource: HCISWWA -2003 Presentation 32 of 40 Learning and test sample. Querying diagnostic results. Devices: multiple services “Service” “Device” Labelled data Learning sample Test sample “Service” Diagnostic model w1w1w1w1 w2w2w2w2 w3w3w3w3 w4w4w4w4 w5w5w5w5 Evaluation and Result integration mechanism … Labelled data Device will support service composition in form of ensembles using own models of service quality estimation. Service composition is made with goal of increasing diagnostic performance.

33. SmartResource: HCISWWA -2003 Presentation 33 of 40 Services: multiple devices “Service” Diagnostic model …“Device” Labelled data “Device” “Device” …“Device” “Device” “Device” 1 n Device-specific diagnostic model Device Class-specific diagnostic model Service builds classification model; many techniques are possible, e.g.:  own model for each device  one model from several devices of same type (provide device experience exchange)

34. SmartResource: HCISWWA -2003 Presentation 34 of 40 Results of Networking Decentralized environment that integrates many devices, many services, many human experts and supports : Establishment of new peer-to-peer links through NoticeBoards, advertisement mechanism Semantic based discovery of necessary network components Service Interaction ”One service – many devices” Interaction ”One device – many services” Exchange of contact lists between neigbor peers

35. SmartResource: HCISWWA -2003 Presentation 35 of 40 Device-to-Device “opinion” exchange Device Device 1 Device 2 Service 1 Service 2 trust = 100 trust = 2 6 1 ? ? 4 8 Device will be able to derive service quality estimates basing on analysis of ”opinions” of other devices and trust to them. Service quality evaluations

36. SmartResource: HCISWWA -2003 Presentation 36 of 40 Service-to- Service “model” exchange and integration Diagnostic models exchange Diagnostic models integration entails creation of a more complex model extension or a service with new diagnostic model

37. SmartResource: HCISWWA -2003 Presentation 37 of 40 CertificationCertification 5 3 4 Certifying party Device Service 1 Service 2 Service 3 6 1 2 Own evaluations Support for certification authorities in the network. Certificates gained by services will be used by devices for optimal service search and selection. Device makes its decision taking into account also its own service quality evaluations. trust

38. SmartResource: HCISWWA -2003 Presentation 38 of 40 Maintenance “executive” services Device Service Control Support for maintenance services that can influence on device state and perform maintenance actions upon it (automated control system, maintenance personnel). They complete the minimal working set of maintenance system components. data diagnosis control

39. SmartResource: HCISWWA -2003 Presentation 39 of 40 Business Models Certifying party Device Service Noticeboard owner ? New players are possible 1-day advertisement = 300 € certification = 3000 € service cost = 10€/hour 1000 new service addresses = 40€ opinion cost = 80€ expert support = 40€/hour service teaching = 45€/min search service = 80€/item platform hosting = 5€/day platform package = 3000€

40. SmartResource: HCISWWA -2003 Presentation 40 of 40 Concluding Remark Among recent initiatives aimed at development of adoption of open information standards for operations and maintenance and implementation of interoperable cooperative industrial environments are: MIMOSA (Machinery Information Management Open System Alliance)[1]. The project consortium pretends to build an open, industry-built, robust Enterprise Application Integration and condition-based maintenance specifications.[1] PROTEUS[2], funded by industrial companies and led with a goal to develop a generic maintenance-oriented platform for industry.[2] These initiatives are very expensive, labor and resource consuming, and still does not attempt to apply and benefit from the Semantic Web technology. We believe however that without comprehensive metadata description framework, ontologies and open knowledge/semantics representation standards their results will be just next consortium-wide standards, rather than comprehensive, flexible and extensible framework. [1] http://www.mimosa.org/ [1] [2] http://www.proteus-iteaproject.com/[2]