Bridging WSRF services and the EGEE middleware Pedro Andrade - CERN, Geneva

Agenda Digital Libraries and the DILIGENT Project - 45´´ Q&A DILIGENT Process Management Service Example - 10´´ DILIGENT DIS and Keeper Service Example - 15´´ Open Discussion 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions Evolution DLs today Requirements 22/09/2018 DILIGENT N° 004260

Evolution 2005 - … 1996 knowledge management user = producer&consumer digital texts repository + centralized catalogue search service 1996 few to many: One of the main concern is regarding the concept of many in Grid and DL environment. In Grid "many" is about "hundreds" while in DL "many" is "thousands". institution to virtual organisation: Usually this means an evolution from structured to unstructured organisation. The top of the evolution is the concept of community (completely unstructured), in between you have the "virtual Enterprise". So you have a chain Institution>enterprise>organisation>community user = consumer few large institutions many virtual organizations 22/09/2018 DILIGENT N° 004260

DLs today The demand for DLs has recently grown considerably DLs are perceived as a necessary instrument to support communication and collaboration among the members of communities of interest The maintenance and management of a DL requires high investments and specialized personnel, multimedia handling requires high computational resources 22/09/2018 DILIGENT N° 004260

DLs today Many of the organizations that demand the usage of Digital Libraries are small, distributed, and dynamic DLs are often used to support temporary activities: Courses (training courses, technology transfer courses, etc) Exhibitions, Conferences Projects … DL services are required by many application domains: e-Health, e-Learning, e-Government, e-Science Corporate Knowledge Management 22/09/2018 DILIGENT N° 004260

Requirements Live documents New information objects: Pollution map Table summarizing data from millions of satellite measures Graph reporting an analytical trend of certain information extracted from a vast amount of data 22/09/2018 DILIGENT N° 004260

Requirements The creation and handling of these documents require: Access to many different, large, heterogeneous information sources Use of specialized services High computing power Small virtual dynamic organizations usually do not have the instruments for satisfying these requirements Controlled resources sharing is the only possible solution: Information sources Services Computers and storage Technical staff Expectation from the Grid: Can support a cost-effective DL operational model based on transient, flexible, coordinated “sharing of resources” Addresses the main DL architecture requirements: distribution, openness, interoperability, scalability, controlled sharing, availability, security, quality, …. Provides a basic common infrastructure for serving several different application domains Offers high storage and computing capabilities that enable the provision of powerful functionality on multimedia content e.g. images and videos 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions Objective DILIGENT impact Project numbers Partners Activities The grid is the Technical solution because: high computing and storage capabilities for managing and processing a wide variety of information objects controlled sharing of resources 22/09/2018 DILIGENT N° 004260

Objective Develop a Digital Library test-bed infrastructure that allows members of dynamic virtual research organizations to create on-demand transient digital libraries based on shared computing, storage, multimedia, multi-type content and application resources In the bottom line DILIGENT will: Develop a platform for creation, maintenance and delivery of DLs (I.e. DLMS) which will act on the Grid /EGEE /gLite by developing and integrating several component services. Demonstrate the potential of the platform through two distinct user-scenarios. 22/09/2018 DILIGENT N° 004260

DILIGENT Impact The DILIGENT service infrastructure will dramatically change the DL development model used by distributed and dynamic organisations and communities Using DILIGENT, these organisations and communities will be able to setup their own DL: Even if they do not have enough financial and human resources When and for the time they need it Defining their own specific requirements Exploiting Grid-based services Profiting form an vast amount of storage and computational resources Sharing resources in a collaborative and efficient way 22/09/2018 DILIGENT N° 004260

Project Numbers Duration: 36 Months Start/end date: September 2004 – August 2007 Effort: 1024 p/m Cost: 9.546.561 € EU funding: 6.300.000 € 15% Technological development Validation Activities Innovation Activities 24% 61% 22/09/2018 DILIGENT N° 004260

Partners Universities/Research Institutes: FhG/IPSI - Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. SNS - Scuola Normale Superiore di Pisa UNIBAS - University of Basel UoA - University of Athens USG - University of Strathclyde International Organizations: CERN - European Organization for Nuclear Research ESA - European Space Agency Administrative Coordinator: ERCIM - European Research Consortium for Informatics and Mathematics Scientific Coordinator: CNR-ISTI - Institute of Information Science and Technology of the Italian National Research Council Commercial Companies: ENG - Engineering Ingegneria Informatica SpA FAST - Fast Search & Transfer ASA 4D SOFT - 4D SOFT Software Development RAI - Italian National Broadcaster 22/09/2018 DILIGENT N° 004260

Activities Experimentation with user communities Technology integration and developemnt Exploitation and sustainability Feedback to the constituent technologies Technology Integration Design and implementation of the DILIGENT services Testing and integration actvitites Experimentation with User Communities Identify user requirements and validate the DILIGENT testbed Feedback to the constituent technologies Provide feedback to the research communities that are developing the EGEE and other Grid infrastructures as well as the DL community Exploitation, Dissemenation and Sustainability Report the experience acquired and demonstrate the results achieved to a large spectrum of target user communities Analyze and aggregate the DILIGENT target communities, define a business model to produce a sustainability plan Manage the project Collaboration with other projects and initiatives 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions DILIGENT user communities ImpECt user community Arte user community 22/09/2018 DILIGENT N° 004260

DILIGENT User Communities Collect requirements and validate the built technology Earth Science Domain Represented by ESA Goal: Improve accessibility, interoperability and usability of environmental data, models, tools and algorithms Cultural Heritage Domain Represented by SNS and RAI Goal: Stimulate collaborative multidisciplinary scientific research acndosye the test-bed will be demonstrated and validated by two complementary real-life application scenarios: one from the cultural heritage domain and one from the environmental e-Science domain. The first scenario will show how the DILIGENT infrastructure, by simplifying the processing of multimedia artefacts and offering support for collaborative multidisciplinary studies, can contribute to favour faster progress in research and better quality of education, also in research areas traditionally far from the most advanced computing technologies; the second scenario will show how this knowledge infrastructure, by integrating distributed multi-type data sources with specialised data handling services, can improve accessibility, interoperability and usability of environmental data, models, tools and instruments 22/09/2018 DILIGENT N° 004260

ImpECt User Community Data: Services: Use cases: Automatically generated, annotated and relatively large Wide variety of content types (maps, satellite data, etc) Services: Complex, taylored image processing Composed services ... Use cases: Dynamic report generation Organization of conferences Handling of information in case of environmental accidents 22/09/2018 DILIGENT N° 004260

ARTE User Community Data: Services: Use cases: Retrodigitized, manually annotated and relatively small Image, video material (from RAI) and (structured) text Services: Feature extraction for improving retrieval Composed services (processes management) ... Use cases: Multidisciplinary research working on images and videos Courses, Workshop Exhibition Catalogue 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions Design Process Overview Services description 22/09/2018 DILIGENT N° 004260

Design Process User Activities Base Technology Analysis Functional Specification Service Specification Testing Strategy and Plan Unified Process UP is a software engineering process that provides a disciplined approach to assign tasks and responsibilities within a development organization ensures the production of high-quality software that meets the needs of its end-users, within a predictable schedule and budget simplifies the interactions between the different project activities formally identifying the communication models, diagrams, and timing UP allows to correctly manage the project technical life cycle Experimentation Service Design Architectural Specification 22/09/2018 DILIGENT N° 004260

Overview 22/09/2018 DILIGENT N° 004260 From a conceptual point of view the services that we are implementing and that compose the DILIGENT infrastructure are organized in a layered perspective. These services also interact directly with gLite… the picture is only a layered view of the system 22/09/2018 DILIGENT N° 004260

DL Creation and Management 22/09/2018 DILIGENT N° 004260

DL Creation and Management Objective: provide the functionality necessary to aggregate and manage a collection of resources for efficient usage under the control of a Virtual Digital Library system 22/09/2018 DILIGENT N° 004260

DL Creation and Management DILIGENT Information System (DIS): Aggregation and storage of information about DILIGENT resources Provide details information about the resources state through query and subscription interfaces Integration and exploitation of gLite information service Triggered actions and information caching Keeper: Dynamic services deployment Distributed hosting node management Deployed services monitoring DILIGENT Information Service Diligent Information Service (DIS) provides mechanisms for: gathering, storing and supplying information about the resources needed by DILIGENT services, monitoring the resources state information. It allows users and services to discover what resources are part of a Virtual Organization (VO) and to monitor those resources; it provides query and subscription interfaces to arbitrarily detailed resource information; it provides a trigger interface that can be configured to take action when pre-configured conditions are met; it archives information to allow historical query execution. Keeper “The Keeper service has to instantiate the set of services belonging to a VDL and to manage them assuring the characteristics of QoS required by the VDL Definition criteria”. Starting from the above definition, the Keeper is primarily responsible for the creation of a new VDL by instantiating its resources (i.e. Running instances of its services) and by authorizing its users. It must also guarantee the overall set of the VDL functionalities at any time by dynamically relocating resources and periodically checking their status. 22/09/2018 DILIGENT N° 004260

DL Creation and Management VDL Generator (VDL): Users/communities support Dynamic composition and configuration of DLs Dynamic Virtual Organization Support (DVOS): Robust and flexible security framework Advanced authentication and authorization model Notification support Dynamic aggregation of resources and users/groups Broker & Matchmaker (BMM): Optimal resources allocation VDL Generator The VDL Generator Service is the service that enables users/communities to create their own DLs. It allows to define a set of criteria that specify the expected characteristics of the new DL; starting from them, it identifies the set of services required to provide the requested features. In particular, the VDL Generator Service: · Supports users/communities in specifying the criteria that characterize the new DL, e.g. the information space, the required functionalities, the characteristics of these functionalities; · Selects the appropriate pool of services and information sources required to implement a DL that satisfies the specified criteria; · Notifies to the DL Management service (see Section 6.5.2.1) the identified services. DVOS DVOS includes a set of services all tailored to support dynamic VO. The DVOS area covers functionalities related to Resource Management (Section 4.3 of D1.1.1), to VO Management (Section 4.4 of D1.1.1), to Users and Group Management (Section 4.5 of D1.1.1) and to Notification Management (Section 4.6 of D1.1.1). Other nonfunctional requirements covered by DVOS are Authentication and Authorization issues as described in the D.1.1.1 Test-bed Functional Specification. Broker & MatchMaker Service The Broker & MatchMaker Service (BMM) is in charge of supporting the Keeper service in deploying a new Digital Library on a set of DILIGENT Hosting Nodes (DHNs). In particular, once the Keeper has identified the set of packages needed to build a new DL the BMM’s job is to identify a set DHNs to be used as target hosts for the deployment. 22/09/2018 DILIGENT N° 004260

DL Creation and Management 22/09/2018 DILIGENT N° 004260

Content and Metadata Management 22/09/2018 DILIGENT N° 004260

Content and Metadata Management Objective: provide access and storage of information objects, management of related metadata, encryption and decryption of information objects for security reasons, and information object annotations management. 22/09/2018 DILIGENT N° 004260

Content and Metadata Management Content Management Distributed content storage support Advanced replication mechanism Flexible storage model Support for arbitrary object types content, metadata, features, collections, archives, indexes, etc Support for object relationships contains, characterizes, extracted-from, indexed-by, etc Integration of 3rd party data sources (archive migration) Metadata Management Efficient support for arbitrarily structured metadata and transparent support for multiple metadata schemas and mappings 22/09/2018 DILIGENT N° 004260

Content and Metadata Management Content Security Deals with the specialties of protecting multimedia content Include authenticity, integrity and confidentiality Grid-enabled watermarking algorithms for different media types Annotation Provides interactive functionality for the management of manually authored, subjective, and context-dependent metadata items about DILIGENT information objects, primarily multimedia documents. 22/09/2018 DILIGENT N° 004260

Content and Metadata Management 22/09/2018 DILIGENT N° 004260

Content and Metadata Management 22/09/2018 DILIGENT N° 004260

Index and Search Management 22/09/2018 DILIGENT N° 004260

Index and Search Management Objective: design and implement an open, scalable, highly-featured, distributed search engine enhanced by the usage of various types of indices. 22/09/2018 DILIGENT N° 004260

Index and Search Management Index Service Build and consume inverted and forward indices on metadata for search performance enhancement Search Service Locate content by searching in metadata and extracted features Optimisation in search execution Support Full Text, Similarity Based and Exact Match searches Feature Extraction Process documents to extract feature descriptors 22/09/2018 DILIGENT N° 004260

Index and Search Management Personalization Service Provide personalized information retrieval Manage user profiles Content Source Description and Selection Filter content sources to maximize search relevance/cost ratio Automatically generate source description Data Fusion Fuse (merge) the ranked content result sets into a single ranked result set 22/09/2018 DILIGENT N° 004260

Index and Search Management 22/09/2018 DILIGENT N° 004260

Process Management 22/09/2018 DILIGENT N° 004260

Process Management Objectives: Use existing services as building blocks for more complex, value-added workflows (processes/compound services) Dynamic scheduling of service requests Define structured processes to be executed on the Grid Efficient usage of the Grid resources available More Objectives: Use existing services as building blocks for more complex, value-added workflows (processes) Processes themselves are invoked like normal services A process may be composed of „simple“ services and/or other processes Distributed execution of the processes Dynamic scheduling of service requests Efficient use of the resources in the infrastructure Scalability and fault-tolerance Transactions: Support atomic execution of processes Guarantee „all or nothing“ wrt process execution 22/09/2018 DILIGENT N° 004260

Process Management Design and Verification Execution and Reliability Check whether a process is defined correctly Provide a graphical UI for specification and analysis of processes Execution and Reliability Distributed execution process on the Grid Avoid centralized workflow engine - avoid bottleneck Dynamic allocation of resources Automate process Optimisation Structural process modifications to maximise parallelism of execution Optimise Grid resources utilisation 22/09/2018 DILIGENT N° 004260

Process Management 22/09/2018 DILIGENT N° 004260

Application Specific Functionality 22/09/2018 DILIGENT N° 004260

Application Specific Functionality Objective: Integrate DILIGENT elements under an application-specific context by adding all the custom elements expected. It also aims to prove the concept of DILIGENT as a versatile and powerful open DLMS able to capture specific community needs. 22/09/2018 DILIGENT N° 004260

Application Specific Functionality Portal: Hosts the graphical user interface (UI) of the core services provided by the DILIGENT platform Every user interface element is a portlet or a part of a portlet The Portlet Hosting Platform: support for all JSR168 portlet containers currently using uPortal and GridShpere optionally support of Web Services for Remote Portlets (WSRP) Visualization: Provides application-specific visualization services for the DILIGENT user communities WSRP Web Services for Remote Portlets (WSRP) is a specification that defines how to leverage SOAP-based Web services that generate mark-up fragments within a portal application. By defining a set of common interfaces, WSRP allows portals to display remotely running portlets inside their pages without requiring any additional programming by the portal developers. To the end-user, it appears that the portlet is running locally within their portal, but in reality the portlet resides in a remotely running portlet container, and interaction occurs through the exchange of SOAP messages. By using WSRP within Service-Oriented Architecture, presentation-oriented portlet applications can be discovered and reused without engaging in additional development or deployment activities. 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions Key concepts EGEE in DILIGENT Globus in DILIGENT Status and Plans Examples 22/09/2018 DILIGENT N° 004260

Key Concepts The DILIGENT design is service oriented in order to provide as many reusable components as possible for other e-applications that could be created on top of the basic DILIGENT infrastructure DILIGENT system is design and relies on the OGSA concepts of Grid Services and its related implementations Service Oriented Arhitecture (SOA) composed by a lot of services, that exploit the functionality of gLite Middleware. 22/09/2018 DILIGENT N° 004260

Key Concepts DILIGENT is going to be built by integrating Digital Library services on the infrastructure developed by the Enabling Grids for E-sciencE project (EGEE) DILIGENT enhances existing Grid services with the functionality needed to support the complex services interactions required to build, operate and maintain transient virtual digital libraries 22/09/2018 DILIGENT N° 004260

Key Concepts gLite GT4 WMS SE LFC R-GMA VOMS … Java WS-Core WS-MDS WS AuthN AuthZ User-Community Specific Applications Digital Library Creation and Management Content and Metadata Management Index and Search Process gLite Middleware WSRF 22/09/2018 DILIGENT N° 004260

EGEE in DILIGENT DILIGENT has a WP dedicated to collaborate and provide feedback to the EGEE project Many kinds of interactions: Participation in several technical meetings and tutorials/trainings Work with EGEE Technical Coordination Group workgroups Contribution in EGEE conferences, User Forum, etc Subscription to EGEE mailing lists Feedback was given about EGEE middleware and infrastructures (only PPS for now) DILIGENT requirements, status and plans 22/09/2018 DILIGENT N° 004260

EGEE in DILIGENT DILIGENT adopts the EGEE middleware as the grid middleware solution: The DILIGENT application will be composed by: services provided by the DILIGENT project services provided by the gLite distribution Both DILIGENT and gLite services will be deployed on the same infrastructure DILIGENT exploits the gLite components: DILIGENT services are designed to include or wrap “pieces of gLite software” 22/09/2018 DILIGENT N° 004260

EGEE in DILIGENT Data Management: SE/LFC/FTS Job Management: WMS/CE/WN Security: VOMS Information System: RGMA 22/09/2018 DILIGENT N° 004260

EGEE in DILIGENT As soon as the DILIGENT middleware reaches a stable point, DILIGENT will join the EGEE production infrastructure Diligent Nodes EGEE Nodes Information Service Information pull 22/09/2018 DILIGENT N° 004260

Globus in DILIGENT DILIGENT has adopted many OASIS specifications: WSRF, WSN, WS-Addressing and WS-Security Most DILIGENT services are compliant with these specifications The following WSRF and WSN specifications are being used: WS-ResourceProperties (WSRF-RP) WS-ResourceLifetime (WSRF-RL) WS-BaseFaults (WSRF-BF) WS-ServiceGroup (WSRF-SG) WS-BaseNotification (WSN) WS-Topics (WSN) 22/09/2018 DILIGENT N° 004260

Globus in DILIGENT DILIGENT hosting nodes must be equipped with a service container capable to host and support: Classical Web services - compliant with basic WS specifications WSRF Web services - compliant with WSRF specifications The following GT4 components are used: Java WS Core (as hosting environment for the DILIGENT services) WS-MDS Credential Management WS AuthN and AuthZ The DILIGENT container must have at least the following characteristics: Implementation of SOAP over HTTP as a message transport protocol, and both transport-level and WS-Security message-level security for all communications Implementation of WS-Addressing, WSRF, and WS-Notification standards 22/09/2018 DILIGENT N° 004260

Globus in DILIGENT Java WS-Core Credential Management WS AuthN and AuthZ Credential Management The following components are used by the Authentication Support (DVOS): SimpleCA: library used to creates End Entity Credentials for users do not have their own official credentials MyProxy: service used to store middle and short-live credentials of users and services. WS-MDS 22/09/2018 DILIGENT N° 004260

Status and Plans release candidate Final release candidate Beta release candidate Alfa release Final release Beta release Alfa development started Functionalities Since the development started only recently there are still different levels of understanding and knowledge Sep 2005 Nov 2005 Jan 2006 Mar 2006 May 2006 Jul 2006 Sep 2006 Nov 2006 Jan 2007 Mar 2007 May 2007 Jul 2007 22/09/2018 DILIGENT N° 004260

Outline Digital Library Community DILIGENT Project User Communities Architecture Technical Solution Conclusions 22/09/2018 DILIGENT N° 004260

Conclusions DILIGENT services provide state-of-the-art, reusable, scalable, persistent and dynamic software components empowered by the: Compliance with OGSA standards Adoption of gLite as the grid middleware Large-scale infrastructure provided by EGEE DILIGENT test-bed provides an highly efficient shared DL infrastructure for distributed communities which: Allows on-demand shared usage of DL Resources (computing, storage, content, services) Supports multi-type content and application Is validated in two areas - Cultural Heritage and Earth Science 22/09/2018 DILIGENT N° 004260

Questions ?? Thank you !!! http://www.diligentproject.org pedro.andrade@cern.ch 22/09/2018 DILIGENT N° 004260

Backup Slides 22/09/2018 DILIGENT N° 004260

DILIGENT Infrastructures DILIGENT has 2 independent infrastructures up and running all gLite services (r1.4.1): Development infrastructure Testing infrastructure Infrastructures are geographically distributed, linking 6 sites in Athens, Budapest, Darmstadt, Pisa, Innsbruck and Rome Running gLite experimentation tests since July 2005: Data upload and job performance gLiteWrappers: Storage Broker WMS Broker 22/09/2018 DILIGENT N° 004260

Technical Solution [cont.] EGEE – gLite Middleware Architecture and Planning Design of grid services Global security architecture OGSA and WSRF specifications – Java WSCore WS-Resource (Stateful Web Service) WS-ResourceProperties (View of the Web Service State) WS-Addressing (WS-Resource access mechanisms) Aggregator Framework (WS-ResourceProperties Collector) Analysis and take-up of the base technology started before the official project kick-off. All software and specification have been analyzed and took-up. Glite Reseases: Release 1.0 - 21 April 2005 Release 1.1 – 30 April 2005 Release 1.2 – 5 August 2005 Release 1.3 – 11 August 2005 Release 1.4 – 16 September 2005 22/09/2018 DILIGENT N° 004260