1. Bridging WSRF services and the EGEE middleware
Pedro Andrade - CERN, Geneva

2. Agenda Digital Libraries and the DILIGENT Project - 45´´
Q&A
DILIGENT Process Management Service Example - 10´´
DILIGENT DIS and Keeper Service Example - 15´´
Open Discussion
22/09/2018
DILIGENT N°

3. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
22/09/2018
DILIGENT N°

4. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
Evolution
DLs today
Requirements
22/09/2018
DILIGENT N°

5. Evolution 2005 - … 1996 knowledge management user = producer&consumer
digital texts repository +
centralized catalogue
search service
1996
few to many: One of the main concern is regarding the concept of many in Grid and DL environment. In Grid "many" is about "hundreds" while in DL "many" is "thousands".
institution to virtual organisation: Usually this means an evolution from structured to unstructured organisation. The top of the evolution is the concept of community (completely unstructured), in between you have the "virtual Enterprise". So you have a chain Institution>enterprise>organisation>community
user = consumer
few large institutions
many virtual organizations
22/09/2018
DILIGENT N°

6. DLs today The demand for DLs has recently grown considerably
DLs are perceived as a necessary instrument to support communication and collaboration among the members of communities of interest
The maintenance and management of a DL requires high investments and specialized personnel, multimedia handling requires high computational resources
22/09/2018
DILIGENT N°

7. DLs today
Many of the organizations that demand the usage of Digital Libraries are small, distributed, and dynamic
DLs are often used to support temporary activities:
Courses (training courses, technology transfer courses, etc)
Exhibitions, Conferences
Projects …
DL services are required by many application domains:
e-Health, e-Learning, e-Government, e-Science
Corporate Knowledge Management
22/09/2018
DILIGENT N°

8. Requirements Live documents New information objects: Pollution map
Table summarizing data from millions of satellite measures
Graph reporting an analytical trend of certain information extracted from a vast amount of data
22/09/2018
DILIGENT N°

9. Requirements The creation and handling of these documents require:
Access to many different, large, heterogeneous information sources
Use of specialized services
High computing power
Small virtual dynamic organizations usually do not have the instruments for satisfying these requirements
Controlled resources sharing is the only possible solution:
Information sources
Services
Computers and storage
Technical staff
Expectation from the Grid:
Can support a cost-effective DL operational model based on transient, flexible, coordinated “sharing of resources”
Addresses the main DL architecture requirements: distribution, openness, interoperability, scalability, controlled sharing, availability, security, quality, ….
Provides a basic common infrastructure for serving several different application domains
Offers high storage and computing capabilities that enable the provision of powerful functionality on multimedia content e.g. images and videos
22/09/2018
DILIGENT N°

10. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
Objective
DILIGENT impact
Project numbers
Partners
Activities
The grid is the Technical solution because:
high computing and storage capabilities for managing and processing a wide variety of information objects
controlled sharing of resources
22/09/2018
DILIGENT N°

11. Objective
Develop a Digital Library test-bed infrastructure that allows members of dynamic virtual research organizations to create on-demand transient digital libraries based on shared computing, storage, multimedia, multi-type content and application resources
In the bottom line DILIGENT will:
Develop a platform for creation, maintenance and delivery of DLs (I.e. DLMS) which will act on the Grid /EGEE /gLite by developing and integrating several component services.
Demonstrate the potential of the platform through two distinct user-scenarios.
22/09/2018
DILIGENT N°

12. DILIGENT Impact
The DILIGENT service infrastructure will dramatically change the DL development model used by distributed and dynamic organisations and communities
Using DILIGENT, these organisations and communities will be able to setup their own DL:
Even if they do not have enough financial and human resources
When and for the time they need it
Defining their own specific requirements
Exploiting Grid-based services
Profiting form an vast amount of storage and computational resources
Sharing resources in a collaborative and efficient way
22/09/2018
DILIGENT N°

13. Project Numbers Duration: 36 Months
Start/end date: September 2004 – August 2007
Effort: p/m
Cost: €
EU funding: €
15%
Technological development
Validation Activities
Innovation Activities
24%
61%
22/09/2018
DILIGENT N°

14. Partners Universities/Research Institutes:
FhG/IPSI - Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
SNS - Scuola Normale Superiore di Pisa
UNIBAS - University of Basel
UoA - University of Athens
USG - University of Strathclyde
International Organizations:
CERN - European Organization for Nuclear Research
ESA - European Space Agency
Administrative Coordinator:
ERCIM - European Research Consortium for Informatics and Mathematics
Scientific Coordinator:
CNR-ISTI - Institute of Information Science and Technology of the Italian National Research Council
Commercial Companies:
ENG - Engineering Ingegneria Informatica SpA
FAST - Fast Search & Transfer ASA
4D SOFT - 4D SOFT Software Development
RAI - Italian National Broadcaster
22/09/2018
DILIGENT N°

15. Activities Experimentation with user communities
Technology integration and developemnt
Exploitation and
sustainability
Feedback to the constituent technologies
Technology Integration
Design and implementation of the DILIGENT services
Testing and integration actvitites
Experimentation with User Communities
Identify user requirements and validate the DILIGENT testbed
Feedback to the constituent technologies
Provide feedback to the research communities that are developing the EGEE and other Grid infrastructures as well as the DL community
Exploitation, Dissemenation and Sustainability
Report the experience acquired and demonstrate the results achieved to a large spectrum of target user communities
Analyze and aggregate the DILIGENT target communities, define a business model to produce a sustainability plan
Manage the project
Collaboration with other projects and initiatives
22/09/2018
DILIGENT N°

16. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
DILIGENT user communities
ImpECt user community
Arte user community
22/09/2018
DILIGENT N°

17. DILIGENT User Communities
Collect requirements and validate the built technology
Earth Science Domain
Represented by ESA
Goal: Improve accessibility, interoperability and usability of environmental data, models, tools and algorithms
Cultural Heritage Domain
Represented by SNS and RAI
Goal: Stimulate collaborative multidisciplinary scientific research
acndosye
the test-bed will be demonstrated and validated by two complementary real-life application scenarios: one from the cultural heritage domain and one from the environmental e-Science domain. The first scenario will show how the DILIGENT infrastructure, by simplifying the processing of multimedia artefacts and offering support for collaborative multidisciplinary studies, can contribute to favour faster progress in research and
better quality of education, also in research areas traditionally far from the most advanced computing technologies; the second scenario will show how this knowledge infrastructure, by integrating distributed multi-type data sources with specialised data handling services, can improve accessibility, interoperability and usability of environmental data, models, tools and instruments
22/09/2018
DILIGENT N°

18. ImpECt User Community Data: Services: Use cases:
Automatically generated, annotated and relatively large
Wide variety of content types (maps, satellite data, etc)
Services:
Complex, taylored image processing
Composed services
...
Use cases:
Dynamic report generation
Organization of conferences
Handling of information in case of environmental accidents
22/09/2018
DILIGENT N°

19. ARTE User Community Data: Services: Use cases:
Retrodigitized, manually annotated and relatively small
Image, video material (from RAI) and (structured) text
Services:
Feature extraction for improving retrieval
Composed services (processes management)
...
Use cases:
Multidisciplinary research working on images and videos
Courses, Workshop
Exhibition Catalogue
22/09/2018
DILIGENT N°

20. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
Design Process
Overview
Services description
22/09/2018
DILIGENT N°

21. Design Process User Activities Base Technology Analysis
Functional Specification
Service Specification
Testing Strategy and Plan
Unified Process
UP is a software engineering process that
provides a disciplined approach to assign tasks and responsibilities within a development organization
ensures the production of high-quality software that meets the needs of its end-users, within a predictable schedule and budget
simplifies the interactions between the different project activities formally identifying the communication models, diagrams, and timing
UP allows to correctly manage the project technical life cycle
Experimentation
Service Design
Architectural Specification
22/09/2018
DILIGENT N°

22. Overview 22/09/2018 DILIGENT N° 004260
From a conceptual point of view the services that we are implementing and that compose the DILIGENT infrastructure are organized in a layered perspective.
These services also interact directly with gLite… the picture is only a layered view of the system
22/09/2018
DILIGENT N°

23. DL Creation and Management
22/09/2018
DILIGENT N°

24. DL Creation and Management
Objective: provide the functionality necessary to aggregate and manage a collection of resources for efficient usage under the control of a Virtual Digital Library system
22/09/2018
DILIGENT N°

25. DL Creation and Management
DILIGENT Information System (DIS):
Aggregation and storage of information about DILIGENT resources
Provide details information about the resources state through query and subscription interfaces
Integration and exploitation of gLite information service
Triggered actions and information caching
Keeper:
Dynamic services deployment
Distributed hosting node management
Deployed services monitoring
DILIGENT Information Service
Diligent Information Service (DIS) provides mechanisms for:
gathering, storing and supplying information about the resources needed by DILIGENT services,
monitoring the resources state information.
It allows users and services to discover what resources are part of a Virtual Organization (VO) and to monitor those resources; it provides query and subscription interfaces to arbitrarily detailed resource information; it provides a trigger interface that can be configured to take action when pre-configured conditions are met; it archives information to allow historical query execution.
Keeper
“The Keeper service has to instantiate the set of services belonging to a VDL and to manage them assuring the characteristics of QoS required by the VDL Definition criteria”. Starting from the above definition, the Keeper is primarily responsible for the creation of a new VDL by instantiating its resources (i.e. Running instances of its services) and by authorizing its users. It must also guarantee the overall set of the VDL functionalities at any time by dynamically relocating resources and periodically checking their status.
22/09/2018
DILIGENT N°

26. DL Creation and Management
VDL Generator (VDL):
Users/communities support
Dynamic composition and configuration of DLs
Dynamic Virtual Organization Support (DVOS):
Robust and flexible security framework
Advanced authentication and authorization model
Notification support
Dynamic aggregation of resources and users/groups
Broker & Matchmaker (BMM):
Optimal resources allocation
VDL Generator
The VDL Generator Service is the service that enables users/communities to create their own DLs. It allows to define a set of criteria that specify the expected characteristics of the new DL; starting from them, it identifies the set of services required to provide the requested features. In particular, the VDL Generator Service:
· Supports users/communities in specifying the criteria that characterize the new DL, e.g. the information space, the required functionalities, the characteristics of these functionalities;
· Selects the appropriate pool of services and information sources required to implement a DL that satisfies the specified criteria;
· Notifies to the DL Management service (see Section ) the identified services.
DVOS
DVOS includes a set of services all tailored to support dynamic VO. The DVOS area covers functionalities related to Resource Management (Section 4.3 of D1.1.1), to VO Management (Section 4.4 of D1.1.1), to Users and Group Management (Section 4.5 of D1.1.1) and to Notification Management (Section 4.6 of D1.1.1). Other nonfunctional requirements covered by DVOS are Authentication and Authorization issues as described in the D Test-bed Functional Specification.
Broker & MatchMaker Service
The Broker & MatchMaker Service (BMM) is in charge of supporting the Keeper service in deploying a new Digital Library on a set of DILIGENT Hosting Nodes (DHNs). In particular, once the Keeper has identified the set of packages needed to build a new DL the BMM’s job is to identify a set DHNs to be used as target hosts for the deployment.
22/09/2018
DILIGENT N°

27. DL Creation and Management
22/09/2018
DILIGENT N°

28. Content and Metadata Management
22/09/2018
DILIGENT N°

29. Content and Metadata Management
Objective: provide access and storage of information objects, management of related metadata, encryption and decryption of information objects for security reasons, and information object annotations management.
22/09/2018
DILIGENT N°

30. Content and Metadata Management
Content Management
Distributed content storage support
Advanced replication mechanism
Flexible storage model
Support for arbitrary object types
content, metadata, features, collections, archives, indexes, etc
Support for object relationships
contains, characterizes, extracted-from, indexed-by, etc
Integration of 3rd party data sources (archive migration)
Metadata Management
Efficient support for arbitrarily structured metadata and transparent support for multiple metadata schemas and mappings
22/09/2018
DILIGENT N°

31. Content and Metadata Management
Content Security
Deals with the specialties of protecting multimedia content
Include authenticity, integrity and confidentiality
Grid-enabled watermarking algorithms for different media types
Annotation
Provides interactive functionality for the management of manually authored, subjective, and context-dependent metadata items about DILIGENT information objects, primarily multimedia documents.
22/09/2018
DILIGENT N°

32. Content and Metadata Management
22/09/2018
DILIGENT N°

33. Content and Metadata Management
22/09/2018
DILIGENT N°

34. Index and Search Management
22/09/2018
DILIGENT N°

35. Index and Search Management
Objective: design and implement an open, scalable, highly-featured, distributed search engine enhanced by the usage of various types of indices.
22/09/2018
DILIGENT N°

36. Index and Search Management
Index Service
Build and consume inverted and forward indices on metadata for search performance enhancement
Search Service
Locate content by searching in metadata and extracted features
Optimisation in search execution
Support Full Text, Similarity Based and Exact Match searches
Feature Extraction
Process documents to extract feature descriptors
22/09/2018
DILIGENT N°

37. Index and Search Management
Personalization Service
Provide personalized information retrieval
Manage user profiles
Content Source Description and Selection
Filter content sources to maximize search relevance/cost ratio
Automatically generate source description
Data Fusion
Fuse (merge) the ranked content result sets into a single ranked result set
22/09/2018
DILIGENT N°

38. Index and Search Management
22/09/2018
DILIGENT N°

39. Process Management
22/09/2018
DILIGENT N°

40. Process Management Objectives:
Use existing services as building blocks for more complex, value-added workflows (processes/compound services)
Dynamic scheduling of service requests
Define structured processes to be executed on the Grid
Efficient usage of the Grid resources available
More
Objectives:
Use existing services as building blocks for more complex, value-added workflows (processes)
Processes themselves are invoked like normal services
A process may be composed of „simple“ services and/or other processes
Distributed execution of the processes
Dynamic scheduling of service requests
Efficient use of the resources in the infrastructure
Scalability and fault-tolerance
Transactions: Support atomic execution of processes
Guarantee „all or nothing“ wrt process execution
22/09/2018
DILIGENT N°

41. Process Management Design and Verification Execution and Reliability
Check whether a process is defined correctly
Provide a graphical UI for specification and analysis of processes
Execution and Reliability
Distributed execution process on the Grid
Avoid centralized workflow engine - avoid bottleneck
Dynamic allocation of resources
Automate process
Optimisation
Structural process modifications to maximise parallelism of execution
Optimise Grid resources utilisation
22/09/2018
DILIGENT N°

42. Process Management
22/09/2018
DILIGENT N°

43. Application Specific Functionality
22/09/2018
DILIGENT N°

44. Application Specific Functionality
Objective: Integrate DILIGENT elements under an application-specific context by adding all the custom elements expected. It also aims to prove the concept of DILIGENT as a versatile and powerful open DLMS able to capture specific community needs.
22/09/2018
DILIGENT N°

45. Application Specific Functionality
Portal:
Hosts the graphical user interface (UI) of the core services provided by the DILIGENT platform
Every user interface element is a portlet or a part of a portlet
The Portlet Hosting Platform:
support for all JSR168 portlet containers
currently using uPortal and GridShpere
optionally support of Web Services for Remote Portlets (WSRP)
Visualization:
Provides application-specific visualization services for the DILIGENT user communities
WSRP
Web Services for Remote Portlets (WSRP) is a specification that defines how to leverage
SOAP-based Web services that generate mark-up fragments within a portal application. By
defining a set of common interfaces, WSRP allows portals to display remotely running
portlets inside their pages without requiring any additional programming by the portal
developers. To the end-user, it appears that the portlet is running locally within their portal,
but in reality the portlet resides in a remotely running portlet container, and interaction
occurs through the exchange of SOAP messages. By using WSRP within Service-Oriented
Architecture, presentation-oriented portlet applications can be discovered and reused
without engaging in additional development or deployment activities.
22/09/2018
DILIGENT N°

46. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
Key concepts
EGEE in DILIGENT
Globus in DILIGENT
Status and Plans
Examples
22/09/2018
DILIGENT N°

47. Key Concepts
The DILIGENT design is service oriented in order to provide as many reusable components as possible for other e-applications that could be created on top of the basic DILIGENT infrastructure
DILIGENT system is design and relies on the OGSA concepts of Grid Services and its related implementations
Service Oriented Arhitecture (SOA) composed by a lot of services, that exploit the functionality of gLite Middleware.
22/09/2018
DILIGENT N°

48. Key Concepts
DILIGENT is going to be built by integrating Digital Library services on the infrastructure developed by the Enabling Grids for E-sciencE project (EGEE)
DILIGENT enhances existing Grid services with the functionality needed to support the complex services interactions required to build, operate and maintain transient virtual digital libraries
22/09/2018
DILIGENT N°

49. Key Concepts gLite GT4 WMS SE LFC R-GMA VOMS … Java WS-Core WS-MDS
WS AuthN AuthZ
User-Community Specific Applications
Digital Library Creation
and Management
Content and Metadata
Management
Index and Search
Process
gLite Middleware
WSRF
22/09/2018
DILIGENT N°

50. EGEE in DILIGENT
DILIGENT has a WP dedicated to collaborate and provide feedback to the EGEE project
Many kinds of interactions:
Participation in several technical meetings and tutorials/trainings
Work with EGEE Technical Coordination Group workgroups
Contribution in EGEE conferences, User Forum, etc
Subscription to EGEE mailing lists
Feedback was given about
EGEE middleware and infrastructures (only PPS for now)
DILIGENT requirements, status and plans
22/09/2018
DILIGENT N°

51. EGEE in DILIGENT
DILIGENT adopts the EGEE middleware as the grid middleware solution:
The DILIGENT application will be composed by:
services provided by the DILIGENT project
services provided by the gLite distribution
Both DILIGENT and gLite services will be deployed on the same infrastructure
DILIGENT exploits the gLite components:
DILIGENT services are designed to include or wrap “pieces of gLite software”
22/09/2018
DILIGENT N°

52. EGEE in DILIGENT Data Management: SE/LFC/FTS Job Management: WMS/CE/WN
Security: VOMS
Information System: RGMA
22/09/2018
DILIGENT N°

53. EGEE in DILIGENT
As soon as the DILIGENT middleware reaches a stable point, DILIGENT will
join the EGEE production infrastructure
Diligent Nodes
EGEE Nodes
Information Service
Information pull
22/09/2018
DILIGENT N°

54. Globus in DILIGENT DILIGENT has adopted many OASIS specifications:
WSRF, WSN, WS-Addressing and WS-Security
Most DILIGENT services are compliant with these specifications
The following WSRF and WSN specifications are being used:
WS-ResourceProperties (WSRF-RP)
WS-ResourceLifetime (WSRF-RL)
WS-BaseFaults (WSRF-BF)
WS-ServiceGroup (WSRF-SG)
WS-BaseNotification (WSN)
WS-Topics (WSN)
22/09/2018
DILIGENT N°

55. Globus in DILIGENT
DILIGENT hosting nodes must be equipped with a service container capable to host and support:
Classical Web services - compliant with basic WS specifications
WSRF Web services - compliant with WSRF specifications
The following GT4 components are used:
Java WS Core (as hosting environment for the DILIGENT services)
WS-MDS
Credential Management
WS AuthN and AuthZ
The DILIGENT container must have at least the following characteristics:
Implementation of SOAP over HTTP as a message transport protocol, and both transport-level and WS-Security message-level security for all communications
Implementation of WS-Addressing, WSRF, and WS-Notification standards
22/09/2018
DILIGENT N°

56. Globus in DILIGENT Java WS-Core Credential Management
WS AuthN and AuthZ
Credential Management
The following components are used by the Authentication Support (DVOS): SimpleCA: library used to creates End Entity Credentials for users do not have their own official credentials MyProxy: service used to store middle and short-live credentials of users and services.
WS-MDS
22/09/2018
DILIGENT N°

57. Status and Plans release candidate Final release candidate Beta
release candidate Alfa
release Final
release Beta
release Alfa
development started
Functionalities
Since the development started only recently there are still different levels of understanding and knowledge
Sep 2005
Nov 2005
Jan 2006
Mar 2006
May 2006
Jul 2006
Sep 2006
Nov 2006
Jan 2007
Mar 2007
May 2007
Jul 2007
22/09/2018
DILIGENT N°

58. Outline Digital Library Community DILIGENT Project User Communities
Architecture
Technical Solution
Conclusions
22/09/2018
DILIGENT N°

59. Conclusions
DILIGENT services provide state-of-the-art, reusable, scalable, persistent and dynamic software components empowered by the:
Compliance with OGSA standards
Adoption of gLite as the grid middleware
Large-scale infrastructure provided by EGEE
DILIGENT test-bed provides an highly efficient shared DL infrastructure for distributed communities which:
Allows on-demand shared usage of DL Resources (computing, storage, content, services)
Supports multi-type content and application
Is validated in two areas - Cultural Heritage and Earth Science
22/09/2018
DILIGENT N°

60. Questions ?? Thank you !!! http://www.diligentproject.org
22/09/2018
DILIGENT N°

61. Backup Slides
22/09/2018
DILIGENT N°

62. DILIGENT Infrastructures
DILIGENT has 2 independent infrastructures up and running all gLite services (r1.4.1):
Development infrastructure
Testing infrastructure
Infrastructures are geographically distributed, linking 6 sites in Athens, Budapest, Darmstadt, Pisa, Innsbruck and Rome
Running gLite experimentation tests since July 2005:
Data upload and job performance
gLiteWrappers:
Storage Broker
WMS Broker
22/09/2018
DILIGENT N°

63. Technical Solution [cont.]
EGEE – gLite Middleware
Architecture and Planning
Design of grid services
Global security architecture
OGSA and WSRF specifications – Java WSCore
WS-Resource (Stateful Web Service)
WS-ResourceProperties (View of the Web Service State)
WS-Addressing (WS-Resource access mechanisms)
Aggregator Framework (WS-ResourceProperties Collector)
Analysis and take-up of the base technology started before the official project kick-off.
All software and specification have been analyzed and took-up.
Glite Reseases:
Release April 2005
Release 1.1 – 30 April 2005
Release 1.2 – 5 August 2005
Release 1.3 – 11 August 2005
Release 1.4 – 16 September 2005
22/09/2018
DILIGENT N°