EMI Service Registry (EMIR) JRA1, Infrastructure

Slides:



Advertisements
Similar presentations
TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.
Advertisements

The CA MDB Revised May © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced.
Web Service Ahmed Gamal Ahmed Nile University Bioinformatics Group
General introduction to Web services and an implementation example
Connect. Communicate. Collaborate Click to edit Master title style MODULE 1: perfSONAR TECHNICAL OVERVIEW.
Web Servers How do our requests for resources on the Internet get handled? Can they be located anywhere? Global?
Wireless LAN Topology Visualiser Project Supervisor: Dr Arkady Zaslavsky Project Team Members: Jignesh Rambhia Robert Mark Bram Tejas Magia.
ΗΛΕΚΤΡΟΝΙΚΟ ΕΜΠΟΡΙΟ Web Services Overview Mary Grammatikou 9/06/2009.
Peoplesoft: Building and Consuming Web Services
06 | Implementing Web APIs Jon Galloway | Tech Evangelist Christopher Harrison | Head Geek.
Introduction to UDDI From: OASIS, Introduction to UDDI: Important Features and Functional Concepts.
Evolution to CIMI Charles (Cal) Loomis & Mohammed Airaj LAL, Univ. Paris-Sud, CNRS/IN2P3 29 August 2013.
TeraGrid Information Services John-Paul “JP” Navarro TeraGrid Grid Infrastructure Group “GIG” Area Co-Director for Software Integration and Information.
M i SMob i S Mob i Store - Mobile i nternet File Storage Platform Chetna Kaur.
OASIS ebXML Registry Standard Open Forum 2003 on Metadata Registries 10:30 – 11:15 January 20, 2003 Kathryn Breininger The Boeing Company Chair, OASIS.
GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.
San Diego Supercomputer Center SDSC Storage Resource Broker Data Grid Automation Arun Jagatheesan et al., San Diego Supercomputer Center University of.
Scalable Systems Software Center Resource Management and Accounting Working Group Face-to-Face Meeting October 10-11, 2002.
Application code Registry 1 Alignment of R-GMA with developments in the Open Grid Services Architecture (OGSA) is advancing. The existing Servlets and.
1 Schema Registries Steven Hughes, Lou Reich, Dan Crichton NASA 21 October 2015.
National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Management of Distributed Data Reagan W. Moore.
EMI is partially funded by the European Commission under Grant Agreement RI Using GLUE 2.0 in Practice with EMI Registry (EMIR) Shiraz Memon, Jülich.
An information and monitoring system for static and dynamic information about grid resources, applications, networks … RDBMS Servlet aware of API during.
EMI is partially funded by the European Commission under Grant Agreement RI Discovering Infrastructures with EMI Registry (EMIR) Emidio Giorgio.
1 Java Servlets l Servlets : programs that run within the context of a server, analogous to applets that run within the context of a browser. l Used to.
1 Registry Services Overview J. Steven Hughes (Deputy Chair) Principal Computer Scientist NASA/JPL 17 December 2015.
Clusterpoint Margarita Sudņika ms RDBMS & NoSQL Databases & tables → Document stores Columns, rows → Schemaless documents Scales UP → Scales UP.
OASIS ebXML Registry Standard Open Forum 2003 on Metadata Registries 10:30 – 11:15 January 20, 2003 Kathryn Breininger The Boeing Company Chair, OASIS.
Google Code Libraries Dima Ionut Daniel. Contents What is Google Code? LDAPBeans Object-ldap-mapping Ldap-ODM Bug4j jOOR Rapa jongo Conclusion Bibliography.
EMI is partially funded by the European Commission under Grant Agreement RI EMI Registry (EMIR) Shiraz Memon, Ivan Marton, Gabor Szigeti, Laurence.
Orion Contextbroker PROF. DR. SERGIO TAKEO KOFUJI PROF. MS. FÁBIO H. CABRINI PSI – 5120 – TÓPICOS EM COMPUTAÇÃO EM NUVEM
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI GLUE 2: Deployment and Validation Stephen Burke egi.eu EGI OMB March 26 th.
1 OASIS BDX TC - March BDX Technical Committee Addressing Mechanism or BDX Technical Committee Addressing Mechanism or "how do I find where to send.
EMI is partially funded by the European Commission under Grant Agreement RI EMI Status And Plans Laurence Field, CERN Towards an Integrated Information.
Argus EMI Authorization Integration
The Holmes Platform and Applications
Outline Introduction and motivation, The architecture of Tycho,
Architecture Review 10/11/2004
Latency of EMIR in Multi-Tier Deployment
Gridpp37 – 31/08/2016 George Ryall David Meredith
The Operations Portal and the Grid Operations Interoperability
MSB Integration Guide.
Web Development Web Servers.
TeraGrid Information Services Developer Introduction
Data Bridge Solving diverse data access in scientific applications
Automate Custom Solutions Deployment on Office 365 and Azure
Understanding SOAP and REST calls The types of web service requests
HMA Identity Management Status
MongoDB Er. Shiva K. Shrestha ME Computer, NCIT
Node.js Express Web Services
ARC GLUE2 Server-Side Status and Plans
Towards an Integrated Information system: the EMI view
CHAPTER 3 Architectures for Distributed Systems
Grid Metadata Management
OGSA Data Architecture Scenarios
Ashish Pandit IT Architect, Middleware & Integration Services
Wsdl.
WEB API.
Ashutosh Rana Rahul Nori 7/17/2018
Tech introduction.
A Component-based Architecture for Mobile Information Access
Introduction to Web Services
RELATIONAL GRID MONITORING ARCHITECHTURE
WEB SERVICES From Chapter 19, Distributed Systems
While the audience is gathering. During breaks etc
WCF Data Services and Silverlight
European Statistical System Metadata Handler ESS MH (Super) Providers
Chengyu Sun California State University, Los Angeles
SDMX IT Tools SDMX Registry
Presentation transcript:

EMI Service Registry (EMIR) JRA1, Infrastructure 10/14/11 EMI Service Registry (EMIR) JRA1, Infrastructure Shiraz Memon (FZJ), Ivan Marton (NIIF), Gabor Szigeti (NIIF)

Outline Motivation Goals EMIR Highlights Architecture 10/14/11 Outline Motivation Goals EMIR Highlights Architecture Information Model REST API Security EMI Integration Scenario Performance Analysis Implementation Remarks Status report Oct 14, 2011 EMI Registry 2

Motivation No common Index Service 10/14/11 Motivation No common Index Service Site/Top level BDII ARC ISIS UNICORE Registry No common “service” to store the service information Various Service record formats Service publish and discovery interfaces Oct 14, 2011 EMI Registry 3

Goals Common Index Service to Publish and Discover the Services 10/14/11 Goals Common Index Service to Publish and Discover the Services Unified Information Model Support for federations (Natural to existing Infrastructures) Inter-Federation Lookups Multi-Federation service discovery Common Security Model Scalable and Performance DCI lookup Multi DCI lookups Oct 14, 2011 EMI Registry 4 4

Unified Index Service: EMI Service Registry (EMIR) 10/14/11 Unified Index Service: EMI Service Registry (EMIR) Discover all the EMI Services (ARC, gLite, UNICORE, dCache) Hierarchies and Federations Common Access Point to Service Discovery (even at each hierarchical level) REST-ful Interface to Publish and Discover the Services Service Information: GLUE 2.0 Based Service Entity Bulding hier. Registries Informa. model Oct 14, 2011 EMI Registry 5 5

10/14/11 System Architecture Oct 14, 2011 EMI Registry 6

10/14/11 Oct 14, 2011 EMI Registry 7 7

Service Information Model 10/14/11 Service Information Model Oct 14, 2011 EMI Registry 8

GLUE 2.0: Information Model 10/14/11 GLUE 2.0: Information Model Service General Information Meta-data (Last-update, Publisher/Owner, Lifetime) Endpoint Access Information Location Physical Location Geo-Coordinates Contact Additional Attributes (can be added on the fly) TTL for every Service Entry Oct 14, 2011 EMI Registry 9

JSON Rendering Developed within EMI 10/14/11 JSON Rendering Developed within EMI Used for Publishing and Discovering the Services [ { "Service_Name": “EMI Execution Service", "Service_CreationTime": {"$date": "2011-10-16T11:47:24Z"}, "Service_Type": “eu.emi.es", "Service_Capability": [“activity submission", “activity creation"], "Service_QualityLevel": "production", "Service_Endpoint_URL": "http://eu-emi.eu/emi-es", "Service_Endpoint_Technology": “SOAP over HTTP", "Service_Endpoint_InterfaceName": “ActivityManagement", "Service_Endpoint_InterfaceVersion": ["1.0“ ], "Service_Endpoint_WSDL": "http://eu-emi.eu/emi-es ?wsdl", "Service_Endpoint_HealthState": "ok", "Service_Endpoint_HealthStateInfo": “it is OK, I can see it", "Service_Endpoint_ServingState": "production", "Service_Endpoint_StartTime": {"$date": "2011-07-21T11:47:24Z"}, "Service_Endpoint_DowntimeAnnounce": {"$date": "2011-07-21T11:47:24Z"}, "Service_Endpoint_DowntimeStart": {"$date": "2011-07-21T11:47:24Z"}, "Service_Endpoint_DowntimeEnd": {"$date": "2011-07-21T11:47:24Z"}, "Service_Endpoint_QualityLevel": "production", "Service_ExpireOn": {"$date": "2011-07-21T11:47:24Z"}, "Service_UpdateOn": {"$date": "2011-07-21T11:47:24Z"}, "Service_OwnerDN": “CN=Shiraz Memon,O=FZJ,OU=JSC” } ] Oct 14, 2011 EMI Registry 10 10

Querying the XML documents (1/2) 10/14/11 Querying the XML documents (1/2) < !-- Document Start -- > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <ns2:QueryResult count="1" xmlns:ns2="http://www.eu-emi.eu/emiregistry" xmlns="http://schemas.ogf.org/glue/2009/03/spec_2.0_r1"> <Service BaseType="Entity" CreationTime="2011-07-21T13:47:24.111+02:00" Validity="12313"> <Name>ComputingService</Name> <Extensions/> <Type>job-management</Type> <QualityLevel>production</QualityLevel> <Location> <Address>A Street 1</Address> <Country>Germany</Country> <PostCode>53119</PostCode> <Latitude>53.3</Latitude> <Longitude>4.0</Longitude> </Location> <Contact> <Detail>http://contactlink</Detail> <Type>sysadmin</Type> </Contact> <Type>developer</Type> Add alternative word Oct 14, 2011 EMI Registry 11 11

Querying the XML documents (2/2) 10/14/11 Querying the XML documents (2/2) <Endpoint BaseType="Entity" CreationTime="2011-07-21T13:47:24.111+02:00"> <Name>ComputingService</Name> <URL>http://eu-emi.eu/emies</URL> <Capability>Activity Management</Capability> <Capability>Activity Submission</Capability> <Technology>technology</Technology> <InterfaceName></InterfaceName> <InterfaceVersion>1.0</InterfaceVersion> <HealthState>ok</HealthState> <HealthStateInfo>Its OK</HealthStateInfo> <ServingState>production</ServingState> <StartTime>2011-07-21T13:47:24.111+02:00</StartTime> <DowntimeAnnounce>2011-07-21T13:47:24.111+02:00</DowntimeAnnounce> <DowntimeStart>2011-07-21T13:47:24.111+02:00</DowntimeStart> <DowntimeEnd>2011-07-21T13:47:24.111+02:00</DowntimeEnd> </Endpoint> </Service> </ns2:QueryResult> < !-- Document End -- > Oct 14, 2011 EMI Registry 12

Database Technology MongoDB: NOSQL database 10/14/11 Database Technology MongoDB: NOSQL database Supports Native JSON format Dynamic Indexing Replication Horizontal scalability with sharding (Similar to Google’s BigTable) Available for most of the major OSs Change title to mongodb Oct 14, 2011 EMI Registry 13 13

REST-ful API: Publishing & Querying 10/14/11 REST-ful API: Publishing & Querying Oct 14, 2011 EMI Registry 14

Service Registration /serviceadmin Register 10/14/11 Service Registration /serviceadmin Register -> HTTP POST - /serviceadmin - Body: JSON <- OK Read -> HTTP GET – /serviceadmin?Service_Endpoint_URL=service_url <- JSON document Update (full replace) -> HTTP PUT - /serviceadmin Delete -> HTTP DELETE - /serviceadmin?Service_Endpoint_URL=service_url Oct 14, 2011 EMI Registry 15

Service Querying Showing all the services 10/14/11 Service Querying Showing all the services /services /query -> HTTP GET <- OK, JSON Returns the 100 services, skipping the first 20 /query?limit=100&skip=20 Returns 100 services skipping the first 20 records with specific name /query?limit=100&skip=20&ServiceName=ComputingService Returns 100 services skipping the first 20 records, filtered by “name”, as GLUE 2.0 XML document /query.xml?limit=100&skip=20&ServiceName=ComputingService <- OK, XML Add enumeration Oct 14, 2011 EMI Registry 16 16

Interacting with EMIR Register/Update Query 10/14/11 Interacting with EMIR Register/Update curl -v -H "Content-Type: application/json" -X POST (PUT) -d @info.json / http://emiregistry1.grid.niif.hu:54321/serviceadmin Query curl -v -X GET http://emiregistry1.grid.niif.hu:54321/services/query curl -v -X GET http://emiregistry1.grid.niif.hu:54321/services/query.xml Delete curl -v -X DELETE http://emiregistry1.grid.niif.hu:54321/serviceadmin?Service_Endpoint _URL=1 Header Method Data Registry URL Oct 14, 2011 EMI Registry 17 17

10/14/11 Federations Oct 14, 2011 EMI Registry 18

Hierarchical Structure 10/14/11 Hierarchical Structure <<Global>> GSR Country Publish <<Parent>> DSR State Publish <<Child>> DSR HPC Center Events are propagated to the top level registries Handling Failures Domain publish Liveness at the GSR ∞ Level of Hierarchy S Oct 14, 2011 EMI Registry 19 19

Failure 1: Parent Goes Offline 10/14/11 EMIR Hierarchy Static Construction Events (changes in the registry) are sent directly to the Parent as they occur Failure 1: Parent Goes Offline Caching the occurred events in between Send as soon as the Parent comes online Failure 2: Child found a Parent (got newly added) Send whole local information to the parent in Chunks Events are propagated to the top level registries Handling Failures Oct 14, 2011 EMI Registry 20 20

Global Registries Replicated Registries 10/14/11 Global Registries Replicated Registries Data Synchronization at Global level Why? Global Service Discovery Failover Robustness Scalability Load Balancing Offers query interface Where to find GSRs? Publicly known list <<Global>> GSR <<Global>> GSR <<Global>> GSR <<Global>> GSR Oct 14, 2011 EMI Registry 21

10/14/11 Security Oct 14, 2011 EMI Registry 22

ARGUS based access control Attribute Source: VOMS-SAML 10/14/11 Goals ARGUS based access control Attribute Source: VOMS-SAML EMI Common Authentication Library (CAL) Oct 14, 2011 EMI Registry 23

Currently Supported Access Control 10/14/11 Currently Supported SSL/TLS based authentication – requires valid X.509 credentials Attribute Source: User Map File Defines “Users” with associated “roles” (Service Provider, Service Consumer/User, Administrator) Access Control Access Control Policies e.g. allows a user with “serviceowner” role can register and manage/control his registrations defined in a XML file containing raw XACML policies Oct 14, 2011 EMI Registry 24 24

10/14/11 Performance Analysis Oct 14, 2011 EMI Registry 25

Single Registry Instance without Security 1 GB JVM Memory 4 Cores 10/14/11 System Environment Single Registry Instance without Security 1 GB JVM Memory 4 Cores ~3 GB for MongoDB Oct 14, 2011 EMI Registry 26 26

Performance Analysis: Single Threaded Response Time (Without Security) 10/14/11 Performance Analysis: Single Threaded Response Time (Without Security) Oct 14, 2011 EMI Registry 27 27

Performance Analysis: Multithreaded Response Time 10/14/11 Performance Analysis: Multithreaded Response Time Oct 14, 2011 EMI Registry 28 28

Implementation Remarks 10/14/11 Implementation Remarks JAVA 6 Maven 2 JAX-RS (Java Standard for REST-ful WS) Jetty 7 Web Server MongoDB Oct 14, 2011 EMI Registry 29 29

Status Report Implementation Documentation DSR: 10/14/11 Status Report Implementation DSR: Provided: Most of the basic features already implemented Missing: Argus and/or VOMS-SAML support GSR: P2P synchronization/replication will be available by EMI 2 Clients: ARC, UNICORE (under development) Documentation Technical Document has been revised to reflect the implementation Comparative Survey Document is progressing Development Plan on EMI TWiki Packaging EPEL distribution, will be available by EMI 2 Source and Binary Tarball ETICS Integration Oct 14, 2011 EMI Registry 30 30

Service Registrants from Service providers SAGA-SD API 10/14/11 Further Extensions Service Registrants from Service providers SAGA-SD API GUI to administer and browse registry(s) Triggers, etc… Oct 14, 2011 EMI Registry 31 31

Twiki: https://twiki.cern.ch/twiki/bin/view/EMI/ EMIRegistry 10/14/11 Links Twiki: https://twiki.cern.ch/twiki/bin/view/EMI/ EMIRegistry Source Code on GitHub (public repo.): https://github.com/eu-emi/emiregistry Oct 14, 2011 EMI Registry 32 32

10/14/11 Questions ??? Oct 14, 2011 EMI Registry 33

10/14/11 Thanks! Oct 14, 2011 EMI Registry 34