Stato degli Science Gateway di Catania Roberto Barbera Univ. of Catania and INFN Riunione tecnica sul portale general purpose e sugli science gateway di Catania – Catania, 7 Marzo 2012

Summary of standards adopted  The framework for Science Gateways developed at Catania is fully web-based and adopts official worldwide standards and protocols, through their most common implementations  These are:  The JSR 168 and JSR 286 standards (also known as "portlet 1.0" and "portlet 2.0" standards)JSR 168JSR 286  The OASIS Security Assertion Markup Language (SAML) standard and its Shibboleth and SimpleSAMLphp implementationsOASISSecurity Assertion Markup Language ShibbolethSimpleSAMLphp  The Lightweight Direct Access Protocol, and its OpenLDAP implementationOpenLDAP  The Cryptographic Token Interface Standard (PKCS#11) standard and its Cryptoki implementationCryptographic Token Interface Standard  The Open Grid Forum (OGF) Simple API for Grid Applications (SAGA) standard and its JSAGA implementationOpen Grid ForumSimple API for Grid ApplicationsJSAGA 2

Our reference model Science Gateway Science Gateway App. 1 App. 2 App. N Embedded Applications Administrator Power User Basic User Users from different organisations having different roles and privileges Standard-based (SAGA) middleware-independent Grid Engine Standard-based (SAGA) middleware-independent Grid Engine 3

AuthN & AuthZ Schema AuthorisationAuthorisation Science Gateway GrIDP (“catch-all”) GrIDP (“catch-all”) IDPCT (“catch- all”) IDPCT (“catch- all”) IDP_y LDAP Register to a Service 2. Sign in Authentication Social Networks’ Bridge IdP

The Grid IDentity Pool (GrIDP) ( This is a “catch-all” Identity Federation

eduGAIN ( All the Science Gateways are registered as Service Providers of eduGAIN 6

7 Grid Engine Users Tracking DB Science GW Interface SAGA/JSAGA API Job Engine Data Engine Users Track & Monit. Science GW 1 Science GW 2 Science GW 3 Grid MWs Liferay Portlets eToken Server DONEBy end of April Catania Grid Engine By mid April DONE 7

Job Engine - Architecture WT Worker Threads for Job Submission WT Worker Threads for Status Checking USER TRACKING DB MONITORING MODULE GRID INFRASTRUCTURE(S) Job Queue WT Job Submission Job Check Status/ Get Output 8

9 Job Engine - Features  The Job Engine has been designed with the following features in mind: FeatureDescriptionStatus Middleware Independent Capacity to submit job to resources running different middleware DONE EasinessCreate code to run applications on the grid in a very short time DONE ScalabilityManage a huge number of parallel job submissions fully exploiting the HW of the machine where the Job Engine is installed DONE PerformanceHave a good response timeDONE Accounting & Auditing Register every grid operation performed by the usersDONE Fault ToleranceHide middleware failure to end usersALMOST DONE WorkflowProviding a way to easily create and run workflowsIN PROGRESS

10 Job Engine – Scalability 40,000 jobs submitted in parallel ! Time to submit 10,000 jobs (h) Job submission time (h)  Submission time scales linearly with number of jobs  >10,000 jobs a hour 10

Job Engine – Performances 11

Both sequential and MPI-enabled jobs successfully executed Tests with Globus planned 12 Job Engine – Middleware interoperability

Job Engine – Accounting & Auditing  A powerful accounting & auditing system is included in the Job Engine  It is fully compliant with EGI VO Portal Policy and EGI Grid Security Traceability and Logging Policy  The following values are stored in the DB for each job submitted:  User ID  Job Submission timestamp  Job Done timestamp  Application name  Job ID  Robot certificate ID  VO name  Execution site (name, latitude, longitude) 13

Catania Science Gateways in numbers Overall usage (arb. units) 14

Data Engine – Requirements  A file browser shows Grid files in a tree  File system exposed by the Science Gateway is virtual  Easy transfers from/to Grid (through the SG at the moment) are done in a few clicks  Users do not need to care about how and where their files are really located 15

Data Engine – Usage Workflow Sign in eTokenServer User Track. DB DOGS DB 5. File Upload 3. Proxy request 4. Proxy transfer 6. Update DB 7. Upload on Grid 7. Tracking 2. Upload request 16

DOGS: Data On Grid Services – Back-end implementation 17  JSAGA API used to transfer data from/to storage elements  Hibernate to manage the VFS collecting information on files stored on Grid; any changes/actions in the user view affect the VFS  MySQL as underlying RDBMS  An additional component has been developed in order to keep track of each transaction in the users tracking DB

DOGS: Data On Grid Services – Front-end implementation  A portlet has been created wit access provided only to federated users with given roles and privileges  The portlet view component includes elFinder, a web-based file manager developed in Javascript using jQuery UI for a dynamic and user friendly interface 

Data Engine in action (1/2) 19

Data Engine in action (2/2) «Share» to be added soon 20

 Middleware-dependent  Does not support all m/w in Italy  Data management missing  Non-customisable (by definition)  Personal certificates needed (until online CA becomes operational)  Support for workflows included  Job submission quite complex (for non expert users)  VO selection procedure exposed to end users  Non user-driven development  User base unclear General-purpose portal vs. Catania Science Gateway General-purpose portal Catania Science Gateway 21  Middleware-independent  Supports all m/w in Italy  Data management (almost) included  Highly & quickly customisable  No personal certificates needed (online CA can be used in future)  Support for workflows in progress  Job submission straightforward (details hidden to end users)  VO selection transparent to end users and managed via LDAP roles  User-driven development  Rapidly adopted and widely used

 Parallelamente al portale general purpose, IGI adotta il modello di Science Gateway di Catania e crea una Task Force allocando a Catania un congruo numero di FTE sul bilancio 2012  In cambio, il gruppo di Catania: Proposta di collaborazione (1/2) (durata: 1 anno, rinnovabile) 22

 Condividerà con il portale general purpose tutte quelle componenti ritenute di reciproco interesse/importanza  Opererà per tutti gli utenti di IGI i servizi della federazione «catch-all» GrIDP nelle more che questi si registrino a IDEM (n.b.: GrIDP include già l’IdP dell’INFN)  Creerà uno Science Gateway specifico per IGI registrandolo sia in IDEM che in eduGAIN e gestirà sia l’autenticazione che l’autorizzazione a tale portale per conto di IGI  S’incaricherà della produzione del materiale per promuovere l’adozione dello Science Gateway e permettere alle varie comunità di proporre nuove applicazioni  S’incaricherà, in parallelo, dello sviluppo delle portlet per le nuove applicazioni da integrare nello Science Gateway e della formazione di nuovi sviluppatori di portlet, mettendo a disposizione di IGI tutto il materiale di training già creato  Farà da tramite con gli sviluppatori di JSAGA e si farà carico dello sviluppo degli eventuali nuovi adaptor che si rendessero necessari  Includerà negli Science Gateway già esistenti l’infrastruttura IGI in modo da incrementarne l’utilizzo  Migrerà ad IGI le comunità nazionali già supportate (agricoltura, beni culturali, salute, scienze della terra, ecc.) aumentandone rapidamente il numero di utenti Proposta di collaborazione (2/2) (durata: 1 anno, rinnovabile) 23