DIRAC as a Service R. Graciani 1, V. Méndez 2, T. Fifield 3, A. Tsaregordtsev 4 1 University of Barcelona 2 University Autónoma of Barcelona 3 University of Melbourne 4 Centre de Physique des Particules de Marseille

Introduction ISGC 2014, TaipeiDIRAC as a Service2

The project ISGC 2014, TaipeiDIRAC as a Service3  Is a software project that provides a ready to use framework for distributed computing.  Started on 2002 to solve the LHCb use case and in 2010 it became general purpose, targeting other communities.  Today CERN, CNRS and UB have signed am agreement to create the DIRAC Consortium  “to confirm the collaboration between the partners … and their willingness to ensure the steady development and continuous support of the DIRAC software”.  User driven development cycle: bottom-up approach Presentation on Fri

The software ISGC 2014, TaipeiDIRAC as a Service4  Provides a complete middleware solution to build a dedicated grid: resource integration  Includes a large number of connectors to most commonly used grid middlewares, cloud managers, batch systems and remote storage solutions: interoperability  A complete framework to build a dedicated Virtual Research Environments: user interface  A platform for applications executing on a distributed environment: HTC PaaS

ISGC 2014, TaipeiDIRAC as a Service5 Tutorial on next Session

Makes it simple 6ISGC 2014, TaipeiDIRAC as a Service

ISGC 2014, TaipeiDIRAC as a Service7

DIRAC User Interfaces ISGC 2014, TaipeiDIRAC as a Service8

Interfaces 9  For most users  Web portal  Command line tools  For advanced users  Python API  RESTful interface  Science Gateways  Mobile Apps ISGC 2014, TaipeiDIRAC as a Service

Web Portal  Fully re-written in the last 18 month  Design goals:  Improve modularity + extendibility  Allow use in a single browser tab  Keep similar look and feel  Improve the usability  Robust framework  Based on previous experience (3 rd version, 10 years)  New tools and technologies ISGC 2014, Taipei10DIRAC as a Service

 Client side (browser):  Javascript  Heavy use of ExtJS 4.2.X  HTML  CSS  Google Charts  Server Side (web server)  Python  Tornado  DIRAC client tools New Web Framework ISGC 2014, Taipei11DIRAC as a Service

 Tornado  well tested framework for web servers ( scale UP )  Feedback:  asynch. Server methods  Multi-process (scale UP)  NGINX  Serves JavaScript (caching)  Redirects to Tornado  Load balancing for the same installation (scale UP)  Can connect to different “tornados” for different URLs (scale DOWN)  State less architecture:  DNS load balancing (scale UP) Designed for scalability ISGC 2014, Taipei12DIRAC as a Service

UML Diagram ISGC 2014, Taipei13DIRAC as a Service

Designed for modularity  One browser tab per user ID  Instance/Setup/User/Group  Common Framework  Provides general functionality and widget toolbox  Navigation and settings  Integrates several views:  Desktop like display (Default)  Tabs like display (being developed by LHCb)  User Profile Management (save, load, share, etc. states) ISGC 2014, Taipei14DIRAC as a Service

Designed for modularity  Applications  Clear recipe for developing new applications  Provide the specific functionality to the user  Can be completely isolated  Accounting  Can be connected to other applications  Job Launchpad -> Job Monitor  Must provide a well defined interface to the Framework  Can be integrated in third party portals (proven in CTA SG) ISGC 2014, Taipei15DIRAC as a Service

Web Portal 16 Launchpad Proxy Upload Certificate Authentication Job&Pilot Browsing Job&Pilot Monitor Configuration Browsing Accounting … or Admin ISGC 2014, TaipeiDIRAC as a Service

Coding new Applications  Developer must be familiar with underlying technologies (python, javascript, ExtJS)  Developer must have working experience with concurrent programming (AJAX, web sockets)  Server side is “similar” to other DIRAC services, but it has “synchronous” and “asynchronous” methods. No direct access to DBs, must use service interface.  Client side is well encapsulated, well defined interface to the framework:  Load (to instantiate and “load” a given state)  Save (returning the current state, to be saved)  Framework makes info available via a Global variable  Each application can define its own internal layout  ISGC 2014, Taipei17DIRAC as a Service

✔ Job monitor ✔ Pilot monitor ✔ Accounting ✔ Configuration manager ✔ File Catalog ✔ System administration ✔ Proxy Upload ✔ Job Launchpad ✔ Registry/Proxy Manager  Resource Manager ✔ Transformation Manager  Virtual Machine Manager ✔ System Monitor ✔ Request Monitor ✚ LHCb specific ones ✚ CTA specific ones ✚ Belle II specific ones ✚ others Coded Applications ISGC 2014, Taipei18DIRAC as a Service

REST Interface ISGC 2014, TaipeiDIRAC as a Service19

Why? ISGC 2014, TaipeiDIRAC as a Service20  Provide a language neutral interface for third party clients connecting to DIRAC:  Credential delegation: Oauth2  Query and request encoding: JSON  Query protocol: REST over HTTP  Being used by CTA SG (InSilicoLab/CYFRONET)  JSAGA connector to DIRAC (IN2P3)  It covers: Job Management & Accounting  Currently working on FC interface (CTA)

The authentication process ISGC 2014, TaipeiDIRAC as a Service21

New Components ISGC 2014, TaipeiDIRAC as a Service22  Server that exports the RESTful API.  It also generates and exchanges OAuth tokens.  A DIRAC Web Portal module that allows users to validate tokens.  Different privileges and life times can be granted.  A credential store where tokens and privileges are kept while tracking of all operations done to and with any token.  All RESTful servers check with the credential store to see if received tokens are still valid.

Running a DIRAC Service ISGC 2014, TaipeiDIRAC as a Service23

The Current Actors LHCb/WLCG Clouds EGI Fedcloud FG Clouds 24ISGC 2014, TaipeiDIRAC as a Service

Human Teams User Communities Scientific Director & Liaisons DIRAC Admins DIRAC Developers Resource Providers 25 User Communities Resource Providers ISGC 2014, TaipeiDIRAC as a Service

Proposal to EGI.eu ISGC 2014, TaipeiDIRAC as a Service26

Getting Started ISGC 2014, TaipeiDIRAC as a Service27

Getting to the users ISGC 2014, TaipeiDIRAC as a Service28

Moving ahead ISGC 2014, TaipeiDIRAC as a Service29

Looking for sustainability ISGC 2014, TaipeiDIRAC as a Service30

ISGC 2014, TaipeiDIRAC as a Service The different activities Production Instance Production Instance Maintenance R + D DIRAC Releasing DIRAC Releasing VRE Releasing VRE Releasing Code & Doc Coordination DIRAC Consortium Developers Operations Admins 31 Large Research Infrastructures Small/Medium Communities

Three main activities ISGC 2014, TaipeiDIRAC as a Service32  Develop framework and Core functionality  Scaling to the new challenges  Ensure that technology is followed  Develop Extensions for particular use cases  Closely follow up use cases and requirements  Integrate with third party solutions  Run services  Dedicated or shared  Define Computing Model  All these can be provided as a Service

Three main funding schemes ISGC 2014, TaipeiDIRAC as a Service33  Large RI projects using DIRAC (+ other TP)  Common DIRAC (+ other TP) + Infrastructure funding request  Other RI will have to contribute in a different manner  Pay for Development + Support  VREs developed together with user communities (+ other TP)  H2020 provides means to startup via EINFRA call (start with some communities + open calls)  Focus on dissemination and training  Pay for the Training  Providing a general purpose Service: DaaS  Together with EGI.eu + NGIs  Connect commercial users to commercial resources  Pay for the Usage

Summary ISGC 2014, TaipeiDIRAC as a Service34

ISGC 2014, TaipeiDIRAC as a Service35  DIRAC project has evolved and it is able to provide high- value services to scientists and beyond.  DIRAC software has modular and extendible design based on a flexible framework.  DIRAC supports big science and the long tail  Synergies, Cost-Savings, best Practices, share Experience  Shared activities: development, training, services  Looking for the long term sustainability  Partnership with users  Pay for the service  Do you have other working models?