1. Multi-community e-Science service connecting grids & clouds R. Graciani 1, V. Méndez 2, T. Fifield 3, A. Tsaregordtsev 4 1 University of Barcelona 2 University Autónoma of Barcelona 3 University of Melbourne 4 Centre de Physique des Particules de Marseille

2. Agenda ISGC 2014, TaipeiMulti-community with Grids & Clouds 2  The DIRAC project  Multiple-community platform  Scientific computing platform  High level services for scientist and others  Transparent integration of Grids & Clouds  Conclusions & Outlook

3. The project ISGC 2014, TaipeiMulti-community with Grids & Clouds 3  Is a software project that provides a ready to use framework for distributed computing.  Started on 2002 to solve the LHCb use case and in 2010 it became general purpose, targeting other communities.  Today CERN, CNRS and UB have signed am agreement to create the DIRAC Consortium  “to confirm the collaboration between the partners … and their willingness to ensure the steady development and continuous support of the DIRAC software”.  User driven development cycle: bottom-up approach Presentation on Fri

4. The software ISGC 2014, TaipeiMulti-community with Grids & Clouds 4  Provides a complete middleware solution to build a dedicated grid: resource integration  Includes a large number of connectors to most commonly used grid middlewares, cloud managers, batch systems and remote storage solutions: interoperability  A complete framework to build a dedicated Virtual Research Environments: user interface  A platform for applications executing on a distributed environment: HTC PaaS

5. http://diracgrid.org ISGC 2014, TaipeiMulti-community with Grids & Clouds 5 Presentation on Thu

6. Tutorial on Thu Multi-Community ISGC 2014, TaipeiMulti-community with Grids & Clouds 6  A single DIRAC instance is able to host multiple virtual research environments  This allows economies of scale in two levels  Software management  Operational effort  Examples multi-communities installations:  DIRAC 4 FranceGrilles @ IN2P3 (France)  DIRAC 4 IberGrid @ UB (Spain-Portugal)  DIRAC 4 INFN @ Bari/CNAF (Italy, 2 instances)  DIRAC 4 IHEP @ IHEP (China)  DIRAC 4 GridPP @ Imperial College  …  Providing web interface and service API

7. ISGC 2014, TaipeiMulti-community with Grids & Clouds 7 Human Teams User Communities Scientific Director & Liaisons DIRAC Admins DIRAC Developers Resource Providers User Communities Resource Providers

8. Multi-Community Platform ISGC 2014, TaipeiMulti-community with Grids & Clouds 8 ~ 1.500

9. Multi-Community Platform ISGC 2014, TaipeiMulti-community with Grids & Clouds 9 ~ 1.500

10. DIRAC 4 EGI ISGC 2014, TaipeiMulti-community with Grids & Clouds 10  Single access point to e-Infrastructure  Go beyond national level  Reduce the operational costs  Improve in resilience  Incorporate all resources/users under EGI.eu umbrella  Grid + FedCloud + EDGF  Extend to SSO, beyond users with x509 grid certificates in VO  Support third party portals  Partnership among NGIs  France, Spain, Poland, Italy, …  Commercial cloud providers More Details on Wed

11. Platform 4 Scientific Computing (I) ISGC 2014, TaipeiMulti-community with Grids & Clouds 11  Community Management tools  Groups  Privileges  Policies  Support third party group management services (Ie VOMS)  Adaptive workload management  User priority  Community wide policies  Status of the resources  Community wide data management policies

12. Platform 4 Scientific Computing (II) ISGC 2014, TaipeiMulti-community with Grids & Clouds 12  Parametric jobs  bulk submissions  Job templates  similar jobs  Workflows to associate applications with  data driven productions  File Catalog:  Metadata: facilitate identifying your data  Replica: locate your data  Provenance: track the source of the data  Connection from application specific portals:  DIRAC as a Service More Details next Session

13. Platform 4 Scientific Computing (III) ISGC 2014, TaipeiMulti-community with Grids & Clouds 13  High level services for scientist enable to define computing tasks in a resource neutral way decoupled from resource provisioning  Security infrastructure  Digital certificate authentication  Single and Multi-Community portal  Resource usage Accounting for providers, communities, groups and users  CPU, Jobs, WallTime  Storage  Transfers

14. DIRAC as VRE (I) ISGC 2014, TaipeiMulti-community with Grids & Clouds 14

15. ISGC 2014, TaipeiMulti-community with Grids & Clouds 15 DIRAC as VRE (II) Launchpad Proxy Upload Certificate Authentication Job&Pilot Browsering Job&Pilot Monitor Configuration Browsering Accounting … or Admin

16. ISGC 2014, TaipeiMulti-community with Grids & Clouds 16 DIRAC as VRE (III) Android App Looking for beta testers

17. Grids & Clouds ISGC 2014, TaipeiMulti-community with Grids & Clouds 17  Aggregation of different resource providers  gLite: EGI  VDT: OSG  ARC: NDGF sites, RAL, etc.

18. Grids & Clouds ISGC 2014, TaipeiMulti-community with Grids & Clouds 18  Resource allocation policies  Elastic VM launching driven by workload  Static VM launching for dedicated resources  VMs are stopped when no longer needed  Aggregation of resource providers  Amazon, OCCI, OpenNebula, OpenStack, CloudStack  Golden images with dynamic context  Preconfigured ad-hoc images  Fully interoperable with Grid resources

19. Clusters Volunteer Computing ISGC 2014, TaipeiMulti-community with Grids & Clouds 19  Torque  Condor  OAR  SLURM  LSF  EDGI  BOINC Grids & Clouds & Others

20. 20  Aggregation of storage resources  Access to standard grid storage elements  Vanilla DIRAC storage element  Others: xrootd, sftp,…  Integration of multiple data catalogs  LFC (replica)  DIRAC File Catalog (replica & provenance & metadata) ISGC 2014, TaipeiMulti-community with Grids & Clouds

21. ISGC 2014, Taipei21Multi-community with Grids & Clouds API/REST/Web/CLI Job Manager Job Monitor Optimization Mind Job Sanity Optimizer Job Path Optimizer Job Path Optimizer Input Data Optimizer Scheduling Optimizer TaskQueue DB Job DB Matcher Job State Update VM Scheduler Pilot Scheduler VM Cloud A VM Cloud B Pilot Grid A Pilot Grid B Pilot Site A Sandbox Job Agent 1 11 1 1 1 11 1 2 3 3 4 5 6 7 2 2 2 2 2 2343311 2 1 Resource Reservation 1 Workload Submission 1 Workload Execution 1 Result Retrieval

22. Conclusions & Outlook ISGC 2014, TaipeiMulti-community with Grids & Clouds 22  DIRAC supports advance distributed computing for scientific communities: DIRAC as a Service  VRE extensions  Connection from third party VRE  Allows multiple communities on a single instance  Reduce installation and operational cost  Provides transparent integration of different resources: DIRAC as a Platform  Grids, Clouds, Volunteer, etc.  Computing & Catalog & Storage  From National instances towards a Global instance You are invited to join