Presentation is loading. Please wait.

Presentation is loading. Please wait.

AliE Pablo Saiz/CERN P.Buncic, J-E. Revsbech R.Piskac, V.Sego, L. Aphecetche ALICE Collaboration ALICE Environment on the GRID.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "AliE Pablo Saiz/CERN P.Buncic, J-E. Revsbech R.Piskac, V.Sego, L. Aphecetche ALICE Collaboration ALICE Environment on the GRID."— Presentation transcript:

1 AliE n @GRID Pablo Saiz/CERN P.Buncic, J-E. Revsbech R.Piskac, V.Sego, L. Aphecetche ALICE Collaboration ALICE Environment on the GRID

2 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch2 Content Alice at LHC Alice Computing Model Building AliEn AliEn Components Deploying AliEn AliEn Roadmap Conclusions

3 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch3 ALICE @ LHC

4 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch4 CERN - LHC

5 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch5 Construction

6 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch6 Problem Typical next generation HEP experiment Large scale simulation & reconstruction effort Heavily distributed processing and event storage  ~1000 scientists in ~100 of institutions Complex analyses of distributed data Large files (one event up to 2GB) 10^9 files/year (x n, n>2) 2 PB/year Experiment lifetime 20-25 years GRID Widely accepted as a solution

7 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch7 Alice Use Cases Simulation, Data Challenges & Reconstruction Centrally managed production of background events Distributed processing and event storage Event mixing Not necessarily centrally managed Once background events exist, the subsequent requests for event mixing must be routed to the location which holds required input Analysis Using AliEn API, PROOF will locate optimal site(s) for macro execution, try to execute it in parallel, collect the output and return it to the user (or register it in the catalogue)

8 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch8 ALICE Computing Model

9 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch9 AliE n @GRID AliROOT ROOT User Simulation, Reconstruction, Calibration, Analysis System GUI Persistent IO Utility libs World Interfaces & Distributed computing environment C++ anything Nice! I only have to learn C++

10 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch10 Challenge Can we provide, building on top of available public domain and open source components and standards, a functional distributed computing infrastructure to community of ALICE users which will remain operational even if underlying technologies keep changing?

11 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch11 Building AliEn

12 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch12 Building AliEn

13 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch13 Open Source Components SASL/OpenSSL/OpenCA as authentication protocol Globus/GSS as an implementation of authentication compatible with other Grid projects CONDOR ClassAds language for job description (compatible with EU DataGrid) OpenLDAP for configuration management Apache for Web Portal MySQL as relational database backend Bbftp as file transfer protocols

14 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch14 Gluing it together… Already existing pieces of code (NA49 file catalogue) in perl5 Good interface to different databases Easy Web integration Simple Object Access Protocol (also known as Service Oriented Access Protocol) Good Perl implementation (SOAP::Lite) on client and server side Possibility to provide client access from many different platforms and languages (Java,C,C++…) Provides standard means to invoke procedures (services) in distributed environment

15 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch15 Components AliEn Services Modules & libraries

16 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch16 “Web of Services”

17 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch17 Statistics… SLOCCount is Open Source Software/Free Software, licensed under the FSF GPL.Please credit this data as "generated using 'SLOCCount' by David A. Wheeler."

18 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch18 Statistics… SLOCCount is Open Source Software/Free Software, licensed under the FSF GPL.Please credit this data as "generated using 'SLOCCount' by David A. Wheeler.“

19 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch19 Benefits of development based on OpenSource components are more than obvious… AliEn vs OpenSource

20 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch20 AliEn Components

21 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch21 AliEn SASL implementation SASL is the Simple Authentication and Security Layer, a method for adding authentication support to connection-based protocols AliEn now has perl module with implementation GSSAPI This allows us to use all SASL authentication schemes old AliEn authentication (token, AFS password, SSH) X509 certificates Globus/GSI (credential delegation) AliEn distribution includes necessary Globus/MDS/GSI software This allows us to develop secure Peer-To-Peer File Transfers based on machine/protocol/user certificates and LDAP based configuration management

22 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch22 Authentication Client Proxy Server Database LDAP Request methods List of methods SASL Authentication Checking if user exists Data X509(AliEn/Globus) PKI/RSA (ssh) Token (AliEn) AFS password

23 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch23 File catalogue ALICE USERS ALICE SIM Tier1 ALICE LOCAL |--./ | |--cern.ch/ | | |--user/ | | | |--a/ | | | | |--admin/ | | | | | | | | | |--aliprod/ | | | | | |--f/ | | | | |--fca/ | | | | | |--p/ | | | | |--psaiz/ | | | | | |--as/ | | | | | | | | |--dos/ | | | | | | | | |--local/ |--simulation/ | |--2001-01/ | | |--V3.05/ | | | |--Config.C | | | |--grun.C | |--36/ | | |--stderr | | |--stdin | | |--stdout | | |--37/ | | |--stderr | | |--stdin | | |--stdout | | |--38/ | | |--stderr | | |--stdin | | |--stdout | | | | | |--b/ | | | | |--barbera/ Files, commands (job specification) as well as job input and output and metadata are stored in the catalogue

24 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch24 Command Interface

25 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch25 GUI: AliEn Xfiles

26 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch26 Web Portal http://alien.cern.ch Generic Web portal Virtual Organizations Alice Atlas NA49 Demo Mammogrid

27 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch27 Task Queue “Pull” rather than “push”architecture

28 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch28 AliEnTasksCEs alien job-submit job.jdl Broker Yes: Select Match ? No: Next CE contacts CPUServer and presents its own ClassAd, Resource Broker will match them against job ClassAds and select the most appropriate job to run on that CE Resource Broker

29 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch29 Resource Broker Resource Broker Optimizer

30 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch30 Class Ads &JDL Requirements = ( other.Type == "machine" ) &&(member(other.Packages,"AliRoot") ); Packages = "AliRoot"; Arguments = "--round 2002-02 --run 00071 --event 269 --version v3.08.02 -–grun G+F"; Executable = "/Alice/bin/AliRoot.sh"; InputFile = { "LF:/alice/simulation/2002-02/v3.08.02/00071/Config.C", "LF:/alice/simulation/2002-02/v3.08.02/00071/grun.C" }; Type = "Job"; An Example – JDL file to run Alice Simulation job:

31 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch31 Class Ads &JDL Requirements = ( other.Type == “Job" ); Type = “machine"; Host = “alienx.cern.ch”; CE =“Alice::CERN::LXBATCH”; Packages = { "AliRoot“, “ROOT”, “AliRoot::3.08.02” }; CloseSE = { “Alice::CERN::Castor”, “Alice::CERN::File”, “Alice::CERN::scratch” }; Class Ads of CE:

32 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch32 Computer Local Center GRID CENTER Computer Monitoring In order to develop and deploy more refined Resource Broker we need monitoring framework Frequent data updates, large data volume for large number of computers The idea is to implement hierarchy of clients and servers where each client (child) maintains the history of measurements reports the summary information to upper layer (parent) using SOAP protocol

33 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch33 Deploying AliEn

34 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch34 First implementation of Alice World Computing Model AliEn@GRID

35 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch35 Production Summary 5682 events validated, 118 failed (2%) Up to 300 concurrently running jobs worldwide (5 weeks) 5 TB of data generated and stored at the sites with mass storage capability (CERN 73%,CCIN2P3 14%, LBL, 14%, OSC 1%) GSI, Karlsruhe, Dubna, Nantes, Budapest, Bologna, Zagreb, Birmingham, Utrecht, Calcutta in addition ready by now 13 clusters, 9 sites 10^5 CPU hours

36 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch36 AliEn Roadmap

37 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch37 AliEn as a meta-GRID AliEn User Interface AliEn stackiVDGL stackEDG stack

38 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch38 Roadmap… Optimization and test suite PROOF interface & support for interactive jobs EDG interface GRID partitioning Queue optimization (based on AliEn monitoring) Implementation of Web services SOAP (Simple Object Access Protocol) WSDL (Web Services Description Language) UDDI (Universal Description Discovery & Integration) Virtual datasets

39 AliE n @GRID 6/4/2015Pablo.Saiz@cern.ch39 Summary AliEn framework is a lightweight, simplified but functionally equivalent alternative to full blown GRID based on standard components (SOAP, Web services) It has been tested in production will be continuously developed with aim to provide long term stable interface to GRID(s) for Alice users AliEn will used to provide GRID component for MammoGRID – 3 year, 2M Euro project funded by EC, starting in September Summary of AliEn features (visit http://alien.cern.ch) Authentication module which supports various authentication methods (Globus/GSI) Distributed file catalogue built on top of RDBMS with user interface that mimics the file system Secure file transport and replication Service Task queue which holds commands to be executed in the system and Resource Broker Configuration and Information Service Computing and Storage elements Metadata catalogue Monitoring framework C/C++/perl API Web portal

Download ppt "AliE Pablo Saiz/CERN P.Buncic, J-E. Revsbech R.Piskac, V.Sego, L. Aphecetche ALICE Collaboration ALICE Environment on the GRID."

Similar presentations

1 ALICE Grid Status David Evans The University of Birmingham GridPP 14 th Collaboration Meeting Birmingham 6-7 Sept 2005.

1 ALICE Grid Status David Evans The University of Birmingham GridPP 14 th Collaboration Meeting Birmingham 6-7 Sept 2005.
1 ALICE Grid Status David Evans The University of Birmingham GridPP 16 th Collaboration Meeting QMUL 27-29 June 2006.

1 ALICE Grid Status David Evans The University of Birmingham GridPP 16 th Collaboration Meeting QMUL June 2006.
Data Management Expert Panel - WP2. WP2 Overview.

Data Management Expert Panel - WP2. WP2 Overview.
1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.

1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.
1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.

1 Software & Grid Middleware for Tier 2 Centers Rob Gardner Indiana University DOE/NSF Review of U.S. ATLAS and CMS Computing Projects Brookhaven National.
1 Grid services based architectures Growing consensus that Grid services is the right concept for building the computing grids; Recent ARDA work has provoked.

1 Grid services based architectures Growing consensus that Grid services is the right concept for building the computing grids; Recent ARDA work has provoked.
23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.

23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.
Workload Management Workpackage Massimo Sgaravatto INFN Padova.

Workload Management Workpackage Massimo Sgaravatto INFN Padova.
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.

Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.

DataGrid Kimmo Soikkeli Ilkka Sormunen. What is DataGrid? DataGrid is a project that aims to enable access to geographically distributed computing power.
Workload Management Massimo Sgaravatto INFN Padova.

Workload Management Massimo Sgaravatto INFN Padova.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
EU 2nd Year Review – 04-05 Jan. 2003 – WP9 WP9 Earth Observation Applications Demonstration Pedro Goncalves :

EU 2nd Year Review – Jan – WP9 WP9 Earth Observation Applications Demonstration Pedro Goncalves :
AliEn Tutorial MODEL 2009 4 th May, 2009. 2 May 2009 Installation of the AliEn software AliEn and the GRID Authentication File Catalogue.

AliEn Tutorial MODEL th May, May 2009 Installation of the AliEn software AliEn and the GRID Authentication File Catalogue.
- 1 - Grid Programming Environment (GPE) Ralf Ratering Intel Parallel and Distributed Solutions Division (PDSD)

- 1 - Grid Programming Environment (GPE) Ralf Ratering Intel Parallel and Distributed Solutions Division (PDSD)
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.

 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.

CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.
AliEn uses bbFTP for the file transfers. Every FTD runs a server, and all the others FTD can connect and authenticate to it using certificates. bbFTP implements.

AliEn uses bbFTP for the file transfers. Every FTD runs a server, and all the others FTD can connect and authenticate to it using certificates. bbFTP implements.
5 November 2001F Harris GridPP Edinburgh 1 WP8 status for validating Testbed1 and middleware F Harris(LHCb/Oxford)

5 November 2001F Harris GridPP Edinburgh 1 WP8 status for validating Testbed1 and middleware F Harris(LHCb/Oxford)
ARGONNE  CHICAGO Ian Foster Discussion Points l Maintaining the right balance between research and development l Maintaining focus vs. accepting broader.

ARGONNE  CHICAGO Ian Foster Discussion Points l Maintaining the right balance between research and development l Maintaining focus vs. accepting broader.

Similar presentations

Ads by Google