ALICE: le strategie per l’analisi Massimo Masera Dipartimento di Fisica Sperimentale e INFN sezione di Torino Workshop CCR e INFN-GRID 2009

Outline The approach of the ALICE experiment to analysis tasks:  Analysis framework  Implications for the Tier-1 and Tier-2 computing centres Issues:  Lessons from the previous talk  Stability of the services  Human resources: Management of the sites Support to the users WITHIN the ALICE national community Interactive analysis  Analysis with PROOF  PROOF facilities: use of virtualization within a Tier-2 centre: A protoype 213/05/2009ALICE analysis

Analysis in ALICE The proposed sub-title for this talk is:  Un rappresentante senior dell'esperimento descrive il piano con cui la collaborazione italiana si sta preparando ad utilizzare le risorse ai T2 e T3 per l'analisi dei dati prodotti ad LHC… (it goes on, but it is enough for now) There are 2 points which deserve a preliminary comment  “Italian collaboration”. Well, the plan to analyze LHC data is unique for the whole experiment. There is no room, within the ALICE computing rules, for approaches, which are not integrated with the ALICE offline framework. Nevertheless: The Italian community has to organize the human resources to manage the computing centres and to support the users. This is critical, but it is not peculiar to the analysis part only. Interactive analysis. We are planning to deploy an analysis facility to be operated with PROOF. 313/05/2009ALICE analysis

Analysis in ALICE There are 2 points which deserve a preliminary comment (cont’d)  How do we plan to use Tier-3  Within INFN-Grid, Tier-3 has been a forbidden word so far (often a forbidden dream): A too easy way to keep normal users away from GRID solutions; A way to make everybody happy with a local computing farm to play with.  In the ALICE computing model there is no a specific role for Tier-3  in the model are considered “community oriented” contributions, open to the whole collaboration  Physics results can be published only if obtained on the GRID, with input/output available on the GRID  Local computing resources are hence considered “private” and intended only for the development phase of the code and of the analysis tasks  very small communities/single physicists Sometimes a desktop does the job Small farms proved to be very effective for these purposes 413/05/2009ALICE analysis

Analysis in ALICE There are 2 points which deserve a preliminary comment (cont’d)  How do we plan to use Tier-3  PROOF clusters do not fit in the MONARC naming scheme and are not operated as LCG centres  However the only PROOF cluster mentioned in our computing model, the CERN Analys Facility, proved to be very effective and it is quite popular in the ALICE community.  PROOF analysis facilities have been deployed in Germany (GSI) and are in the final stage of deployment in France (Lyon). They are opened (at least nominally) to any member of the collaboration.  We are planning to build PROOF clusters also also in Italy, intended as virtual facilities largely based on existing hardware in Tier-2 centres.  PROOF clusters are somehow the ALICE reinterpretation of the Tier-3 concept They need to be integrated in the production framework and opened to potentially every ALICE user 513/05/2009

Analysis framework The analysis framework has been introduced by A. Dainese in the previous talk The framework is common to both scheduled and chaotic analysis 6 The analysis code is organized in classes which inherit from AliAnalysisTask The analysis is steered by a AliAnalysisManager object The goal is to be able to develop with a single macro the analysis Locally (1 st phase) Possibly on a PROOF cluster (test) On the GRID (production) It works with AOD as well ALICE analysis

Scheduled analysis The first analysis step has ESD (Event Summary Data) as input. It is a scheduled analysis procedure. Analyses which are carried out on ESDs are done where ESDs are stored: mainly in Tier-1 sites for RAW ESDs and Tier-2 for MC ESDs. The output can range from simple histograms to AODs (Analysis Object Datasets), which in turn can be used as input for subsequent analysis passes AODs are replicated in at least 3 Tier-2 centres (more depending on popularity) to optimize data access and availability 713/05/2009ALICE analysis

Scheduled analysis Analysis tasks to be included in scheduled analyses must obey to basically two validation criteria:  Physics-wise: validation is done within the Physics Working Groups  Framework-wise: the code must run seamlessly on the GRID with a released version of AliRoot. Different tasks are executed together, grouped in the so called analysis trains 8 Scheduled analysis passes are steered by the core offline team on the sites contributing to ALICE Priorities defined by the Physics Board 13/05/2009ALICE analysis

Chaotic analysis Steered by small groups of physicists (at the limit single physicists) Done on Tier-2 sites: AODs as input Same framework as for scheduled analysis  Essentially shorter trains sent by individual users Tier-2 resources will be devoted (actually are) with priority to the user analysis The management of user priorities is not an issue (single task queue for all the jobs)  Priorities are managed at central services level Results of the analysis can be directly accessed from a root session running on a ALICE user’s laptop  Only alien-client is needed  Done with TGrid class  File accessed via xrootd 913/05/2009ALICE analysis

Chaotic analysis The analysis framework provides tools to allow the development of an analysis macro on local resources and its usage on a PROOF cluster or on the GRID Local CPU resources do not mean local data: ALICE catalogue and data can be accessed from virtually any PC All the AODs corresponding to a given dataset are replicated on several Tier-2. There are no different roles for different Tier-2’s. Data localization in principle is not relevant  In some cases we asked to replicate on Italian T2’s data of particular interest for our community, mostly to have the opportunity to test our sites performance  In any case, Physics Working Groups are naturally spread  We need to support ALICE users from any country participating in the experiment 1013/05/2009ALICE analysis

Issues for INFN Tier-x sites Lessons from the previous talk:  The analysis framework is being tested by physicists in view of its use with real data  Results are positive: no urgent need (and time is running out) for new features Problems like file collocation issues will be addressed at the level of the central catalogue With recent xrootd, the global redirector solves this problem  Stability and reliability of the sites and of local ALICE services are essential to run an analysis in an effective way. 1113/05/2009ALICE analysis

Issues for INFN Tier-x sites Lessons from the previous talk:  Human resources, also within the collaboration, are the main asset for a successful usage of the resources As a national community we are aiming to form a group of collaboration members able to monitor and to some extent manage Tier-2 activities This group should contribute also to the user support. We had a first “hands-on” meeting last January in Torino.  We decided to exploit the existing INFN-GRID ticketing system An ALICE support unit has been created A couple of people/site has been registered as supporter 1213/05/2009ALICE analysis

Interactive analysis There are several tasks which would greatly benefit from an interactive approach. For instance:  Validation of the analysis algorithms on a sizeable, but limited data set  Tuning of multidimensional data selection cuts  In general analysis items for which a prompt response is critical, typically in view of a major analysis pass The CERN Analysis Facility is highly appreciated by ALICE physicists, but it can accommodate a limited number of users In Italy we cannot afford to build a static facility for interactive use Using the XEN hypervisor it is possible to build a fully virtual Analysis Facility on the same Worker Nodes that compose an existing LCG Grid Farm  this maximize HW usage A small protoype is being tested in Torino 13 XEN dom0 LCG WorkerNode PROOF slave 13/05/2009 Preliminary results presented at ACAT08 and at CHEP09

root Remote PROOF Cluster Data root Client – Local PC ana.C stdout/result node1 node2 node3 node4 ana.C root PROOF Schema Data Proof master Proof slave Result Data Result Data Result 13/05/200914ALICE analysis

“LCG” Configuration 15 LCG CE WN PROOF Master  Start with an LCG CE farm  Virtualize the Worker Nodes and add a second Virtual Machine on each of them, running a Parallel Interactive platform like PROOF  Both environments are sandboxed and independent  Xen can dynamically allocate resources (both CPU priority and memory) to either machine, no reboot or restart needed  Normal operation: PROOF slaves are “dormant” (minimal memory allocation, very low CPU priority) 13/05/2009ALICE analysis

“PROOF” Configuration 16 LCG CE WN PROOF Master Slave  When needed, resource can be moved to from virtual LCG WNs to virtual PROOF slaves with minimal latency  Grid batch job on the WN ideally never completely stops, only slows down: non- CPU-intensive I/0 operations can go on and do not timeout  As the demand for interactive access increases, resources can be added either by shrinking further the WNs or by “waking up” more PROOF slaves  As soon as everybody goes home, resources can be moved back to the WNs, that resume batch processing at full speed 13/05/2009ALICE analysis

The prototype 13/05/2009ALICE analysis17 Hardware 4x HP ProLiant 360DL, dual quad-core, plus one head node for access, management and monitoring Separate physical 146GB SAS disk for each virtual machine performance isolation Private network with NAT to outside world (currently including storage) Software Linux CentOS 5.1 with kernel on dom0 Xen 3.0 gLite 3.1 Worker Node suite PROOF/Scalla Custom monitoring & management tools

Moving resources 13/05/2009ALICE analysis18  Moving resources from the Wn to the PROOF slave, as seen by the WN  About 1.5 minutes to complete transition (but the slave can be started almost immediately)  Moving resources from the Wn to the PROOF slave, as seen by the WN  About 1.5 minutes to complete transition (but the slave can be started almost immediately)  CPU efficiency (CPU time/wall clock time) for regular ALICE MC production/reconstruction jobs, in three different resource configurations  Jobs become increasingly I/O-bound as swap activity increases  No abnormal job terminations observed  CPU efficiency (CPU time/wall clock time) for regular ALICE MC production/reconstruction jobs, in three different resource configurations  Jobs become increasingly I/O-bound as swap activity increases  No abnormal job terminations observed

Performance 13/05/2009ALICE analysis19  Parallel interactive analysis on PROOF nodes  Regular load on WNs (mix of memory- hungry ALICE production jobs, ALICE user jobs and some jobs from other VOs)  PROOF scaling OK (saturates only when # of workers = # of CPUs)  Parallel interactive analysis on PROOF nodes  Regular load on WNs (mix of memory- hungry ALICE production jobs, ALICE user jobs and some jobs from other VOs)  PROOF scaling OK (saturates only when # of workers = # of CPUs)  PROOF processing rate in several resource configuration  Separate physical disks ensure minimal impact from swapping  System behave as expected: Grid jobs go on running, even though very slowly, and resources are efficiently exploited by PROOF  PROOF processing rate in several resource configuration  Separate physical disks ensure minimal impact from swapping  System behave as expected: Grid jobs go on running, even though very slowly, and resources are efficiently exploited by PROOF

to-do list Develop an automatic system for resource allocation Investigate the optimal resource allocation policy Test for diret access to local SE  Local network optimization Accounting:  Interface with standard accounting system 13/05/2009ALICE analysis20

Conclusions INFN sites will participate in the ALICE analysis effort, according to the computing model The analysis framework is being used by real users  it has the functionality to deal with real LHC data Issues:  Storage solution must be stable and efficient. See S. Bagnasco’s talk for a discussion of the existing solutions suitable for ALICE (data accessed with xrootd)  Human resources: growing involvement of ALICE members in production and user support activities. Needed to have an adequate service level We are planning to devote dynamically a limited amount of resources (at T2) to PROOF 13/05/2009ALICE analysis21

13/05/2009ALICE analysis22 Siamo attrezzati per l’analisi