1 User Analysis Workgroup Update  All four experiments gave input by mid December  ALICE by document and links  Very independent.

Slides:



Advertisements
Similar presentations
Applications Area Issues RWL Jones GridPP13 – 5 th June 2005.
Advertisements

4/2/2002HEP Globus Testing Request - Jae Yu x Participating in Globus Test-bed Activity for DØGrid UTA HEP group is playing a leading role in establishing.
Storage Issues: the experiments’ perspective Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 9 September 2008.
Implementing Finer Grained Authorization in the Open Science Grid Gabriele Carcassi, Ian Fisk, Gabriele, Garzoglio, Markus Lorch, Timur Perelmutov, Abhishek.
ALICE Operations short summary and directions in 2012 Grid Deployment Board March 21, 2011.
ALICE Operations short summary and directions in 2012 WLCG workshop May 19-20, 2012.
Large scale data flow in local and GRID environment V.Kolosov, I.Korolko, S.Makarychev ITEP Moscow.
LCG Milestones for Deployment, Fabric, & Grid Technology Ian Bird LCG Deployment Area Manager PEB 3-Dec-2002.
CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.
INTRODUCTION The GRID Data Center at INFN Pisa hosts a big Tier2 for the CMS experiment, together with local usage from other HEP related/not related activities.
Overview of day-to-day operations Suzanne Poulat.
11/30/2007 Overview of operations at CC-IN2P3 Exploitation team Reported by Philippe Olivero.
Your university or experiment logo here Caitriana Nicholson University of Glasgow Dynamic Data Replication in LCG 2008.
LCG Plans for Chrsitmas Shutdown John Gordon, STFC-RAL GDB December 10 th, 2008.
INFSO-RI Enabling Grids for E-sciencE DAGs with data placement nodes: the “shish-kebab” jobs Francesco Prelz Enzo Martelli INFN.
WLCG Service Report ~~~ WLCG Management Board, 1 st September
November SC06 Tampa F.Fanzago CRAB a user-friendly tool for CMS distributed analysis Federica Fanzago INFN-PADOVA for CRAB team.
Cracow Grid Workshop October 2009 Dipl.-Ing. (M.Sc.) Marcus Hilbrich Center for Information Services and High Performance.
CCRC’08 Weekly Update Jamie Shiers ~~~ LCG MB, 1 st April 2008.
Enabling Grids for E-sciencE System Analysis Working Group and Experiment Dashboard Julia Andreeva CERN Grid Operations Workshop – June, Stockholm.
Maarten Litmaath (CERN), GDB meeting, CERN, 2006/02/08 VOMS deployment Extent of VOMS usage in LCG-2 –Node types gLite 3.0 Issues Conclusions.
Dan Tovey, University of Sheffield User Board Overview Dan Tovey University Of Sheffield.
What is SAM-Grid? Job Handling Data Handling Monitoring and Information.
Caitriana Nicholson, CHEP 2006, Mumbai Caitriana Nicholson University of Glasgow Grid Data Management: Simulations of LCG 2008.
CASTOR evolution Presentation to HEPiX 2003, Vancouver 20/10/2003 Jean-Damien Durand, CERN-IT.
1 User Analysis Workgroup Discussion  Understand and document analysis models  Best in a way that allows to compare them easily.
WLCG Grid Deployment Board, CERN 11 June 2008 Storage Update Flavia Donno CERN/IT.
US LHC OSG Technology Roadmap May 4-5th, 2005 Welcome. Thank you to Deirdre for the arrangements.
1 Andrea Sciabà CERN Critical Services and Monitoring - CMS Andrea Sciabà WLCG Service Reliability Workshop 26 – 30 November, 2007.
SLACFederated Storage Workshop Summary For pre-GDB (Data Access) Meeting 5/13/14 Andrew Hanushevsky SLAC National Accelerator Laboratory.
INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.
XROOTD AND FEDERATED STORAGE MONITORING CURRENT STATUS AND ISSUES A.Petrosyan, D.Oleynik, J.Andreeva Creating federated data stores for the LHC CC-IN2P3,
Site Manageability & Monitoring Issues for LCG Ian Bird IT Department, CERN LCG MB 24 th October 2006.
The CMS Top 5 Issues/Concerns wrt. WLCG services WLCG-MB April 3, 2007 Matthias Kasemann CERN/DESY.
Testing and integrating the WLCG/EGEE middleware in the LHC computing Simone Campana, Alessandro Di Girolamo, Elisa Lanciotti, Nicolò Magini, Patricia.
DIRAC Pilot Jobs A. Casajus, R. Graciani, A. Tsaregorodtsev for the LHCb DIRAC team Pilot Framework and the DIRAC WMS DIRAC Workload Management System.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Monitoring of the LHC Computing Activities Key Results from the Services.
LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006.
MND review. Main directions of work  Development and support of the Experiment Dashboard Applications - Data management monitoring - Job processing monitoring.
Author - Title- Date - n° 1 Partner Logo WP5 Status John Gordon Budapest September 2002.
1. 2 Overview Extremely short summary of the physical part of the conference (I am not a physicist, will try my best) Overview of the Grid session focused.
Distributed Data Access Control Mechanisms and the SRM Peter Kunszt Manager Swiss Grid Initiative Swiss National Supercomputing Centre CSCS GGF Grid Data.
INFSO-RI Enabling Grids for E-sciencE gLite Certification and Deployment Process Markus Schulz, SA1, CERN EGEE 1 st EU Review 9-11/02/2005.
Distributed Physics Analysis Past, Present, and Future Kaushik De University of Texas at Arlington (ATLAS & D0 Collaborations) ICHEP’06, Moscow July 29,
Enabling Grids for E-sciencE INFSO-RI Enabling Grids for E-sciencE Gavin McCance GDB – 6 June 2007 FTS 2.0 deployment and testing.
Site Services and Policies Summary Dirk Düllmann, CERN IT More details at
LHCC Referees Meeting – 28 June LCG-2 Data Management Planning Ian Bird LHCC Referees Meeting 28 th June 2004.
Status of gLite-3.0 deployment and uptake Ian Bird CERN IT LCG-LHCC Referees Meeting 29 th January 2007.
Meeting with University of Malta| CERN, May 18, 2015 | Predrag Buncic ALICE Computing in Run 2+ P. Buncic 1.
Monitoring the Readiness and Utilization of the Distributed CMS Computing Facilities XVIII International Conference on Computing in High Energy and Nuclear.
EGEE-III INFSO-RI Enabling Grids for E-sciencE VO Authorization in EGEE Erwin Laure EGEE Technical Director Joint EGEE and OSG Workshop.
Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES The Common Solutions Strategy of the Experiment Support group.
II EGEE conference Den Haag November, ROC-CIC status in Italy
SAM architecture EGEE 07 Service Availability Monitor for the LHC experiments Simone Campana, Alessandro Di Girolamo, Nicolò Magini, Patricia Mendez Lorenzo,
VO Box discussion ATLAS NIKHEF January, 2006 Miguel Branco -
WLCG Accounting Task Force Update Julia Andreeva CERN GDB, 8 th of June,
Grid Deployment Technical Working Groups: Middleware selection AAA,security Resource scheduling Operations User Support GDB Grid Deployment Resource planning,
SLACFederated Storage Workshop Summary Andrew Hanushevsky SLAC National Accelerator Laboratory April 10-11, 2014 SLAC.
CERN LCG1 to LCG2 Transition Markus Schulz LCG Workshop March 2004.
Site notifications with SAM and Dashboards Marian Babik SDC/MI Team IT/SDC/MI 12 th June 2013 GDB.
Vendredi 27 avril 2007 Management of ATLAS CC-IN2P3 Specificities, issues and advice.
Availability of ALICE Grid resources in Germany Kilian Schwarz GSI Darmstadt ALICE Offline Week.
Claudio Grandi INFN Bologna Workshop congiunto CCR e INFNGrid 13 maggio 2009 Le strategie per l’analisi nell’esperimento CMS Claudio Grandi (INFN Bologna)
Jean-Philippe Baud, IT-GD, CERN November 2007
Xiaomei Zhang CMS IHEP Group Meeting December
LHCb Computing Model and Data Handling Angelo Carbone 5° workshop italiano sulla fisica p-p ad LHC 31st January 2008.
Olof Bärring LCG-LHCC Review, 22nd September 2008
Simulation use cases for T2 in ALICE
Ákos Frohner EGEE'08 September 2008
The LHCb Computing Data Challenge DC06
Presentation transcript:

1 User Analysis Workgroup Update  All four experiments gave input by mid December  ALICE by document and links  Very independent from underlying infrastructure  Shares, allocation, access control by Alien framework  Production like analysis with “Train” concept  End User analysis by non grid means “PROOF” and via Alien task queues  LHCb by mail and links  Not very detailed at the moment  End user work requires very little storage ( desktop scale )  CMS by a document with concrete requirements  In terms of missing functionality  In terms of storage requirements  In terms of split between production and analysis use of CPU  ATLAS provided a set of links  Two ways to access the resources.

2 General  The analysis models of the experiments depend to a large degree on their own grid systems that are layered on top of the provided infrastructure  ALICE is in this respect very independent and flexible and has simple requirements  Since most complex aspects are handled in the Alien layer  The models look on a high level similar, but the different implementations impact the infrastructure in different ways  Calibration data access, data access control  In addition several different systems are used by the larger experiments  Pilot/WMS based submissions  Which makes it difficult to give universally applicable advice to sites  And have a generic set of requirements for services

3 General  All experiments have exercised their frameworks  It is not clear how close this is to the activity level when we have beam  For data access there is no clear metric to measure a T2s capability  Data size, number of files accesses, parallel active users is too simple  Each system impacts the fabric a different way  The internal structure of files has shown to have a large impact on the I/O efficiency ( which is still under investigation)  It might be instructive to take a look at some of CMSs requests and observed issues

4 CMS  Strong concept of locality  Each physicist is assigned to one or more T2s  User data is tied to these sites  Transfers with experiment’s tools only after registering the files in a global catalogue  Jobs are sent by CRAB to the data  In addition official working groups manage their own data  However, this locality cannot currently be enforced by the underlying infrastructure  It is enforced indirectly by the experiments tools

5 CMS T2 storage  T2 storage  >20TB temporary space for production, controlled by prod team  30TB space for centrally managed official data sets  N* 30TB for each official physics group  Each group can have multiple sites  Each site can host multiple groups  Regional (local) user spaces  Managed regionally  SRM based stage-out space ( future )  T3 space at the site  Complex ACLs and quotas are required  VOMS can express this  By using groups for locality and roles below that level  Complex mapping at the fabric level

6 CMS T2 CPU  50/50 % share between production and all other activities  With a timeframe for equalization of shares under load of less than a week  Split and prioritization on a granularity of analysis groups and taking locality into account  Currently VOMS is only used for prod/analysis distinction  And at some places to express locality  The full implementation with the current infrastructure will be very cumbersome and put a significant load on the information system  Prioritization: For individual users based on recent usage  Requires fixed mappings between users and local ids to work  For each and every VOMS identity of the user  Even more accounts….  Related: lcgadmin mappings and priorities

7 CMS ISSUES  Storage systems reliability  Improved SRM APIs and tools ( bulk ops)  Quotas, accounting and ACLs for space  SE and CE scalability  SRM operations  CE high load figures  Should have improved with recent mods ( see GDB)  LCG will not handle the large number of groups and roles that are required to implement the desired behavior  Lack of support for multi user pilot jobs ( parallel to push model)  SCAS/glexec soon, GUMS/glexec in use  Batch scheduler configuration appears to be not inline with agreements  Lcgadmin mapping  Except for the more divers resource allocation requests the reported issues apply as much to analysis as they do to production tasks.

8 Summary  On an abstract level the different analysis models look similar  But the used tools are very different and by the way they are layered individually on the infrastructure create different requirements and constraints  Job flow, data transport and access, calibration data access  The analysis specific problems that are encountered are mostly specific to the way the infrastructure is used ( stressed).  Data access, catalogue operations  Rollout experience supports this observation  It is very difficult to estimate the true scale of activity after LHC start  In terms of users, “grid users” etc.  There is a shift from Push to Pilot  Which will make a huge difference for the job management  Is there a similar move for data access ( xrootd)

9 Summary  For resource configuration issues direct communication between the T2s and their experiment seems to be more efficient  The common problem domain is storage specific  And solutions depend on the SE’s implementation  And the details of the experiments access  There measurements of the systems have started  The analysis use cases are better handled within the scope of the SEs