GridPP Deployment Status Steve Traylen 28th October 2004 GOSC Face to Face, NESC, UK

Contents Middleware components of the GridPP Production System Status of the current operational Grid Future plans and challenges Summary GridPP 2 – From Prototype to Production

CMSLHCbATLASALICE 1 Megabyte (1MB) A digital photo 1 Gigabyte (1GB) = 1000MB A DVD movie 1 Terabyte (1TB) = 1000GB World annual book production 1 Petabyte (1PB) = 1000TB Annual production of one LHC experiment 1 Exabyte (1EB) = 1000 PB World annual information production The physics driver 40 million collisions per second After filtering, collisions of interest per second 1-10 Megabytes of data digitised for each collision = recording rate of Gigabytes/sec collisions recorded each year = ~10 Petabytes/year of data The LHC

The UK response GridPP GridPP – A UK Computing Grid for Particle Physics 19 UK Universities, CCLRC (RAL & Daresbury) and CERN Funded by the Particle Physics and Astronomy Research Council (PPARC) GridPP1 - Sept £17m "From Web to Grid GridPP2 – Sept £16(+1)m "From Prototype to Production"

Current context of GridPP

Our grid is working … NorthGrid **** Daresbury, Lancaster, Liverpool, Manchester, Sheffield SouthGrid * Birmingham, Bristol, Cambridge, Oxford, RAL PPD, Warwick ScotGrid * Durham, Edinburgh, Glasgow LondonGrid *** Brunel, Imperial, QMUL, RHUL, UCL

… and is part of LCG Resources are being used for data challenges Within the UK we have some VO/experiment Memorandum of Understandings in place Tier-2 structure is working well

Scale GridPP prototype Grid > 1,000 CPUs –500 CPUs at the Tier-1 at RAL > 500 CPUs at 11 sites across UK organised in 4 Regional Tier-2s > 500 TB of storage > 800 simultaneous jobs Integrated with international LHC Computing Grid (LCG) > 5,000 CPUs > 4,000 TB of storage > 85 sites around the world > 4,000 simultaneous jobs monitored via Grid Operations Centre (RAL) CPUsFree CPUs Run Jobs Wait Jobs Avail TBUsed TBMax CPUAve. CPU Total (hyperthreading enabled on some sites)

Operational status (October)

VOs active

Who is directly involved? NumberPositionStatus General 1Production managerIn place and engaged 1Applications expertIdentified but not formally engaged 2Tier-1 /deployment expertIn place and fully engaged 4Tier-2 coordinatorsIn place and functioning well 0.5VO managementWill be part time but not yet in place 9.0Hardware supportPost allocated but not yet filled Specialist 1Data and storage managementExisting expert 1Work load managementExisting expert 1Security officerNot yet recruited 1NetworkingStarting in September

Past upgrade experience at RAL Previously utilisation of new resources grew steadily over weeks or months.

Tier-1 update th July 2004 Hardware Upgrade With the Grid we see a much more rapid utilisation of newly deployed resources.

The infrastructure developed in EDG/GridPP1 Job submission Python – default Java – GUI APIs (C++,J,P) Batch workers Storage Element Gatekeeper (PBS Scheduler) GridFTP Server NFS, Tape, Castor User Interface Computing Element Resource broker (C++ Condor MM libraries, Condor-G for submission) Replica catalogue per VO (or equiv.) Berkely Database Information Index AA server (VOMS) UI JDL Logging & Book keeping MySQL DB – stores job state info

Common Grid Components LCG uses middleware common to other Grid Projects. –VDT (v1.1.14) Globus Gatekeeper. Globus MDS. GlueCE Information Provider. Used by NGS, Grid3 and Nordugrid. Preserving this core increases chances of inter grid interoperability.

Extra Grid Components LCG extends VDT with fixes and the deployment of other grid services. This is only done when there is a shortfall or performance issue with the existing middleware. Most are grid wide services for LCG rather than extra components for sites to install. –Minimise conflicts between grids. –Not always true – see later.

LCG PBSJobManager Motivation –Standard Globus JobManager starts one perl process per job, queued or running. One user can destroy a Gatekeeper easily. –Also assumes a shared /home file system is present. Not scalable to 1000s of nodes. NFS a single failure point. –The Resource Broker must poll jobs indivdually.

LCG PBSJobManager Solution –LCG jobmanager stages files to batch worker with scp and GridFTP. Creates new problems though. –Even harder to debug and there is more to go wrong. –MPI jobs more difficult though an rsync work around exists.

LCG PBSJobManager Solution –JobManager starts up a GridMonitor on the gatekeeper. –One GridMonitor per Resource Broker is started currently. –Resource Broker communicates with the monitor instead of polling jobs individually. –Moving this to one GridMonitor per user is possible. Currently deployed at almost all GridPP sites.

Storage in LCG Currently there are three active solutions. –GridFTP servers, the so called ClassicSE –SRM interfaces at CERN, IHEP(Russia), DESY and RAL (this week). –edg-se – Only one as a front end the atlas data store tape system at RAL. The edg-rm and lcg-* commands abstract the end user from these interfaces.

Storage - SRM SRM = Storage Resource Manager. Motivation –Sites need to move files around and reorganise data dynamically. –The end user wants/requires a consistent name space for their files. –End users want to be able to reserve space this space as well. SRM will in time be the preferred solution supported within LCG.

SRM Deployment Current storage solution for LCG is dCache with an SRM interface. Produced by DESY and FNAL. This is currently deployed at RAL in a test state and is slipping into production initially for the CMS experiment. Expectation is that dCache with SRM will provide a solution for many sites. –Edinburgh, Manchester, Oxford all keen to deploy.

SRM/dCache at RAL

Resource Broker Allows selection of and submission to sites based on what they publish into the information system. Queues are published with –Queue lengths –Software available. –Authorised VOs or individual DNs. The RB can query the replica catalogue to run at a site with a particular file. Three RBs are deployed in the UK.

L&B L&B = Logging and Bookkeeping Service Jobs publish their Grid State to L&B. –Either by calling commands installed on batch worker. –Or by GridFTPing the job wrapper back. The second requires no software on batch workers but the first gives better feedback.

Application Installation with LCG Currently a sub VO of software managers owns an NFS mounted space. –Software area managed by jobs. –Software validated in process. –The drop a status file on to the file which is published by the site. With the RB –End users match jobs to tagged sites. –SW managers install SW at non tagged sites. This is being extended to allow DTEAM to install grid clients SW on WNs.

R-GMA Developed by GridPP within both EDG and now EGEE. Takes the role of a grid enabled SQL database. Example applications include CMS and D to publish their job bookkeeping. Can also be used to transport the Glue values and allows SQL lookups of Glue. R-GMA is deployed at most UK HEP sites. RAL currently runs the single instance of the R- GMA registry.

Next LCG Release LCG 2_3_0 is due now. –Built entirely on SL3 (RHE3 clone). RH73 still an option. –Many stability improvements. –Addition of accounting solution. –Easier addition of VOs. –Addition of DCache/SRM. and lots more… This release will last into next year. Potentially the last release before gLite components appear.

There are still challenges Middleware validation Meeting experiment requirements with the Grid Distributed file (and sub-file) management Experiment software distribution Production accounting Encouraging an open sharing of resources Security Smoothing deployment and service upgrades.

Middleware validation CERTIFICATION TESTING Integrate Basic Functionality Tests Run tests C&T suites Site suites Run Certification Matrix Release candidate tag APP INTEGR Certified release tag DEVELOPMENT & INTEGRATION UNIT & FUNCTIONAL TESTING Dev Tag JRA1 HEP EXPTS BIO-MED OTHER TBD APPS SW Installation DEPLOYMENT PREPARATION Deployment release tag DEPLOY SA1 SERVICES PRE-PRODUCTION PRODUCTION Production tag Is starting to be addressed through a Certification and Testing testbed… RAL is involved with both JRA1 and Pre Production systems.

ATLAS Data Challenge to validate world-wide computing model Packaging, distribution and installation: Scale: one release build takes 10 hours produces 2.5 GB of files Complexity: 500 packages, Mloc, 100s of developers and 1000s of users –ATLAS collaboration is widely distributed: 140 institutes, all wanting to use the software –needs push-button easy installation.. Physics Models Monte Carlo Truth Data MC Raw Data Reconstruction MC Event Summary Data MC Event Tags Detector Simulation Raw Data Reconstruction Data Acquisition Level 3 trigger Trigger Tags Event Summary Data ESD Event Summary Data ESD Event Tags Calibration Data Run Conditions Trigger System Step 1: Monte Carlo Data Challenges Step 1: Monte Carlo Data Challenges Step 2: Real Data Software distribution

Summary The Large Hadron Collider data volumes make Grid computing a necessity GridPP1 with EDG developed a successful Grid prototype GridPP members have played a critical role in most areas – security, work load management, information systems, monitoring & operations. GridPP involvement continues with the Enabling Grids for e-SciencE (EGEE) project – driving the federating of Grids As we move towards a full production service we face many challenges in areas such as deployment, accounting and true open sharing of resources

Useful links GRIDPP and LCG: GridPP collaboration Grid Operations Centre (inc. maps) The LHC Computing Grid Others PPARC The EGEE project The European Data Grid final review