Collaboration Meeting GridPP Overview Tony Doyle 11 January 2006 Collaboration Meeting

OR The Icemen Cometh

Collaboration Meeting Outline High level LHC, Expt. and Grid Plans.. 2005 Outturn and the Goldilocks Problem.. “Some of the challenges for next year” gLite is not too late.. Pulling together on the good ship Grid Coming together to enable a discovery Plans and Resolutions (from the year of the rooster to the year of the dog..) 11 January 2006 Collaboration Meeting

Collaboration Meeting A. When Cometh the LHC? “Main objectives are to terminate installation in February 2007 and enable first collisions in summer 2007” Lyn Evans 11 January 2006 Collaboration Meeting

B. When Cometh the Detectors? e.g. ATLAS “with good will and great efforts from everybody we can be confident that the Technical Coordination Team will manage to have ATLAS installed by June 2007” 11 January 2006 Collaboration Meeting

Collaboration Meeting C. When Cometh the Grid? Service Challenges – UK deployment plans End point April ’07 Context: The virtual LHC Computing Centre 11 January 2006 Collaboration Meeting

Collaboration Meeting Grid Overview Aim: by 2008 (full year’s data taking) CPU ~100MSi2k (100,000 CPUs) Storage ~80PB - Involving >100 institutes worldwide Build on complex middleware being developed in advanced Grid technology projects, both in Europe (Glite) and in the USA (VDT) Prototype went live in September 2003 in 12 countries Extensively tested by the LHC experiments in September 2004 197 sites, 13,797 CPUs, 5PB storage in September 2005 11 January 2006 Collaboration Meeting

2005 Metrics and Quality Assurance: wider concerns Target Current status Q2 2006 Target values Number of Users ~ 1000 ≥ 3000 Number of sites 120 50 Number of CPU ~12000 9500 at month 15 Number of Disciplines 6 ≥ 5 Multinational 24 ≥ 15 countries 11 January 2006 Collaboration Meeting

Collaboration Meeting 2005 Tier-1 GOC Accounting 11 January 2006 Collaboration Meeting

2005 Grid and Non-Grid Tier-1/A CPU Use Fraction ~50% 11 January 2006 Collaboration Meeting

2005 Grid and Non-Grid Tier-1/A CPU Use OC user Users! 70% “Target” OC! 11 January 2006 Collaboration Meeting

Collaboration Meeting 2005 Tier-1 Efficiency overall efficiency = S (CPU times) / S (wall times) Jan ’05 Pre-Grid Efficiency High Apr ’05 Remote data access problems Dec ’05 General Improvements Mixed view from “VOs” Need to test data access I/O-bound jobs possible Good to have this data! 11 January 2006 Collaboration Meeting

Some of the challenges for next year See Jamie’s talk Castor 2 Good progress, rapid reaction to problems But some way still to go with testing - stressing reliability, performance Can only be done with participation of experiments Distribution to other sites being planned Distributed database services Architecture and plan agreed now But still to deploy pilot services - timing is worryingly tight End-to-end testing of the DAQ-T0-T1 chain recording, calibration and alignment, reconstruction, distribution Full Tier-1 work load testing – Recording, reprocessing, ESD distribution, analysis, Tier-2 support Understanding the CERN Analysis Facility batch analysis interactive analysis Startup scenarios Schedule may be better known after next spring’s Chamonix meeting

Collaboration Meeting High Level View from recent LHCC review is ~ Service Challenges - OK (established) Throughput - not OK (not sufficiently tested) Baseline services – OK (defined, not completely established) Practical steps – OK (we need to improve communication) MoUs – OK (we need to sign off, covered last meting) Concerns: Significant delay in gLite, Castor2, distributed data management, database services all are late Middleware and experiment connections are too weak Analysis models are untested 11 January 2006 Collaboration Meeting

Collaboration Meeting gLite Stack Requirement Storage Element Basic File Transfer Reliable File Transfer Catalogue Services Data Management tools Compute Element Workload Management VO Agents VO Membership Services DataBase Services Posix-like I/O Application Software Installation Tools Job Monitoring Reliable Messaging Information System 15 Baseline Services for a functional Grid We rely upon gLite components This middleware builds upon VDT (Globus and Condor) and meets the requirements of all the basic scientific use cases: Green (amber) areas are (almost) agreed as part of the shared generic middleware stack by each of the application areas Red are areas where generic middleware competes with application-specific software. www.glite.org 11 January 2006 Collaboration Meeting

Collaboration Meeting gLite Pack 11 January 2006 Collaboration Meeting

Middleware Re-engineering A series of gLite releases have been produced (1.1, 1.2, 1.3, and 1.4) Driven by application and deployment needs Focus on defect fixing gLite deployed on a Pre-Production Service and made available for application use Independent evaluation by NGS gLite components also available via VDT (US) gLite components deployed on the infrastructure Emphasis is now on release of gLite 1.5 Will continue… see Steve Fisher’s talk EGEE phase 2 starts in April 2006 www.glite.org 11 January 2006 Collaboration Meeting

Some of the challenges for next year File transfers Good initial progress (except dCache->DPM- currently) But some way still to go with testing - stressing reliability, performance Can only be done with participation of experiments Distribution to other sites being planned Distributed VO services Plan agreed – T1 will sign off and then VO boxes may be deployed by T2s But still to deploy pilot services - CMS (OK) LHCb (OK) ATLAS ALICE End-to-end testing of the T0-T1-T2 chain MC production, reconstruction, distribution Full Tier-1 work load testing Recording, reprocessing, ESD distribution, analysis, Tier-2 support Understanding the “Analysis Facility” batch analysis @ T1 and T2 interactive analysis Startup scenarios Schedule is known at high level and defined for Service Challenges – testing time ahead (in many ways) 11 January 2006 Collaboration Meeting

Collaboration Meeting Themes for 2006 Think how you can help by either: measuring throughput for experiments; testing gLite, providing feedback; working practically with experiments; running your analysis on the Grid; helping Grid adoption (in other fields). OR combine elements of 1-5. Emphasis on end-to-end (vertical) integration. 11 January 2006 Collaboration Meeting

Pulling together with the experiments? Hopefully the effort in pulling the Grid boat out is more equal.. However many discoveries made in Grid circles are currently being re-discovered by experiment users Succinct user documentation will help 11 January 2006 Collaboration Meeting

Pulling together with the experiments? There are currently not enough users in the Grid School for the Gifted Having smart users helps (the current ones are) The system may be too complex, requiring too much work by the user? Or the (virtual) help desk may not be enough? Or the documentation may be misleading? 11 January 2006 Collaboration Meeting

2005: Functional Tests 2006: File Transfers number of sites passing the SFT tests Log data lost total grid sites Successful Year of Functional Tests, with bar raised throughout the year. “Functional” ≠ “Performant”. Need to test network-file transfers -file placement-file collection transfers, working with experiments. 11 January 2006 Collaboration Meeting

Collaboration Meeting Come together.. Physics discovery requires many elements to work.. The Icemen Cometh 2009 2008 2007 2008-09 “Physics discovery” 2006-07 “Performance Testing” 2004-05 “Functional Testing” 2006 2005 2004 11 January 2006 Collaboration Meeting

Collaboration Meeting A (Light) Summary Priors indicate a Higgs mass 114.4 < mH [GeV] <219 The Grid Service will be launched in April 2007 The Detectors will be complete in June 2007 The LHC will provide first collisions in Summer 2007 These will enable data analyses such that the Higgs will be discovered on May 29th 2009 probably… large corrections If the Higgs particle is discovered, the Grid will be one of three major components 11 January 2006 Collaboration Meeting

Collaboration Meeting 2005: The Year of the Tier-1 “World's biggest grid seeks secrets of the universe” “It's in Didcot and it's running on open source” Silicon.Com Published: Thursday 24 November 2005 http://www.silicon.com/publicsector/0,3800010403,39154492,00.htm 11 January 2006 Collaboration Meeting

Collaboration Meeting Summary This meeting focuses on the challenges of a New Year The Old Year was the year of the (Tier-1) rooster Tier-1/A utilisation was too low (at the start) and too high (in the end), but just right overall? Tier-2 utilisation was too low (throughout) The New Year is the year of the (Tier-2) dog? The seamless vision of a T0-T1-T2 structure hidden behind a transparent Grid requires (relatively rapid) testing The New Year will be better, if we resolve to make (Many) T1 T2, Grid Expt. Performance Tests we have (but they must all ultimately be successful) 11 January 2006 Collaboration Meeting