Grids for the LHC Paula Eerola Lund University, Sweden Four Seas Conference Istanbul 5-10 September 2004 Acknowledgement: much of the material is from.

Slides:



Advertisements
Similar presentations
CERN STAR TAP June 2001 Status of the EU DataGrid Project Fabrizio Gagliardi CERN EU-DataGrid Project Leader June 2001
Advertisements

Jorge Gasós Grid Technologies Unit European Commission The EU e Infrastructures Programme Workshop, Beijing, June 2005.
EU DataGrid progress Fabrizio Gagliardi EDG Project Leader
An open source approach for grids Bob Jones CERN EU DataGrid Project Deputy Project Leader EU EGEE Designated Technical Director
S.L.LloydATSE e-Science Visit April 2004Slide 1 GridPP – A UK Computing Grid for Particle Physics GridPP 19 UK Universities, CCLRC (RAL & Daresbury) and.
Particle physics – the computing challenge CERN Large Hadron Collider –2007 –the worlds most powerful particle accelerator –10 petabytes (10 million billion.
Resources for the ATLAS Offline Computing Basis for the Estimates ATLAS Distributed Computing Model Cost Estimates Present Status Sharing of Resources.
HP Puerto-Rico – 9 February CERN and the LHC Computing Grid Ian Bird IT Department CERN, Geneva, Switzerland HP Puerto Rico 9 February 2004
Last update: 02/06/ :05 LCG les robertson - cern-it 1 The LHC Computing Grid Project Preparing for LHC Data Analysis NorduGrid Workshop Stockholm,
LCSC October The EGEE project: building a grid infrastructure for Europe Bob Jones EGEE Technical Director 4 th Annual Workshop on Linux.
Grid activities in Sweden Paula Eerola IT seminar, Vetenskapsrådet,
Nordic Data Grid Facility NDGF – Paula Eerola, paula.eerola [at] hep.lu.se paula.eerola [at] hep.lu.sepaula.eerola [at] hep.lu.se 1st Iberian.
EGEE is proposed as a project funded by the European Union under contract IST The EGEE International Grid Infrastructure and the Digital Divide.
1 Global Grid Efforts Richard Cavanaugh University of Florida.
The LHC Computing Grid – February 2008 The Worldwide LHC Computing Grid Dr Ian Bird LCG Project Leader 15 th April 2009 Visit of Spanish Royal Academy.
Grid and High Energy Physics Paula Eerola Lunarc, Artist’s view on Grid, by Ursula Wilby, Sydsvenskan
1 Developing Countries Access to Scientific Knowledge Ian Willers CERN, Switzerland.
GridPP Steve Lloyd, Chair of the GridPP Collaboration Board.
RomeWorkshop on eInfrastructures 9 December LCG Progress on Policies & Coming Challenges Ian Bird IT Division, CERN LCG and EGEE Rome 9 December.
Frédéric Hemmer, CERN, IT DepartmentThe LHC Computing Grid – October 2006 LHC Computing and Grids Frédéric Hemmer IT Deputy Department Head October 10,
CERN TERENA Lisbon The Grid Project Fabrizio Gagliardi CERN Information Technology Division May, 2000
Andrew McNab - Manchester HEP - 5 July 2001 WP6/Testbed Status Status by partner –CNRS, Czech R., INFN, NIKHEF, NorduGrid, LIP, Russia, UK Security Integration.
Advanced Computing Services for Research Organisations Bob Jones Head of openlab IT dept CERN This document produced by Members of the Helix Nebula consortium.
Frédéric Hemmer, CERN, IT Department The LHC Computing Grid – June 2006 The LHC Computing Grid Visit of the Comité d’avis pour les questions Scientifiques.
LCG and HEPiX Ian Bird LCG Project - CERN HEPiX - FNAL 25-Oct-2002.
Grid Applications for High Energy Physics and Interoperability Dominique Boutigny CC-IN2P3 June 24, 2006 Centre de Calcul de l’IN2P3 et du DAPNIA.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Related Projects Dieter Kranzlmüller Deputy.
The LHC Computing Grid – February 2008 The Worldwide LHC Computing Grid Dr Ian Bird LCG Project Leader 25 th April 2012.
Ian Bird LCG Deployment Manager EGEE Operations Manager LCG - The Worldwide LHC Computing Grid Building a Service for LHC Data Analysis 22 September 2006.
DataTAG Research and Technological Development for a Transatlantic Grid Abstract Several major international Grid development projects are underway at.
SouthGrid SouthGrid SouthGrid is a distributed Tier 2 centre, one of four setup in the UK as part of the GridPP project. SouthGrid.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE – paving the way for a sustainable infrastructure.
INFSO-RI Enabling Grids for E-sciencE Plan until the end of the project and beyond, sustainability plans Dieter Kranzlmüller Deputy.
Bob Jones Technical Director CERN - August 2003 EGEE is proposed as a project to be funded by the European Union under contract IST
1 The LHC Computing Grid – February 2007 Frédéric Hemmer, CERN, IT Department LHC Computing and Grids Frédéric Hemmer Deputy IT Department Head January.
CERN IT Department CH-1211 Genève 23 Switzerland Visit of Professor Karel van der Toorn President University of Amsterdam Wednesday 10 th.
EGEE is a project funded by the European Union under contract IST Middleware Planning for LCG/EGEE Bob Jones EGEE Technical Director e-Science.
GridPP Building a UK Computing Grid for Particle Physics Professor Steve Lloyd, Queen Mary, University of London Chair of the GridPP Collaboration Board.
Ian Bird LHC Computing Grid Project Leader LHC Grid Fest 3 rd October 2008 A worldwide collaboration.
…building the next IT revolution From Web to Grid…
The LHC Computing Grid – February 2008 The Challenges of LHC Computing Dr Ian Bird LCG Project Leader 6 th October 2009 Telecom 2009 Youth Forum.
Les Les Robertson LCG Project Leader High Energy Physics using a worldwide computing grid Torino December 2005.
Ian Bird LCG Deployment Area Manager & EGEE Operations Manager IT Department, CERN Presentation to HEPiX 22 nd October 2004 LCG Operations.
Grid User Interface for ATLAS & LHCb A more recent UK mini production used input data stored on RAL’s tape server, the requirements in JDL and the IC Resource.
Presentation of the A particle collision = an event Physicist's goal is to count, trace and characterize all the particles produced and fully.
Dieter European Grid Initiative Towards a sustainable production grid infrastructure.
LHC Computing, CERN, & Federated Identities
Data Processing and the LHC Computing Grid (LCG) Jamie Shiers Database Group, IT Division CERN, Geneva, Switzerland
INFSO-RI Enabling Grids for E-sciencE The EGEE Project Owen Appleton EGEE Dissemination Officer CERN, Switzerland Danish Grid Forum.
EGEE is a project funded by the European Union under contract IST EGEE Summary NA2 Partners April
Induction: What is EGEE? –April 26-28, What is EGEE? John Murison, EGEE Training Team EGEE is funded by the European Union under contract IST
DataGrid is a project funded by the European Commission under contract IST rd EU Review – 19-20/02/2004 The EU DataGrid Project Three years.
1 The LHC Computing Grid – April 2007 Frédéric Hemmer, CERN, IT Department The LHC Computing Grid A World-Wide Computer Centre Frédéric Hemmer Deputy IT.
LHC Computing, SPC-FC-CC-C; H F Hoffmann1 CERN/2379/Rev: Proposal for building the LHC computing environment at CERN (Phase 1) Goals of Phase.
The LHC Computing Grid – February 2008 The Worldwide LHC Computing Grid Dr Ian Bird LCG Project Leader 1 st March 2011 Visit of Dr Manuel Eduardo Baldeón.
Dominique Boutigny December 12, 2006 CC-IN2P3 a Tier-1 for W-LCG 1 st Chinese – French Workshop on LHC Physics and associated Grid Computing IHEP - Beijing.
10-Feb-00 CERN HepCCC Grid Initiative ATLAS meeting – 16 February 2000 Les Robertson CERN/IT.
Bob Jones EGEE Technical Director
WLCG Tier-2 Asia Workshop TIFR, Mumbai 1-3 December 2006
The LHC Computing Grid Visit of Mtro. Enrique Agüera Ibañez
SA1 Execution Plan Status and Issues
Ian Bird GDB Meeting CERN 9 September 2003
Grid related projects CERN openlab LCG EDG F.Fluckiger
Long-term Grid Sustainability
EGEE support for HEP and other applications
A high-performance computing facility for scientific research
LHC Computing Grid Project - LCG
Tour of CERN Computer Center
LHC Data Analysis using a worldwide computing grid
LHC Computing, RRB; H F Hoffmann
Presentation transcript:

Grids for the LHC Paula Eerola Lund University, Sweden Four Seas Conference Istanbul 5-10 September 2004 Acknowledgement: much of the material is from Ian Bird, Lepton-Photon Symposium 2003, Fermilab.

2 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Outline Introduction Introduction –What is a Grid? –Grids and high-energy physics? Grid projects Grid projects –EGEE –NorduGrid LHC Computing Grid project LHC Computing Grid project –Using grid technology to access and analyze LHC data Outlook Outlook

3 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Introduction What is a Grid?

4 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 About the Grid WEB: get information on any computer in the world WEB: get information on any computer in the world GRID: get CPU-resources, disk-resources, tape- resources on any computer in the world GRID: get CPU-resources, disk-resources, tape- resources on any computer in the world Grid needs advanced software, middleware, which connects the computers together Grid needs advanced software, middleware, which connects the computers together Grid is the future infrastructure of computing and data management Grid is the future infrastructure of computing and data management

5 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Short history 1996: Start of the Globus project for connecting US supercomputers together (funded by US Defence Advanced Research Projects Agency...) 1996: Start of the Globus project for connecting US supercomputers together (funded by US Defence Advanced Research Projects Agency...) 1998: early Grid testbeds in the USA - supercomputing centers connected together 1998: early Grid testbeds in the USA - supercomputing centers connected together 1998 Ian Foster, Carl Kesselman: 1998 Ian Foster, Carl Kesselman: GRID: Blueprint for a new Computing Infrastructure 2000— PC capacity increases, prices drop  supercomputers become obsolete  Grid focus is moved from supercomputers to PC-clusters 2000— PC capacity increases, prices drop  supercomputers become obsolete  Grid focus is moved from supercomputers to PC-clusters 1990’s – WEB, 2000’s – GRID? 1990’s – WEB, 2000’s – GRID? Huge commercial interests: IBM, HP, Intel, … Huge commercial interests: IBM, HP, Intel, …

6 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Grid prerequisites Powerful PCs are cheap Powerful PCs are cheap PC-clusters are everywhere PC-clusters are everywhere Networks are improving even faster than CPUs Networks are improving even faster than CPUs Network & Storage & Computing exponentials: Network & Storage & Computing exponentials: –CPU performance (# transistors) doubles every 18 months –Data storage (bits per area) doubles every 12 months –Network capacity (bits per sec) doubles every 9 months

7 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Grids and high-energy physics? The Large Hadron Collider, LHC, start 2007 The Large Hadron Collider, LHC, start experiments, ATLAS, CMS, ALICE, LHCb, with physicists from all over the world 4 experiments, ATLAS, CMS, ALICE, LHCb, with physicists from all over the world LHC computing = data processing, data storage, production of simulated data LHC computing = data processing, data storage, production of simulated data LHC computing is of unprecedented scale LHC computing is of unprecedented scale Massive data flow The 4 experiments are accumulating 5-8 PetaBytes of data/year Massive data flow The 4 experiments are accumulating 5-8 PetaBytes of data/year

Paula Eerola Four Seas Conference, Istanbul, 5-10 September Needed capacity Storage – 10 PetaBytes of disk and tape Processing – 100,000 of today’s fastest PCs World-wide data analysis Physicists are located in all the continents Needed capacity Storage – 10 PetaBytes of disk and tape Processing – 100,000 of today’s fastest PCs World-wide data analysis Physicists are located in all the continents Computing must be distributed for many reasons Not feasible to put all the capacity in one place Political, economic, staffing: easier to get funding for resources at home country Faster access to data for all physicists around the world Better sharing of computing resources required by physicists Computing must be distributed for many reasons Not feasible to put all the capacity in one place Political, economic, staffing: easier to get funding for resources at home country Faster access to data for all physicists around the world Better sharing of computing resources required by physicists

Paula Eerola Four Seas Conference, Istanbul, 5-10 September LHC Computing Hierarchy Tier 1 FNAL Center IN2P3 Center INFN Center RAL Center Tier 1 Centres = large computer centers (about 10). Tier 1’s provide permanent storage and management of raw, summary and other data needed during the analysis process. Tier 2 Centres = smaller computer centers (several 10’s). Tier 2 Centres provide disk storage and concentrate on simulation and end-user analysis. Tier2 Center Tier 2 Institute Workstations Physics data cache CERN Center PBs of Disk; Tape Robot ~ MBytes/s Tier 0 Experiment Tier 0= CERN. Tier 0 receives raw data from the Experiments and records them on permanent mass storage. First-pass reconstruction of the data, producing summary data.

10 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Grid technology as a solution Grid technology can provide optimized access to and use of the computing and storage resources Grid technology can provide optimized access to and use of the computing and storage resources Several HEP experiments currently running (Babar, CDF/DO, STAR/PHENIX), with significant data and computing requirements, have already started to deploy grid-based solutions Several HEP experiments currently running (Babar, CDF/DO, STAR/PHENIX), with significant data and computing requirements, have already started to deploy grid-based solutions Grid technology is not yet off-the shelf product  Requires development of middleware, protocols, services,… Grid technology is not yet off-the shelf product  Requires development of middleware, protocols, services,… Grid development and engineering projects: EDG, EGEE, NorduGrid, Grid3,….

Grid projects

12 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 US, Asia, Australia USA NASA Information Power Grid NASA Information Power Grid DOE Science Grid DOE Science Grid NSF National Virtual Observatory NSF National Virtual Observatory NSF GriPhyN NSF GriPhyN DOE Particle Physics Data Grid DOE Particle Physics Data Grid NSF TeraGrid NSF TeraGrid DOE ASCI Grid DOE ASCI Grid DOE Earth Systems Grid DOE Earth Systems Grid DARPA CoABS Grid DARPA CoABS Grid NEESGrid NEESGrid DOH BIRN DOH BIRN NSF iVDGL NSF iVDGL … Asia, Australia Australia: ECOGRID, GRIDBUS,… Australia: ECOGRID, GRIDBUS,… Japan: BIOGRID, NAREGI, … Japan: BIOGRID, NAREGI, … South Korea: National Grid Basic Plan, Grid Forum Korea,… South Korea: National Grid Basic Plan, Grid Forum Korea,… … …

13 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Europe EGEE EGEE NorduGrid NorduGrid EDG, LCG EDG, LCG UK GridPP UK GridPP INFN Grid, Italy INFN Grid, Italy Cross-grid projects in order to link together Grid projects Cross-grid projects in order to link together Grid projects  Many Grid projects have particle physics as the initiator  Other fields are joining in: healthcare, bioinformatics,…  Address different aspects of grids:  Middleware  Infrastructure  Networking, cross-Atlantic interoperation

PARTNERS 70 partners organized in nine regional federations Coordinating and Lead Partner: CERN CENTRAL EUROPE – FRANCE - GERMANY & SWITZERLAND – ITALY - IRELAND & UK - NORTHERN EUROPE - SOUTH-EAST EUROPE - SOUTH-WEST EUROPE – RUSSIA - USA STRATEGY  Leverage current and planned national and regional Grid programmes  Build on existing investments in Grid Technology by EU and US  Exploit the international dimensions of the HEP- LCG programme  Make the most of planned collaboration with NSF CyberInfrastructure initiative A seamless international Grid infrastructure to provide researchers in academia and industry with a distributed computing facility ACTIVITY AREAS SERVICES  Deliver “production level” grid services (manageable, robust, resilient to failure)  Ensure security and scalability MIDDLEWARE  Professional Grid middleware re-engineering activity in support of the production services NETWORKING  Proactively market Grid services to new research communities in academia and industry  Provide necessary education

15 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Create a European-wide Grid Infrastructure for the support of research in all scientific areas, on top of the EU Reseach Network infrastructure (GEANT) Integrate regional grid efforts EGEE: goals and partners 9 regional federations covering 70 partners in 26 countries

16 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 EGEE project Project funded by EU FP6, 32 MEuro for 2 years Project start 1 April 2004 Activities: Grid Infrastructure: Provide a Grid service for science research Next generation of Grid middleware  gLite Dissemination, Training and Applications (initially HEP & Bio)

17 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 EGEE: timeline

18 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Grid in Scandinavia: the NorduGrid Project Nordic Testbed for Wide Area Computing and Data Handling Wide Area Computing and Data Handling

19 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 NorduGrid: original objectives and current status Goals 2001 (project start): Introduce the Grid to Scandinavia Introduce the Grid to Scandinavia Create a Grid infrastructure in Nordic countries Create a Grid infrastructure in Nordic countries Apply available Grid technologies/middleware Apply available Grid technologies/middleware Operate a functional Testbed Operate a functional Testbed Expose the infrastructure to end-users of different scientific communities Expose the infrastructure to end-users of different scientific communities Status 2004: The project has grown world-wide: nodes in Germany, Slovenia, Australia,... The project has grown world-wide: nodes in Germany, Slovenia, Australia, nodes, 3500 CPUs 39 nodes, 3500 CPUs Created own NorduGrid Middleware, ARC (Advanced Resource Connector), which is operating in a stable way Created own NorduGrid Middleware, ARC (Advanced Resource Connector), which is operating in a stable way Applications: massive production of ATLAS simulation and reconstruction Applications: massive production of ATLAS simulation and reconstruction Other applications: AMANDA simulation, genomics, bio-informatics, visualization (for metheorological data), multimedia applications,... Other applications: AMANDA simulation, genomics, bio-informatics, visualization (for metheorological data), multimedia applications,... Status 2004: The project has grown world-wide: nodes in Germany, Slovenia, Australia,... The project has grown world-wide: nodes in Germany, Slovenia, Australia, nodes, 3500 CPUs 39 nodes, 3500 CPUs Created own NorduGrid Middleware, ARC (Advanced Resource Connector), which is operating in a stable way Created own NorduGrid Middleware, ARC (Advanced Resource Connector), which is operating in a stable way Applications: massive production of ATLAS simulation and reconstruction Applications: massive production of ATLAS simulation and reconstruction Other applications: AMANDA simulation, genomics, bio-informatics, visualization (for metheorological data), multimedia applications,... Other applications: AMANDA simulation, genomics, bio-informatics, visualization (for metheorological data), multimedia applications,...

20 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Current NorduGrid status

The LHC Computing Grid, LCG The distributed computing environment to analyse the LHC data lcg.web.cern.ch lcg.web.cern.ch

22 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 LCG - goals Goal: prepare and deploy the computing environment that will be used to analyse the LHC data Phase 1: 2003 – 2005  Build a service prototype  Gain experience in running a production grid service Phase 2: 2006 – 2008  Build and commission the initial LHC computing environment Technical Design Report for Phase 2 LCG full multi-tier prototype batch+interactive service LCG service opens LCG with upgraded m/w, management etc Event simulation productions

23 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 LCG composition and tasks The LCG Project is a collaboration of The LCG Project is a collaboration of –The LHC experiments –The Regional Computing Centres –Physics institutes Development and operation of a distributed computing service Development and operation of a distributed computing service –computing and storage resources in computing centres, physics institutes and universities around the world –reliable, coherent environment for the experiments Support for applications Support for applications –provision of common tools, frameworks, environment, data persistency

24 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Resource targets ´04 CPU (kSI2K) (kSI2K)Disk TB TBSupport FTE FTETapeTBCERN Czech Rep France Germany Holland Italy Japan Poland Russia Taiwan Spain Sweden Switzerland UK USA Total

LCG status Sept ’04 Tier 0 CERN CERN Tier 1 Centres Brookhaven Brookhaven CNAF Bologna CNAF Bologna PIC Barcelona PIC Barcelona Fermilab Fermilab FZK Karlsruhe FZK Karlsruhe IN2P3 Lyon IN2P3 Lyon Rutherford (UK) Rutherford (UK) Univ. of Tokyo Univ. of Tokyo CERN CERN Tier 2 centers South-East Europe: HellasGrid, AUTH, Tel-Aviv, Weizmann South-East Europe: HellasGrid, AUTH, Tel-Aviv, Weizmann Budapest Budapest Prague Prague Krakow Krakow Warsaw Warsaw Moscow region Moscow region Italy Italy …… ……

26 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 LCG status Sept ´04 First production service for LHC experiments operational First production service for LHC experiments operational –Over 70 centers, over 6000 CPUs, although many of these sites are small and cannot run big simulations –LCG-2 middleware – testing, certification, packaging, configuration, distribution and site validation Grid operations centers in RAL and Taipei (+US) – performance monitoring, problem solving – 24x7 globally Grid operations centers in RAL and Taipei (+US) – performance monitoring, problem solving – 24x7 globally Grid call centers in FZK Karlsruhe and Taipei. Grid call centers in FZK Karlsruhe and Taipei. Progress towards inter-operation between LCG, NorduGrid, Grid3 (US) Progress towards inter-operation between LCG, NorduGrid, Grid3 (US)

Outlook EU vision of e-infrastructure in Europe

Paula Eerola Four Seas Conference, Istanbul, 5-10 September Moving towards an e-infrastructure IPv6 Grids GÉANT Grids middleware

Paula Eerola Four Seas Conference, Istanbul, 5-10 September Moving towards an e-infrastructure Grids middleware Grid-empowered e-infrastructure – “all in one” e-Infrastructure

30 Paula Eerola Four Seas Conference, Istanbul, 5-10 September 2004 Summary Huge investment in e-science and Grids in Europe Huge investment in e-science and Grids in Europe –regional, national, cross-national, EU Emerging vision of European-wide e- science infrastructure for research Emerging vision of European-wide e- science infrastructure for research High Energy Physics is a major application that needs this infrastructure today and is pushing the limits of the technology High Energy Physics is a major application that needs this infrastructure today and is pushing the limits of the technology