Jean-Yves Nief, CC-IN2P3 Wilko Kroeger, SCCS/SLAC Adil Hasan, CCLRC/RAL HEPiX, SLAC October 11th – 13th, 2005 BaBar data distribution using the Storage.

Slides:

Advertisements

Similar presentations

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.

Advertisements

The Storage Resource Broker and.

The Storage Resource Broker and.

Peter Berrisford RAL – Data Management Group SRB Services.

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE SAN DIEGO SUPERCOMPUTER CENTER Particle Physics Data Grid PPDG Data Handling System Reagan.

Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.

Web Caching Schemes1 A Survey of Web Caching Schemes for the Internet Jia Wang.

Applying Data Grids to Support Distributed Data Management Storage Resource Broker Reagan W. Moore Ian Fisk Bing Zhu University of California, San Diego.

Data Grid: GRASP Mike Smorul. Grid Retrieval and Search Platform Based on concepts developed in the Earth Science Data Interface (ESDI) developed at the.

1 Andrew Hanushevsky - HEPiX, October 6-8, 1999 Mass Storage For BaBar at SLAC Andrew Hanushevsky Stanford.

Generic policy rules and principles Jean-Yves Nief.

Magda – Manager for grid-based data Wensheng Deng Physics Applications Software group Brookhaven National Laboratory.

IRODS usage at CC-IN2P3 Jean-Yves Nief. Talk overview What is CC-IN2P3 ? Who is using iRODS ? iRODS administration: –Hardware setup. iRODS interaction.

Experiences Deploying Xrootd at RAL Chris Brew (RAL)

CMS Report – GridPP Collaboration Meeting VI Peter Hobson, Brunel University30/1/2003 CMS Status and Plans Progress towards GridPP milestones Workload.

IRODS performance test and SRB system at KEK Yoshimi KEK Building data grids with iRODS 27 May 2008.

Jean-Yves Nief, CC-IN2P3, Lyon First Latin American EELA Workshop April 24th – 26th, 2006 Data distribution and aggregation over geographically distant.

Core SRB Technology for 2005 NCOIC Workshop By Michael Wan And Wayne Schroeder SDSC SDSC/UCSD/NPACI.

Introduction to iRODS Jean-Yves Nief. Talk overview Data management context. Some data management goals: –Storage virtualization. –Virtualization of the.

Jan Storage Resource Broker Managing Distributed Data in a Grid A discussion of a paper published by a group of researchers at the San Diego Supercomputer.

1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong

3rd June 2004 CDF Grid SAM:Metadata and Middleware Components Mòrag Burgon-Lyon University of Glasgow.

Jean-Yves Nief CC-IN2P3, Lyon HEPiX-HEPNT, Fermilab October 22nd – 25th, 2002.

8th November 2002Tim Adye1 BaBar Grid Tim Adye Particle Physics Department Rutherford Appleton Laboratory PP Grid Team Coseners House 8 th November 2002.

DATABASE MANAGEMENT SYSTEMS IN DATA INTENSIVE ENVIRONMENNTS Leon Guzenda Chief Technology Officer.

Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore

PPDG and ATLAS Particle Physics Data Grid Ed May - ANL ATLAS Software Week LBNL May 12, 2000.

BaBar RAL Staff Talks. 4th April 2005 RAL BaBar Staff Talk 2 Outline The Group  Fergus Wilson – Group Leader since Oct 2004  Tim Adye  Emmanuel Olaiya.

BaBar Data Distribution using the Storage Resource Broker Adil Hasan, Wilko Kroeger (SLAC Computing Services), Dominique Boutigny (LAPP), Cristina Bulfon.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

Data Distribution and Management Tim Adye Rutherford Appleton Laboratory BaBar Computing Review 9 th June 2003.

21 st October 2002BaBar Computing – Stephen J. Gowdy 1 Of 25 BaBar Computing Stephen J. Gowdy BaBar Computing Coordinator SLAC 21 st October 2002 Second.

Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.

4/5/2007Data handling and transfer in the LHCb experiment1 Data handling and transfer in the LHCb experiment RT NPSS Real Time 2007 FNAL - 4 th May 2007.

LCG Phase 2 Planning Meeting - Friday July 30th, 2004 Jean-Yves Nief CC-IN2P3, Lyon An example of a data access model in a Tier 1.

NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.

Owen SyngeTitle of TalkSlide 1 Storage Management Owen Synge – Developer, Packager, and first line support to System Administrators. Talks Scope –GridPP.

J.J.Blaising April 02AMS DataGrid-status1 DataGrid Status J.J Blaising IN2P3 Grid Status Demo introduction Demo.

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

Michael Doherty RAL UK e-Science AHM 2-4 September 2003 SRB in Action.

26 September 2000Tim Adye1 Data Distribution Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting 26 th September 2000.

1 e-Science AHM st Aug – 3 rd Sept 2004 Nottingham Distributed Storage management using SRB on UK National Grid Service Manandhar A, Haines K,

Introduction to The Storage Resource.

E-Curator: A Web-based Curatorial Tool Ian Brown, Mona Hess Sally MacDonald, Francesca Millar Yean-Hoon Ong, Stuart Robson Graeme Were UCL Museums & Collections.

11th November 2002Tim Adye1 Distributed Analysis in the BaBar Experiment Tim Adye Particle Physics Department Rutherford Appleton Laboratory University.

Managing Petabytes of data with iRODS at CC-IN2P3

Biomedical Informatics Research Network The Storage Resource Broker & Integration with NMI Middleware Arcot Rajasekar, BIRN-CC SDSC October 9th 2002 BIRN.

CASTOR project status CASTOR project status CERNIT-PDP/DM October 1999.

1 A Scalable Distributed Data Management System for ATLAS David Cameron CERN CHEP 2006 Mumbai, India.

The Storage Resource Broker and.

1 GridFTP and SRB Guy Warner Training, Outreach and Education Team, Edinburgh e-Science.

Distributed Physics Analysis Past, Present, and Future Kaushik De University of Texas at Arlington (ATLAS & D0 Collaborations) ICHEP’06, Moscow July 29,

GDB meeting - Lyon - 16/03/05 An example of data management in a Tier A/1 Jean-Yves Nief.

Lynda : Lyon Neuroimaging Database and Applications (1) Institut des Sciences Cognitives UMR 5015 CNRS ; (2) parallel computing ENS-Lyon ; (3)Centre de.

Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.

Jean-Yves Nief, CC-IN2P3 CC-IN2P3 KEK-CCIN2P3 meeting on Grids. September 11th – 12th, 2006.

Collection-Based Persistent Archives Arcot Rajasekar, Richard Marciano, Reagan Moore San Diego Supercomputer Center Presented by: Preetham A Gowda.

Data Infrastructure in the TeraGrid Chris Jordan Campus Champions Presentation May 6, 2009.

1 eScience Grid Environments th May 2004 NESC - Edinburgh Deployment of Storage Resource Broker at CCLRC for E-science Projects Ananta Manandhar.

IRODS at CC-IN2P3: overview Jean-Yves Nief. Talk overview iRODS in production: –Hardware setup. –Usage. –Prospects. iRODS developpements in Lyon: –Scripts.

11th September 2002Tim Adye1 BaBar Experience Tim Adye Rutherford Appleton Laboratory PPNCG Meeting Brighton 11 th September 2002.

CC-IN2P3 data repositories Jean-Yves Nief. What is CC-IN2P3 ? 04/12/2009CC-IN2P3 data repositories2 Federate computing needs of the french community:

Grid Services for Digital Archive Tao-Sheng Chen Academia Sinica Computing Centre

Management of the Data in Auger Jean-Noël Albert LAL – Orsay IN2P3 - CNRS ASPERA – Oct Lyon.

High Performance Storage System (HPSS) Jason Hick Mass Storage Group HEPiX October 26-30, 2009.

Chapter 1 Characterization of Distributed Systems

An Overview of iRODS Integrated Rule-Oriented Data System

CC-IN2P3 Jean-Yves Nief, CC-IN2P3 HEPiX, SLAC

Presentation transcript:

Jean-Yves Nief, CC-IN2P3 Wilko Kroeger, SCCS/SLAC Adil Hasan, CCLRC/RAL HEPiX, SLAC October 11th – 13th, 2005 BaBar data distribution using the Storage Resource Broker (SRB)

HEPiX Conference, SLAC, October 11th-13th BaBar overview. Large amount of data produced by BaBar:> 800 TB. Using the collaboration Computing Centres: –Data (re)processing (prompt reco, skimming). –Simulation production. –Data analysis  allow foreign collaborators to access data. 6 Tier A: –SLAC stores all the data + simulation: central repository. –Computing tasks divided among all the Tier A sites.

HEPiX Conference, SLAC, October 11th-13th BaBar data organization. SLACCC-IN2P3PadovaRALCNAFGridKa Data processing Data analysis  Lots of data movement + simulation production (~20 sites involved)

HEPiX Conference, SLAC, October 11th-13th Requirements. Short turn-around between data production and analysis. Data must be transfered efficiently and quickly to Computing Centres. Data transfer tools must be: –Reliable. –Robust. –Automated. –Scalable. –Easy to use.

HEPiX Conference, SLAC, October 11th-13th SLAC & CC-IN2P3. First analysis centres opened to the BaBar collaboration. Both held complete copies of data (Objectivity). Now only SLAC old a complete copy of the data. Natural candidates for testing and deployment of grid middleware. SRB: good candidate for data distribution.

HEPiX Conference, SLAC, October 11th-13th What’s SRB ? Storage Resource Broker: developed by SDSC (San Diego). Provides an uniform interface to heterogeneous storage system (disk, tape, databases) for data distributed in multiple sites. Who is using SRB ? –Biology, biomedical applications (e.g: BIRN). –Astrophysics, Earth Sciences (e.g: NASA). –Digital libraries (e.g: NARA). Used world wide: USA, Europe, Asia, Australia.

HEPiX Conference, SLAC, October 11th-13th Some SRB features. Files organized in a logical space (directories, subdirectories …): logical view totally decoupled from the physical location of the files. Replica management. Search for files based on their attributes (metadata). Fine granularity for SRB users’ACLs. Interfaced with various Mass Storage System: –Access to HPSS transparent to users. –Small files management. And many more…

HEPiX Conference, SLAC, October 11th-13th SRB architecture. 1 zone: –1 SRB/MetaCatalog server: contains list of files, physical resources, users registered. –several SRB servers to access the data at their physical location. Site 1 SRB Site 2 SRB Site 3 Application (asking for test1.txt) SRB MCAT (1) (4) (2) test1.txt (3)

HEPiX Conference, SLAC, October 11th-13th SRB BaBar architecture ( ). CC-IN2P3 (Lyon) HPSS/SLAC HPSS/Lyon client SRB SLAC (Stanford, CA) SRB MCAT (1) client (3) (2) 1 Zone (SLAC)

HEPiX Conference, SLAC, October 11th-13th Extra details. Hardware: –SUN servers (Solaris 5.8, 5.9): NetraT 1405, V440. Software: –Oracle 8, 9 for the SLAC MCAT (530,000 files registered). –Multithreaded client application (Python). Advantages: –All the 3 steps controlled by the client application in //  no latency due to prestaging. –Small amount of cache needed on both sides (a few GBs). Drawbacks: –No registration into the MCAT of the files stored in Lyon. Performance: –Up to 3 TB/day from tape to tape.

HEPiX Conference, SLAC, October 11th-13th Fermilab (US)  CERN SLAC (US)  IN2P3 (FR) 1 Terabyte/day SLAC (US)  INFN Padva (IT) Fermilab (US)  U. Chicago (US) CEBAF (US)  IN2P3 (FR) INFN Padva (IT)  SLAC (US) U. Toronto (CA)  Fermilab (US) Helmholtz-Karlsruhe (DE)  SLAC (US) DOE Lab  DOE Lab SLAC (US)  JANET (UK) Fermilab (US)  JANET (UK) Argonne (US)  Level3 (US) Argonne  SURFnet (NL) IN2P3 (FR)  SLAC (US) Fermilab (US)  INFN Padva (IT) ESNET Traffic with one server on both sides (April 2004).

HEPiX Conference, SLAC, October 11th-13th SRB BaBar architecture (since 2005). CC-IN2P3 (Lyon) HPSS/Lyon SRB SLAC (Stanford, CA) SRB MCAT (1) (3) (2) HPSS/SLAC SRB MCAT 2 Zones (SLAC + Lyon)

HEPiX Conference, SLAC, October 11th-13th Extra details. Hardware: –Still the same. Software: –Oracle 10g for the SLAC MCAT. –Oracle 9i for the Lyon MCAT (migration to 10g foreseen). MCATs synchronization: only users and physical resources. Comparison of the MCATs contents to transfer the data. Advantage: –External client can pick up data from SLAC or Lyon without interacting with the other site.

HEPiX Conference, SLAC, October 11th-13th Overall assessment. A lot of time saved for developping applications thanks to the SRB. Transparent access to data: –Very useful in an hybrid environment (disk, tape). –Easy to scale the service (adding new servers). –Not dependent of physical locations changes in the application. Fully automated procedure. Easy for SLAC to recover corrupted data. 132 TB (232,000 files) shipped to Lyon. Up to 3 TB /day from tape to tape (minimum latency).

HEPiX Conference, SLAC, October 11th-13th Future plans. Possibly include RAL as part of the system: –Would reduce the network activity on the SLAC & ESNET. Some tests: –Assessment of web services technologies. SLAC HPSS/SLAC SRB HPSS/Lyon SRB CC-IN2P3 RAL SRB

HEPiX Conference, SLAC, October 11th-13th Summary and outlook. SRB used by BaBar for more than 3 years. Data shipment fully automated. Lightweight development to make it work. Easy administration. More servers on both side: –Increase peak transfer rate to 5 TB/day. –Hundreds of TB foreseen. Scalability: the sky is the limit… well and the money too… Aiming at including RAL.

HEPiX Conference, SLAC, October 11th-13th Never forget… Cannot be achieved without the work of: –The network and system admins. –The database admins. Many thanks to them!