A. Sim, CRD, L B N L 1 Production Data Management Workshop, Mar. 3, 2009 BeStMan and Xrootd Alex Sim Scientific Data Management Research Group Computational.

Slides:

Advertisements

Similar presentations

HEPiX GFAL and LCG data management Jean-Philippe Baud CERN/IT/GD.

Advertisements

A. Sim, CRD, L B N L 1 ANI and Magellan Launch, Nov. 18, 2009 Climate 100: Scaling the Earth System Grid to 100Gbps Networks Alex Sim, CRD, LBNL Dean N.

EGEE is a project funded by the European Union under contract IST Using SRM: DPM and dCache G.Donvito,V.Spinoso INFN Bari

1 SRM-Lite: overcoming the firewall barrier for large scale file replication Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory April, 2007.

1 CHEP 2003 Arie Shoshani Experience with Deploying Storage Resource Managers to Achieve Robust File replication Arie Shoshani Alex Sim Junmin Gu Scientific.

A. Sim, CRD, L B N L 1 Data Management Foundations Workshop, Mar. 3, 2009 Storage in OSG and BeStMan Alex Sim Scientific Data Management Research Group.

Experiences Deploying Xrootd at RAL Chris Brew (RAL)

Open Science Grid Software Stack, Virtual Data Toolkit and Interoperability Activities D. Olson, LBNL for the OSG International.

16 th May 2006Alessandra Forti Storage Alessandra Forti Group seminar 16th May 2006.

Presented by The Earth System Grid: Turning Climate Datasets into Community Resources David E. Bernholdt, ORNL on behalf of the Earth System Grid team.

A. Sim, CRD, L B N L GIN-Data : SRM Island Inter-Op Testing With SRM-TESTER Alex Sim, Vijaya Natarajan Computational Research Division Lawrence Berkeley.

SRM at Clemson Michael Fenn. What is a Storage Element? Provides grid-accessible storage space. Is accessible to applications running on OSG through either.

Introduction to OSG Storage Suchandra Thapa Computation Institute University of Chicago March 19, 20091GSAW 2009 Clemson.

A. Sim, CRD, L B N L 1 US CMS Workshop, Mar. 3, 2009 Berkeley Storage Manager (BeStMan) Alex Sim Scientific Data Management Research Group Computational.

The Disk Resource Manager A Berkeley SRM for Disk Cache Implementation and Testing Experience on the OSG Jorge Luis Rodriguez IBP/Grid Mini-Workshop Apr.

A. Sim, CRD, L B N L 1 Oct. 23, 2008 BeStMan Extra Slides.

Data management in grid. Comparative analysis of storage systems in WLCG.

SDM Center February 2, 2005 Progress on MPI-IO Access to Mass Storage System Using a Storage Resource Manager Ekow J. Otoo, Arie Shoshani and Alex Sim.

StoRM Some basics and a comparison with DPM Wahid Bhimji University of Edinburgh GridPP Storage Workshop 31-Mar-101Wahid Bhimji – StoRM.

INTRODUCTION The GRID Data Center at INFN Pisa hosts a big Tier2 for the CMS experiment, together with local usage from other HEP related/not related activities.

A. Sim, CRD, L B N L 1 OSG Applications Workshop 6/1/2005 OSG SRM/DRM Readiness and Plan Alex Sim / Jorge Rodriguez Scientific Data Management Group Computational.

1 Use of SRMs in Earth System Grid Arie Shoshani Alex Sim Lawrence Berkeley National Laboratory.

Data Management The GSM-WG Perspective. Background SRM is the Storage Resource Manager A Control protocol for Mass Storage Systems Standard protocol:

Xrootd, XrootdFS and BeStMan Wei Yang US ATALS Tier 3 meeting, ANL 1.

SLAC Experience on Bestman and Xrootd Storage Wei Yang Alex Sim US ATLAS Tier2/Tier3 meeting at Univ. of Chicago Aug 19-20,

D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.

Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.

OSG Storage Architectures Tuesday Afternoon Brian Bockelman, OSG Staff University of Nebraska-Lincoln.

Author - Title- Date - n° 1 Partner Logo EU DataGrid, Work Package 5 The Storage Element.

LCG Phase 2 Planning Meeting - Friday July 30th, 2004 Jean-Yves Nief CC-IN2P3, Lyon An example of a data access model in a Tier 1.

Xrootd Update Andrew Hanushevsky Stanford Linear Accelerator Center 15-Feb-05

The Earth System Grid: A Visualisation Solution Gary Strand.

OSG Tier 3 support Marco Mambelli - OSG Tier 3 Dan Fraser - OSG Tier 3 liaison Tanya Levshina - OSG.

The Earth System Grid (ESG) Computer Science and Technologies DOE SciDAC ESG Project Review Argonne National Laboratory, Illinois May 8-9, 2003.

Light weight Disk Pool Manager experience and future plans Jean-Philippe Baud, IT-GD, CERN September 2005.

1 Grid File Replication using Storage Resource Management Presented By Alex Sim Contributors: JLAB: Bryan Hess, Andy Kowalski Fermi: Don Petravick, Timur.

A. Sim, CRD, L B N L 1 SRM Collaboration, Aug SRM implementations and deployments that may affect OSG in the near-term Alex Sim Scientific Data Management.

Computing Sciences Directorate, L B N L 1 CHEP 2003 Standards For Storage Resource Management BOF Co-Chair: Arie Shoshani * Co-Chair: Peter Kunszt ** *

1 User Analysis Workgroup Discussion  Understand and document analysis models  Best in a way that allows to compare them easily.

USATLAS dCache System and Service Challenge at BNL Zhenping (Jane) Liu RHIC/ATLAS Computing Facility, Physics Department Brookhaven National Lab 10/13/2005.

1 SRM-Lite: overcoming the firewall barrier for data movement Arie Shoshani Alex Sim Viji Natarajan Lawrence Berkeley National Laboratory SDM Center All-Hands.

Jens G Jensen RAL, EDG WP5 Storage Element Overview DataGrid Project Conference Heidelberg, 26 Sep-01 Oct 2003.

SDM Center Coupling Parallel IO to SRMs for Remote Data Access Ekow Otoo, Arie Shoshani and Alex Sim Lawrence Berkeley National Laboratory.

1 Use of SRM File Streaming by Gateway Alex Sim Arie Shoshani May 2008.

Padova, 5 October StoRM Service view Riccardo Zappi INFN-CNAF Bologna.

The CMS Top 5 Issues/Concerns wrt. WLCG services WLCG-MB April 3, 2007 Matthias Kasemann CERN/DESY.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Upcoming Features and Roadmap Ricardo Rocha ( on behalf of the.

Super Computing 2000 DOE SCIENCE ON THE GRID Storage Resource Management For the Earth Science Grid Scientific Data Management Research Group NERSC, LBNL.

Andrea Manzi CERN On behalf of the DPM team HEPiX Fall 2014 Workshop DPM performance tuning hints for HTTP/WebDAV and Xrootd 1 16/10/2014.

Scalla As a Full-Fledged LHC Grid SE Wei Yang, SLAC Andrew Hanushevsky, SLAC Alex Sims, LBNL Fabrizio Furano, CERN SLAC National Accelerator Laboratory.

A. Sim, CRD, L B N L 1 SRM Collaboration Meeting, Sep , 2005 SRM v3.0 LBNL Implementation Status Report Scientific Data Management Research Group.

Tier 3 Support and the OSG US ATLAS Tier2/Tier3 Workshop at UChicago August 20, 2009 Marco Mambelli –

DCache/XRootD Dmitry Litvintsev (DMS/DMD) FIFE workshop1Dmitry Litvintsev.

1 Scientific Data Management Group LBNL SRM related demos SC 2002 DemosDemos Robust File Replication of Massive Datasets on the Grid GridFTP-HPSS access.

A. Sim, CRD, L B N L 1 OSG Site Administrators Meeting, Dec. 13, 2007 Berkeley Storage Manager (BeStMan) Alex Sim Scientific Data Management Research Group.

OSG STORAGE AND DATA MOVEMENT Tanya Levshina. Talk Outline  Data movement methods and limitation  Open Science Grid (OSG) Storage  Storage Resource.

New Features of Xrootd SE Wei Yang US ATLAS Tier 2/Tier 3 meeting, University of Texas, Arlington,

OSG STORAGE OVERVIEW Tanya Levshina. Talk Outline  OSG Storage architecture  OSG Storage software  VDT cache  BeStMan  dCache  DFS:  SRM Clients.

Introduction to Storage Element Hsin-Wei Wu Academia Sinica Grid Computing Center, Taiwan.

Open Science Grid Consortium Storage on Open Science Grid Placing, Using and Retrieving Data on OSG Resources Abhishek Singh Rana OSG Users Meeting July.

EMI is partially funded by the European Commission under Grant Agreement RI Future Proof Storage with DPM Oliver Keeble (on behalf of the CERN IT-GT-DMS.

BeStMan/DFS support in VDT OSG Site Administrators workshop Indianapolis August Tanya Levshina Fermilab.

KIT - University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association Xrootd SE deployment at GridKa WLCG.

EGEE Data Management Services

a brief summary for users

The Earth System Grid: A Visualisation Solution

Berkeley Storage Manager (BeStMan)

Data Management cluster summary

DRM Deployment Readiness Plan

INFNGRID Workshop – Bari, Italy, October 2004

Presentation transcript:

A. Sim, CRD, L B N L 1 Production Data Management Workshop, Mar. 3, 2009 BeStMan and Xrootd Alex Sim Scientific Data Management Research Group Computational Research Division Lawrence Berkeley National Laboratory

A. Sim, CRD, L B N L 2 Production Data Management Workshop, Mar. 3, 2009 What is SRM? SRM : Storage Resource ManagerSRM : Storage Resource Manager Well-defined storage management interface specification based on standard Different implementations for underlying storage systems are based on the same SRM specification Provides dynamic space allocation and file management on shared storage components on the Grid Over 300 deployments of different SRMs in the world Managing more than 10 PB

A. Sim, CRD, L B N L 3 Production Data Management Workshop, Mar. 3, 2009BeStMan Light-weight implementation of SRM v2.2Light-weight implementation of SRM v2.2 Works on existing disk storages with posix-like file systems E.g. NFS, GPFS, GFS, NGFS, PNFS, HFS+, PVFS, Lustre, Xrootd, Hadoop, Ibrix Supports multiple partitions Adaptable to other file systems and storages Supports customized plug-in for MSS to stage/archive such as HPSS Easy adaptability and integration to special project environments Supports two modeSupports two mode Full mode : management on request queues and spaces Gateway mode : faster performance without queue/space management Supports multiple transfer protocolsSupports multiple transfer protocols Supports load balancing for multiple transfer servers Scales well with some file systems and storagesScales well with some file systems and storages Xrootd, Hadoop, GPFS, PVFS, etc Works with grid-mapfile or GUMS serverWorks with grid-mapfile or GUMS server Simple installation and easy maintenanceSimple installation and easy maintenance Packaged in VDT using Pacman Who would benefit from BeStMan?Who would benefit from BeStMan? Sites with limited resources and/or limited admin effort

A. Sim, CRD, L B N L 4 Production Data Management Workshop, Mar. 3, 2009 Xrootd and XrootdFS Developed at SLAC and INFN PadovaDeveloped at SLAC and INFN Padova Suite of tools for serving dataSuite of tools for serving data server daemons, clients (ROOT TXNetFile, POSIX), use xrootd protocol Provides high performance file-based accessProvides high performance file-based access Scales on server and client sideScales on server and client side Separate data flow and control flowSeparate data flow and control flow Fault tolerantFault tolerant SecureSecure rootd compatiblerootd compatible Xrootd provides a Posix I/O interfaceXrootd provides a Posix I/O interface in addition to the native XTNetFile class XrootdFS provides a Posix File System layerXrootdFS provides a Posix File System layer Mounts the Xrootd storage cluster to a desktop machine Provides a complete view of storage tree Base on FUSE, implemented using Xrootd Posix I/O library

A. Sim, CRD, L B N L 5 Production Data Management Workshop, Mar. 3, 2009 How it works all together File System Gridftp server BeStMan Gateway Client srmPrepareToGet/Put TURL GridFTP file transfers srmReleaseFiles/srmPutDone in PUT/GET File system Gridftp server BeStMan Gateway Client srmLs/srmRm/srmMkdir/srmRmdir in Ls/Rm/Mkdir/Rmdir XrootdFSXrootdXrootdFSXrootd

A. Sim, CRD, L B N L 6 Production Data Management Workshop, Mar. 3, 2009 Some Use Cases CMSCMS BeStMan Gateway as an SRM frontend for Hadoop at UNL Passed all the automated CMS tests through EGEE SAM at UNL ATLASATLAS BeStMan on NFS BeStMan Gateway on Xrootd/FS, GPFS, Ibrix STARSTAR Data replication between BNL and LBNL HPSS access at BNL and NERSC SRMs in production for over 4 years Part of analysis scenario to move job-generated data files from PDSF/NERSC to remote BNL storage Earth System GridEarth System Grid Serving about 6000 users Over a million files and 170TB of climate data from 5 storage sites (LANL, LLNL, NCAR, NERSC, ORNL) Uses an adapted BeStMan for NCAR’s own MSS

A. Sim, CRD, L B N L 7 Production Data Management Workshop, Mar. 3, 2009Summary BeStMan is an implementation of SRM v2.2.BeStMan is an implementation of SRM v2.2. Great for smaller disk-based storage and file systems BeStMan Gateway mode on some file systems and storage gives scalable performanceBeStMan Gateway mode on some file systems and storage gives scalable performance Install/maintain through VDTInstall/maintain through VDT Works with other SRM v2.2 implementationsWorks with other SRM v2.2 implementations Servers: CASTOR, dCache, DPM, StoRM, SRM/SRB, … Clients: FTS, PhEDEx, glite-url-copy, lcg-cp, srm-copy, srmcp, … In OSG, WLCG/EGEE, ESG, …

A. Sim, CRD, L B N L 8 Production Data Management Workshop, Mar. 3, 2009 Documents and Support BeStManBeStMan XrootdXrootd SRM Collaboration and SRM SpecificationsSRM Collaboration and SRM Specifications Contact and support : and support :