DPM Name Server (DPNS) Namespace Authorization Location of physical files DPM Server Requests queuing and processing Space Management SRM Servers v1.1,

Slides:

Advertisements

Similar presentations

INFSO-RI Enabling Grids for E-sciencE EGEE and gLite Slides by: Erwin Laure EGEE Deputy Middleware Manager.

Advertisements

30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.

DPM Monitoring Wahid Bhimji University of Edinburgh, Apr-101Wahid Bhimji – Files access.

Metadata Progress GridPP18 20 March 2007 Mike Kenyon.

DPM Basics and its status and plans Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010 Apr-101Wahid Bhimji – DPM.

12th EELA TUTORIAL - USERS AND SYSTEM ADMINISTRATORS E-infrastructure shared between Europe and Latin America SRM Installation and Configuration.

INFSO-RI Enabling Grids for E-sciencE Agreement Service for Resource Reservation and Allocation: Overview Tiziana Ferrari, Elisabetta.

EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,

EGEE is a project funded by the European Union under contract IST Using SRM: DPM and dCache G.Donvito,V.Spinoso INFN Bari

Storage: Futures Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 8 October 2008.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT dpm-xrootd v3 Creating Federated Data Stores for the LHC David.

The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.

SRM at Clemson Michael Fenn. What is a Storage Element? Provides grid-accessible storage space. Is accessible to applications running on OSG through either.

DPM CCRC - 1 Research and developments DPM status and plans Jean-Philippe Baud.

CERN, 29 August 2006 Status Report Riccardo Zappi INFN-CNAF, Bologna.

StoRM Some basics and a comparison with DPM Wahid Bhimji University of Edinburgh GridPP Storage Workshop 31-Mar-101Wahid Bhimji – StoRM.

VOMS Alessandra Forti HEP Sysman meeting April 2005.

The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.

INFSO-RI Enabling Grids for E-sciencE DPM Administration Jean-Philippe Baud (Sophie Lemaitre)

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE middleware: gLite Data Management EGEE Tutorial 23rd APAN Meeting, Manila Jan.

Architecture and ATLAS Western Tier 2 Wei Yang ATLAS Western Tier 2 User Forum meeting SLAC April

Enabling Grids for E-sciencE Introduction Data Management Jan Just Keijser Nikhef Grid Tutorial, November 2008.

Maarten Litmaath (CERN), GDB meeting, CERN, 2006/02/08 VOMS deployment Extent of VOMS usage in LCG-2 –Node types gLite 3.0 Issues Conclusions.

Light weight Disk Pool Manager experience and future plans Jean-Philippe Baud, IT-GD, CERN September 2005.

Owen SyngeTitle of TalkSlide 1 Storage Management Owen Synge – Developer, Packager, and first line support to System Administrators. Talks Scope –GridPP.

INFSO-RI Enabling Grids for E-sciencE gLite Data Management and Interoperability Peter Kunszt (JRA1 DM Cluster) 2 nd EGEE Conference,

INFSO-RI Enabling Grids for E-sciencE Enabling Grids for E-sciencE Pre-GDB Storage Classes summary of discussions Flavia Donno Pre-GDB.

US LHC OSG Technology Roadmap May 4-5th, 2005 Welcome. Thank you to Deirdre for the arrangements.

INFSO-RI Enabling Grids for E-sciencE Experiences with LFC and comparison with RNS Erwin Laure Jean-Philippe.

EGEE-II INFSO-RI Enabling Grids for E-sciencE Data Management cluster summary Krzysztof Nienartowicz JRA1 All Hands meeting, Helsinki.

Jens G Jensen RAL, EDG WP5 Storage Element Overview DataGrid Project Conference Heidelberg, 26 Sep-01 Oct 2003.

Derek Ross E-Science Department DCache Deployment at Tier1A UK HEP Sysman April 2005.

1 e-Science AHM st Aug – 3 rd Sept 2004 Nottingham Distributed Storage management using SRB on UK National Grid Service Manandhar A, Haines K,

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE Site Architecture Resource Center Deployment Considerations MIMOS EGEE Tutorial.

SEE-GRID-SCI Storage Element Installation and Configuration Branimir Ackovic Institute of Physics Serbia The SEE-GRID-SCI.

INFSO-RI Enabling Grids for E-sciencE Introduction Data Management Ron Trompert SARA Grid Tutorial, September 2007.

Enabling Grids for E-sciencE EGEE-II INFSO-RI Medical Data Manager 1 Dicom retrieval : overview of the DPM One command line to retrieve a file:

EGEE is a project funded by the European Union under contract IST VO box: Experiment requirements and LCG prototype Operations.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Upcoming Features and Roadmap Ricardo Rocha ( on behalf of the.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Data management in LCG and EGEE David Smith.

Andrea Manzi CERN On behalf of the DPM team HEPiX Fall 2014 Workshop DPM performance tuning hints for HTTP/WebDAV and Xrootd 1 16/10/2014.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks gLite Data Management Components Presenter.

INFSO-RI Enabling Grids for E-sciencE SRMv2.2 in DPM Sophie Lemaitre Jean-Philippe.

SRM-2 Road Map and CASTOR Certification Shaun de Witt 3/3/08.

SESEC Storage Element (In)Security hepsysman, RAL 0-1 July 2009 Jens Jensen.

StoRM: status report A disk based SRM server.

SRM-iRODS Interface Development WeiLong UENG Academia Sinica Grid Computing 1.

EGEE-II INFSO-RI Enabling Grids for E-sciencE Architecture of LHC File Catalog Valeria Ardizzone INFN Catania – EGEE-II NA3/NA4.

CMS User Support and Beijing Site Xiaomei Zhang CMS IHEP Group Meeting March

Bologna, March 30, 2006 Riccardo Zappi / Luca Magnoni INFN-CNAF, Bologna.

Security recommendations DPM Jean-Philippe Baud CERN/IT.

Open Science Grid Consortium Storage on Open Science Grid Placing, Using and Retrieving Data on OSG Resources Abhishek Singh Rana OSG Users Meeting July.

Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.

Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Standard Protocols in DPM Ricardo Rocha.

CERN IT Department CH-1211 Genève 23 Switzerland t DPM status and plans David Smith CERN, IT-DM-SGT Pre-GDB, Grid Storage Services 11 November.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks SE Security Rémi Mollon, Ákos Frohner EGEE'08,

Data Management & Information Systems

Enabling Grids for E-sciencE EGEE-II INFSO-RI The Development of SRM interface for SRB Fu-Ming Tsai Academia Sinica Grid Computing.

EGEE Data Management Services

a brief summary for users

Jean-Philippe Baud, IT-GD, CERN November 2007

DPM Installation Configuration

StoRM: a SRM solution for disk based storage systems

Status of the SRM 2.2 MoU extension

The lightweight Grid-enabled Disk Pool Manager (DPM)

Jean-Philippe Baud - Sophie Lemaitre IT-GD, CERN May 2005

Introduction to Data Management in EGI

Ákos Frohner EGEE'08 September 2008

Data Management cluster summary

INFNGRID Workshop – Bari, Italy, October 2004

Presentation transcript:

DPM Name Server (DPNS) Namespace Authorization Location of physical files DPM Server Requests queuing and processing Space Management SRM Servers v1.1, v2.1 and v2.2 Database (DPM and DPNS) MySQL and Oracle are supported High scalability and availability DPM and DPNS scalability is directly linked to the database (MySQL or Oracle) scalability All servers can be load balanced, except the DPM one Security in DPM GSI Permissions – Ownership ACLs VOMS support Virtual uids/gids (based on VOMS groups and roles) SRM v2.2 Admin srmPing Space Management srmReserveSpace, srmReleaseSpace,... Permissions and ACLs srmSetPermission, srmGetPermission,... Transfer srmPrepareToPut, srmPrepareToGet,... Namespace srmLs, srmMkdir The DPM (Disk Pool Manager) is a secure, lightweight and highly scalable system to manage storage on disks DPM head node Disk Servers The Disk Pool Manager (DPM) Enabling Grids for E-sciencE EGEE-II INFSO-RI The Disk Pool Manager Data Management in gLite Status and Next Steps file DPM disk servers /vo /dpm /domain /home DPM head node CLI, C API, SRM-enabled client, etc. data transfer … Quotas srmCopy NFSv4.1 support Medical Data Management Encryption New features http / https In the namespace Secondary groups support On disk pools Secondary groups intentionally not supported at pool level (so that the group or sub-group who actually uses the space gets accounted for it) But possibility to: (1) restrict a pool to a list of groups (2) add and remove groups later on Web access to DPM storage via http and https The DPM Architecture Largest DPM Storage 80 TB (Glasgow) Largest DPM Name Server million entries (Oracle) Thanks to Gidon Moont (GridPP – Imperial College – London) See DPM target in EGEE Small & medium-sized sites What do sites like? Easy installation Easy administration DPMs in production 125 instances 138 VOs supported (CERN) Akos FROHNER, Jean-Philippe BAUD, David SMITH, Sophie LEMAITRE, Krzysztof NIENARTOWICZ, Lana ABADIE, Rémi MOLLON (1) dpm-addpool --poolname AtlasPool --def_filesize 1G --group atlas, atlas/Role=production (2) dpm-modifypool --poolname AtlasPool --group +atlas/Role=lcgadmin,- atlas/Role=production GridFTP Server Secure RFIO Server The Disk Servers host the actual physical files No bottleneck: direct data transfer from/to the Disk Servers Easy to guarantee space for a given VO Easy space reservation Space can easily be reserved (1) and released (3) by users (limited) administrators (unlimited) (1) dpm-reservespace --gspace 200G --group atlas --token_desc atlas_ESD (2) lcg-cr --vo atlas file:higgs.root --st atlas_ESD –d dpm.cern.ch (3) dpm-releasespace --token_desc atlas_ESD The user space token created in (1) by the DPM administrator can be used by any user in the atlas group. Ex. with lcg_util (2) $ export DPNS_HOST=dpm.cern.ch $ export DPM_HOST=dpm.cern.ch $ rfcp file01 /dpm/cern.ch/home/atlas/data/file01 $ rfrm /dpm/cern.ch/home/atlas/file01 Usage example: