DCache things Paul Millar … on behalf of the dCache team.

Slides:



Advertisements
Similar presentations
30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.
Advertisements

High Throughput Data Program at Fermilab R&D Parag Mhashilkar Grid and Cloud Computing Department Computing Sector, Fermilab Network Planning for ESnet/Internet2/OSG.
ATLAS T1/T2 Name Space Issue with Federated Storage Hironori Ito Brookhaven National Laboratory.
Storage: Futures Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 8 October 2008.
Volunteer Computing Laurence Field IT/SDC 21 November 2014.
WebFTS as a first WLCG/HEP FIM pilot
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Data Grid Web Services Chip Watson Jie Chen, Ying Chen, Bryan Hess, Walt Akers.
QCDgrid Technology James Perry, George Beckett, Lorna Smith EPCC, The University Of Edinburgh.
ILDG Middleware Status Chip Watson ILDG-6 Workshop May 12, 2005.
D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.
Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.
NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.
WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing.
XROOTD AND FEDERATED STORAGE MONITORING CURRENT STATUS AND ISSUES A.Petrosyan, D.Oleynik, J.Andreeva Creating federated data stores for the LHC CC-IN2P3,
CERN IT Department CH-1211 Geneva 23 Switzerland GT HTTP solutions for data access, transfer, federation Fabrizio Furano (presenter) on.
Providing web services to mobile users: The architecture design of an m-service portal Minder Chen - Dongsong Zhang - Lina Zhou Presented by: Juan M. Cubillos.
Tier3 monitoring. Initial issues. Danila Oleynik. Artem Petrosyan. JINR.
Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Upcoming Features and Roadmap Ricardo Rocha ( on behalf of the.
Andrea Manzi CERN On behalf of the DPM team HEPiX Fall 2014 Workshop DPM performance tuning hints for HTTP/WebDAV and Xrootd 1 16/10/2014.
EGI-Engage Data Services and Solutions Part 1: Data in the Grid Vincenzo Spinoso EGI.eu/INFN Data Services.
Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Overview of DMLite Ricardo Rocha ( on behalf of the LCGDM team.
COM: 111 Introduction to Computer Applications Department of Information & Communication Technology Panayiotis Christodoulou.
Wahid Bhimji (Some slides are stolen from Markus Schulz’s presentation to WLCG MB on 19 June Apologies to those who have seen some of this before)
DCache/XRootD Dmitry Litvintsev (DMS/DMD) FIFE workshop1Dmitry Litvintsev.
Andrea Manzi CERN EGI Conference on Challenges and Solutions for Big Data Processing on cloud 24/09/2014 Storage Management Overview 1 24/09/2014.
CERN IT Department CH-1211 Geneva 23 Switzerland t OIS Operating Systems & Information Services CERN IT Department CH-1211 Geneva 23 Switzerland.
Dynamic Federation of Grid and Cloud Storage Fabrizio Furano, Oliver Keeble, Laurence Field Speaker: Fabrizio Furano.
Portlet Development Konrad Rokicki (SAIC) Manav Kher (SemanticBits) Joshua Phillips (SemanticBits) Arch/VCDE F2F November 28, 2008.
J Jensen / WP5 /RAL UCL 4/5 March 2004 GridPP / DataGrid wrap-up Mass Storage Management J Jensen
Security recommendations for dCache
dCache Paul Millar, on behalf of the dCache Team
a brief summary for users
Gene Oleynik, Head of Data Storage and Caching,
Scalable sync-and-share service with dCache
Dynamic Storage Federation based on open protocols
Ricardo Rocha ( on behalf of the DPM team )
Virtualization and Clouds ATLAS position
Federation made simple
StoRM: a SRM solution for disk based storage systems
Vincenzo Spinoso EGI.eu/INFN
Status of the SRM 2.2 MoU extension
WLCG Data Steering Group
BNL Tier1 Report Worker nodes Tier 1: added 88 Dell R430 nodes
Report of Dubna discussion
James Casey, IT-GD, CERN CERN, 5th September 2005
Dynafed, DPM and EGI DPM workshop 2016 Speaker: Fabrizio Furano
AAI … but This talk is about the second 'A': Authorisation.
dCache – protocol developments and plans
Evolution of Internet.
dCache Scientific Cloud
Taming the protocol zoo
Data Federation with Xrootd Wei Yang US ATLAS Computing Facility meeting Southern Methodist University, Oct 11-12, 2011.
Study course: “Computing clusters, grids and clouds” Andrey Y. Shevel
Storage Resource Reporting Proposal
GFAL 2.0 Devresse Adrien CERN lcgutil team
THE STEPS TO MANAGE THE GRID
XROOTd for cloud storage
WLCG Demonstrator R.Seuster (UVic) 09 November, 2016
WLCG and support for IPv6-only CPU
dCache: new and exciting features
Large Scale Test of a storage solution based on an Industry Standard
#01 Client/Server Computing
Data Management cluster summary
Know about 3D Configurator and their Business Advantages.
A Web-Based Data Grid Chip Watson, Ian Bird, Jie Chen,
World Wide Web “WWW”, "Web" or "W3". World Wide Web “WWW”, "Web" or "W3"
World Wide Web “WWW”, "Web" or "W3". World Wide Web “WWW”, "Web" or "W3"
#01 Client/Server Computing
Summary of the dCache workshop
Presentation transcript:

dCache things Paul Millar … on behalf of the dCache team

Agility in a Changing World Adapt to changing environment Adapt to changing access methods Adapt to different users

Flexibility through plugins dCache has always been flexible Very flexible configuration Little known secret, dCache has always been extensible (adding new functionality). Added support for loading plugins when starting If there's something you can't configure, write a plugin!

gPlazma: flexible identity management Old gPlazma wasn't flexible Monolithic operations: either succeed or fail Rewrote to use a plugin approach based on PAM Plugins can cooperate to complete a login step Plan to add support for Federated Identity Part of the LSDMA project Good test of gPlazma flexibility In good shape if WLCG moves away from X.509

gPlazma: flexible identity management LSDMA Non-HEP users: no grid certificate /X.509 Web-based solutions are preferred (portals for metadata-based work), LSDMA WP1 is Federated AAI, dCache has flexible identity management system Plugins allow adding new functionality to work with existing Useful if WLCG moves away from pure X.509.

Federated storage: the idea Client-driven Find all replicas Choose “best” one Open data to best replica Federated storage Open data Make request to central end-point Find all replicas Chooses “best” one Reply with redirect Follow redirect The idea: Access the data from wherever is “closest” Don't be explicit: always use a “central” endpoint, which redirects client to (hopefully) the best choice Currently for read-only (e.g., recovery) Benefits: Move logic to central components Easier to keep up-to-date Allows stateful decision Needs a protocol with redirection support: xrootd and HTTP NB. needs a network protocol that supports redirection: xrootd or HTTP NB. file may have different paths in different storage systems

Federated storage: the implementation xrootd-based Experiment-driven CMS pushing, ATLAS are investigating xrootd door has plugins: Allows global-to- local path mapping Plugin for CMS and ATLAS developed Experiments are involved in plugins HTTP-based Collaboration between DESY and CERN IT- GT/DMS Allows use of industry standard software Can browse files Currently in the early stages Prototype software written Initial performance tests are encouraging

dCache scientific cloud Easy way to store, use and share data Form new collaborations Currently: Users get their own area HTTP/WebDAV for direct access to files, Use web-browser for managing stored files, drag-and-drop support Data is available on compute resources (via NFS) Files are available for copying (via GridFTP) Active collaboration with Globus Online Longer-term: Provide DropBox-like storage that allows scientific analysis User sees only their own data (and what's shared)

Conclusion dCache is agile software flexibility achieved through plugins more than SRM, more than grid storage dCache is back-bone of HEP analysis ~50% of data stored in dCache Get involved! If you don't have dCache, install it If you have dCache, what else can you get it to do?

Image credits Skateboard: Tim Drivas (from flickr) Horses: Verity Borthwick (from flickr) Keyboard: Who Are You? psd (from flickr)