Your university or experiment logo here Storage and Data Management - Background Jens Jensen, STFC.

Slides:



Advertisements
Similar presentations
30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.
Advertisements

GridPP3 Storage Perspective, Achievements, Challenges Jens Jensen, STFC RAL GridPP20 TCD Dublin, March 2008.
Applications Area Issues RWL Jones GridPP13 – 5 th June 2005.
Storage Workshop Summary Wahid Bhimji University Of Edinburgh On behalf all of the participants…
User Board - Supporting Other Experiments Stephen Burke, RAL pp Glenn Patrick.
Jens G Jensen Atlas Petabyte store Supporting Multiple Interfaces to Mass Storage Providing Tape and Mass Storage to Diverse Scientific Communities.
J Jensen CCLRC RAL Data Management AUZN (mostly about SRM though) GGF 16, Athens J Jensen.
Jens G Jensen CCLRC/RAL hepsysman 2005Storage Middleware SRM 2.1 issues hepsysman Oxford 5 Dec 2005.
Wahid Bhimji SRM; FTS3; xrootd; DPM collaborations; cluster filesystems.
LCG Tiziana Ferrari - SC3: INFN installation status report 1 Service Challenge Phase 3: Status report Tiziana Ferrari on behalf of the INFN SC team INFN.
Data & Storage Management TEGs Summary of recommendations Wahid Bhimji, Brian Bockelman, Daniele Bonacorsi, Dirk Duellmann GDB, CERN 18 th April 2012.
Storage Issues: the experiments’ perspective Flavia Donno CERN/IT WLCG Grid Deployment Board, CERN 9 September 2008.
Data Transfer Efficiency - leave no byte unchurned Jens Jensen Rutherford Appleton Laboratory GridPP26, U Sussex, March 2011.
Wahid Bhimji Andy Washbrook And others including ECDF systems team Not a comprehensive update but what ever occurred to me yesterday.
Jan 2010 Current OSG Efforts and Status, Grid Deployment Board, Jan 12 th 2010 OSG has weekly Operations and Production Meetings including US ATLAS and.
CASTOR Upgrade, Testing and Issues Shaun de Witt GRIDPP August 2010.
Wahid Bhimji University of Edinburgh P. Clark, M. Doidge, M. P. Hellmich, S. Skipsey and I. Vukotic 1.
GridPP Steve Lloyd, Chair of the GridPP Collaboration Board.
INFSO-RI Enabling Grids for E-sciencE SRMv2.2 experience Sophie Lemaitre WLCG Workshop.
Heads in the cloud? GSM-WG at OGF31, Taipei Jens Jensen, RAL.
SRM 2.2: status of the implementations and GSSD 6 th March 2007 Flavia Donno, Maarten Litmaath INFN and IT/GD, CERN.
News from the HEPiX IPv6 Working Group David Kelsey (STFC-RAL) GridPP35, Liverpool 11 Sep 2015.
PhysX CoE: LHC Data-intensive workflows and data- management Wahid Bhimji, Pete Clarke, Andrew Washbrook – Edinburgh And other CoE WP4 people…
Storage Wahid Bhimji DPM Collaboration : Tasks. Xrootd: Status; Using for Tier2 reading from “Tier3”; Server data mining.
Wahid, Sam, Alastair. Now installed on production storage Edinburgh: srm.glite.ecdf.ed.ac.uk  Local and global redir work (port open) e.g. root://srm.glite.ecdf.ed.ac.uk//atlas/dq2/mc12_8TeV/NTUP_SMWZ/e1242_a159_a165_r3549_p1067/mc1.
Data and Storage Evolution in Run 2 Wahid Bhimji Contributions / conversations / s with many e.g.: Brian Bockelman. Simone Campana, Philippe Charpentier,
Your university or experiment logo here GridPP Storage Future Jens Jensen GridPP workshop RHUL, April 2010.
GLUE 2 Open Issues in Storage Information Providers 16 th May 2014.
GridPP Dirac Service The 4 th Dirac User Workshop May 2014 CERN Janusz Martyniak, Imperial College London.
1 LHCb on the Grid Raja Nandakumar (with contributions from Greig Cowan) ‏ GridPP21 3 rd September 2008.
1 User Analysis Workgroup Discussion  Understand and document analysis models  Best in a way that allows to compare them easily.
INFSO-RI Enabling Grids for E-sciencE Enabling Grids for E-sciencE Pre-GDB Storage Classes summary of discussions Flavia Donno Pre-GDB.
WLCG Grid Deployment Board, CERN 11 June 2008 Storage Update Flavia Donno CERN/IT.
WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing.
Your university or experiment logo here The European Landscape John Gordon GridPP24 RHUL 15 th April 2010.
LCG Storage Accounting John Gordon CCLRC – RAL LCG Grid Deployment Board September 2006.
Jens G Jensen RAL, EDG WP5 Storage Element Overview DataGrid Project Conference Heidelberg, 26 Sep-01 Oct 2003.
Federated Identity Management for HEP David Kelsey STFC – RAL Nijmegen workshop 22 June 2012.
Storage Interfaces Introduction Wahid Bhimji University of Edinburgh Based on previous discussions with Working Group: (Brian Bockelman, Simone Campana,
Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland t DBCF GT Upcoming Features and Roadmap Ricardo Rocha ( on behalf of the.
Julia Andreeva on behalf of the MND section MND review.
The GridPP DIRAC project DIRAC for non-LHC communities.
Andrea Manzi CERN On behalf of the DPM team HEPiX Fall 2014 Workshop DPM performance tuning hints for HTTP/WebDAV and Xrootd 1 16/10/2014.
WLCG Service Report ~~~ WLCG Management Board, 31 st March 2009.
GridPP storage status update Joint GridPP Board Deployment User Experiment Update Support Team, Imperial 12 July 2007,
Storage Interfaces and Access pre-GDB Wahid Bhimji University of Edinburgh On behalf of all those who participated.
UKI-LT2-RHUL ATLAS STEP09 report Duncan Rand on behalf of the RHUL Grid team.
SESEC Storage Element (In)Security hepsysman, RAL 0-1 July 2009 Jens Jensen.
INFSO-RI Enabling Grids for E-sciencE Enabling Grids for E-sciencE Storage Element Model and Proposal for Glue 1.3 Flavia Donno,
Testing Infrastructure Wahid Bhimji Sam Skipsey Intro: what to test Existing testing frameworks A proposal.
Storage Interfaces and Access: Interim report Wahid Bhimji University of Edinburgh On behalf of WG: Brian Bockelman, Philippe Charpentier, Simone Campana,
Grid Deployment Board 5 December 2007 GSSD Status Report Flavia Donno CERN/IT-GD.
The Grid Storage System Deployment Working Group 6 th February 2007 Flavia Donno IT/GD, CERN.
The GridPP DIRAC project DIRAC for non-LHC communities.
WLCG Service Report Jean-Philippe Baud ~~~ WLCG Management Board, 24 th August
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks EGEE Operations: Evolution of the Role of.
Andrea Manzi CERN EGI Conference on Challenges and Solutions for Big Data Processing on cloud 24/09/2014 Storage Management Overview 1 24/09/2014.
An Analysis of Data Access Methods within WLCG Shaun de Witt, Andrew Lahiff (STFC)
Outcome should be a documented strategy Not everything needs to go back to square one! – Some things work! – Some work has already been (is being) done.
User Domain Storage Elements SURL  TURL LFC Domain (LCG File Catalogue) SA1 – Data Grid Interoperation Enabling Grids for E-sciencE EGEE-III INFSO-RI
CMS data access Artem Trunov. CMS site roles Tier0 –Initial reconstruction –Archive RAW + REC from first reconstruction –Analysis, detector studies, etc.
HEPiX IPv6 Working Group David Kelsey (STFC-RAL) GridPP33 Ambleside 22 Aug 2014.
J Jensen / WP5 /RAL UCL 4/5 March 2004 GridPP / DataGrid wrap-up Mass Storage Management J Jensen
Riccardo Zappi INFN-CNAF SRM Breakout session. February 28, 2012 Ingredients 1. Basic ingredients (Fabric & Conn. level) 2. (Grid) Middleware ingredients.
WLCG IPv6 deployment strategy
Storage Interfaces and Access: Introduction
Gfal/lcg-util -> Gfal2/gfal2-util
Taming the protocol zoo
SRM2 Migration Strategy
GLUE 2 Support in gLite Data Management
Presentation transcript:

Your university or experiment logo here Storage and Data Management - Background Jens Jensen, STFC

PICS FROM YESTERDAY guid:723ce86b-8e2f-4b6e-97cb-db2fb1622a66

Overview “Support the deployment and operation of Grid storage and data management middleware to enable every GridPP site to meet the needs of the its users and the site’s obligations” Storage and Data Managament Group –Three supporters – Wahid Bhimji (Ed), Sam Skipsey (Gla), Brian Davies (RAL) –“hard core” of ~15 people attending weekly meetings –~70-80 people on mailing list (mostly T2s), from 10 dif. countries –Blog – –Wiki – –Contributions to WLCG (eg TEGs) “Finding” the Higgs was useful –“We did that – and we support other research, too.”

Technologies SRM –DPM –StoRM –dCache –(CASTOR) Storage information systems –CIP and GIPs in BDIIs Data movers –FTS (Catalogues) –(LFC)

 Technologies WebDAV (RFC4918) – HEP case is read only –(Study by C.T.Cirstea, Eindhoven) –Uses HTTP (RFC2616), adds a few commands –HEP: only using PROP commands NFS4 –Local access protocol –Instead of RFIO and DCAP Future of SRM... –Data access –Data transfer –Data control –Data information –“Reserved space”

Services for VOs Help getting “onto the grid” –Often local (like Campus Champions, except more ad hoc) –Supporting VO data model Syncat Performance tuning –Storage systems Large VOs and small VOs –ATLAS – fairly well represented –CMS – usu. via sites? –LHCb – “no data at T2s” –ALICE in T1/experiment meetings –Non-LHC VOs – Wedn. meetings

Stuff Lots of expertise in the group –How do we best make use of it? DPM support –See Wahid’s presentation –A Plan exists Information Systems –Accuracy of publishing –The need for EGI-compliant GLUE 2 publishing Supporting the experiments Storage “federations” Future data management –Lots of EU funded projects (and proposals) –Cloud storage – people will ask...