02-June-2008Fabrizio Furano - Data access and Storage: new directions1.

Slides:



Advertisements
Similar presentations
Potential Data Access Architectures using xrootd OSG All Hands Meeting Harvard University March 7-11, 2011 Andrew Hanushevsky, SLAC
Advertisements

ATLAS/LHCb GANGA DEVELOPMENT Introduction Requirements Architecture and design Interfacing to the Grid Ganga prototyping A. Soroko (Oxford), K. Harrison.
Xrootd and clouds Doug Benjamin Duke University. Introduction Cloud computing is here to stay – likely more than just Hype (Gartner Research Hype Cycle.
Outline Network related issues and thinking for FAX Cost among sites, who has problems Analytics of FAX meta data, what are the problems  The main object.
1 Status of the ALICE CERN Analysis Facility Marco MEONI – CERN/ALICE Jan Fiete GROSSE-OETRINGHAUS - CERN /ALICE CHEP Prague.
Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google∗
PROOF: the Parallel ROOT Facility Scheduling and Load-balancing ACAT 2007 Jan Iwaszkiewicz ¹ ² Gerardo Ganis ¹ Fons Rademakers ¹ ¹ CERN PH/SFT ² University.
The LCG File Catalog (LFC) Jean-Philippe Baud – Sophie Lemaitre IT-GD, CERN May 2005.
The Next Generation Root File Server Andrew Hanushevsky Stanford Linear Accelerator Center 27-September-2004
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
Scalla/xrootd Andrew Hanushevsky, SLAC SLAC National Accelerator Laboratory Stanford University 19-May-09 ANL Tier3(g,w) Meeting.
Scalla/xrootd Andrew Hanushevsky SLAC National Accelerator Laboratory Stanford University 29-October-09 ATLAS Tier 3 Meeting at ANL
Xrootd, XrootdFS and BeStMan Wei Yang US ATALS Tier 3 meeting, ANL 1.
Scalla/xrootd Introduction Andrew Hanushevsky, SLAC SLAC National Accelerator Laboratory Stanford University 6-April-09 ATLAS Western Tier 2 User’s Forum.
July-2008Fabrizio Furano - The Scalla suite and the Xrootd1 cmsd xrootd cmsd xrootd cmsd xrootd cmsd xrootd Client Client A small 2-level cluster. Can.
11-July-2008Fabrizio Furano - Data access and Storage: new directions1.
D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.
FAX Redirection Topology Wei Yang 7/16/121. Redirector hardware at CERN Redundant redirectors for EU, UK, DE, FR – Redundant (the “+” sign below) VMs.
Introduction to HDFS Prasanth Kothuri, CERN 2 What’s HDFS HDFS is a distributed file system that is fault tolerant, scalable and extremely easy to expand.
July-2008Fabrizio Furano - The Scalla suite and the Xrootd1.
 CASTORFS web page - CASTOR web site - FUSE web site -
Xrootd Update Andrew Hanushevsky Stanford Linear Accelerator Center 15-Feb-05
Redirector xrootd proxy mgr Redirector xrootd proxy mgr Xrd proxy data server N2N Xrd proxy data server N2N Global Redirector Client Backend Xrootd storage.
Status & Plan of the Xrootd Federation Wei Yang 13/19/12 US ATLAS Computing Facility Meeting at 2012 OSG AHM, University of Nebraska, Lincoln.
CERN IT Department CH-1211 Genève 23 Switzerland Internet Services Xrootd explained
Introduction to HDFS Prasanth Kothuri, CERN 2 What’s HDFS HDFS is a distributed file system that is fault tolerant, scalable and extremely easy to expand.
CERN IT Department CH-1211 Genève 23 Switzerland t Scalla/xrootd WAN globalization tools: where we are. Now my WAN is well tuned! So what?
AliEn AliEn at OSC The ALICE distributed computing environment by Bjørn S. Nilsen The Ohio State University.
Scalla/xrootd Andrew Hanushevsky, SLAC SLAC National Accelerator Laboratory Stanford University 08-June-10 ANL Tier3 Meeting.
Scalla Advancements xrootd /cmsd (f.k.a. olbd) Fabrizio Furano CERN – IT/PSS Andrew Hanushevsky Stanford Linear Accelerator Center US Atlas Tier 2/3 Workshop.
Data & Storage Services CERN IT Department CH-1211 Genève 23 Switzerland t DSS XROOTD news New release New features.
JAliEn Java AliEn middleware A. Grigoras, C. Grigoras, M. Pedreira P Saiz, S. Schreiner ALICE Offline Week – June 2013.
AliEn central services Costin Grigoras. Hardware overview  27 machines  Mix of SLC4, SLC5, Ubuntu 8.04, 8.10, 9.04  100 cores  20 KVA UPSs  2 * 1Gbps.
ALICE DATA ACCESS MODEL Outline 05/13/2014 ALICE Data Access Model 2  ALICE data access model  Infrastructure and SE monitoring.
Scalla In’s & Out’s xrootdcmsd xrootd /cmsd Andrew Hanushevsky SLAC National Accelerator Laboratory OSG Administrator’s Work Shop Stanford University/SLAC.
SRM Space Tokens Scalla/xrootd Andrew Hanushevsky Stanford Linear Accelerator Center Stanford University 27-May-08
Scalla As a Full-Fledged LHC Grid SE Wei Yang, SLAC Andrew Hanushevsky, SLAC Alex Sims, LBNL Fabrizio Furano, CERN SLAC National Accelerator Laboratory.
Federated Data Stores Volume, Velocity & Variety Future of Big Data Management Workshop Imperial College London June 27-28, 2013 Andrew Hanushevsky, SLAC.
11-June-2008Fabrizio Furano - Data access and Storage: new directions1.
Latest Improvements in the PROOF system Bleeding Edge Physics with Bleeding Edge Computing Fons Rademakers, Gerri Ganis, Jan Iwaszkiewicz CERN.
Bestman & Xrootd Storage System at SLAC Wei Yang Andy Hanushevsky Alex Sim Junmin Gu.
Data transfers and storage Kilian Schwarz GSI. GSI – current storage capacities vobox LCG RB/CE GSI batchfarm: ALICE cluster (67 nodes/480 cores for batch.
09-Apr-2008Fabrizio Furano - Scalla/xrootd status and features1.
DCache/XRootD Dmitry Litvintsev (DMS/DMD) FIFE workshop1Dmitry Litvintsev.
Dynamic and Scalable Distributed Metadata Management in Gluster File System Huang Qiulan Computing Center,Institute of High Energy Physics,
A System for Monitoring and Management of Computational Grids Warren Smith Computer Sciences Corporation NASA Ames Research Center.
New Features of Xrootd SE Wei Yang US ATLAS Tier 2/Tier 3 meeting, University of Texas, Arlington,
Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Solutions for WAN data access: xrootd and NFSv4.1 Andrea Sciabà.
PRIN STOA-LHC: STATUS BARI BOLOGNA-18 GIUGNO 2014 Giorgia MINIELLO G. MAGGI, G. DONVITO, D. Elia INFN Sezione di Bari e Dipartimento Interateneo.
CERN IT Department CH-1211 Genève 23 Switzerland t Xrootd LHC An up-to-date technical survey about xrootd- based storage solutions.
The ALICE Analysis -- News from the battlefield Federico Carminati for the ALICE Computing Project CHEP 2010 – Taiwan.
Kilian Schwarz ALICE Computing Meeting GSI, October 7, 2009
a brief summary for users
Global Data Access – View from the Tier 2
Blueprint of Persistent Infrastructure as a Service
SCTP Handoff for Cluster Servers
Xrootd explained Cooperation among ALICE SEs
XRootD Release 4.5 And Beyond
Data access and Storage
Data Federation with Xrootd Wei Yang US ATLAS Computing Facility meeting Southern Methodist University, Oct 11-12, 2011.
FDR readiness & testing plan
GFAL 2.0 Devresse Adrien CERN lcgutil team
Brookhaven National Laboratory Storage service Group Hironori Ito
Ákos Frohner EGEE'08 September 2008
Scalla/XRootd Advancements
Support for ”interactive batch”
CS 345A Data Mining MapReduce This presentation has been altered.
INFNGRID Workshop – Bari, Italy, October 2004
CS533 Concepts of Operating Systems Class 11
Presentation transcript:

02-June-2008Fabrizio Furano - Data access and Storage: new directions1

02-June-2008Fabrizio Furano - Data access and Storage: new directions2

02-June-2008Fabrizio Furano - Data access and Storage: new directions3

02-June-2008Fabrizio Furano - Data access and Storage: new directions4

02-June-2008Fabrizio Furano - Data access and Storage: new directions5

02-June-2008Fabrizio Furano - Data access and Storage: new directions6 Courtesy of Gerardo Ganis (CERN PH-SFT)

02-June-2008Fabrizio Furano - Data access and Storage: new directions7 Courtesy of Gerardo Ganis (CERN PH-SFT)

02-June-2008Fabrizio Furano - Data access and Storage: new directions8

02-June-2008Fabrizio Furano - Data access and Storage: new directions9

02-June-2008Fabrizio Furano - Data access and Storage: new directions10

02-June-2008Fabrizio Furano - Data access and Storage: new directions11 Pre-xfer data “locally” Pre-xfer data “locally” Remote access Remote access Remote access+ Remote access+ Data Processing Data access Overhead Need for potentially useless replicas And a huge Bookkeeping! Overhead Need for potentially useless replicas And a huge Bookkeeping! Latency Wasted CPU cycles But easy to understand Latency Wasted CPU cycles But easy to understand Interesting! Efficient practical Interesting! Efficient practical

Application 02-June-2008Fabrizio Furano - Data access and Storage: new directions12 Client1 Server Client2 Client3 TCP (control)‏ Clients still see One Physical connection per server Clients still see One Physical connection per server TCP(data)‏ Async data gets automaticall y splitted Async data gets automaticall y splitted

02-June-2008Fabrizio Furano - Data access and Storage: new directions13

09-Apr-2008Fabrizio Furano - Scalla/xrootd status and features14 *Federico Carminati, The ALICE Computing Status and Readiness, LHCC, November 2007

09-Apr-2008Fabrizio Furano - Scalla/xrootd status and features15 *Federico Carminati, The ALICE Computing Status and Readiness, LHCC, November 2007

02-June-2008Fabrizio Furano - Data access and Storage: new directions16

02-June-2008Fabrizio Furano - Data access and Storage: new directions17

02-June-2008Fabrizio Furano - Data access and Storage: new directions18

02-June-2008Fabrizio Furano - Data access and Storage: new directions19 cmsd xrootd Prague NIHAM … any other cmsd xrootd CERN cmsd xrootd ALICE global redirector (alirdr) all.role meta manager all.manager meta alirdr.cern.ch:1312root://alirdr.cern.ch/Includes CERN, GSI, and others xroot clusters Meta Managers can be geographically replicated Can have several in different places for region- aware load balancing Meta Managers can be geographically replicated Can have several in different places for region- aware load balancing cmsd xrootd GSI all.manager meta alirdr.cern.ch:1312 all.role manager

cmsd xrootd GSI 02-June-2008Fabrizio Furano - Data access and Storage: new directions20 cmsd xrootd Prague NIHAM … any other cmsd xrootd CERN cmsd xrootd ALICE global redirector all.role meta manager all.manager meta alirdr.cern.ch:1312 all.role manager Note: the security hats could require you use xrootd native proxy support But missing a file? Ask to the global metamgr Get it from any other collaborating cluster all.manager meta alirdr.cern.ch:1312 Local clients work normally

02-June-2008Fabrizio Furano - Data access and Storage: new directions21

02-June-2008Fabrizio Furano - Data access and Storage: new directions22

02-June-2008Fabrizio Furano - Data access and Storage: new directions23

02-June-2008Fabrizio Furano - Data access and Storage: new directions24

02-June-2008Fabrizio Furano - Data access and Storage: new directions25

02-June-2008Fabrizio Furano - Data access and Storage: new directions26

02-June-2008Fabrizio Furano - Data access and Storage: new directions27 Redirector xrootd:1094 Name Space xrootd:2094RedirectorHost ClientHost opendir create mkdir mv rm rmdir xrootd POSIX Client Kernel User Space Appl POSIX File System Interface FUSE FUSE/Xroot Interface Should run cnsd on servers to capture non-FUSE events And keep the FS namespace!

02-June-2008Fabrizio Furano - Data access and Storage: new directions28

02-June-2008Fabrizio Furano - Data access and Storage: new directions29

02-June-2008Fabrizio Furano - Data access and Storage: new directions30