Towards Storage On-Demand on Petabyte Grids A case for: Quotas and Reserve-ahead, Timed Leases and Contracts Frank Wuerthwein UC San Diego Abhishek Singh.

Slides:

Advertisements

Similar presentations

Open Science Grid Living on the Edge: OSG Edge Services Framework Kate Keahey Abhishek Rana.

Advertisements

Applications Area Issues RWL Jones GridPP13 – 5 th June 2005.

Global Grid Forum GridWorld GGF15 Boston USA October Abhishek Singh Rana and Frank Wuerthwein UC San Diegowww.opensciencegrid.org The Open Science.

Ted Hesselroth USCMS T3 Meeting Abhishek Singh Rana and Frank Wuerthwein UC San Diego Ted Hesselroth STAF/STAX OSG Storage Demo of.

Role Based VO Authorization Services Ian Fisk Gabriele Carcassi July 20, 2005.

1 User Analysis Workgroup Update  All four experiments gave input by mid December  ALICE by document and links  Very independent.

Implementing Finer Grained Authorization in the Open Science Grid Gabriele Carcassi, Ian Fisk, Gabriele, Garzoglio, Markus Lorch, Timur Perelmutov, Abhishek.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Infrastructure overview Arnold Meijster &

Ted Hesselroth Nordugrid 2007 September 24-28, 2007 Abhishek Singh Rana and Frank Wuerthwein UC San Diego The Open Science Grid Ted Hesselroth Fermilab.

Distributed Application Management Using PLuSH Jeannie Albrecht, Christopher Tuttle, Alex C. Snoeren, and Amin Vahdat UC San Diego CSE {jalbrecht, ctuttle,

On Developing Data Grid Workflows using Storage Resource Broker (SRB) and Kepler Tim H. Wong - UC Davis Efrat Frank - SDSC Bertram Ludäscher - UC Davis.

VGrids Virtual Grids in MiG. Idea Virtual organizations are a central point in the Grid model VO’s share –Resources –Files –Credits A VO is a set of users.

Consorzio COMETA - PI2S2 Project UNIONE EUROPEA SAGE – Storage Accounting for Grid Environments in gLite Fabio Scibilia Consorzio.

National Computational Science National Center for Supercomputing Applications National Computational Science Alliance Setup Package Requirements Jim Basney.

Grid Information Systems. Two grid information problems Two problems  Monitoring  Discovery We can use similar techniques for both.

OSG Public Storage and iRODS

ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)

A. Sim, CRD, L B N L GIN-Data : SRM Island Inter-Op Testing With SRM-TESTER Alex Sim, Vijaya Natarajan Computational Research Division Lawrence Berkeley.

Polish Infrastructure for Supporting Computational Science in the European Research Space QoS provisioning for data-oriented applications in PL-Grid D.

Ted Hesselroth USATLAS Tier 2 and Tier 3 Workshop November 29, 2007 Abhishek Singh Rana and Frank Wuerthwein UC San Diego Installing and Using SRM-dCache.

WP9 – Earth Observation Applications – n° 1 Experiences with Testbed1, plans and objectives for Testbed 2 Testbed retreat th August 2002

Jan Storage Resource Broker Managing Distributed Data in a Grid A discussion of a paper published by a group of researchers at the San Diego Supercomputer.

Maarten Litmaath (CERN), EGEE User Forum, CERN, 2006/03/02 (v3) Use of the SRM interface Use case What is the SRM? –Who develops it? –Is it a standard?

YAN, Tian On behalf of distributed computing group Institute of High Energy Physics (IHEP), CAS, China CHEP-2015, Apr th, OIST, Okinawa.

The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Dataset Caitlin Minteer & Kelly Clynes.

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center National Partnership for Advanced.

Monitoring in EGEE EGEE/SEEGRID Summer School 2006, Budapest Judit Novak, CERN Piotr Nyczyk, CERN Valentin Vidic, CERN/RBI.

Concept: Well-managed provisioning of storage space on OSG sites owned by large communities, for usage by other science communities in OSG. Examples –Providers:

11/30/2007 Overview of operations at CC-IN2P3 Exploitation team Reported by Philippe Olivero.

Movement of MC Data using SAM & SRM Interface Manoj K. Jha (INFN- Bologna) Gabriele Compostella, Antonio Cuomo, Donatella Lucchesi,, Simone Pagan (INFN.

Global Grid Forum GridWorld GGF15 Boston USA October Abhishek Singh Rana and Frank Wuerthwein UC San Diegowww.opensciencegrid.org The Open Science.

A Test Framework for Movement of MC Data using SAM & SRM Interface Manoj K. Jha (INFN- Bologna) Antonio Cuomo, Donatella Lucchesi, Gabriele Compostella,

3-Jul-02D.P.Kelsey, Security1 Security meetings Report to EDG PTB 3 Jul 2002 David Kelsey CLRC/RAL, UK

Ruth Pordes November 2004TeraGrid GIG Site Review1 TeraGrid and Open Science Grid Ruth Pordes, Fermilab representing the Open Science.

Role Based VO Authorization Services Ian Fisk Gabriele Carcassi July 20, 2005.

1 LHCb File Transfer framework N. Brook, Ph. Charpentier, A.Tsaregorodtsev LCG Storage Management Workshop, 6 April 2005, CERN.

VO Privilege Activity. The VO Privilege Project develops and implements fine-grained authorization to grid- enabled resources and services Started Spring.

OSG Abhishek Rana Frank Würthwein UCSD.

EUROPEAN UNION Polish Infrastructure for Supporting Computational Science in the European Research Space Operational Architecture of PL-Grid project M.Radecki,

EGEE is a project funded by the European Union under contract IST Data Management Gaps Krzysztof Nienartowicz Gavin McCance EGEE JRA1 Data.

David Adams ATLAS ATLAS distributed data management David Adams BNL February 22, 2005 Database working group ATLAS software workshop.

Auditing Project Architecture VERY HIGH LEVEL Tanya Levshina.

UC-12 L1 Flight Track over Ground Sites.

Author - Title- Date - n° 1 Partner Logo WP5 Status John Gordon Budapest September 2002.

Ákos FROHNER – DataGrid Security n° 1 Security Group TODO

EGEE-III INFSO-RI Enabling Grids for E-sciencE VO Authorization in EGEE Erwin Laure EGEE Technical Director Joint EGEE and OSG Workshop.

Ted Hesselroth, OSG Site Administrators Meeting, December 13, 2007 Abhishek Singh Rana and Frank Wuerthwein UC San Diego dCache in OSG 1.0 and SRM 2.2.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The.

1 DIRAC Project Status A.Tsaregorodtsev, CPPM-IN2P3-CNRS, Marseille 10 March, DIRAC Developer meeting.

Abhishek Singh Rana and Frank Wuerthwein UC San Diegowww.opensciencegrid.org The Open Science Grid ConsortiumCHEP 2006 Mumbai INDIA February gPLAZMA:

Open Science Grid Consortium Storage on Open Science Grid Placing, Using and Retrieving Data on OSG Resources Abhishek Singh Rana OSG Users Meeting July.

Riccardo Zappi INFN-CNAF SRM Breakout session. February 28, 2012 Ingredients 1. Basic ingredients (Fabric & Conn. level) 2. (Grid) Middleware ingredients.

UCS D OSG Summer School 2011 Overlay systems OSG Summer School An introduction to Overlay systems Also known as Pilot systems by Igor Sfiligoi University.

HPDC Grid Monitoring Workshop June 25, 2007 Grid monitoring from the VO/user perspectives Shava Smallen.

EGEE Data Management Services

Classic Storage Element

EGEE SA2 / TERENA NRENs & Grids joint workshop

Abhishek Singh Rana UC San Diego

AAA from HEP* Perspective

A Replica Location Service

CREAM-CE/HTCondor site

QoS in the Tier1 batch system(LSF)

Job workflow Pre production operations:

Data Management cluster summary

Developing approaches to learning skills Step 1Objective strand: Approaches to learning skill: Step 2Explicit learning experience: Step 3Implicit learning.

GIN-Data : SRM Island Inter-Op Testing

lundi 25 février 2019 FTS configuration

INFNGRID Workshop – Bari, Italy, October 2004

QPTM – Imbalance Trades/Storage Transfers

The LHCb Computing Data Challenge DC06

Presentation transcript:

Towards Storage On-Demand on Petabyte Grids A case for: Quotas and Reserve-ahead, Timed Leases and Contracts Frank Wuerthwein UC San Diego Abhishek Singh Rana UC San Diego

Towards Storage On-Demand on Petabyte Grids A case for: Quotas and Reserve-ahead, Timed Leases and Contracts Frank Wuerthwein UC San Diego Abhishek Singh Rana UC San Diego

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids3 Case List Priority 0 Reserve-ahead: File transfer level (~Implicit Reservation) Priority 1 Bulk Quotas: VO level (~Explicit Partitioning) Fine Quotas: Individual User level (~generally Oversubscribed) Priority 2 Reserve-ahead (from a Quota): User level (~Explicit Reservation) Reserve-ahead (from a Quota): VO level (~Explicit Reservation) Priority 3 Hierarchical and Cumulative Exhaustion of Quotas Priority 4 Storage space accrual and Time Timed Leases & Contracts: VO/User level Timed Leases & Contracts: Grid/Enterprise level

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids4 Quota Based on agreements between a Site and VOs Assigned by a Site Ceiling on storage space to a VO or User Oversubscription possible Timeless configuration Based on needs of Users/Jobs Requested by a User/Job Can only be a subset of Quota Timed lease Reserve-ahead

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids5 Owner/Primary VO Storage at a Site Reserve-ahead: File transfer level Peer VO1Peer VO2 Needed primarily for robust transfers. Independent of quota as file size << quota.

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids6 Storage at a Site Bulk Quotas: VO level Owner/Primary VO

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids7 Storage at a Site Bulk Quotas: VO level Peer VO1Peer VO2 Owner/Primary VO

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids8 Storage at a Site Bulk Quotas: VO level Peer VO1Peer VO2 Owner/Primary VO Need bulk quotas for VOs to co-exist on SE

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids9 Storage at a Site Fine Quotas: User level Peer VO1Peer VO2 Owner/Primary VO

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids10 Owner/Primary VO Storage at a Site Fine Quotas: User level Peer VO1Peer VO2 ?

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids11 Owner/Primary VO Storage at a Site Fine Quotas: User level Peer VO1Peer VO2 Need user level quota For users to co-exist.

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids12 Owner/Primary VO Storage at a Site Reserve-ahead: User level Peer VO1Peer VO2

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids13 Owner/Primary VO Storage at a Site Reserve-ahead: VO level Peer VO1Peer VO2

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids14 Owner/Primary VO Storage at a Site Hierarchical and Cumulative Exhaustion of Quotas Peer VO1Peer VO2 R1 Used = U1 Reserved = R1 Exhausted = Used + Reserved Available = Total – Exhausted U1

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids15 Owner/Primary VO Storage at a Site Hierarchical and Cumulative Exhaustion of Quotas Peer VO1Peer VO2 R1 Used = U1 Reserved = R1 Exhausted = Used + Reserved Available = Total – Exhausted U1

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids16 Owner/Primary VO Storage at a Site Hierarchical and Cumulative Exhaustion of Quotas Peer VO1Peer VO2 R11 Used = U11+U12 Reserved = R11 + R12 Exhausted = Used + Reserved Available = Total – Exhausted U11 R12 U12

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids17 Storage space accrual and Time Leases & Contracts: VO/User level Owner/Primary VO Storage at a Site Peer VO1Peer VO2 R11 Used = U11+ U12 Reserved = R11 + R12 Exhausted = Used + Reserved Available = Total – Exhausted U11 R12 U12 R-peer1 U-peer1 f(t)

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids18 Storage space accrual and Time Leases & Contracts: Grid/Enterprise level f(t)

Frank Wuerthwein & Abhishek Singh Rana, UC San Diego Towards Storage On-Demand on Petabyte Grids19 Storage space accrual and Time Leases & Contracts: Grid/Enterprise level f(t)