Presentation is loading. Please wait.

Presentation is loading. Please wait.

EMI INFSO-RI-261611 EMI proposal for a Storage Accounting Record standard Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the EMI StAR team.

Published byMarlee Priestly Modified over 9 years ago

Similar presentations

Presentation on theme: "EMI INFSO-RI-261611 EMI proposal for a Storage Accounting Record standard Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the EMI StAR team."— Presentation transcript:

1 EMI INFSO-RI-261611 EMI proposal for a Storage Accounting Record standard Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the EMI StAR team

2 EMI INFSO-RI-261611 EMI StAR team – Paul Millar – Ralph Müller-Pfefferkorn – Zsolt Molnár – Riccardo Zappi – Jon Kerr Nilsen Henrik Thostrup Jensen (NDGF) Who? Jon Kerr Nilsen 2

3 EMI INFSO-RI-261611 Why storage accounting? Storage accounting vs. job accounting Processing model Storage accounting work in EMI and OGF StAR proposal Prototype and first feedback Way forward Outline Jon Kerr Nilsen 3

4 EMI INFSO-RI-261611 Why storage accounting? Storage resources are shared between organizations – more than one owner Developing a storage infrastructure – We need to know how much storage space is used, by which group/user, on which storage media – To know where to put the money when increasing the storage space – To know who to ask for the money to increase the storage space Basis for billing – Storage is expensive – Some non-academic resource owners may not like to give it away for free Jon Kerr Nilsen 4

5 EMI INFSO-RI-261611 Accounting Accounting is the recording and summarizing of the consumption of a resource by an individual user or a group of users Typically used to find out who uses how much of a set of resources Typically not used to find out how individual resource components are used Jon Kerr Nilsen 5

6 EMI INFSO-RI-261611 Storage vs. job accounting Job accounting Job accounting is the recording and summarizing of the consumption of computing resources Recording wall-time, cpu-time, etc. – strictly increasing values. When a job finishes it stays finished Typically measured per job – information readily available from local batch system Standardized through UR1.0UR1.0 Storage accounting Storage accounting is the recording and summarizing of the consumption of a storage resource Storage usage varies in time – recorded usage is only relevant in a given time frame Data is rarely in finished state and can live for a long time Typically recorded through (in)frequent usage snapshots – high fluctuation rate No standardized way to record storage usage in a distributed environment Jon Kerr Nilsen 6

7 EMI INFSO-RI-261611 Storage vs. job summary Jon Kerr Nilsen 7 Storage records must be treated different than compute records wrt. aggregation Storage records need special treatment – Only valid between measurement time and expiration time – Can be invalidated if newer record is registered for same resource – Nontrivial processing model

8 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 8

9 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 9

10 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 10

11 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 11

12 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 12

13 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 13

14 EMI INFSO-RI-261611 Processing model Records can overlap in time – cannot simply sum up to identify total consumption Must ask for a certain point in time and sum up the most recent, still-valid records To get usage per time, you must choose a set of timestamps and repeat process per time stamp – numerical integration Jon Kerr Nilsen 14

15 EMI INFSO-RI-261611 From EMI Description of Work: – EMI will address known UR issues and extend accounting records to include VO-aware storage usage accounting – The refined and extended standard UR format, service interface and data transport protocol will be implemented in gLite, ARC, dCache and UNICORE – EMI will thus contribute to the definition of UR 2.0 of OGF Storage accounting in EMI Jon Kerr Nilsen 15

16 EMI INFSO-RI-261611 “Next Generation Usage Records” OGF session at OGF31 – Discussed the way forward for UR – A properly defined storage record and an improved compute record are good steps towards UR2.0 – Lessons learned in storage records can be ported back into compute records (e.g. long-lived services) – EMI will work actively to have standard ready before end of EMI Being a highly active participant, EMI will co-chair the OGF UR-WG Storage accounting in OGF Jon Kerr Nilsen 16

17 EMI INFSO-RI-261611 EMI is finalizing a proposal for a storage accounting record (StAR) Contacted and received input from organizations and potential user groups – NDGF, OGF, EGI, OSG, INFN, CMS, ATLAS, … Current draft: StAR-EMI-tech-doc-v7.pdfStAR-EMI-tech-doc-v7.pdf Final version to be proposed to OGF as a standard Will be implemented by EMI storage providers StAR Jon Kerr Nilsen 17

18 EMI INFSO-RI-261611 To be able to arrive to a useful record in record time we need to limit the scope Many interesting storage-related features to record – I/O (like in Amazon S3) – Metadata access (stat,ls) – Available space – File name –... Scope of StAR is limited to consumption of storage space What StAR is not Jon Kerr Nilsen 18

19 EMI INFSO-RI-261611 Resource – Fields describing the system the resource was consumed on –StorageSystem, StorageShare Resource consumption – How much of the described resource has been used –ResourceCapacityUsed, LogicalCapacityUsed Consumption details – Fields describing what is consumed –StorageMedia, StorageClass, DirectoryPath, … Identity – Describes the person or group accountable for the consumption –SubjectIdentity block ( UserIdentity, Group, GroupAttribute, …) Record identity and duration StAR structure Jon Kerr Nilsen 19

20 EMI INFSO-RI-261611 Local example Jon Kerr Nilsen 20 <sr:StorageUsageRecord xmlns:sr="http://eu-emi.eu/namespaces/2011/02/storagerecord"> <sr:RecordIdentity sr:createTime="2010-11-09T09:06:52Z" sr:recordId="host.example.org/sr/87912469269276"/> host.example.org johndoe ssd 42 2010-10-11T09:31:40Z PT3600S 913617

21 EMI INFSO-RI-261611 <sr:StorageUsageRecord xmlns:sr="http://eu-emi.eu/namespaces/2011/02/storagerecord"> <sr:RecordIdentity sr:createTime="2010-11-09T09:06:52Z" sr:recordId="host.example.org/sr/87912469269276"/> host.example.org binarydataproject.example.org ukusers tape 42 2010-10-11T09:31:40Z PT3600S 913617 Grid example Jon Kerr Nilsen 21

22 EMI INFSO-RI-261611 NDGF already has working prototype – Using SGAS accounting service and data from NDGF dCache Records inserted in SGAS, parsed and put into PostgreSQL DB Querying data model with measure points and valid duration doable for anyone comfortable with putting joins in queries Hackish record generation (simple SQL into dCache DB) Still limited graphing possibilities Prototype implemented Jon Kerr Nilsen 22

23 EMI INFSO-RI-261611 Prototype implemented Jon Kerr Nilsen 23

24 EMI INFSO-RI-261611 Prototype implemented Jon Kerr Nilsen 24 Usage (TB)

25 EMI INFSO-RI-261611 Information available from records has limitations (we knew that) Good for measuring actual usage Good for measuring logical usage/VO Logical usage/site/VO is more tricky Measuring and aggregating logical usage is not straight-forward Free space could be useful as well but this is tricky for same reasons as logical usage First experiences from NDGF Jon Kerr Nilsen 25

26 EMI INFSO-RI-261611 The NDGF dCache system is composed of seven sites, each storing a lot of data NDGF replicates all data which has been accessed within the last 30 days to two sites NDGF needs usage per VO and per site Local usage issue Jon Kerr Nilsen 26

27 EMI INFSO-RI-261611 Local usage issue If a file is stored on site A and site B for a VO C, then what exactly is reported as logical usage for any of the sites for that VO? Cannot say from record if file* is a replica or a logical file May need hierarchy in the record to get clear answer – More complex, less intuitive and more error prone record – Different storage systems may have different hierarchies – generic data model will be REALLY complicated We probably don’t want to go there Jon Kerr Nilsen 27

28 EMI INFSO-RI-261611 Definition of StAR is a first step to establish common storage accounting record A step towards a common usage record (UR2.0) Next jump (two steps in one go) – Propose StAR to OGF to start standardization process Will take time Will include further discussions May lead to changes in StAR EMI will take active part in this process Draft has been made available through UR-WG mailing list Will be made available in SourceForge – Implement StAR record into EMI storage middleware EMI ends in two years Want to gain user experience before end of EMI Work already started outside EMI Comments and input more than welcome throughout the process! Way forward Jon Kerr Nilsen 28

29 EMI INFSO-RI-261611 THANK YOU! Contact: emi-jra1-data-sar@eu-emi.euemi-jra1-data-sar@eu-emi.eu Web page: http://twiki.cern.ch/twiki/bin/view/EMI/StorageAccountinghttp://twiki.cern.ch/twiki/bin/view/EMI/StorageAccounting Current StAR draft: http://twiki.cern.ch/twiki/pub/EMI/StorageAccounting/StAR-EMI- tech-doc-v7.pdf http://twiki.cern.ch/twiki/pub/EMI/StorageAccounting/StAR-EMI- tech-doc-v7.dochttp://twiki.cern.ch/twiki/pub/EMI/StorageAccounting/StAR-EMI- tech-doc-v7.pdfhttp://twiki.cern.ch/twiki/pub/EMI/StorageAccounting/StAR-EMI- tech-doc-v7.doc Jon Kerr Nilsen 29

Download ppt "EMI INFSO-RI-261611 EMI proposal for a Storage Accounting Record standard Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the EMI StAR team."

Similar presentations

Polish Infrastructure for Supporting Computational Science in the European Research Space EUROPEAN UNION Services and Operations in Polish NGI M. Radecki,

Polish Infrastructure for Supporting Computational Science in the European Research Space EUROPEAN UNION Services and Operations in Polish NGI M. Radecki,
EMI INFSO-RI-261611 Towards URF 2.0 From a storage perspective Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the EMI StAR team.

EMI INFSO-RI Towards URF 2.0 From a storage perspective Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the EMI StAR team.
Copyright © 2001 Bolton Institute Faculty of Technology - 34.4 Multimedia Integration and Applications Lecture 7: Prototype Review Damien Markey.

Copyright © 2001 Bolton Institute Faculty of Technology Multimedia Integration and Applications Lecture 7: Prototype Review Damien Markey.
NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.

NextGRID & OGSA Data Architectures: Example Scenarios Stephen Davey, NeSC, UK ISSGC06 Summer School, Ischia, Italy 12 th July 2006.
Lecture 7: Prototype Review Damien Markey. Lecture 6: Prototype Review What makes a prototype successful Why a prototype is never a failure Review criteria.

Lecture 7: Prototype Review Damien Markey. Lecture 6: Prototype Review What makes a prototype successful Why a prototype is never a failure Review criteria.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.

LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.
Storage Accounting John Gordon, STFC GDB June 2012.

Storage Accounting John Gordon, STFC GDB June 2012.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) Grid Engine Riccardo Rotondo

The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) Grid Engine Riccardo Rotondo
ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, 17.07.12, JINR, Dubna.

ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.
Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 GPGPU Accounting John Gordon STFC 09/04/2013 EGI CF – Accounting and Billing1.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI GPGPU Accounting John Gordon STFC 09/04/2013 EGI CF – Accounting and Billing1.
INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.

INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
EGEE-III INFSO-RI-222667 Enabling Grids for E-sciencE www.eu-egee.org EGEE and gLite are registered trademarks Felix Ehm CERN IT-GD EGEE 2008 GLUE 2.0.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Felix Ehm CERN IT-GD EGEE 2008 GLUE 2.0.
Kerberos Named after a mythological three-headed dog that guards the underworld of Hades, Kerberos is a network authentication protocol that was designed.

Kerberos Named after a mythological three-headed dog that guards the underworld of Hades, Kerberos is a network authentication protocol that was designed.
Lecture2: Database Environment Prepared by L. Nouf Almujally & Aisha AlArfaj 1 Ref. Chapter2 College of Computer and Information Sciences - Information.

Lecture2: Database Environment Prepared by L. Nouf Almujally & Aisha AlArfaj 1 Ref. Chapter2 College of Computer and Information Sciences - Information.
Web: Minimal Metadata for Data Services Through DIALOGUE Neil Chue Hong AHM2007.

Web: Minimal Metadata for Data Services Through DIALOGUE Neil Chue Hong AHM2007.
ATLAS Detector Description Database Vakho Tsulaia University of Pittsburgh 3D workshop, CERN 14-Dec-2004.

ATLAS Detector Description Database Vakho Tsulaia University of Pittsburgh 3D workshop, CERN 14-Dec-2004.
240-Current Research Easily Extensible Systems, Octave, Input Formats, SOA.

240-Current Research Easily Extensible Systems, Octave, Input Formats, SOA.
User Interface Design & Usability for the Web Card Sorting You should now have a basic idea as to content requirements, functional requirements and user.

User Interface Design & Usability for the Web Card Sorting You should now have a basic idea as to content requirements, functional requirements and user.
1 User Analysis Workgroup Discussion  Understand and document analysis models  Best in a way that allows to compare them easily.

1 User Analysis Workgroup Discussion  Understand and document analysis models  Best in a way that allows to compare them easily.

Similar presentations

Ads by Google