1. LCG Storage Accounting John Gordon CCLRC – RAL LCG Grid Deployment Board September 2006

2. LCG John Gordon – GDB 6 September 2006 Since July  Following my update session at the July GDB, There has been some progress but only limited deployment

3. LCG John Gordon – GDB 6 September 2006 Status  A Storage Accounting schema has been defined  A single storage accounting sensor at Edinburgh queries the BDII once per day to gather information on current storage usage from all UK sites  This sensor populates an R-GMA archiver with the gathered information  StorageRecords table  The information is aggregated per VO/Site/Month as in APEL  Value stored is average over the number of measurements  Sum(measurements)/N(measurements)  APEL-like visualisation interface has been developed  Grids, countries, regions  Only tables for country and region implemented so far  http://goc02.grid-support.ac.uk/accountingDisplay/

4. LCG John Gordon – GDB 6 September 2006 StorageRecords Schema rgma> describe StorageRecords +------------------+--------------+-------------+-------------+ | Column name | Type | Primary key | Can be NULL | +------------------+--------------+-------------+-------------+ | RecordIdentity | VARCHAR(255) | Yes | No | | ResourceIdentity | VARCHAR(255) | No | Yes | | Grid | VARCHAR(50) | No | Yes | | ExecutingSite | VARCHAR(50) | No | Yes | | VO | VARCHAR(50) | No | Yes | | SpaceUsed | INTEGER | No | Yes | | SpaceAvailable | INTEGER | No | Yes | | Total | INTEGER | No | Yes | | Unit | VARCHAR(50) | No | Yes | | SEArchitecture | VARCHAR(50) | No | Yes | | Type | VARCHAR(50) | No | Yes | | EventDate | DATE | No | No | | EventTime | TIME | No | No | | MeasurementDate | DATE | No | No | | MeasurementTime | TIME | No | No | +------------------+--------------+-------------+-------------+

5. LCG John Gordon – GDB 6 September 2006 StorageRecords Examples +------------------+--------------+-------------+----------------------------------------+ | Column name | Type | Pkey | NULL?| Example | +------------------+--------------+-------------+----------------------------------------+ | RecordIdentity | VARCHAR(255) | Yes | No | dcache.gridpp.rl.ac.uk atlas 2006-08-13 20:00:17 | | ResourceIdentity | VARCHAR(255) | No | Yes | dcache.gridpp.rl.ac.uk | | Grid | VARCHAR(50) | No | Yes | EGEE | | ExecutingSite | VARCHAR(50) | No | Yes | RAL-LCG2 | | VO | VARCHAR(50) | No | Yes | atlas | | SpaceUsed | INTEGER | No | Yes | 9717308 | | SpaceAvailable | INTEGER | No | Yes | 687036 | | Total | INTEGER | No | Yes | 10404344 | | Unit | VARCHAR(50) | No | Yes | MB | | SEArchitecture | VARCHAR(50) | No | Yes | multidisk | | Type | VARCHAR(50) | No | Yes | permanent | | EventDate | DATE | No | No | 2006-08-13 | | EventTime | TIME | No | No | 20:00:17 | | MeasurementDate | DATE | No | No | 2006-08-13 | | MeasurementTime | TIME | No | No | 19:00:17 | +------------------+--------------+-------------+-----------------------------------------+

6. LCG John Gordon – GDB 6 September 2006 Information Providers  The Castor GIP is under test  Local BDII so not included here  A replacement DPM GIP is under development  A dCache GIP has been developed  Used by RAL T1 and T2  Not sure if it will be officially released.  Chimera later this year will allow better info  There is a GIP for Classic SE  but propose not to account classic SEs as Grid storage should be SRM-aware  Deployment of these has been later than planned

7. LCG John Gordon – GDB 6 September 2006 dCache (1) +---------------------+--------+-----------+----------------+---------+ | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total | +---------------------+--------+-----------+----------------+---------+ | UKI-SOUTHGRID-RALPP | cdf | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | cms | 166576 | 1167395 | 1333971 | | UKI-SOUTHGRID-RALPP | esr | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | ilc | 53 | 33296 | 33349 | | UKI-SOUTHGRID-RALPP | ops | 0 | 16674 | 16674 | | UKI-SOUTHGRID-RALPP | t2k | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | hone | 12759 | 270709 | 283468 | | UKI-SOUTHGRID-RALPP | lhcb | 0 | 1333971 | 1333971 | | UKI-SOUTHGRID-RALPP | na48 | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | zeus | 3535 | 279933 | 283468 | | UKI-SOUTHGRID-RALPP | alice | 0 | 16674 | 16674 | | UKI-SOUTHGRID-RALPP | atlas | 523340 | 810631 | 1333971 | | UKI-SOUTHGRID-RALPP | babar | 211547 | 1122424 | 1333971 | | UKI-SOUTHGRID-RALPP | cedar | 0 | 16674 | 16674 | | UKI-SOUTHGRID-RALPP | dteam | 173520 | -140172 | 33348 | | UKI-SOUTHGRID-RALPP | dzero | 0 | 266794 | 266794 | | UKI-SOUTHGRID-RALPP | magic | 0 | 16674 | 16674 | | UKI-SOUTHGRID-RALPP | minos | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | pheno | 2120 | 31228 | 33348 | | UKI-SOUTHGRID-RALPP | ralpp | 0 | 66698 | 66698 | | UKI-SOUTHGRID-RALPP | biomed | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | fusion | 0 | 33349 | 33349 | | UKI-SOUTHGRID-RALPP | geant4 | 0 | 16674 | 16674 | | UKI-SOUTHGRID-RALPP | planck | 0 | 16674 | 16674 | +---------------------+--------+-----------+----------------+---------+

8. LCG John Gordon – GDB 6 September 2006 dCache (2) +----------------+--------+-----------+----------------+---------+ | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total | +----------------+--------+-----------+----------------+---------+ | UKI-LT2-IC-HEP | cms | 4177738 | 3568092 | 7745830 | | UKI-LT2-IC-HEP | dteam | 1560864 | 1197938 | 2758802 | | UKI-LT2-IC-HEP | atlas | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | alice | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | lhcb | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | sixt | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | zeus | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | na48 | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | calice | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | ilc | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | biomed | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | babar | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | dzero | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | hone | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | pheno | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | t2k | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | geant4 | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | gin | 108198 | 116196 | 224394 | | UKI-LT2-IC-HEP | ltwo | 0 | 0 | 0 | | UKI-LT2-IC-HEP | ops | 108198 | 116196 | 224394 | +----------------+--------+-----------+----------------+---------+

9. LCG John Gordon – GDB 6 September 2006 Multiple SRM +---------------+--------+-----------+----------------+-----------+----------+ | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Type | Total | +---------------+--------+-----------+----------------+-----------+----------+ | RAL-LCG2 | cdf | 0 | 0 | permanent | 0 | | RAL-LCG2 | cms | 12993009 | 5301774 | permanent | 18294783 | | RAL-LCG2 | esr | 15 | 102386 | permanent | 102401 | | RAL-LCG2 | ilc | 498 | 101903 | permanent | 102401 | | RAL-LCG2 | ops | 0 | 2050039 | permanent | 2050039 | | RAL-LCG2 | t2k | 0 | 102401 | permanent | 102401 | | RAL-LCG2 | hone | 312936 | 732185 | permanent | 1045121 | | RAL-LCG2 | lhcb | 6482263 | 7302824 | permanent | 13785087 | | RAL-LCG2 | zeus | 48913 | 53488 | permanent | 102401 | | RAL-LCG2 | alice | 0 | 2012159 | permanent | 2012159 | | RAL-LCG2 | atlas | 11457691 | 1342813 | permanent | 12800504 | | RAL-LCG2 | babar | 30284 | 72117 | permanent | 102401 | | RAL-LCG2 | cedar | 0 | 102401 | permanent | 102401 | | RAL-LCG2 | dteam | 128390 | 1921649 | permanent | 2050039 | | RAL-LCG2 | dzero | 0 | 0 | permanent | 0 | | RAL-LCG2 | magic | 0 | 102401 | permanent | 102401 | | RAL-LCG2 | minos | 182 | 2114379 | permanent | 2114561 | | RAL-LCG2 | pheno | 29713 | 72688 | permanent | 102401 | | RAL-LCG2 | biomed | 581 | 101820 | permanent | 102401 | | RAL-LCG2 | cms | 0 | 19531 | permanent | 19531 | | RAL-LCG2 | fusion | 0 | 102401 | permanent | 102401 | | RAL-LCG2 | geant4 | 0 | 102401 | permanent | 102401 | | RAL-LCG2 | lhcb | 0 | 19531 | permanent | 19531 | | RAL-LCG2 | babar | 0 | 19531 | permanent | 19531 | | RAL-LCG2 | cms | 41943040 | 0 | permanent | 41943040 | | RAL-LCG2 | dteam | 0 | 19531 | permanent | 19531 | | RAL-LCG2 | ops | 524288 | 0 | permanent | 524288 | | RAL-LCG2 | lhcb | 20971520 | 0 | permanent | 20971520 | | RAL-LCG2 | alice | 2097152 | 0 | permanent | 2097152 | | RAL-LCG2 | atlas | 8388608 | 0 | permanent | 8388608 | | RAL-LCG2 | dteam | 524288 | 0 | permanent | 524288 | | RAL-LCG2 | minos | 10485760 | 0 | permanent | 10485760 | +---------------+--------+-----------+----------------+-----------+----------+ Tape SRM

10. LCG John Gordon – GDB 6 September 2006

11. LCG John Gordon – GDB 6 September 2006

12. LCG John Gordon – GDB 6 September 2006

13. LCG John Gordon – GDB 6 September 2006 Problems  Old information providers publish info per Storage Area, not per VO  Cannot add up all SA/VO to get site storage  Published numbers GB not NB  Just changing to TB  Tape not well understood

14. LCG John Gordon – GDB 6 September 2006 Old dCache GIP +----------------------+--------+-----------+----------------+---------+ | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total | +----------------------+--------+-----------+----------------+---------+ | UKI-SOUTHGRID-OX-HEP | atlas | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | alice | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | lhcb | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | cms | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | dteam | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | sixt | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | babar | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | zeus | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | cdf | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | biomed | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | pheno | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | ilc | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | hone | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | dzero | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | oxg | 781250 | 2304687 | 3085937 | | UKI-SOUTHGRID-OX-HEP | ops | 781250 | 2304687 | 3085937 | +----------------------+--------+-----------+----------------+---------+

15. LCG John Gordon – GDB 6 September 2006 Issues (1)  GLUE publishes space used and space available.  For MoUs etc we are interested in space allocated  There have been discussions about changes to GLUE for Storage (at the SRM workshop?)  or derive allocated=used+available at client or server  Haven’t solved the double counting of available space in a StorageArea shared between VOs  But we are not really interested in available space for accounting  But this is important if we use it as above  Don’t have an SRM yet for the storage on WNs  Reporting tape needs more thought

16. LCG John Gordon – GDB 6 September 2006 Issues (2)  We can’t do user-level accounting via the BDII  We can probably support a FQAN owning a GlueSA if GLUE allows this in future  We can support future storagetypes as long as they are published in GLUE  But not differentiate between raw data and ESD (eg)  Current storage information in BDII isn’t that reliable  Start a programme of checking

17. LCG John Gordon – GDB 6 September 2006 Status  So if we currently have  a schema, a repository, collection, and visualisation  Then we have a framework for storage accounting  …but little good data.

18. LCG John Gordon – GDB 6 September 2006 Plan  Continue developing the prototype portal  Graphical views  Storage(time, VO, site),  used, allocated, available  Populate it with data from sites running the new GIPs  Check the correctness of the data published  Change to TB  Ask for comments and feedback

19. LCG John Gordon – GDB 6 September 2006 Can I have comments and feedback, please?