LCG Storage Accounting John Gordon CCLRC – RAL LCG Grid Deployment Board September 2006

LCG John Gordon – GDB 6 September 2006 Since July  Following my update session at the July GDB, There has been some progress but only limited deployment

LCG John Gordon – GDB 6 September 2006 Status  A Storage Accounting schema has been defined  A single storage accounting sensor at Edinburgh queries the BDII once per day to gather information on current storage usage from all UK sites  This sensor populates an R-GMA archiver with the gathered information  StorageRecords table  The information is aggregated per VO/Site/Month as in APEL  Value stored is average over the number of measurements  Sum(measurements)/N(measurements)  APEL-like visualisation interface has been developed  Grids, countries, regions  Only tables for country and region implemented so far 

LCG John Gordon – GDB 6 September 2006 StorageRecords Schema rgma> describe StorageRecords | Column name | Type | Primary key | Can be NULL | | RecordIdentity | VARCHAR(255) | Yes | No | | ResourceIdentity | VARCHAR(255) | No | Yes | | Grid | VARCHAR(50) | No | Yes | | ExecutingSite | VARCHAR(50) | No | Yes | | VO | VARCHAR(50) | No | Yes | | SpaceUsed | INTEGER | No | Yes | | SpaceAvailable | INTEGER | No | Yes | | Total | INTEGER | No | Yes | | Unit | VARCHAR(50) | No | Yes | | SEArchitecture | VARCHAR(50) | No | Yes | | Type | VARCHAR(50) | No | Yes | | EventDate | DATE | No | No | | EventTime | TIME | No | No | | MeasurementDate | DATE | No | No | | MeasurementTime | TIME | No | No |

LCG John Gordon – GDB 6 September 2006 StorageRecords Examples | Column name | Type | Pkey | NULL?| Example | | RecordIdentity | VARCHAR(255) | Yes | No | dcache.gridpp.rl.ac.uk atlas :00:17 | | ResourceIdentity | VARCHAR(255) | No | Yes | dcache.gridpp.rl.ac.uk | | Grid | VARCHAR(50) | No | Yes | EGEE | | ExecutingSite | VARCHAR(50) | No | Yes | RAL-LCG2 | | VO | VARCHAR(50) | No | Yes | atlas | | SpaceUsed | INTEGER | No | Yes | | | SpaceAvailable | INTEGER | No | Yes | | | Total | INTEGER | No | Yes | | | Unit | VARCHAR(50) | No | Yes | MB | | SEArchitecture | VARCHAR(50) | No | Yes | multidisk | | Type | VARCHAR(50) | No | Yes | permanent | | EventDate | DATE | No | No | | | EventTime | TIME | No | No | 20:00:17 | | MeasurementDate | DATE | No | No | | | MeasurementTime | TIME | No | No | 19:00:17 |

LCG John Gordon – GDB 6 September 2006 Information Providers  The Castor GIP is under test  Local BDII so not included here  A replacement DPM GIP is under development  A dCache GIP has been developed  Used by RAL T1 and T2  Not sure if it will be officially released.  Chimera later this year will allow better info  There is a GIP for Classic SE  but propose not to account classic SEs as Grid storage should be SRM-aware  Deployment of these has been later than planned

LCG John Gordon – GDB 6 September 2006 dCache (1) | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total | | UKI-SOUTHGRID-RALPP | cdf | 0 | | | | UKI-SOUTHGRID-RALPP | cms | | | | | UKI-SOUTHGRID-RALPP | esr | 0 | | | | UKI-SOUTHGRID-RALPP | ilc | 53 | | | | UKI-SOUTHGRID-RALPP | ops | 0 | | | | UKI-SOUTHGRID-RALPP | t2k | 0 | | | | UKI-SOUTHGRID-RALPP | hone | | | | | UKI-SOUTHGRID-RALPP | lhcb | 0 | | | | UKI-SOUTHGRID-RALPP | na48 | 0 | | | | UKI-SOUTHGRID-RALPP | zeus | 3535 | | | | UKI-SOUTHGRID-RALPP | alice | 0 | | | | UKI-SOUTHGRID-RALPP | atlas | | | | | UKI-SOUTHGRID-RALPP | babar | | | | | UKI-SOUTHGRID-RALPP | cedar | 0 | | | | UKI-SOUTHGRID-RALPP | dteam | | | | | UKI-SOUTHGRID-RALPP | dzero | 0 | | | | UKI-SOUTHGRID-RALPP | magic | 0 | | | | UKI-SOUTHGRID-RALPP | minos | 0 | | | | UKI-SOUTHGRID-RALPP | pheno | 2120 | | | | UKI-SOUTHGRID-RALPP | ralpp | 0 | | | | UKI-SOUTHGRID-RALPP | biomed | 0 | | | | UKI-SOUTHGRID-RALPP | fusion | 0 | | | | UKI-SOUTHGRID-RALPP | geant4 | 0 | | | | UKI-SOUTHGRID-RALPP | planck | 0 | | |

LCG John Gordon – GDB 6 September 2006 dCache (2) | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total | | UKI-LT2-IC-HEP | cms | | | | | UKI-LT2-IC-HEP | dteam | | | | | UKI-LT2-IC-HEP | atlas | | | | | UKI-LT2-IC-HEP | alice | | | | | UKI-LT2-IC-HEP | lhcb | | | | | UKI-LT2-IC-HEP | sixt | | | | | UKI-LT2-IC-HEP | zeus | | | | | UKI-LT2-IC-HEP | na48 | | | | | UKI-LT2-IC-HEP | calice | | | | | UKI-LT2-IC-HEP | ilc | | | | | UKI-LT2-IC-HEP | biomed | | | | | UKI-LT2-IC-HEP | babar | | | | | UKI-LT2-IC-HEP | dzero | | | | | UKI-LT2-IC-HEP | hone | | | | | UKI-LT2-IC-HEP | pheno | | | | | UKI-LT2-IC-HEP | t2k | | | | | UKI-LT2-IC-HEP | geant4 | | | | | UKI-LT2-IC-HEP | gin | | | | | UKI-LT2-IC-HEP | ltwo | 0 | 0 | 0 | | UKI-LT2-IC-HEP | ops | | | |

LCG John Gordon – GDB 6 September 2006 Multiple SRM | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Type | Total | | RAL-LCG2 | cdf | 0 | 0 | permanent | 0 | | RAL-LCG2 | cms | | | permanent | | | RAL-LCG2 | esr | 15 | | permanent | | | RAL-LCG2 | ilc | 498 | | permanent | | | RAL-LCG2 | ops | 0 | | permanent | | | RAL-LCG2 | t2k | 0 | | permanent | | | RAL-LCG2 | hone | | | permanent | | | RAL-LCG2 | lhcb | | | permanent | | | RAL-LCG2 | zeus | | | permanent | | | RAL-LCG2 | alice | 0 | | permanent | | | RAL-LCG2 | atlas | | | permanent | | | RAL-LCG2 | babar | | | permanent | | | RAL-LCG2 | cedar | 0 | | permanent | | | RAL-LCG2 | dteam | | | permanent | | | RAL-LCG2 | dzero | 0 | 0 | permanent | 0 | | RAL-LCG2 | magic | 0 | | permanent | | | RAL-LCG2 | minos | 182 | | permanent | | | RAL-LCG2 | pheno | | | permanent | | | RAL-LCG2 | biomed | 581 | | permanent | | | RAL-LCG2 | cms | 0 | | permanent | | | RAL-LCG2 | fusion | 0 | | permanent | | | RAL-LCG2 | geant4 | 0 | | permanent | | | RAL-LCG2 | lhcb | 0 | | permanent | | | RAL-LCG2 | babar | 0 | | permanent | | | RAL-LCG2 | cms | | 0 | permanent | | | RAL-LCG2 | dteam | 0 | | permanent | | | RAL-LCG2 | ops | | 0 | permanent | | | RAL-LCG2 | lhcb | | 0 | permanent | | | RAL-LCG2 | alice | | 0 | permanent | | | RAL-LCG2 | atlas | | 0 | permanent | | | RAL-LCG2 | dteam | | 0 | permanent | | | RAL-LCG2 | minos | | 0 | permanent | | Tape SRM

LCG John Gordon – GDB 6 September 2006

LCG John Gordon – GDB 6 September 2006

LCG John Gordon – GDB 6 September 2006

LCG John Gordon – GDB 6 September 2006 Problems  Old information providers publish info per Storage Area, not per VO  Cannot add up all SA/VO to get site storage  Published numbers GB not NB  Just changing to TB  Tape not well understood

LCG John Gordon – GDB 6 September 2006 Old dCache GIP | ExecutingSite | VO | SpaceUsed | SpaceAvailable | Total | | UKI-SOUTHGRID-OX-HEP | atlas | | | | | UKI-SOUTHGRID-OX-HEP | alice | | | | | UKI-SOUTHGRID-OX-HEP | lhcb | | | | | UKI-SOUTHGRID-OX-HEP | cms | | | | | UKI-SOUTHGRID-OX-HEP | dteam | | | | | UKI-SOUTHGRID-OX-HEP | sixt | | | | | UKI-SOUTHGRID-OX-HEP | babar | | | | | UKI-SOUTHGRID-OX-HEP | zeus | | | | | UKI-SOUTHGRID-OX-HEP | cdf | | | | | UKI-SOUTHGRID-OX-HEP | biomed | | | | | UKI-SOUTHGRID-OX-HEP | pheno | | | | | UKI-SOUTHGRID-OX-HEP | ilc | | | | | UKI-SOUTHGRID-OX-HEP | hone | | | | | UKI-SOUTHGRID-OX-HEP | dzero | | | | | UKI-SOUTHGRID-OX-HEP | oxg | | | | | UKI-SOUTHGRID-OX-HEP | ops | | | |

LCG John Gordon – GDB 6 September 2006 Issues (1)  GLUE publishes space used and space available.  For MoUs etc we are interested in space allocated  There have been discussions about changes to GLUE for Storage (at the SRM workshop?)  or derive allocated=used+available at client or server  Haven’t solved the double counting of available space in a StorageArea shared between VOs  But we are not really interested in available space for accounting  But this is important if we use it as above  Don’t have an SRM yet for the storage on WNs  Reporting tape needs more thought

LCG John Gordon – GDB 6 September 2006 Issues (2)  We can’t do user-level accounting via the BDII  We can probably support a FQAN owning a GlueSA if GLUE allows this in future  We can support future storagetypes as long as they are published in GLUE  But not differentiate between raw data and ESD (eg)  Current storage information in BDII isn’t that reliable  Start a programme of checking

LCG John Gordon – GDB 6 September 2006 Status  So if we currently have  a schema, a repository, collection, and visualisation  Then we have a framework for storage accounting  …but little good data.

LCG John Gordon – GDB 6 September 2006 Plan  Continue developing the prototype portal  Graphical views  Storage(time, VO, site),  used, allocated, available  Populate it with data from sites running the new GIPs  Check the correctness of the data published  Change to TB  Ask for comments and feedback

LCG John Gordon – GDB 6 September 2006 Can I have comments and feedback, please?