EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Storage Accounting John Gordon, STFC OMB August 2013

EGI-InSPIRE RI Outline History Timeline Current state Evidence of testing Current issues Open Issues Portal Views Reports?

EGI-InSPIRE RI EMI StAR EMI recognised the requirement for storage accounting EMI created a proposal for a storage accounting record (StAR) ‘Finalised’ in June 2011 Revised Autumn 2012 to include the WLCG requirements I described at the GDB last June. Published version: Been through public comment in OGF and just resubmitted for approval implemented by EMI storage providers in EMI 3 3 EMI INFSO-RI

EGI-InSPIRE RI Original EMI StAR timeline May 2011: StAR definition agreement Oct. 2011: Each SE to provide an implementation plan May 2012: Each SE to implement accounting sensors June 2012: Accounting record publishers (APEL and/or DGAS) to publish storage records December 2012: Each SE to test and deploy their accounting sensors (for EMI 3 release) January-April 2013: Bug-fixing in EMI 3 RC’s April 2013: Accounting sensors released in EMI 3 4 EMI INFSO-RI

EGI-InSPIRE RI Actual EMI Timeline February 2013 – first test storage records appeared from dcache and dpm February-March 2013 checking and revising. 11 March – Release of EMI 3.0 Any changes required should be available in update 1.

EGI-InSPIRE RI Status of StAR implementations Dcache – published from a test system. Dpm - published from a test system. StoRM – did not make EMI 3.0 cutoff - ??? NGI-Italy is publishing for all INFN sites using data from the BDII. EMI produced tests to verify that StAR records agree with standard –But that doesn’t address requirements on semantics. StAR publishing uses the SSM2 infrastructure asthe EMI-3 cpu client, and some other publishers.

EGI-InSPIRE RI

EGI-InSPIRE RI Issues from Testing No GOCDB Sitename – may be artifact of test sites. No ResourceCapacityAllocated – just Used All fixed in EMI-3 production releases of DPM and dcache

EGI-InSPIRE RI Other Issues Frequency of publishing –Sites should publish once per day. They can collect data as frequently as they wish and publish daily averages if they wish. –Repository could average daily records into months, like CPU but will start by passing everything to the portal. What is to be viewed in the portal?

EGI-InSPIRE RI Accounting Portal Summarise data by Site, VO, Month (Average, min, max) Views at any point in tree of –Tier1, Tier2, Country, NGI View Average Storage for a time period and VO selection, for any two of (site/country/Tier2), VO, Date E.g. Country(date), VO(Site), VO(date), Site(date) (similar to CPU Portal Views.) WARNING! Examples do not show real data.

EGI-InSPIRE RI Portal View for Storage

EGI-InSPIRE RI Storage per country per month

EGI-InSPIRE RI Storage used by LHC VOs at French sites.

EGI-InSPIRE RI Storage for Alice in France Allocated and used.

EGI-InSPIRE RI

EGI-InSPIRE RI

EGI-InSPIRE RI The views shown were broadly accepted by WLCG. Does EGI have anything else to be added? Portal team working on a prototype.

EGI-InSPIRE RI Reports When storage accounting is deployed at most sites then:- –The Tier1 monthly report can be autofilled with storage data. –The Tier2 report can be extended to cover storage as well as cpu. –EGI Quarterly Metrics can be pre-filled with storage data

EGI-InSPIRE RI Next Steps Verify that storage providers are publishing consistent records. Start publishing real data from a variety of sites Extend BDII data to other countries/storage solutions Wider coverage of storage solutions –StoRM, OSG, xrootd? Finalise what is shown in portal Implement Portal view and reports. Investigate merged reports (CPU+Storage)

EGI-InSPIRE RI Conclusions Storage Accounting is making (slow) progress. Are there suggestions of other views or reports? Questions?

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Cloud Accounting John Gordon, STFC

EGI-InSPIRE RI Federated Cloud Task Force Cloud Accounting work is done within the EGI Federated Cloud Task Force. –Scenario 4 Accounting The TF brings together cloud providers of various flavours and the main testbed use case is for a user to run VMs across different providers. The accounting use case is to record the use of these VMs.

EGI-InSPIRE RI Cloud Accounting The APEL team proposed an initial usage record which was implemented in all types of cloud and URs published by SSM1 to a central repository. Prototype Portal View Exists This summer the publishing was reviewed –Certain providers were actioned to correct certain values in their records –Rename some Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula – v3 of UR –Move to SSM2

EGI-InSPIRE RI

EGI-InSPIRE RI Next Steps Implement v3 UR everywhere Migrate to SSM2 (same time) Demo at EGI-TF Nothing to roll out in production to NGIs yet (all within testbed) –Scope for more NGIs to join testbed?