1. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Storage Accounting John Gordon, STFC OMB August 2013

2. www.egi.eu EGI-InSPIRE RI-261323 Outline History Timeline Current state Evidence of testing Current issues Open Issues Portal Views Reports?

3. www.egi.eu EGI-InSPIRE RI-261323 EMI StAR EMI recognised the requirement for storage accounting EMI created a proposal for a storage accounting record (StAR) ‘Finalised’ in June 2011 Revised Autumn 2012 to include the WLCG requirements I described at the GDB last June. Published version: http://cds.cern.ch/record/1452920?ln=en Been through public comment in OGF and just resubmitted for approval implemented by EMI storage providers in EMI 3 3 EMI INFSO-RI-261611

4. www.egi.eu EGI-InSPIRE RI-261323 Original EMI StAR timeline May 2011: StAR definition agreement Oct. 2011: Each SE to provide an implementation plan May 2012: Each SE to implement accounting sensors June 2012: Accounting record publishers (APEL and/or DGAS) to publish storage records December 2012: Each SE to test and deploy their accounting sensors (for EMI 3 release) January-April 2013: Bug-fixing in EMI 3 RC’s April 2013: Accounting sensors released in EMI 3 4 EMI INFSO-RI-261611

5. www.egi.eu EGI-InSPIRE RI-261323 Actual EMI Timeline February 2013 – first test storage records appeared from dcache and dpm February-March 2013 checking and revising. 11 March – Release of EMI 3.0 Any changes required should be available in update 1.

6. www.egi.eu EGI-InSPIRE RI-261323 Status of StAR implementations Dcache – published from a test system. Dpm - published from a test system. StoRM – did not make EMI 3.0 cutoff - ??? NGI-Italy is publishing for all INFN sites using data from the BDII. EMI produced tests to verify that StAR records agree with standard –But that doesn’t address requirements on semantics. StAR publishing uses the SSM2 infrastructure asthe EMI-3 cpu client, and some other publishers.

7. www.egi.eu EGI-InSPIRE RI-261323

8. www.egi.eu EGI-InSPIRE RI-261323 Issues from Testing No GOCDB Sitename – may be artifact of test sites. No ResourceCapacityAllocated – just Used All fixed in EMI-3 production releases of DPM and dcache

9. www.egi.eu EGI-InSPIRE RI-261323 Other Issues Frequency of publishing –Sites should publish once per day. They can collect data as frequently as they wish and publish daily averages if they wish. –Repository could average daily records into months, like CPU but will start by passing everything to the portal. What is to be viewed in the portal?

10. www.egi.eu EGI-InSPIRE RI-261323 Accounting Portal Summarise data by Site, VO, Month (Average, min, max) Views at any point in tree of –Tier1, Tier2, Country, NGI View Average Storage for a time period and VO selection, for any two of (site/country/Tier2), VO, Date E.g. Country(date), VO(Site), VO(date), Site(date) (similar to CPU Portal Views.) WARNING! Examples do not show real data.

11. www.egi.eu EGI-InSPIRE RI-261323 Portal View for Storage

12. www.egi.eu EGI-InSPIRE RI-261323 Storage per country per month

13. www.egi.eu EGI-InSPIRE RI-261323 Storage used by LHC VOs at French sites.

14. www.egi.eu EGI-InSPIRE RI-261323 Storage for Alice in France Allocated and used.

15. www.egi.eu EGI-InSPIRE RI-261323

16. www.egi.eu EGI-InSPIRE RI-261323

17. www.egi.eu EGI-InSPIRE RI-261323 The views shown were broadly accepted by WLCG. Does EGI have anything else to be added? Portal team working on a prototype.

18. www.egi.eu EGI-InSPIRE RI-261323 Reports When storage accounting is deployed at most sites then:- –The Tier1 monthly report can be autofilled with storage data. –The Tier2 report can be extended to cover storage as well as cpu. –EGI Quarterly Metrics can be pre-filled with storage data

19. www.egi.eu EGI-InSPIRE RI-261323 Next Steps Verify that storage providers are publishing consistent records. Start publishing real data from a variety of sites Extend BDII data to other countries/storage solutions Wider coverage of storage solutions –StoRM, OSG, xrootd? Finalise what is shown in portal Implement Portal view and reports. Investigate merged reports (CPU+Storage)

20. www.egi.eu EGI-InSPIRE RI-261323 Conclusions Storage Accounting is making (slow) progress. Are there suggestions of other views or reports? Questions?

21. www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE www.egi.eu EGI-InSPIRE RI-261323 Cloud Accounting John Gordon, STFC

22. www.egi.eu EGI-InSPIRE RI-261323 Federated Cloud Task Force Cloud Accounting work is done within the EGI Federated Cloud Task Force. –Scenario 4 Accounting The TF brings together cloud providers of various flavours and the main testbed use case is for a user to run VMs across different providers. The accounting use case is to record the use of these VMs.

23. www.egi.eu EGI-InSPIRE RI-261323 Cloud Accounting The APEL team proposed an initial usage record which was implemented in all types of cloud and URs published by SSM1 to a central repository. Prototype Portal View Exists This summer the publishing was reviewed –Certain providers were actioned to correct certain values in their records –Rename some Cloud Accounting Usage Record fields to fields used by Venus-C and OpenNebula – v3 of UR –Move to SSM2

24. www.egi.eu EGI-InSPIRE RI-261323

25. www.egi.eu EGI-InSPIRE RI-261323 Next Steps Implement v3 UR everywhere Migrate to SSM2 (same time) Demo at EGI-TF Nothing to roll out in production to NGIs yet (all within testbed) –Scope for more NGIs to join testbed?