User Accounting Integration Spreading the Net. John Gordon, STFC EGI-EUDAT-PRACE Workshop EGI Technical Forum, Prague, September 2012 8/2/2019

Outline Collection of cpu records Integration of more infrastructures Storage Accounting Including EU-DAT and PRACE?

APEL 2011 The APEL accounting infrastructure started in production in 2004. initially developed for the LHC Computing Grid project (WLCG) it then adopted as central accounting solution for the EGEE and then EGI European-focussed but worldwide e-infrastructure. Currently APEL holds data from 1.7*10**9 jobs from 364 sites in 50 countries.

5 This shows the Usage of the WLCG Tier1 Centres by the LHC VOs. It is interesting because the accounting from the 12 Tier1s is sent by 7 different routes to a single place which makes possible the aggregated view at the bottom

APEL collects cpu usage data from 274 sites by the APEL clent software and from a further 90 where data is collected by other software and published to the central APEL repository. It is thus a single worldwide point of reference for all accounting data for a range of VOs(LHC and other international VOs as well as regional and national VOs). These non-APEL sites include alternative middleware stacks(ARC) within EGI and e-infrastructures within EGI (Italy, NorduGrid) and outside (Open Science Grid in US). The data from systems/services not using the APEL Client were doing direct database inserts.

APEL – 2013 External clients EGI Accounting Portal New APEL client http://goc-accounting.grid-support.ac.uk New APEL client DBunloader MySQL CPU JobRecords CPU Summaries New APEL client EGI Message Brokers EGI Message Brokers External clients EGI Accounting Portal Record loader Receiving SSM Sending SSM

External Clients for CPU Open Science Grid CERN (including local jobs) DGAS (IGI) SGAS (NorduGrid, Finnish NGI, Swiss NGI) Grid-SAFE (IGE) Sites (NIKHEF, CC-IN2P3) Projects (MAPPER, EDGI) ARC CE (direct to APEL) Unicore

New Types of Accounting Recent changes in the APEL Infrastructure have made it straightforward to receive and process additional types of usage records. There is a prototype cloud/VM consumer running and in use (see demo this week) There is a consumer running for storage records (not yet used)

APEL – new system External clients Storage clients http://goc-accounting.grid-support.ac.uk New APEL client DBunloader MySQL CPU JobRecords CPU Summaries StorageRecords StorageSummaries New APEL client EGI Message Brokers EGI Message Brokers External clients EGI Accounting Portal Record loader Storage clients Receiving SSM Storage loader Sending SSM

Spreading the Net? By the end of the year, the EGI CPU accounting will cover more: Middleware Sites Infrastructures Storing data from the EGI Federated Cloud Task Force And it should be receiving test storage records.

PRACE? Discussions with PRACE Instigated by MAPPER which uses both EGI and PRACE Collecting their data in one place allows a single view of their usage. PRACE uses Grid-SAFE to collect data from sites Unclear how/if this differs from IGE Grid-SAFE Could IGE technique be used to publish PRACE data to EGI? Restricted to consenting VOs.

EU-DAT? EU-DAT services hold data Implement storage accounting sensors for storage solutions used by EU-DAT. Which ones? Data Usage? Who uses a site’s data?, from where? Gathering requirements for a new usage record. Would like EU-DAT input.