1. User Accounting Integration Spreading the Net.
John Gordon, STFC
EGI-EUDAT-PRACE Workshop
EGI Technical Forum,
Prague, September 2012
8/2/2019

2. Outline Collection of cpu records Integration of more infrastructures
Storage Accounting
Including EU-DAT and PRACE?

3. APEL 2011
The APEL accounting infrastructure started in production in initially developed for the LHC Computing Grid project (WLCG) it then adopted as central accounting solution for the EGEE and then EGI European-focussed but worldwide e-infrastructure.
Currently APEL holds data from 1.7*10**9 jobs from 364 sites in 50 countries.

5. 5 This shows the Usage of the WLCG Tier1 Centres by the LHC VOs.
It is interesting because the accounting from the 12 Tier1s is sent by 7 different routes to a single place which makes possible the aggregated view at the bottom

6. APEL collects cpu usage data from 274 sites by the APEL clent software and from a further 90 where data is collected by other software and published to the central APEL repository.
It is thus a single worldwide point of reference for all accounting data for a range of VOs(LHC and other international VOs as well as regional and national VOs).
These non-APEL sites include alternative middleware stacks(ARC) within EGI and e-infrastructures within EGI (Italy, NorduGrid) and outside (Open Science Grid in US).
The data from systems/services not using the APEL Client were doing direct database inserts.

7. APEL – 2013 External clients EGI Accounting Portal New APEL client
New APEL client
DBunloader
MySQL
CPU JobRecords
CPU Summaries
New APEL client
EGI Message Brokers
EGI Message Brokers
External
clients
EGI Accounting Portal
Record loader
Receiving SSM
Sending SSM

8. External Clients for CPU
Open Science Grid
CERN (including local jobs)
DGAS (IGI)
SGAS (NorduGrid, Finnish NGI, Swiss NGI)
Grid-SAFE (IGE)
Sites (NIKHEF, CC-IN2P3)
Projects (MAPPER, EDGI)
ARC CE (direct to APEL)
Unicore

9. New Types of Accounting
Recent changes in the APEL Infrastructure have made it straightforward to receive and process additional types of usage records.
There is a prototype cloud/VM consumer running and in use (see demo this week)
There is a consumer running for storage records (not yet used)

10. APEL – new system External clients Storage clients
New APEL client
DBunloader
MySQL
CPU JobRecords
CPU Summaries
StorageRecords
StorageSummaries
New APEL client
EGI Message Brokers
EGI Message Brokers
External
clients
EGI Accounting Portal
Record loader
Storage
clients
Receiving SSM
Storage loader
Sending SSM

11. Spreading the Net?
By the end of the year, the EGI CPU accounting will cover more:
Middleware
Sites
Infrastructures
Storing data from the EGI Federated Cloud Task Force
And it should be receiving test storage records.

12. PRACE? Discussions with PRACE
Instigated by MAPPER which uses both EGI and PRACE
Collecting their data in one place allows a single view of their usage.
PRACE uses Grid-SAFE to collect data from sites
Unclear how/if this differs from IGE Grid-SAFE
Could IGE technique be used to publish PRACE data to EGI? Restricted to consenting VOs.

13. EU-DAT? EU-DAT services hold data
Implement storage accounting sensors for storage solutions used by EU-DAT.
Which ones?
Data Usage? Who uses a site’s data?, from where?
Gathering requirements for a new usage record.
Would like EU-DAT input.