1. APEL as a Global Accounting Repository
Alison Packer
STFC
2/25/2019
APEL Accounting Repository – EGI Technical Forum (September 2012)

2. Introduction APEL Central Repository redevelopment
Other Accounting Clients
APEL Client Development
Regional APEL Server

3. APEL – old system EGI Accounting Portal APEL client BROKER APEL client
ActiveMQ
APEL client
APEL client
MySQL
CONSUMER
MySQL
MySQL
Summaries
created
External
clients

4. New System - Deployment Stage 1
CONSUMER
MySQL
APEL client
BROKER
ActiveMQ
External
clients
MySQL
MySQL
Summaries created
MySQL
CPU JobRecords
CPU Summaries
External
clients
EGI Message Brokers
Record loader
Receiving SSM
EGI Accounting Portal
Sending SSM

5. New System - Deployment Stage 2
APEL client
BROKER
ActiveMQ
CONSUMER
DBunloader
MySQL
EGI Message Brokers
MySQL
CPU JobRecords
CPU Summaries
Converted JobRecords
New APEL client
External
clients
EGI Message Brokers
Record loader
Receiving SSM
EGI Accounting Portal
Sending SSM

6. APEL – new system External clients EGI Accounting Portal
nagios pub/sync
New APEL client
DBunloader
MySQL
CPU JobRecords
CPU Summaries
New APEL client
EGI Message Brokers
EGI Message Brokers
External
clients
EGI Accounting Portal
Record loader
Receiving SSM
Sending SSM

7. External clients - production
CERN publishing Job Records
OSG publishing Summary Records
CERN Job Records summarised on new system, including local jobs
New system records included in Summaries retrieved by Accounting Portal

8. External Clients - testing
INFN/DGAS have tested sending JobRecords and Summary records, ready to move to production.
IN2P3 have successfully tested sending Summary records, but still some date errors to resolve.
NDGF/SGAS have successfully tested sending Summary records, ready to move to production.

9. New External Clients SGAS (CH & Finland) – testing
Nikhef – successfully tested, ready for production
IGE/GridSafe – testing
QosCoSGrid/MAPPER – testing
EDGI – have a plan, awaiting testing
UNICORE – successfully tested, implementing new SSM protocol

10. New Server - development
New Summaries:
By CE, to enable nagios monitoring by CE as well as by site
Accounting of parallel and multicore jobs
CAR support (XML)
Additional types of accounting records
Publishing to Portal plus additional consumers

11. EMI 3 Client - parsers Support for
LSF (versions 5,6,7,8 )
PBS (version independent),
GridEngine (version independent)
Condor – is this still required?
Parser code complete and tested with LSF, PBS and GridEngine logs.
SLURM – we can do this if we have a collaborating production site

12. EMI 3 Client – in development
Support for local jobs.
Accounting of parallel and multicore jobs, Job Record modelled on CAR, and includes NodeCount and Processors fields.
Publishing summaries from client.

13. APEL Regionalisation
Once migration to new system complete a regional version of the repository will be provided.
NGI can then collect data from APEL clients, create summaries and publish – to local portal and central APEL repository.

14. APEL - summary Migration ongoing
Wider range of sites publishing in production
Additional accounting types (cloud, storage)
EMI 3 client supporting a range of new features
Regional APEL server