1. Cristina del Cano Novales STFC - RAL
APEL Accounting
Cristina del Cano Novales
STFC - RAL

2. Introduction APEL (Accounting Processor for Event Logs)
Accounts for jobs after they’ve finished
Deployed on the WLCG production Grid in late 2005
Collected over 250M individual job records
Accounted for
~800M CPU hours
~33M CPU days
~91000 CPU years
EGEE'09 Barcelona

3. Introduction
Used by WLCG and EGEE projects.
EGEE'09 Barcelona

4. Introduction
APEL client maintained by the APEL team at RAL (Rutherford Appleton Laboratory UK)
Source code - CVS repository at CERN
Built / packaged with ETICS system
Distributed as part of gLite (CE + MON box)
Accounting Servers at RAL
Accounting Portal at CESGA (Spain)
EGEE'09 Barcelona

5. Functionality APEL client has two components APEL parser
Parses batch system log files:
CONDOR – history
PBS –
SGE – accounting
LSF - lsb.acct
Parses globus gatekeeper and messages log files
Parses grid-jobmap log files (blah)
Reads CE benchmark values from site GIIS
Records this information into a local MySQL database
EGEE'09 Barcelona

6. Functionality APEL Publisher
Combines the entries from different log files into a full Accounting Record
Publishes records to a Flexible Archiver
Missing records (based on last successful published date)
All records
Records in a time interval (GAP)
It’s possible to use the APEL publisher just as a transport mechanism for records created with a different tool and converted into APEL schema
EGEE'09 Barcelona

7. Standards OGF-UR OGF-RUS Compliant with Usage Record v1.0
With extensions
ExecutingSite
Benchmark values …
APEL contribution to the Experiences document in UR-WG
Comparison between APEL schema and the Usage Record
OGF-RUS
Long term plan of implementing RUS for APEL
Not high priority at the moment
EGEE'09 Barcelona

8. Interoperability APEL receiving data from other tools DGAS SGAS OSG
Collecting data with own tool and publishing summaries to APEL Accounting Server
IN2P3-CC
IN2P3-CC-T2
Collecting data with own tool and publishing individual records with the APEL Publisher
EGEE'09 Barcelona

9. Roadmap Replacement of RGMA with Apache ActiveMQ (Dec 09)
Other tools working with ActiveMQ
Infrastructure (cluster of central brokers at CERN)
Reliable / robust
Distribution of APEL accounting system to regions (Apr 10)
Keeping a Central Accounting Repository / Central Accounting Portal for cross-region queries
Regions to decide whether to host own APEL instance
No changes required in the client
EGEE'09 Barcelona

10. Roadmap Replacement of RGMA with Apache ActiveMQ
Client side implemented
Testing in progress
Same authorization/authentication as before
X.509 host certificate
Access Control List in broker
Not dependent of gLite
No significant changes for user
Server side in progress
EGEE'09 Barcelona

11. Roadmap Distribution of APEL Accounting Other work in progress
Redesign of system required
Working with Accounting Portal for distribution of the two tools together
Other work in progress
Improving reliability of system
Archiving of historical records (only 13 month-old data in live database)
RUS (long term)
EGEE'09 Barcelona

12. CENTRAL ACCOUNTING PORTAL
EGI
Region B
APEL (Parser + Publisher) installed at all sites
Regional Accounting Server
Sites publish to Regional Archiver
Regional Accounting Server publishes summaries to Central Accounting Server
APEL
REGION B
REGIONAL ARCHIVER
REGIONAL DATABASE
REGIONAL PORTAL
Region A
APEL (Parser + Publisher) installed at all sites
No Regional Accounting Server
Sites publish directly to Central Accounting Server
APEL
REGION A
CENTRAL ARCHIVER
CENTRAL DATABASE
CENTRAL ACCOUNTING PORTAL
EGEE'09 Barcelona

13. CENTRAL ACCOUNTING PORTAL
EGI
Region C
Own accounting tool
Own accounting system
Region to publish summaries to the Central Accounting System
Region D
Own accounting tool
Own accounting system
Region to publish individual records to the Central Accounting System
CENTRAL ARCHIVER
CENTRAL DATABASE
CENTRAL ACCOUNTING PORTAL
REGIONAL DB
OTHER TOOL
REGION C
OTHER TOOL
REGION D
EGEE'09 Barcelona

14. EGI
EGEE'09 Barcelona

15. Links APEL Home Accounting deliverable document Accounting Portal
Accounting deliverable document
Accounting Portal
EGEE'09 Barcelona

16. Questions
EGEE'09 Barcelona