Some Title from the Headrer and Footer, 19 April Overview Requirements Current Design Work in Progress

Some Title from the Headrer and Footer, 19 April Requirements Would like to measure the usage of LCG resources Consumption of resources per VO can be mapped to a set of LCG sites where usage is based as a function of time Accounting data needs to be aggregated across all the resources available to LCG

Some Title from the Headrer and Footer, 19 April Requirements Cont … Minimise administration of accounting log files at each site Be based on existing or emerging grid accounting standards Permit accounting records to be collected from any batch system Be capable of processing accounting records by either running on the CE at the site or by running on a host at the GOC site using copies of the accounting logs there Should be modelled on the Portable Batch System (PBS) and the Globus GateKeeper (GK) logging

Some Title from the Headrer and Footer, 19 April Current Design Design is split into two bits  GOC deployment Runs the famous ‘Flexible Archiver’  Site deployment Runs the Apel Log Processor Text book Secondary Producer application

Some Title from the Headrer and Footer, 19 April Topology MySql Apel Log Processor LCG Site MON Box PBS/GK records Data Streamed MON Box Flexible Archiver GOC MySql Accounting data stored locally

Some Title from the Headrer and Footer, 19 April The Accounting Schema Flexible Archiver uses an Accounting schema based on a GGF recommendation Contains one large table called LCGRecords LCGRecords table holds information about when a job was started, finished, what CPU resources were used and who submitted it Tuples conforming to this schema are stitched together by the Apel Log Processor and then published using a Primary Producer

Some Title from the Headrer and Footer, 19 April Apel Log Processor The main work horse of the Apel application Parses three types of logs  PBS Event Logs produced by the CE  GK Logs produced by the GK  Message Logs produced by the GK Log file types maybe distributed across a site Flexible enough to handle different site topologies  For example, sites may combine the CE and GK on one host  Can run off site – especially useful if processing is to be carried out at the GOC Logs are periodically parsed using a cron job  Once set-up, administration is straight forward

Some Title from the Headrer and Footer, 19 April Apel Log Processor Cont … Parsed log data is published into a table matching its log type Data is stored on the Site’s Mon Box using a Latest Primary Producer After the log processing is complete, a further operation is carried out to piece data into the LcgRecord schema This operation is a huge SQL Join query in disguise Stiched-up LcgRecords are then streamed to the GOC Flexy Archiver A record of processed log files is also made to prevent old files being re-processed in subsequent runs

Some Title from the Headrer and Footer, 19 April The Flexible Friend A Secondary Producer deployed on the GOC scoops up all LcgRecords published from each site Large cache of data for billing/statistics etc Work is being carried out by Dave Kant to visualize this data set

Some Title from the Headrer and Footer, 19 April Current Status & Future Work Apel application currently going through certification and testing  Conducted by Di Qing at CERN  Will provide support to fix any problems as they occur Currently a lot of interest to support different logging formats  Frederic Schaer of In2p3 is currently implementing a plug-in for their own site which uses BQS  CERN Tier 1/0 uses LSF based batch processing - would like a bespoke plug-in to keep them happy Centre runs 1000’s of computing resources Current Timeline??

Some Title from the Headrer and Footer, 19 April Future Improvements Currently, each site uses a Mon Box to store its accounting data locally But, why not just directly connect to the local MySQL database?  We then whip out the various reference to the Latest Primary Producer  Replace with the standard JDBC interface  Then publish the LcgRecords using a Streaming Primary Producer  Voila - we get rid of the Mon Box  Now all we need to do is work out where to put the MySQL installation! Probably won’t make the code go faster But might be useful for code re-use when implementing a plug-in for LSF  For example, do we follow the same flawed approach or take the plunge and refactor?