Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)

Enabling Grids for E-sciencE Grid Accounting - Overview Metering: Collection of usage metrics on computational resources. Accounting: Storage of usage metrics for further analysis. Usage Analysis: Production of reports from the available records. Pricing: Assign and manage prices for computational resources. Billing: Assign a cost to each user for his operations on the Grid. Account Balancing: Exchanging credits between users.

Enabling Grids for E-sciencE Usage Metering Usage metering is done by lightweight sensors (e.g. Gianduia) installed on Computing Elements. These sensors parse event logs produced by LRMS and build Usage Records, containing information about the unique ID of the user, the resource and the job. The Usage Records are then signed with the user’s certificate proxy and passed to the Accounting layer.

Enabling Grids for E-sciencE Accounting The usage of Grid Resources by users is registered in appropriate servers, called Home Location Registers (HLRs) that manage both user and resource accounts. Accounting records can be stored on an arbitrary number of independent HLRs. Each HLR keeps the records of all jobs submitted by each of its registered users or resources. Accounting requires accurate usage metering, but not necessarily resource pricing and billing.

Enabling Grids for E-sciencE Usage Analysis Different types of users are interested in different views of the usage records, for example: – A user will simply want to know how he used the grid resources. – A site manager needs to know who used his resources and how. – A VO manager needs to trace what the VO users are doing on the Grid. So there is the necessity for a system to analyses them and produce reports.

Enabling Grids for E-sciencE Pricing Resource pricing is done by dedicated Price Authorities (PAs) that may use different pricing algorithms (manual or dynamic settings). Prices can be established by an arbitrary number of independent PAs and are expressed in a virtual currency called Grid Credits. The way prices are set can help the workload balancing of the grid, lowering the prices of idle resources and raising those of overloaded ones. The job cost is determined (by the HLR service) from resource prices and usage records.

Enabling Grids for E-sciencE Billing The crediting and debiting process is initiated by Automatic Transaction Managers (ATMs) that are deployed on every HLR server. The economic transaction information is queued to a Transation Queue such that it may be processed asynchronously. The Transaction Manager processes each transaction stored in the queue by forwarding it to an HLR client that sends a payment check to the Resource HLR and waits for a receipt. Only at the end the HLR client debits the user account.

Enabling Grids for E-sciencE Account Balancing The account balancing provided by DGAS is intentionally generic. It may be used for different use cases, such as: – Redistribution of credits earned by a VO's resources to the VOs users (for balanced resource sharing between VOs). – Billing/charging of users after resource usage. – Credit/quota acquisition by users before resource usage. – Monitoring of overall resource consumption by users and resource contribution by owners. The purpose of DGAS is not to define (or limit) the economic interactions between users and resource owners, but to provide the necessary means to enable them.

Enabling Grids for E-sciencE DGAS DGAS is a fully distributed Grid accounting toolkit, implemented by a network of servers. On the server side it contains: – Price Authority (PA) – Home Location Register (HLR) – High Availability Daemon (HAD) The client side is composed by: – Gianduia – CEPushD – ceServerd – HAD

Enabling Grids for E-sciencE DGAS Server Side Price Authority – It is used to set the prices of the resources, either with a static or dynamic policy. Home Location Registers – Responsible for keeping the accounting information for users and resources. High Availability Daemon – Used to monitor the status of the services. – In case of failure it restarts the daemon avoiding long down periods due to service failures.

Enabling Grids for E-sciencE DGAS Client Side Gianduia – It is installed on a Computing Element in order to collect the usage records of the executed user jobs. CEPushD – This daemon sends the files created by Gianduia to the HLRs for accounting. ceServerd – Collects information transmitted from the Worker Nodes on which the jobs are running High Availability Daemon

Enabling Grids for E-sciencE DGAS Architecture A simplified view of DGAS within the WMS context.

Enabling Grids for E-sciencE DGAS Deployment DGAS is implemented by a network of server and it can be deployed according to various schemas. The suggested configuration is: – A User HLR per VO (The user specifies the address of his HLR in the job JDL). – One or more Resource HLRs (The site manager specifies the address of the resource HLR with a configuration file on the CE node). – The Gianduia metering system installed on the Computing Elements. – One or more Price Authorities (There should be one PA for every Resource HLR). PA are only needed if economic accounting is desired.

Enabling Grids for E-sciencE Storage accounting (the future) Actually, DGAS supports only CPU usage records, but we want to bill users for the storage space they use. We should modify our architecture in order to trace all user activities involving file creation or deletion. Problems to solve: – Capture all file creation (and deletion) whether they come from gLite I/O, GridFTP or any other source. – Modify the Usage Records format in order to contain storage data. – Extend the accounting DB to collect this Storage Records. – Modify the PAs in order to set prices for disk space. – Fix “dgas-check-balance” tool to show this information

Enabling Grids for E-sciencE Further information Distributed Grid Accounting System (DGAS) – Rosario M. Piro (INFN Turin) – EGEE JRA1 Middleware CVS Repository –