1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org EGEE is a project funded by the European Union under contract INFSO-RI-508833 Grid Accounting in EGEE using DGAS current practices Andrea Guarise – INFN, Italy June 8th, Poznan, TNC 2005

2. Enabling Grids for E-sciencE INFSO-RI-508833 Introduction A generic Grid accounting process involves many subsequent phases that can be divided in: Metering: collection of usage metrics on computational resources. Accounting: storage of such metrics for further analysis. Usage Analysis: Production of reports from the available records. Pricing: Assign and manage prices for computational resources. Billing: Assign a cost to user operations on the Grid and charge them. In this presentation we briefly describe these steps and give a quick overview of DGAS, the accounting middleware of the EGEE project.

3. Enabling Grids for E-sciencE INFSO-RI-508833 Metering The metering phase in Grid accounting is probably the most important of the whole process. During the metering phase the user payload on a resource needs to be correctly measured, and unambiguously assigned to the Grid User that directly or indirectly requested it to the Grid. This requires the system to collect information from the operating system (or the LRMS for batch jobs) and from the grid middleware. These information forms the Usage Record for the user process. This usage record must comprehend at least the Grid Unique Identifier for the Grid User, the Grid Resource and the Grid Job, plus the metrics of the resource consumption.

4. Enabling Grids for E-sciencE INFSO-RI-508833 Accounting Once collected on the computational resources, the usage records needs to be properly archived in databases for further analysis. The minimum set of requirements for the accounting service at this level is: ● Information should be available to the Users responsible for the payload, to the Site Managers of the Grid Resources and to the VO administrator of the user, but not to other people. Or, in other words, information must be confidential. ● Usage records must be sent encrypted and signed to the accounting services. Signing the Usage Records with the user credential assures that they become irrefutable when users have to be charged. ● A distributed architecture is essential, as well as reliable and fault tolerant communication mechanisms.

5. Enabling Grids for E-sciencE INFSO-RI-508833 Usage Analysis Information archived in the Accounting databases are rather complex. Not all the ‘users’ are interested in all of them. So there is the necessity for a system that analyses them and produce reports. Different types of users are interested in different views of the usage records, for example: ● A user will simply want to know how he used the grid resources. ● A site manager needs to know who used his resources and how. For example the percentage of usage per VO. Then he needs to be able to track a single job, for example because it caused some problems. ● A VO manager needs to trace what the VO users are doing on the Grid.

6. Enabling Grids for E-sciencE INFSO-RI-508833 Pricing & billing Resource owners may want to charge the users, thus it is necessary to establish a cost for the service furnished to the user. A cost is usually computed according to a price assigned to the unit of usage of a computing resource and to the usage measured for the same resource. Thus a service responsible for managing the resource prices and communicating them to all the partners is needed. The way prices are set contributes to the creation of an economic market that deeply influences the behaviour of the Grid as a dynamic system. Once the resource consumption is known and a price is assigned to the computational resources, it is possible to define a cost that can be charged to the Grid User. The final cost applied to the User is influenced also by policy issues like discounts or offers.

7. Enabling Grids for E-sciencE INFSO-RI-508833 DGAS The Data Grid Accounting System was originally developed within the EU Datagrid Project and is now being maintained and re- engineered within the EU EGEE Project. The Purpose of DGAS is to implement Resource Usage Metering, Accounting and Account Balancing (through resource pricing) in a fully distributed Grid environment. It is conceived to be distributed, secure and extensible. The system is designed in order for Usage Metering, Accounting and Account Balancing (through resource pricing) to be indipendent layers. Usage Metering Usage accounting Account balancing, resource pricing, (billing) usage records accounting data Usage Analysis

8. Enabling Grids for E-sciencE INFSO-RI-508833 DGAS accounting architecture A simplified view of DGAS within the WMS context.

9. Enabling Grids for E-sciencE INFSO-RI-508833 DGAS: Metering Usage Metering on Computing Elements is done by lightweight sensors installed on the Comuting Elements. These sensors parse PBS/LSF/Torque event logs to built Usage Records that can be passed to the accounting layer. For a reliable accounting of resource usage (essential for billing) it is important that the collected data is unequivocally associated to the unique grid ID of the user (certificate subject/DN), the resource (CE ID) as well as the job (global job ID). A process, completely transparent to the Grid User collects the necessary information needed by the Accounting. These, and the corresponding metrics are sent via an encrypted channel to the Accounting System signed with the user credentials.

10. Enabling Grids for E-sciencE INFSO-RI-508833 Metering: Gianduia

11. Enabling Grids for E-sciencE INFSO-RI-508833 DGAS: Accounting The usage of Grid Resources by Grid Users is registered in appropriate servers, called Home Location Registers (HLRs) where both users and resources are registered. In order to achieve scalability, accounting records can be stored on an arbitrary number of independent HLRs. At least one HLR per VO is foreseen, although a finer granularity is possible. Each HLR keeps the records of all grid jobs submitted or executed by each of its registered users or resources, thus being able to furnish usage information with many granularity levels: Per user or resource, per group of users or resources, per VO. Accounting requires usage metering, but not necessarily resource pricing and billing.

12. Enabling Grids for E-sciencE INFSO-RI-508833 DGAS deployment VO1 VO2 VO3 site1 site2 site3 HLR 1HLR 2HLR 3HLR 5HLR 4 CE APEL Aggregate site accounting

13. Enabling Grids for E-sciencE INFSO-RI-508833 Balancing and Resource Pricing Resource pricing is done by dedicated Price Authorities (PAs) that may use different pricing algorithms: manual setting of fixed prices, dynamical determination of prices according to the state of a resource. In order to achieve scalability, prices can be established by an arbitrary number of independent PAs. At least one PA per VO is foreseen (VOs will want to retain control on the pricing of their resources). Price algorithms are dynamically linked by the PA server and can be re-implemented according to the resource owners' needs. The job cost is determined (by the HLR service) from resource prices and usage records. Account balancing is done by exchanging virtual credits between the User HLR and the Resource HLR.

14. Enabling Grids for E-sciencE INFSO-RI-508833 What about billing/charging? The Account Balancing provided by DGAS is intentionally generic. It may be used for different use cases, such as: > Monitoring of overall resource consumption by users and resource contribution by owners. > Redistribution of credits earned by a VO's resources to the VOs users (for balanced resource sharing between VOs). > Billing/charging of users after resource usage. > Credit/quota acquisition by users before resource usage. The purpose of DGAS is not to define (and hence limit) the economic interactions between users and resource owners, but to provide the necessary means to enable them.

15. Enabling Grids for E-sciencE INFSO-RI-508833 Example of economic accounting VO 1 HLR 1 CE VO 2 HLR 2 WMS PA 2 CE Usage Metering Usage Accounting Account Balancing/Resource Pricing Job Flow Check of economic authorization

16. Enabling Grids for E-sciencE INFSO-RI-508833 References ● Further information and documentation about DGAS can be found at: http://www.to.infn.it/grid/accounting