DGAS A.Guarise April 19th, Athens www.eu-egee.org EGEE is a project funded by the European Union under contract INFSO-RI-508833

DataGrid Accounting System The Data Grid Accounting System was originally developed within the EU Datagrid Project and is now being maintained and re engineered within the EU EGEE Project. The Purpose of DGAS is to implement Resource Usage Metering, Accounting and Billing in a fully distributed Grid environment. It is conceived to be distributed, secure and extensible. The system is designed in order for Usage Metering, Accounting and Billing to be indipendent layers.

Usage Metering and Accounting The usage of Grid Resources by Grid Users is registered in appropriate servers, known as HLRs (Home Location Registers) where both users and resources are registered. In order to achieve scalability, there can be many independent HLRs. At least one HLR per VO are foreseen, although a finer granularity is possible. Each HLR keeps the records of every grid job executed by each of its registered users or resources, thus being able to furnish usage information with many granularity levels: Per user or resource, per group of users or resources, per VO.

Economic Accounting The DGAS software is concieved mainly as an Economic Accounting system. Within this approach, grid users and grid resources are treated as consumers and producers of goods like computational power or storage capacity. The consumers can use these goods paying an apropriate amount of Grid Credits to the producers. The idea behind this approach is that an Economic Accounting system may have a major advantage over a plain, bookkeeping style accounting system: it deploys a virtual economic market over the grid so that the natural behviour of this market can be used to drive the workload management system in balancing the resource load.

Example of economic accounting Check Economic Authorization VO 1 VO 2 WMS HLR 1 HLR 2 The user submits a job to the WMS (edg-job-submit) The WMS selects the CE (matchmaking) and contact the User HLR to check if the user has a valid account and enough credits to pay the job or if he hasn’t exceed his usage quotas. If this authorization is passed the job is submitted to the CE. Once the job is finished, the Usage Records collected by the metering system are sent to the User HLR where the records are stored and, if economic accounting is required by the resource, the job cost is computed. Then the User HLR contacts the Resource HLR and instantiates a transaction. The user account is debited, the resource account is credited. In any moment the HLR administrator of VO2 can redistribute the resource earnings among the VO users. CE CE

HLR: Available Information For each job executed on a CE, the following information are available querying the HLR: GridJobID User Certificate Subject Local user Id Grid Unique Computing Element Id User VO Cpu Time Wall clock time Physical memory Virtual Memory Job assigned nodes (cpus) In principle all information available to the LRMS can be published in the HLR server.

HLR: Standard queries An administrator that has local access to the HLR server can retrieve many information about the status of the accounts: For example: edg-wl-dgas-hlr-queryUser –Ua Retrieves the information about the account of all the registered users. edg-wl-dgas-hlr-queryUser –UaT Retrieves also all the transactions performed for each user. edg-wl-dgas-hlr-queryUser –U –e guariseATto.infn.it |guarise|andrea.guariseATto.infn.it|Andrea Guarise|/C=IT/O=INFN…|torino|infngrid|10000|0|0|wall_t ime=0,cpu_time=0,job_number=0| Retrieves only information about the user with the given email address Resource Queries Similar queries are available for resources accounts via the command: edg-wl-dgas-hlr-queryResource

HLR: Advanced queries It is also possible to perform more complex queries to get, for example, statistics of the usage of a set of resources, or to have a snapshot of the amount of work performed by users dividing them by groups or VOs. Statistics of all the user jobs accounted by an HLR: edg-wl-dgas-hlrAdvancedQuery –Q userAggregate Jobs| totCost| totCpuTime|totWallTime|totMem(MB)|totVMem(MB)| 15| 0| 21550| 48551| 22| 25| Statistics of all the user jobs accounted for the group ‘torino’: edg-wl-dgas-hlrAdvancedQuery –Q userAggregate –g torino 5| 0| 9201| 15539| 7| 8| Aggregates are available for resources, groups, VOs. It is also possible to have job lists and specify a time interval for the queries.

Metering Infrastructure: GIANDUIA DGAS furnishes the APIs that can be used by each site to develop its own metering infrastructure. So that a site can freely decide how to measure its own Usage Records. Two APIs are available. These both provide a C++ library and a CLI tool. Their purpose is to push the information to the User HLR once the Usage Record for a job is available. These two API only differs in terms of the schema adopted to describe the Usage Records: The first implements a simple, proprietary, schema. The second implements the GGF URWG schema, more powerful, but rather complex to manage. A minimum set of information need to be available in the usage record to make grid accounting possible: The User HLR Location (comes with the job JDL) A valid User Proxy (Needed to secure the communication) The CE ID (Needed to unambiguously assign the Usage) The Grid Job Id (For grid accounting the LRMS job id is not enough)

Metering Infrastructure: GIANDUIA GIANDUIA workflow (Gianduia Is A Nice Distributed Usage-metering Infrastructure for Accounting) WN HLR_LOCATION USER_PROXY CE_ID LRMS_JOB_ID GRID_JOB_ID jobWrapper CE LRMS LOG gianduia conf User HLR (HLR_LOCATION) GIANDUIA gianduiottiBox

References Further information and documentation about DGAS can be found at: http://www.to.infn.it/grid/accounting