1. DGAS Basic Management A.Guarise
February 24th, Frascati, Italy
DGAS Basic Management A.Guarise
EGEE is a project funded by the European Union under contract INFSO-RI

2. DataGrid Accounting System
The Data Grid Accounting System was originally developed within the EU Datagrid Project and is now being maintained and re engineered within the EU EGEE Project.
The Purpose of DGAS is to implement Resource Usage Metering, Accounting and Billing in a fully distributed Grid environment. It is conceived to be distributed, secure and extensible.
The system is designed in order for Usage Metering, Accounting and Billing to be indipendent layers.

3. Usage Metering and Accounting
The usage of Grid Resources by Grid Users is registered in appropriate servers, known as HLRs (Home Location Registers) where both users and resources are registered.
In order to achieve scalability, there can be many independent HLRs. At least one HLR per VO are foreseen, although a finer granularity is possible.
Each HLR keeps the records of every grid job executed by each of its registered users or resources, thus being able to furnish usage information with many granularity levels:
Per user or resource, per group of users or resources, per VO.

4. Economic Accounting
The DGAS software is concieved mainly as an Economic Accounting system. Within this approach, grid users and grid resources are treated as consumers and producers of goods like computational power or storage capacity. The consumers can use these goods paying an apropriate amount of Grid Credits to the producers.
The idea behind this approach is that an Economic Accounting system may have a major advantage over a plain, bookkeeping style accounting system: it deploys a virtual economic market over the grid so that the natural behviour of this market can be used to drive the workload management system in balancing the resource load.

5. Example of economic accounting
Check Economic
Authorization
VO 1
VO 2
WMS
HLR 1
HLR 2
The user submits a job to the WMS (edg-job-submit)
The WMS selects the CE (matchmaking) and contact the User HLR to check if the user has a valid account and enough credits to pay the job or if he hasn’t exceed his usage quotas.
If this authorization is passed the job is submitted to the CE.
Once the job is finished, the Usage Records collected by the metering system are sent to the User HLR where the records are stored and, if economic accounting is required by the resource, the job cost is computed.
Then the User HLR contacts the Resource HLR and instantiates a transaction. The user account is debited, the resource account is credited.
In any moment the HLR administrator of VO2 can redistribute the resource earnings among the VO users. CE CE

6. HLR Server Installation
There are actually two ‘flavours’ of the DGAS software available. The one coming from the EGEE gLite software and the one coming from the INFNGrid middleware. The two ‘flavours’ are very similar, the only differences being in some changes in command names.
In order to install an HLR server using the INFNGrid ‘flavour’ of DGAS (actually available for RH7.3 and SLC3) the needed rpms are:
edg-wl-common-api_slc_3_0_ i486.rpm
edg-wl-common-utilities_slc_3_0_ i486.rpm
edg-wl-services-common_slc_3_0_ i486.rpm
edg-wl-dgas-hlr-server_slc_3_0_ i486.rpm
edg-wl-dgas-hlr-server-admin_slc_3_0_ i486.rpm
edg-wl-dgas-hlr-ui_slc_3_0_ i486.rpm \
edg-wl-dgas-pa-paClient_slc_3_0_ i486.rpm

7. HLR Server configuration
The HLR server has a configuration file:
/opt/edg/etc/edg-wl-dgas-hlr.conf
The main parameters needed to tune up the server are:
Parameter
Default Value
Description
hlr_def_log
/opt/edg/var/log/hlrd.log
TCP listener log file
hlr_qmgr_def_log
/opt/edg/var/log/hlr_qmgrd.log
Transaction daemon log file
hlr_def_port
56568
HLR listening port
hlr_gridmapfile
/etc/grid-security/grid-mapfile
Host grid-mapfile
hlr_def_lock
/opt/edg/var/hlr.lock
Tcp listener lock file
hlr_qmgr_def_lock
/opt/edg/var/hlr_qmgr.lock
Transaction daemon lock file

8. HLR Database initialisation
The HLR stores all the critical information in a persistent database based upon MySQL.
Before starting up a DGAS installation for the first time it is necessary to initialise the databases used by the HLR.
There is a script that automates this process:
/opt/edg/sbin/edg-wl-hlrd-dbcreate
The script creates two databases named ‘hlr’ and ‘hlr_tmp’ and populate them with a set of predefined tables.

9. HLR Server Startup
The HLR server is actually composed by three separate daemons, illustrated in the table below.
The startup script: /etc/rc.d/init.d/edg-wl-hlrd understands the usual start, stop, restart and status targets.
Daemon name
Description
edg-wl-dgas-hlrd
The TCP listener, waits for incoming requests and calls the engine needed to fulfill the client command.
edg-wl-dgas-hlr-tqd
The daemon responsible for managing accounting transactions with other HLRs.
edg-wl-dgas-hlr-had.pl
Daemon responsible for restarting the server in case of failures.

10. PA Server Installation
A Price Authority is a service responsible to establish prices for resources, store them and allow to retrieve the price assigned to a resource in a given moment in time.
The installation of a PA server is conceptually identical to that of an HLR, since the underlying code is mainly the same. Usually it should be deployed on the same node that runs the HLR server that manages the resources of a site.
The RPMs needed by a PA server are (INFNGrid flavour):
edg-wl-common-api_slc_3_0_ i486.rpm
edg-wl-common-utilities_slc_3_0_ i486.rpm
edg-wl-services-common_slc_3_0_ i486.rpm
edg-wl-dgas-pa-server_slc_3_0_ i486.rpm
edg-wl-dgas-pa-paClient_slc_3_0_ i486.rpm

11. PA Server configuration
The PA server has a configuration file:
/opt/edg/etc/edg-wl-dgas-pa.conf
The Parameters needed to tune up the server are:
Parameter
Default Value
Description
pa_def_log
/opt/edg/var/log/pad.log
PA log file
pa_def_lock
/opt/edg/var/pa.lock
PA lock file
pa_def_port
56567
PA listening port
pricing_scheme
dynamic | manual
Selects if prices are inserted by the administrator or computed automatically by the system.
pa_price_dll_name
libdgas_paPriceAlgDyn.so
Name of the DLL plugin used for dynamic price computation.
def_price_minTTL
3600
Time (in seconds) after wich a price expires.

12. PA Server Startup
The PA server is actually composed by two separate daemons, illustrated in the table below.
The startup script: /etc/rc.d/init.d/edg-wl-pad understands the usual start, stop, restart and status targets.
Daemon name
Description
edg-wl-dgas-pad
The TCP listener, wait for incoming requests and calls the engine needed to fulfill the client command.
edg-wl-dgas-hlr-had.pl
Daemon responsible for restarting the server in case of failures. It is the same command used by the HLR server.

13. HLR : Registering Accounts (1)
Users need to have a valid DGAS account on an HLR in order for their jobs to be accounted.
To manually create an user account the command is:
edg-wl-dgas-hlr-addUser -u guarise –g torino –f infngrid -e –c “/C=IT/O=INFN/OU=Personal Certificate/L=Torino/CN=Andrea
This creates an account for the user Andrea Guarise, with dgas uid ‘guarise’, assigning him to the ‘torino’ group of the infngrid Virtual Organisation.
As the command suggests, it is possible to assign users to VOs and to groups inside a VO. There are therefore commands to create such groups:
The command used to create the torino group would be:
edg-wl-dgas-hlrAddGroup –g torino –d “infngrid group torino” –f infngrid

14. HLR : Registering Accounts (2)
Then the infngrid VO account can be created with the command:
edg-wl-dgas-hlr-addFund –d “INFNGrid VO” –f infngrid
Also Grid Resources need to have an account on an HLR server, the command to manually register a resource is:
edg-wl-dgas-hlr-addResource –r to001 –e –d grid002.to.infn.it –c “grid002.to.infn.it:2119/jobmanager-pbs-short” –g torino –f infngrid
This command creates an account for the CE with CEId:
grid002.to.infn.it:2119/jobmanager-pbs-short
As you can see also the CE can be seen as an account part of a group, that, as in the example, can be a group contaqining users as well.

15. HLR : Registering Accounts (3)
Since manually inserting a huge amount of user and resource accounts is clearly not feasible. Two CLI tools are furnished for bulk registrations as well.
To register in a single step all the Users belonging to a given VO LDAP server the command is:
dgas_hlr_voImport.pl –v “ldap://grid- vo.nikhef.nl/o=alice,de=eu-datagrid,dc=org” –V alice
This command creates the accounts for all the users registered in the ALICE VO. It also creates group accounts using the domain of the addresses so that users belonging to the VO are automatically grouped by site too.

16. HLR : Registering Accounts (4)
To register in a single step all the Resources belonging to a given bdII server the command is:
dgas_hlr_bdiiImport.pl –s “ibm140.cnaf.infn.it” –p 2170
This command creates the accounts for all the resources registered in the ibm140 bdII. It creates group accounts using the domain of the ce hostname and uses the Mds-Vo-Name ldap field to assign the resource to a VO.
This is the default behaviour of the command, some tuning is available with other CLI options, that can be seen via the ‘-h’ command option.

17. HLR and PA: Basic operations
Both the HLR and PA TCP listener can be remotely queried for their status, the command is:
edg-wl-dgas-pingClient –s “grid005.to.infn.it:56568:” –t 1
Available Engines: UI:ATM:BANK:JOBAUTH:USERAUTH:URWG:PING
Ping requests: 2
ATM requests: 539/523
jobAuth requests: 36/0
It needs to be executed with a valid user proxy.
All the daemons write logs of their activities. Each log entry contains information about the Engine that originates the log stamp and information about the connection responsible for the service call, such as hostname of the machine originating the request and the subject of the certificate that authenticates the client connection.
Every daemon writes a lock file that contains the pid of the corresponding daemon process.

18. HLR: Standard queries
An administrator that has local access to the HLR server can retrieve many information about the status of the accounts:
For example:
edg-wl-dgas-hlr-queryUser –Ua
Retrieves the information about the account of all the registered users.
edg-wl-dgas-hlr-queryUser –UaT
Retrieves also all the transactions performed for each user.
edg-wl-dgas-hlr-queryUser –U –e guariseATto.infn.it
|guarise|andrea.guariseATto.infn.it|Andrea Guarise|/C=IT/O=INFN…|torino|infngrid|10000|0|0|wall_t ime=0,cpu_time=0,job_number=0|
Retrieves only information about the user with the given address
Resource Queries
Similar queries are available for resources accounts via the command: edg-wl-dgas-hlr-queryResource

19. HLR: Advanced queries
It is also possible to perform more complex queries to get, for example, statistics of the usage of a set of resources, or to have a snapshot of the amount of work performed by users dividing them by groups or VOs.
Statistics of all the user jobs accounted by an HLR:
edg-wl-dgas-hlrAdvancedQuery –Q userAggregate
Jobs| totCost| totCpuTime|totWallTime|totMem(MB)|totVMem(MB)|
15| | | | | |
Statistics of all the user jobs accounted for the group ‘torino’:
edg-wl-dgas-hlrAdvancedQuery –Q userAggregate –g torino
5| | | | | |
Aggregates are available for resources, groups, VOs. It is also possible to have job lists and specify a time interval for the queries.

20. Metering Infrastructure: GIANDUIA
DGAS furnishes the APIs that can be used by each site to develop its own metering infrastructure. So that a site can freely decide how to measure its own Usage Records.
Two APIs are available. These both provide a C++ library and a CLI tool. Their purpose is to push the information to the User HLR once the Usage Record for a job is available.
These two API only differs in terms of the schema adopted to describe the Usage Records: The first implements a simple, proprietary, schema. The second implements the GGF URWG schema, more powerful, but rather complex to manage.
A minimum set of information need to be available in the usage record to make grid accounting possible:
The User HLR Location (comes with the job JDL)
A valid User Proxy (Needed to secure the communication)
The CE ID (Needed to unambiguously assign the Usage)
The Grid Job Id (For grid accounting the LRMS job id is not enough)

21. Metering Infrastructure: GIANDUIA
GIANDUIA workflow
(Gianduia Is A Nice Distributed Usage-metering Infrastructure for Accounting) WN
HLR_LOCATION
USER_PROXY
CE_ID
LRMS_JOB_ID
GRID_JOB_ID
jobWrapper CE
LRMS
LOG
gianduia
conf
User HLR
(HLR_LOCATION)
GIANDUIA
gianduiottiBox

22. GIANDUIA: Installation & configuration
Gianduia needs to be installed on CE/WN Nodes, but soon only CE installation will be necessary. The RPMs needed from the INFNGrid Distribution are:
CE:
edg-wl-common-api_slc_3_0_ i486.rpm
edg-wl-common-utilities_slc_3_0_ i486.rpm
edg-wl-dgas-hlr-ATMClient_slc_3_0_ i486.rpm
WN:
edg-wl-dgas-wn-ceServiceClient_slc_3_0_ i486.rpm
The configuration files used by gianduia are:
/opt/edg/etc/edg-wl-dgas-ATMClient.conf
It is used by the daemon that pushes the UR to the user HLR.
/opt/edg/etc/edg-wl-dgas-gianduia.conf
It is used by the gianduia daemon, responsible for retrieving LRMS accounting log entries and appending them to the job UR file.

23. GIANDUIA: Installation & configuration
/opt/edg/etc/edg-wl-dgas-ATMClient.conf
res_acct_PA_id = “hlr02.to.infn.it:56567” -- the PA where the resource is registered
res_acct_bank_id = “hlr02.to.infn.it:56568” – the HLR where the resource is registered
economicAccounting = “no|yes” -- tells if the site manager requires job payment or not.
lockFileName = “/opt/…/edg-wl-dgas-cliend.lock” -- lock file
/opt/edg/etc/edg-wl-dgas-gianduia.conf
lockFileName = “/opt/…/edg-wl-dgas-gianduia.lock” – lockfile
chocolateBox = “/opt/…/dgasRawBox” – directory where the JW puts (via a tcp channel) the files needed by gianduia.
gianduiottiBox = “/opt/…/dgasURBox” – directory where gianduia stores the complete UR for each job.

24. References
Further information and documentation about DGAS can be found at: