EGEE is a project funded by the European Union under contract INFSO-RI-508833 DGAS Grid accounting L.Gaido on behalf of A.Guarise LCG Workshop November.

Slides:



Advertisements
Similar presentations
Overview of local security issues in Campus Grid environments Bruce Beckles University of Cambridge Computing Service.
Advertisements

Legacy code support for commercial production Grids G.Terstyanszky, T. Kiss, T. Delaitre, S. Winter School of Informatics, University.
Andrew McNab - EDG Access Control - 14 Jan 2003 EU DataGrid security with GSI and Globus Andrew McNab University of Manchester
CoreGRID Workpackage 5 Virtual Institute on Grid Information and Monitoring Services Authorizing Grid Resource Access and Consumption Erik Elmroth, Michał.
Security Issues in Physics Grid Computing Ian Stokes-Rees OeSC Security Working Group 14 June 2005.
GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.
EGEE is a project funded by the European Union under contract IST The way ahead Alistair Mills Grid Deployment Group
FESR Consorzio COMETA Grid Introduction and gLite Overview Corso di formazione sul Calcolo Parallelo ad Alte Prestazioni (edizione.
FP6−2004−Infrastructures−6-SSA E-infrastructure shared between Europe and Latin America Information System (IS) Valeria Ardizzone.
Frascati, October 9th, Accounting in DataGrid Initial Architecture Albert Werbrouck Frascati, October 9, 2001.
Frascati, October 5th, Accounting in DataGrid Preliminary Proposal and basis for discussion Stefano Barale Frascati, October.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
Summary of Accounting Discussion at the GDB in Bologna Dave Kant CCLRC, e-Science Centre.
A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.
SEE-GRID-SCI SEE-GRID-SCI Operations Procedures and Tools Antun Balaz Institute of Physics Belgrade, Serbia The SEE-GRID-SCI.
1 OSG Accounting Service Requirements Matteo Melani SLAC for the OSG Accounting Activity.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks David Kelsey RAL/STFC,
INFSO-RI Enabling Grids for E-sciencE OSG-LCG Interoperability Activity Author: Laurence Field (CERN)
Conference name Company name INFSOM-RI Speaker name The ETICS Job management architecture EGEE ‘08 Istanbul, September 25 th 2008 Valerio Venturi.
E-infrastructure shared between Europe and Latin America FP6−2004−Infrastructures−6-SSA gLite Information System Pedro Rausch IF.
1 Andrea Sciabà CERN Critical Services and Monitoring - CMS Andrea Sciabà WLCG Service Reliability Workshop 26 – 30 November, 2007.
LCG Accounting John Gordon Grid Deployment Board 13 th January 2004.
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract INFSO-RI Grid Accounting.
LCG workshop on Operational Issues CERN November, EGEE CIC activities (SA1) Accounting: current status
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Using GStat 2.0 for Information Validation.
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.
DTI Mission – 29 June LCG Security Ian Neilson LCG Security Officer Grid Deployment Group CERN.
Last update 21/01/ :05 LCG 1Maria Dimou- cern-it-gd Current LCG User Registration, VO management and Authorisation Procedures VOMS workshop
Last update 31/01/ :41 LCG 1 Maria Dimou Procedures for introducing new Virtual Organisations to EGEE NA4 Open Meeting Catania.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
INFSO-RI Enabling Grids for E-sciencE Policy management and fair share in gLite Andrea Guarise HPDC 2006 Paris June 19th, 2006.
EGEE is a project funded by the European Union under contract IST Information and Monitoring Services within a Grid R-GMA (Relational Grid.
EGEE is a project funded by the European Union under contract INFSO-RI Grid accounting with GridICE Sergio Fantinel, INFN LNL/PD LCG Workshop November.
Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore
Accounting in LCG Dave Kant CCLRC, e-Science Centre.
The National Grid Service User Accounting System Katie Weeks Science and Technology Facilities Council.
Accounting in DataGrid HLR software demo Andrea Guarise Milano, September 11, 2001.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Practical using WMProxy advanced job submission.
EGEE is a project funded by the European Union under contract IST Experiment Software Installation toolkit on LCG-2
INFSO-RI Enabling Grids for E-sciencE gLite Information System: R-GMA Tony Calanducci INFN Catania gLite tutorial at the EGEE User.
INFSO-RI Enabling Grids for E-sciencE DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006.
Open Science Grid OSG Accounting System Matteo Melani SLAC 9/28/05 Joint OSG and EGEE Operations Workshop.
INFSO-RI Enabling Grids for E-sciencE NPM Security Alistair K Phipps (NeSC) JRA4 Face To Face, CERN, Geneva.
EGEE is a project funded by the European Union under contract IST New VO Integration Fabio Hernandez ROC Managers Workshop,
EGEE-II INFSO-RI Enabling Grids for E-sciencE Overview of gLite, the EGEE middleware Mike Mineter Training Outreach Education National.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid is a Bazaar of Resource Providers and.
EGEE is a project funded by the European Union under contract IST Aims and organization of the Biomedical VO Yannick Legré CNRS/IN2P3 NA4/SA1.
1/3/2006 Grid operations: structure and organization Cristina Vistoli INFN CNAF – Bologna - Italy.
HLRmon Enrico Fattibene INFN-CNAF 1EGI-TF Lyon, France19-23 September 2011.
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
DGAS Accounting – toward national grid infrastructures HPDC workshop on Monitoring, Logging and Accounting, (MLA) in production Grids 10/06/2009, Munich.
DGAS A.Guarise April 19th, Athens
OGF PGI – EDGI Security Use Case and Requirements
DGAS Basic Management A.Guarise
Practical: The Information Systems
GGF OGSA-WG, Data Use Cases Peter Kunszt Middleware Activity, Data Management Cluster EGEE is a project funded by the European.
The EU DataGrid – Information and Monitoring Services
Accounting at the T1/T2 Sites of the Italian Grid
CRC exercises Not happy with the way the document for testbed architecture is progressing More a collection of contributions from the mware groups rather.
October 11th, CNAF GDB Meeting
Giuseppe Patania Nov, Martina Franca (Ta)‏
THE STEPS TO MANAGE THE GRID
EGEE Middleware: gLite Information Systems (IS)
DGAS Today and tomorrow
HLRmon accounting portal
Neopay Practical Guides #2 PSD2 (Should I be worried?)
Information System (BDII)
Presentation transcript:

EGEE is a project funded by the European Union under contract INFSO-RI DGAS Grid accounting L.Gaido on behalf of A.Guarise LCG Workshop November 2004, CERN

LCG Workshop, November DGAS: Overview DGAS (DataGrid Accounting System) is a grid infrastructure that implements two accounting approaches:  Resource Usage Accounting  Economic Accounting The first one means collecting and aggregating the resource usage records performed by grid users over grid resources. The second one means assigning a price to grid resources and a cost to the operations performed by users on the grid and charge for them. DGAS was designed as a a true ‘Grid’ accounting infrastructure. So it aims to be distributed, scalable, secure and fault tolerant and to provide the needed tools to VOs, resource managers and single users.

LCG Workshop, November DGAS: Simplified Workflow The user submits the job and concurrently authorises its accounting The WMS submits the job to the CE A metering system on the CE collects the Usage Records The metering system sends the usage records to the accounting server where the user is registered (HLR). The UR are signed with user credentials and encrypted. If Economic Accounting is needed a Price Authority is queried to retrieve the CE price. The price can either be set by the resource’s administrator or automatically computed using ad-hoc algorithms. If Economic Accounting is needed a cost is assigned to the job. The usage records are archived on the user’s HLR, and on the HLR where the CE is registered. The Economic Accounting can be turned on by the resource administrator when needed. In this case a cost will be assigned to the jobs executed on the CE. Therefore the user account will be charged and the resource (CE) one credited for the corresponding amount. If Economic Accounting is turned off only the Usage Records for the job will be reported to the Accounting System.

LCG Workshop, November DGAS: the VO’s point of view The basic accounting needs for a VO manager are:  Be able to track the grid usage for each user belonging to the Virtual Organization.  Be able to track the grid usage for group of users belonging to the Virtual Organization.  Be able to get statistics of the grid usage for the whole VO. The Economic Accounting needs:  Be able to assign individual quotas to users.  Be able to track grid money transactions. Users belonging to a Virtual Organisation need to be registered on a DGAS server as well. With this registration an account for the user is created on a DGAS accounting server, the so-called HLR (Home Location Register). This can be seen as a virtual bank account for that user on the grid.  The user accounts on an HLR can be created by importing the information from an LDAP VO server or manually inserted using command line tools.

LCG Workshop, November DGAS: the VO’s point of view The minimum granularity of the information archived within the DGAS RDMBS is the single grid job. The useful information available for each job is:  GridJobId.  DN of the user who submitted the job.  CEId of the resource where the job was executed.  the URL of the accounting server where the resource is registered; the usage record is archived for the resource administrator’s use.  Usage Records for the job (currently: CpuTime, WallClockTime, Physical and Virtual used Memory).  The cost (if any) charged for the job. The queries that the VO manager can perform on the RDBMS are:  List of the jobs submitted by a given user over a given period of time. The retrieved information consist in a list of records with the mentioned field.  Aggregation of the usage records for the job submitted by a single user, a group of users or by the entire VO (over a given period of time).The information available with these type of queries is: Total number of submitted jobs, used cpuTime, wallClockTime, Phisical and Virtual Memory, Total cost charged.

LCG Workshop, November DGAS: the Site Manager’s point of view The basic accounting needs for a Site Manager are:  Be able to track the usage of his/her resources performed by Grid Users.  Be able to get statistics of the usage of his/her resources or a group of them. The Economic Accounting needs:  Be able to assign prices to his/her resources. Also for the Grid resources, there is the need to register the resources on an HLR service.  The resource accounts on an HLR can be created by importing the information from an LDAP BDII server or manually inserted using command line tools.

LCG Workshop, November DGAS: the Site Manager’s point of view The minimum granularity is that of the single grid job. The available information for each job is:  GridJobId.  DN of the user who submitted the job.  CEId of the resource where the job ran.  the URL of the accounting server where the resource is registered; the usage record is archived for the resource administrator’s use  Usage Records of this job (currently CpuTime, WallClockTime, Physical and virtual memory used).  The cost (if any) charged for the job. The queries that the Site manager can perform on the RDBMS are:  List of the jobs submitted to a given CE over a given period of time.  Aggregated usage records for the job submitted to a single CE, a group of CE or the entire set of resources (over a given period of time). The information available with these type of queries are: the total number of submitted jobs, the cpuTime, the wallClockTime, the Phisical and Virtual Memory and the total cost charged in that period.

LCG Workshop, November DGAS: the User’s point of view The basic accounting needs for a User are:  Be able to track the usage of his/her jobs submitted to Grid Resources.  Be sure of the confidentiality of his/her work. The Economic Accounting needs:  Be able to check the cost of his/her jobs.  Be able to check the price of a Grid resource before submitting a job.  Be sure that the usage records charged to him originate from his/her jobs. Since users will not be granted direct access to the DGAS server where their account are registered, a remote interface is provided. This interface (available on a Grid User Interface) allows them to perform simple queries to the DGAS server:  Retrieve account information, including the aggregate of Usage Records consumed or the costs already charged.  Retrieve information for their jobs with detailed Usage Records (cpuTime, WallTime, memory…), the cost charged for the job, the CEId of the CE and the DGAS server where that CE is registered.

LCG Workshop, November DGAS: the User’s point of view For queries concerning information about a single job or a job for a given user it is important to grant a proper level of authorisation:  A typical user would only be granted access to his/her own accounting information, i.e. he/she will not be able to access other account’s information.  If needed an authorisation mechanism may grant special privileges to some users and allow them to retrieve information regarding other people or resources as well.

LCG Workshop, November Economic Accounting & Brokering DGAS implements the possibility to “virtually” charge the users for their work on the Grid: The cost of a job is a function of the resource usage and the price of the resource. Since the grid resources may have different prices it is possible that the same job submitted to different resources results in different costs charged to the user. A dedicated service to compute (and archive) prices for the Grid resources has been developed. This service is known as Price Authority. Apart from obvious reason (i.e. effectively charging for the grid usage), the economic accounting may also be used to make a better use of the grid by means of algorithms that automatically compute the resource prices; this implies “training” the users (or the broker on their behalf) to choose the cheapest resource. See

LCG Workshop, November DGAS For more information please contact Andrea Guarise