DGAS A.Guarise April 19th, Athens

Slides:



Advertisements
Similar presentations
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks MyProxy and EGEE Ludek Matyska and Daniel.
Advertisements

The Grid Job Monitoring Service Luděk Matyska et al. CESNET, z.s.p.o. Prague Czech Republic.
EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,
CoreGRID Workpackage 5 Virtual Institute on Grid Information and Monitoring Services Authorizing Grid Resource Access and Consumption Erik Elmroth, Michał.
GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.
Accounting Management IACT 918 April 2005 Glenn Bewsell/Gene Awyzio SITACS University of Wollongong.
SOS EGEE ‘06 GGF Security Auditing Service: Draft Architecture Brian Tierney Dan Gunter Lawrence Berkeley National Laboratory Marty Humphrey University.
FESR Consorzio COMETA Grid Introduction and gLite Overview Corso di formazione sul Calcolo Parallelo ad Alte Prestazioni (edizione.
Frascati, October 9th, Accounting in DataGrid Initial Architecture Albert Werbrouck Frascati, October 9, 2001.
Frascati, October 5th, Accounting in DataGrid Preliminary Proposal and basis for discussion Stefano Barale Frascati, October.
:: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: ::::: :: GridKA School 2009 MPI on Grids 1 MPI On Grids September 3 rd, GridKA School 2009.
A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.
Grid Technologies  Slide text. What is Grid?  The World Wide Web provides seamless access to information that is stored in many millions of different.
INFSO-RI Enabling Grids for E-sciencE Workload Management System Mike Mineter
Group 1 : Grid Computing Laboratory of Information Technology Supervisors: Alexander Ujhinsky Nikolay Kutovskiy.
US LHC OSG Technology Roadmap May 4-5th, 2005 Welcome. Thank you to Deirdre for the arrangements.
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract INFSO-RI Grid Accounting.
Glite. Architecture Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware Higher-Level Grid Services are supposed.
LCG workshop on Operational Issues CERN November, EGEE CIC activities (SA1) Accounting: current status
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.
EGEE is a project funded by the European Union under contract IST WS-Based Advance Reservation and Co-allocation Architecture Proposal T.Ferrari,
The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no Workflow repository, user.
Testing and integrating the WLCG/EGEE middleware in the LHC computing Simone Campana, Alessandro Di Girolamo, Elisa Lanciotti, Nicolò Magini, Patricia.
INFSO-RI Enabling Grids for E-sciencE Policy management and fair share in gLite Andrea Guarise HPDC 2006 Paris June 19th, 2006.
EGEE is a project funded by the European Union under contract IST Information and Monitoring Services within a Grid R-GMA (Relational Grid.
Grid Workload Management (WP 1) Massimo Sgaravatto INFN Padova.
EGEE is a project funded by the European Union under contract INFSO-RI Grid accounting with GridICE Sergio Fantinel, INFN LNL/PD LCG Workshop November.
The National Grid Service User Accounting System Katie Weeks Science and Technology Facilities Council.
Accounting in DataGrid HLR software demo Andrea Guarise Milano, September 11, 2001.
HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.
INFSO-RI Enabling Grids for E-sciencE DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006.
HLRmon Enrico Fattibene INFN-CNAF 1EGI-TF Lyon, France19-23 September 2011.
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI John Gordon EGI Virtualisation and Cloud Workshop Amsterdam 13 th May 2011.
Enabling Grids for E-sciencE Agreement-based Workload and Resource Management Tiziana Ferrari, Elisabetta Ronchieri Mar 30-31, 2006.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
DGAS Accounting – toward national grid infrastructures HPDC workshop on Monitoring, Logging and Accounting, (MLA) in production Grids 10/06/2009, Munich.
Panda Monitoring, Job Information, Performance Collection Kaushik De (UT Arlington), Torre Wenaus (BNL) OSG All Hands Consortium Meeting March 3, 2008.
SEE-GRID-SCI WRF-ARW model: Grid usage The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures.
EGEE is a project funded by the European Union under contract INFSO-RI DGAS Grid accounting L.Gaido on behalf of A.Guarise LCG Workshop November.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.
Accessing the VI-SEEM infrastructure
Practical using C++ WMProxy API advanced job submission
Grid2Win Porting of gLite middleware to Windows XP platform
Next Steps.
The EDG Testbed Deployment Details
OGF PGI – EDGI Security Use Case and Requirements
DGAS Basic Management A.Guarise
JRA3 Introduction Åke Edlund EGEE Security Head
How to connect your DG to EDGeS? Zoltán Farkas, MTA SZTAKI
and Alexandre Duarte OurGrid/EELA Interoperability Meeting
GGF OGSA-WG, Data Use Cases Peter Kunszt Middleware Activity, Data Management Cluster EGEE is a project funded by the European.
Accounting at the T1/T2 Sites of the Italian Grid
Grid2Win: Porting of gLite middleware to Windows XP platform
Grid2Win: Porting of gLite middleware to Windows XP platform
Workload Management System
October 11th, CNAF GDB Meeting
Giuseppe Patania Nov, Martina Franca (Ta)‏
Copyright © 2011 Pearson Education, Inc. Publishing as Pearson Addison-Wesley Chapter 2 Database System Concepts and Architecture.
Grid Security M. Jouvin / C. Loomis (LAL-Orsay)
EGEE Middleware: gLite Information Systems (IS)
DGAS Today and tomorrow
HLRmon accounting portal
Information System (BDII)
UR 1.0 Experiences OGF 24, Singapore.
gLite The EGEE Middleware Distribution
Information Services Claudio Cherubino INFN Catania Bologna
Presentation transcript:

DGAS A.Guarise April 19th, Athens www.eu-egee.org EGEE is a project funded by the European Union under contract INFSO-RI-508833

DataGrid Accounting System The Data Grid Accounting System was originally developed within the EU Datagrid Project and is now being maintained and re engineered within the EU EGEE Project. The Purpose of DGAS is to implement Resource Usage Metering, Accounting and Billing in a fully distributed Grid environment. It is conceived to be distributed, secure and extensible. The system is designed in order for Usage Metering, Accounting and Billing to be indipendent layers.

Usage Metering and Accounting The usage of Grid Resources by Grid Users is registered in appropriate servers, known as HLRs (Home Location Registers) where both users and resources are registered. In order to achieve scalability, there can be many independent HLRs. At least one HLR per VO are foreseen, although a finer granularity is possible. Each HLR keeps the records of every grid job executed by each of its registered users or resources, thus being able to furnish usage information with many granularity levels: Per user or resource, per group of users or resources, per VO.

Economic Accounting The DGAS software is concieved mainly as an Economic Accounting system. Within this approach, grid users and grid resources are treated as consumers and producers of goods like computational power or storage capacity. The consumers can use these goods paying an apropriate amount of Grid Credits to the producers. The idea behind this approach is that an Economic Accounting system may have a major advantage over a plain, bookkeeping style accounting system: it deploys a virtual economic market over the grid so that the natural behviour of this market can be used to drive the workload management system in balancing the resource load.

Example of economic accounting Check Economic Authorization VO 1 VO 2 WMS HLR 1 HLR 2 The user submits a job to the WMS (edg-job-submit) The WMS selects the CE (matchmaking) and contact the User HLR to check if the user has a valid account and enough credits to pay the job or if he hasn’t exceed his usage quotas. If this authorization is passed the job is submitted to the CE. Once the job is finished, the Usage Records collected by the metering system are sent to the User HLR where the records are stored and, if economic accounting is required by the resource, the job cost is computed. Then the User HLR contacts the Resource HLR and instantiates a transaction. The user account is debited, the resource account is credited. In any moment the HLR administrator of VO2 can redistribute the resource earnings among the VO users. CE CE

HLR: Available Information For each job executed on a CE, the following information are available querying the HLR: GridJobID User Certificate Subject Local user Id Grid Unique Computing Element Id User VO Cpu Time Wall clock time Physical memory Virtual Memory Job assigned nodes (cpus) In principle all information available to the LRMS can be published in the HLR server.

HLR: Standard queries An administrator that has local access to the HLR server can retrieve many information about the status of the accounts: For example: edg-wl-dgas-hlr-queryUser –Ua Retrieves the information about the account of all the registered users. edg-wl-dgas-hlr-queryUser –UaT Retrieves also all the transactions performed for each user. edg-wl-dgas-hlr-queryUser –U –e guariseATto.infn.it |guarise|andrea.guariseATto.infn.it|Andrea Guarise|/C=IT/O=INFN…|torino|infngrid|10000|0|0|wall_t ime=0,cpu_time=0,job_number=0| Retrieves only information about the user with the given email address Resource Queries Similar queries are available for resources accounts via the command: edg-wl-dgas-hlr-queryResource

HLR: Advanced queries It is also possible to perform more complex queries to get, for example, statistics of the usage of a set of resources, or to have a snapshot of the amount of work performed by users dividing them by groups or VOs. Statistics of all the user jobs accounted by an HLR: edg-wl-dgas-hlrAdvancedQuery –Q userAggregate Jobs| totCost| totCpuTime|totWallTime|totMem(MB)|totVMem(MB)| 15| 0| 21550| 48551| 22| 25| Statistics of all the user jobs accounted for the group ‘torino’: edg-wl-dgas-hlrAdvancedQuery –Q userAggregate –g torino 5| 0| 9201| 15539| 7| 8| Aggregates are available for resources, groups, VOs. It is also possible to have job lists and specify a time interval for the queries.

Metering Infrastructure: GIANDUIA DGAS furnishes the APIs that can be used by each site to develop its own metering infrastructure. So that a site can freely decide how to measure its own Usage Records. Two APIs are available. These both provide a C++ library and a CLI tool. Their purpose is to push the information to the User HLR once the Usage Record for a job is available. These two API only differs in terms of the schema adopted to describe the Usage Records: The first implements a simple, proprietary, schema. The second implements the GGF URWG schema, more powerful, but rather complex to manage. A minimum set of information need to be available in the usage record to make grid accounting possible: The User HLR Location (comes with the job JDL) A valid User Proxy (Needed to secure the communication) The CE ID (Needed to unambiguously assign the Usage) The Grid Job Id (For grid accounting the LRMS job id is not enough)

Metering Infrastructure: GIANDUIA GIANDUIA workflow (Gianduia Is A Nice Distributed Usage-metering Infrastructure for Accounting) WN HLR_LOCATION USER_PROXY CE_ID LRMS_JOB_ID GRID_JOB_ID jobWrapper CE LRMS LOG gianduia conf User HLR (HLR_LOCATION) GIANDUIA gianduiottiBox

References Further information and documentation about DGAS can be found at: http://www.to.infn.it/grid/accounting