INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006.

Slides:



Advertisements
Similar presentations
EGEE-II INFSO-RI Enabling Grids for E-sciencE The gLite middleware distribution OSG Consortium Meeting Seattle,
Advertisements

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Consistency of Accounting Information with.
Accounting Update Dave Kant Grid Deployment Board Nov 2007.
08/11/908 WP2 e-NMR Grid deployment and operations Technical Review in Brussels, 8 th of December 2008 Marco Verlato.
The National Grid Service User Accounting System Katie Weeks Science and Technology Facilities Council.
Introduction on R-GMA Shi Jingyan Computing Center IHEP.
INFSO-RI Enabling Grids for E-sciencE gLite Data Management Services - Overview Mike Mineter National e-Science Centre, Edinburgh.
FESR Consorzio COMETA Grid Introduction and gLite Overview Corso di formazione sul Calcolo Parallelo ad Alte Prestazioni (edizione.
INFSO-RI Enabling Grids for E-sciencE Logging and Bookkeeping and Job Provenance Services Ludek Matyska (CESNET) on behalf of the.
Summary of Accounting Discussion at the GDB in Bologna Dave Kant CCLRC, e-Science Centre.
A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Information System on gLite middleware Vincent.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Security and Job Management.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks WMSMonitor: a tool to monitor gLite WMS/LB.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks David Kelsey RAL/STFC,
Accounting in LCG Dave Kant CCLRC, e-Science Centre.
INFSO-RI Enabling Grids for E-sciencE SA1 and gLite: Test, Certification and Pre-production Nick Thackray SA1, CERN.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Next steps with EGEE EGEE training community.
GDB March User-Level, VOMS Groups and Roles Dave Kant CCLRC, e-Science Centre.
INFSO-RI Enabling Grids for E-sciencE OSG-LCG Interoperability Activity Author: Laurence Field (CERN)
EGEE User Forum Data Management session Development of gLite Web Service Based Security Components for the ATLAS Metadata Interface Thomas Doherty GridPP.
US LHC OSG Technology Roadmap May 4-5th, 2005 Welcome. Thank you to Deirdre for the arrangements.
LCG Accounting John Gordon Grid Deployment Board 13 th January 2004.
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract INFSO-RI Grid Accounting.
INFSO-RI Enabling Grids for E-sciencE G-PBox Auth meeting 13/9/2005 Presenter: Vincenzo Ciaschini.
INFSO-RI Enabling Grids for E-sciencE GridICE: Grid and Fabric Monitoring Integrated for gLite-based Sites Sergio Fantinel INFN.
LCG workshop on Operational Issues CERN November, EGEE CIC activities (SA1) Accounting: current status
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Using GStat 2.0 for Information Validation.
INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.
DTI Mission – 29 June LCG Security Ian Neilson LCG Security Officer Grid Deployment Group CERN.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid2Win : gLite for Microsoft Windows Roberto.
INFSO-RI Enabling Grids for E-sciencE /10/20054th EGEE Conference - Pisa1 gLite Configuration and Deployment Models JRA1 Integration.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Communication tools between Grid Virtual.
LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Storage Accounting for Grid Environments Fabio Scibilia INFN - Catania.
LCG Accounting Update John Gordon, CCLRC-RAL WLCG Workshop, CERN 24/1/2007 LCG.
INFSO-RI Enabling Grids for E-sciencE Policy management and fair share in gLite Andrea Guarise HPDC 2006 Paris June 19th, 2006.
LCG User Level Accounting John Gordon CCLRC-RAL LCG Grid Deployment Board October 2006.
Enabling Grids for E-sciencE EGEE Applications Registry Current status & latest developments Marios Chatziangelou.
Accounting in LCG/EGEE Can We Gauge Grid Usage via RBs? Dave Kant CCLRC, e-Science Centre.
EGEE is a project funded by the European Union under contract INFSO-RI Grid accounting with GridICE Sergio Fantinel, INFN LNL/PD LCG Workshop November.
The National Grid Service User Accounting System Katie Weeks Science and Technology Facilities Council.
HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.
INFN GRID Production Infrastructure Status and operation organization Cristina Vistoli Cnaf GDB Bologna, 11/10/2005.
INFSO-RI Enabling Grids for E-sciencE gLite Information System: R-GMA Tony Calanducci INFN Catania gLite tutorial at the EGEE User.
1 GRID – Stato dell’arte Alessandro Paolini (INFN-CNAF) Workshop della Commissione Calcolo e Reti dell'INFN Laboratori Nazionali del Gran Sasso 10 – 13.
EGEE-II INFSO-RI Enabling Grids for E-sciencE Overview of gLite, the EGEE middleware Mike Mineter Training Outreach Education National.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Accounting Portal Pablo Rey, Javier Lopez.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Grid is a Bazaar of Resource Providers and.
HLRmon Enrico Fattibene INFN-CNAF 1EGI-TF Lyon, France19-23 September 2011.
DGAS Distributed Grid Accounting System INFN Workshop /05/1009, Palau Giuseppe Patania Andrea Guarise 6/18/20161.
Using HLRmon for advanced visualization of resource usage Enrico Fattibene INFN - CNAF ISCG 2010 – Taipei March 11 th, 2010.
LCG Accounting Update John Gordon, CCLRC-RAL 10/1/2007.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Operations automation team presentazione.
Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
The EPIKH Project (Exchange Programme to advance e-Infrastructure Know-How) gLite Grid Introduction Salma Saber Electronic.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
DGAS Accounting – toward national grid infrastructures HPDC workshop on Monitoring, Logging and Accounting, (MLA) in production Grids 10/06/2009, Munich.
Accounting Update Dave Kant, John Gordon RAL Javier Lopez, Pablo Rey Mayo CESGA.
GDB July APEL Accounting Summary Dave Kant Rutherford Appleton Laboratory.
EGEE is a project funded by the European Union under contract INFSO-RI DGAS Grid accounting L.Gaido on behalf of A.Guarise LCG Workshop November.
DGAS A.Guarise April 19th, Athens
Accounting at the T1/T2 Sites of the Italian Grid
October 11th, CNAF GDB Meeting
Giuseppe Patania Nov, Martina Franca (Ta)‏
Leigh Grundhoefer Indiana University
Presentation transcript:

INFSO-RI Enabling Grids for E-sciencE DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006

Enabling Grids for E-sciencE INFSO-RI A Grid accounting infrastructure A generic grid accounting infrastructure should cover the requirements of tree type of customers: Sites, VOs and their users, grid administrators. While fulfilling the requirements it must also be secure, flexible and scalable. Site accounting database Site layer Usage MeteringResource’s layer VO accounting database Users/VO layer grid accounting databasegrid layer Web reports: -Aggregate info -Sites load -Anonymised statistics -Aggregate VO info -Group/Role info -Detailed User info -Job level info -Aggregate site info -VO (with role/group) usage on the site. -Detailed Resource Usage info -Job level info

Enabling Grids for E-sciencE INFSO-RI Metering(1) DGAS provides its own metering system, which is used to retrieve usage records for jobs running on the Computing Elements. It is however always possible to replace DGAS legacy metering system with other metering system using the provided APIs to send Usage Records to the DGAS accounting database (known as HLR). DGAS metering system is logically composed by two distinct set of sensors: –Gianduia: retrieves Usage Records for grid-jobs (With user DN, gridJobId, userVO, VOMS FQAN). –DGAS-outOfBand: retrieves Usage Records for locally submitted jobs or for grid jobs that, for whatever reason, where impossible to retrieve via Gianduia (in this case the usage record will not contain the user certificate’s DN and the grid Job Id, but does contain the user VO).

Enabling Grids for E-sciencE INFSO-RI Metering(2) Recent version of the Blahpd-based gLite Computing Elements, and future versions of the LCG GRAM based Computing Elements will be able to write an accounting log files for Grid Jobs. Such log file will contain, for every Grid Job, the following information: Timestamp, userDN, userFQAN, ceId, jobId, lrmsId, localUser. These information are useful to map the local lrmsId of the job with its grid related information and to the grid identity of the user who submitted the job. For these Computing Element a new, simpler, metering sensor is provided, it is known as dgasURCollector. It can retrieve Usage Records for both the Grid Jobs and the local ones. On such computing elements this metering sensor replaces both Gianduia and outOfBand.

Enabling Grids for E-sciencE INFSO-RI Usage record Grid Job, GianduiaGridJob, URCollectorLocal or Grid, outOfBand gridJobIdYES NO startDateYES endDateYES userGridIdYES NO resourceGridIdYES NO vomsFQANYES NO Virtual OrganisationYES lrmsIdYES localUserIdYES localGroupYES cpuTimeYES wallClockTimeYES Phisycal MemoryYES Virtual MemoryYES SpecInt2KYES NO SpecFloat2KYES NO CostYES NO lrmsHost + lrmsQueueYES

Enabling Grids for E-sciencE INFSO-RI Site HLR The site HLR is a service designed to manage a set of ‘accounts’ for the Computing Elements of a given computing site. For each job executed on a Computing Element (or a on local queue), the Usage Record for that job is stored on the RDBMS of the site HLR. Each site HLR can: Receive Usage Records from the registered Computing Elements. Answer to site manager queries such as: –Datailed job list queries (with many search keys: per user, VO, FQAN,CEId…) –Aggregate usage reports, such as per hour, day, month…, with flexible search criteria. Optionally forward Usage Records to APEL database. Optionally forward Usage Records to a VO specific HLR. Site HLR Site layer Usage MeteringResource’s layer -Aggregate site info -VO (with role/group) usage on the site. -Detailed Resource Usage info -Job level info GOC

Enabling Grids for E-sciencE INFSO-RI VO HLR The VO HLR is a service designed to manage a set of ‘accounts’ for the Users belonging to a Virtual Organisation. The VO HLR receives a copy of the Usage Records for the jobs executed on computing sites that authorise the site HLR to forward the records to the User/VO HLR Each VO HLR can be queried to obtain information about user Jobs (both detailed and aggregate views). Unprivileged users are authorised to acces just to their own records, while authorised people (VO managers), can access to broader set of records. Site HLR Site layer Usage MeteringResource’s layer VO HLR Users/VO layer -Aggregate VO info -Group/Role info -Detailed User info -Job level info -Aggregate site info -VO (with role/group) usage on the site. -Detailed Resource Usage info -Job level info

Enabling Grids for E-sciencE INFSO-RI Deployment DGAS (tag 3_1_3, form gLite 3.1 branch) is currently deployed in INFNGrid and it is the official INFNGrid accouting tool, used as the source of information for the accounting monthly reports to the GOC. Actual INFNGrid deployment uses just the metering system (for both LCG2.7 and gLite3.0 Computing Elements), and sites HLR. The sites that currently have a working HLR are: –INFN-T1* –INFN-Catania –INFN-Padova –INFN-Frascati –INFN-Legnaro –INFN-Milano –INFN-Napoli –INFN-Pisa –INFN-Roma1 –INFN-Torino * (It uses the already existing INFNGrid T1 ad hoc monitoring&metering system known as RedEye, which communicates the job Usage Records to the T1 HLR using the DGAS APIs.)

Enabling Grids for E-sciencE INFSO-RI dgas2apel DGAS Usage Records tables can be converted into APEL LCGRecords table structure. This is done with a translation tool nemed ‘dgas2apel’ that simply translates DGAS Usage Records into a local (to the HLR) instance of the LCGRecords table. This instance of the table can then be sent to the GOC APEL database. This can be performed using the already existing APEL producer that uses RGMA to send the sites LCGRecords table to the GOC. The translation tool is already tested and working. Test activities on the transport using APEL producer should start soon. The aim is to send INFN Grid production records directly to the GOC using this mechanism.

Enabling Grids for E-sciencE INFSO-RI Next steps Currently we’re testing the candidate for the next INFNGrid DGAS release. It shall be a minor update comprehending some improvements in querying functionalities, error reporting and system management. Next step will be to port DGAS to gLite3.0 (it is currently in the 3.1 branch) for integration and certification. The foreseen roadmap will be to start with the adoption of the metering infrastructure and the site HLRs. VO level HLRs, altough being already available, will not be adopted from the beginning but just in a second iteration. Longer term development efforts foreseen are: –Full integration with G-PBox to implement grid usage quotas. –Storage Accounting (currently design and analysis is ongoing, in collaboration with Tri-GRID). –Improved and simplifyied metering system, with full adoption of GGF-URWG Usage Record standard, which is currently being finalised (In collaboration with OSG and APEL). –RUS interface to DGAS (with OMII-EU)

Enabling Grids for E-sciencE INFSO-RI Conclusion. Further information about DGAS can be found at: –DGAS website: –DGAS User Guides: &version= &version=1