LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006.

Slides:



Advertisements
Similar presentations
LCG WLCG Operations John Gordon, CCLRC GridPP18 Glasgow 21 March 2007.
Advertisements

1 User Analysis Workgroup Update  All four experiments gave input by mid December  ALICE by document and links  Very independent.
Accounting in LCG Dave Kant & John Gordon CCLRC, e-Science Centre.
Accounting Update Dave Kant Grid Deployment Board Nov 2007.
Accounting in EGEE … and beyond John Gordon and David Kant CCLRC, e-Science Centre.
08/11/908 WP2 e-NMR Grid deployment and operations Technical Review in Brussels, 8 th of December 2008 Marco Verlato.
Storage Accounting John Gordon, STFC GDB June 2012.
Summary of Accounting Discussion at the GDB in Bologna Dave Kant CCLRC, e-Science Centre.
A.Guarise – F.Rosso 1 Enabling Grids for E-sciencE INFSO-RI Comprehensive Accounting Views on large computing farms. Andrea Guarise & Felice Rosso.
Monitoring in EGEE EGEE/SEEGRID Summer School 2006, Budapest Judit Novak, CERN Piotr Nyczyk, CERN Valentin Vidic, CERN/RBI.
JSPG: User-level Accounting Data Policy David Kelsey, CCLRC/RAL, UK LCG GDB Meeting, Rome, 5 April 2006.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Security and Job Management.
GridPP Deployment & Operations GridPP has built a Computing Grid of more than 5,000 CPUs, with equipment based at many of the particle physics centres.
Enabling Grids for E-sciencE System Analysis Working Group and Experiment Dashboard Julia Andreeva CERN Grid Operations Workshop – June, Stockholm.
Accounting in LCG Dave Kant CCLRC, e-Science Centre.
Some Title from the Headrer and Footer, 19 April Overview Requirements Current Design Work in Progress.
GDB March User-Level, VOMS Groups and Roles Dave Kant CCLRC, e-Science Centre.
WLCG Grid Deployment Board, CERN 11 June 2008 Storage Update Flavia Donno CERN/IT.
LCG Storage Accounting John Gordon CCLRC – RAL LCG Grid Deployment Board September 2006.
LCG Accounting John Gordon Grid Deployment Board 13 th January 2004.
INFSO-RI Enabling Grids for E-sciencE EGEE is a project funded by the European Union under contract INFSO-RI Grid Accounting.
Storage Accounting John Gordon, STFC GDB March 2013.
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
EMI INFSO-RI Accounting John Gordon (STFC) APEL PT Leader.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Using GStat 2.0 for Information Validation.
Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi.
Accounting Update John Gordon and Stuart Pullinger January 2014 GDB.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks John Gordon SA1 Face to Face CERN, June.
APEL Cloud Accounting Status and Plans APEL Team John Gordon.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks APEL CPU Accounting in the EGEE/WLCG infrastructure.
Accounting For Multicore Jobs John Gordon, STFC, UK Scientific Computing Department, APEL Team MB 17 th March 2015.
LCG Accounting Update John Gordon, CCLRC-RAL WLCG Workshop, CERN 24/1/2007 LCG.
LCG User Level Accounting John Gordon CCLRC-RAL LCG Grid Deployment Board October 2006.
GridView - A Monitoring & Visualization tool for LCG Rajesh Kalmady, Phool Chand, Kislay Bhatt, D. D. Sonvane, Kumar Vaibhav B.A.R.C. BARC-CERN/LCG Meeting.
Accounting in LCG/EGEE Can We Gauge Grid Usage via RBs? Dave Kant CCLRC, e-Science Centre.
LCG Accounting/Reporting John Gordon, STFC MB November 9 th 2011.
Accounting in LCG Dave Kant CCLRC, e-Science Centre.
HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.
INFN GRID Production Infrastructure Status and operation organization Cristina Vistoli Cnaf GDB Bologna, 11/10/2005.
INFSO-RI Enabling Grids for E-sciencE DGAS, current status & plans Andrea Guarise EGEE JRA1 All Hands Meeting Plzen July 11th, 2006.
John Gordon Grid Accounting Update John Gordon (for Dave Kant) CCLRC e-Science Centre, UK LCG Grid Deployment Board NIKHEF, October.
Accounting in LCG Dave Kant CCLRC, e-Science Centre.
EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Accounting Portal Pablo Rey, Javier Lopez.
1/3/2006 Grid operations: structure and organization Cristina Vistoli INFN CNAF – Bologna - Italy.
TIFR, Mumbai, India, Feb 13-17, GridView - A Grid Monitoring and Visualization Tool Rajesh Kalmady, Digamber Sonvane, Kislay Bhatt, Phool Chand,
HLRmon Enrico Fattibene INFN-CNAF 1EGI-TF Lyon, France19-23 September 2011.
APEL Architecture Alison Packer. Overview Grid jobs accounting tool APEL Client software - installed in sites (CEs, gLite- APEL node) APEL Server accepts.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Storage Accounting John Gordon, STFC OMB August 2013.
Using HLRmon for advanced visualization of resource usage Enrico Fattibene INFN - CNAF ISCG 2010 – Taipei March 11 th, 2010.
LCG Accounting Update John Gordon, CCLRC-RAL 10/1/2007.
Enabling Grids for E-sciencE INFN Workshop – May 7-11 Rimini 1 Grid Accounting Status at INFN Riccardo Brunetti INFN-TORINO.
John Gordon EMI TF and EGI CF March 2012 Accounting Workshop.
Storage Accounting John Gordon STFC GDB, Lyon 6 th April2011 GDB January 2012.
Enabling Grids for E-sciencE Claudio Cherubino INFN DGAS (Distributed Grid Accounting System)
Accounting Update Dave Kant, John Gordon RAL Javier Lopez, Pablo Rey Mayo CESGA.
GDB July APEL Accounting Summary Dave Kant Rutherford Appleton Laboratory.
15-Jun-04D.P.Kelsey, LCG-GDB-Security1 LCG/GDB Security Update (Report from the LCG Security Group) CERN 15 June 2004 David Kelsey CCLRC/RAL, UK
GridPP37, Ambleside Adrian Coveney (STFC)
John Gordon STFC OMB 26 July 2011
WLCG Resources Reporting
Accounting Portal Pablo Rey, Javier Lopez (CESGA)
Accounting at the T1/T2 Sites of the Italian Grid
Update on Plan for KISTI-GSDC
Raw Wallclock in APEL John Gordon, STFC-RAL
Proposal for obtaining installed capacity
Cristina del Cano Novales STFC - RAL
New Types of Accounting Beyond CPU
HLRmon accounting portal
IPv6 update Duncan Rand Imperial College London
User Accounting Integration Spreading the Net.
Presentation transcript:

LCG WLCG Accounting: Update, Issues, and Plans John Gordon RAL Management Board, 19 December 2006

last update 17/02/ :53 LCG – MB 19Dec06 Overview  Update  WLCG Reporting  APEL Portal,  APEL Sensors  Other Sensors  DGAS  User Level Accounting  Storage Accounting  Future work  Issues  Going Forward

last update 17/02/ :53 LCG – MB 19Dec06 Update  Much of this section was reported to the GDB in December resId=1&materialId=slides&confId=a resId=1&materialId=slides&confId=a057712

last update 17/02/ :53 LCG – MB 19Dec06 WLCG Reporting  WLCG official accounting is currently done manually, only by Tier1s.  By the end of 2006 the T0 and all existing Tier1s, except NorduGrid, are reporting monthly  per VO and per site on normalised CPU time, wallclock time, disk allocated and used, and tape used.  The reports are consolidated, compared to MoU pledges (with some efficiencies assumed), and published via the LCG Bulletin

last update 17/02/ :53 LCG – MB 19Dec06 APEL Portal  The APEL Accounting Portal at RAL has been storing cpu accounting results for WLCG for more than a year.  CESGA has taken over development of the various reports so the EGEE view is now the definitive one  CESGA also monitor which sites are publishing and raise trouble tickets in GGUS when a site fails to publish for 30 days.  Manual checking of published results has revealed some gaps in data. SAM tests for APEL are under development to compare the results stored locally in the RGMA MON box with the central data.

last update 17/02/ :53 LCG – MB 19Dec06 Accounting Portal

last update 17/02/ :53 LCG – MB 19Dec06 APEL Sensors  APEL2 was released in production in gLite Update 10 last Monday. Main new features are:  More reliable publisher which can handle tcp connection timeouts with the archiver.  Encryption of UserDN using a 1024-bit RSA key, ready for user-level accounting  Support for the Blah accounting file on the gLiteCE  gLite 3.0.2u10 also contains patches to the gLite CE to correct erros in the Blah accounting log. The APEL sensors now work correctlky with the gLite CE.

last update 17/02/ :53 LCG – MB 19Dec06 Other Sensors  Not all sites report cpu accounting via the APEL sensors. Some interrogate their own site accounting databases and publish directly using R-GMA. Advice on how to do this is available at support.ac.uk/gridsite/accounting/faq.htmlhttp://goc.grid- support.ac.uk/gridsite/accounting/faq.html DGAS  INFN uses DGAS to collect accounting information and stores it in its own repositories (HLR) for each site. A new development DGAS2APEL, takes information from the site HLR and publishes it via RGMA to the APEL repository. This is deployed in production at 3 INFN sites usage records are being successfully transferred to the central APEL repository.

last update 17/02/ :53 LCG – MB 19Dec06 User Level Accounting  APEL2 encrypts the user DN in the Usage Record. When a site switches on external user publishing the encrypted DN is sent to the central repository where it is decrypted to allow aggregation and then re- encrypted.  A prototype portal has been developed to show information to the roles identified (see GDB talks from October and December). No userDN information will be made available until the relevant policy documents are in place, approved, and signed by the relevant individuals.

last update 17/02/ :53 LCG – MB 19Dec06 User-Level Accounting  Development of a prototype User-level reporting display based on the “Five Actors” described:  VO Resource Manager  VO Member  User  Site Administrator  GOC Developer.  Screen shots demonstrate this in action

last update 17/02/ :53 LCG – MB 19Dec06 VO-Resource Manager  Table shows CPU, WCT and Job Eff. of the Top 10 Anonymised Users  This example shows that the largest WCT User has a job efficiency of 10%…clearly the VO Manager may wish to contact this person?

last update 17/02/ :53 LCG – MB 19Dec06 VO-Resource Manager  Cumulative CPU of the Top 10  Relative Share of Top 10 compared to the VO Total

last update 17/02/ :53 LCG – MB 19Dec06 Site Admin View  The Site Administrator can view usage of anonymous grid users who executed jobs at the site.

last update 17/02/ :53 LCG – MB 19Dec06 User  Each Grid User can interrogate their own accounting data  Tables showing what they did and when  Number of Jobs, CPU and WCT per Month (per VO)  Average Job Efficiency per VO  Accumulative Njobs, CPU and WCT per VO  The sites which executed the jobs, and when they were done The following table shows the distribution of the Total number of Your Jobs grouped by VO and DATE

last update 17/02/ :53 LCG – MB 19Dec06 Storage Accounting  GridPP in the UK has developed storage accounting using values published in GLUE and harvesting them from the BDII.  The results are published and summarised in the same way as cpu and some example visualisations (by CESGA) shown using data from GridPP sites.  A roadmap exists for further development of the portal   This storage accounting has recently been extended to all EGEE sites. OSG are developing their own solution.

last update 17/02/ :53 LCG – MB 19Dec06 Storage Accounting Display  Visualisation of Storage Used per VO for Disk and Tape   Select Resources via a Tree  Select time interval (last year, last month, last week, last day)

last update 17/02/ :53 LCG – MB 19Dec06 Storage Accounting Display  Looking at data for RAL-LCG2  Storage units are 1TB = 10^6 MB  Tape Used + Disk Used = Total Sensor Drop Outs have been fixed Total Used Storage (TB) Tape Used Disk Used

last update 17/02/ :53 LCG – MB 19Dec06 Future work  OGF UR, RUS, WS interfaces - expand

last update 17/02/ :53 LCG – MB 19Dec06 Issues: CPU Reporting  NorduGrid is not reporting any CPU use.  Reporting should be extended to Tier2s.  Correctness of data needs checking  Completeness of data needs checking.  CPU versus wallclock.  How many accounting solutions do we need?  Use of VOMS.  Local versus Grid.

last update 17/02/ :53 LCG – MB 19Dec06 Issues: User Level Accounting We need  Sites to deploy gLite 3.0.2u10 and start publishing encrypted DNs.  The relevant policies to be formulated and approved  Feedback on the reporting suggested at December GDB.

last update 17/02/ :53 LCG – MB 19Dec06 Issues: Storage Accounting  GLUE1.3 introduces new SE reporting concepts.  Are they sufficient for storage accounting?  Can they be implemented across all SEs?  Can we ever account shared space on SEs correctly?

last update 17/02/ :53 LCG – MB 19Dec06 Going Forward I suggest the priorities are:  Introduce T2 reporting using APEL for cpu (now) and for storage (hopefully soon)  Sites to check the data being published for storage.  Rollout DGAS2APEL across INFN so that information from Italy is collected centrally.  Check results from Storage Accounting and develop information providers further.  Persuade NDGF to start publishing Tier1 accounting  Rollout user level accounting