Recent improvements in HLRmon, an accounting portal suitable for national Grids Enrico Fattibene (speaker), Andrea Cristofori, Luciano Gaido, Paolo Veronesi INFN - Italy hlrmon lists.infn.it WORLDCOMP GCA’11 – Las Vegas, July 18 th, 2011

Outline gLite Grid infrastructure Grid accounting issues Accounting layout in the Italian Grid Infrastructure (IGI) HLRmon overview HLRmon main report section Storage accounting prototype Outlook and conclusions 2 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

gLite Grid infrastructure HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th CE WMS BDII Accounting Service Grid site A SE CE Grid site B SE The Computing Element (CE) is the service representing a computing resource The Storage Element (SE) is the storage unit at Grid site level CEs and SEs send information about the available resources and their status to the Berkley Database Information Index (BDII) Grid user can submit jobs:  through the Workload Management System (WMS) that retrieves information on the available resources from the BDII and sends the job to the appropriate CE  directly to the CE The accounting service is responsible for the collection of computing and storage information Grid user

Grid accounting issues Resource usage accounting in Grid is  challenging due to the distributed nature of the infrastructure and the heterogeneous nature of the resources accessed  necessary for its stakeholders, such as site administrators, Virtual Organization (VO) managers, Grid managers, etc. Users need to access accounting information at different levels of aggregation  an exhaustive and easy to use graphical interface is fundamental  many views are needed for different stakeholders HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th

Accounting layout in IGI 5 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011 Distributed Grid Accounting System (DGAS) sensors collect accounting information at Grid site level Site data are sent to site or multisite Home Location Registers (HLRs) A top-level HLR receives Usage Records (URs) from all the Grid HLRs HLRmon retrieves data from the top level HLR and presents them through a web interface. Aggregated data are also sent to the European Grid Infrastructure (EGI) Accounting Portal

HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th HLRmon introduction Developed and funded in the framework of the INFN Grid project Deployed in the Italian production Grid since 2007 Choosen as the accounting portal by gLite-based National Grid Infrastructures (NGIs) and projects  IGI (Italy)  HellasGrid (Greece)  D-Grid (Germany)  W-eNMR project  Cybersar project

HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th HLRmon structure HLRmon collector retrieves URs from the desired DGAS HLR server Data are aggregated, stored in a local MySQL database and presented through a php-based web portal  charts highly customizable by the users  tabular reports available also in xls format A x.509 personal certificate is needed to access the portal The web interface includes:  a main report section providing usage information aggregated in many different ways  a section reporting on resource usage by VOs, aggregated by the corresponding scientific disciplines  a section dedicated to the Worldwide LHC Computing Grid (WLCG) sites and VOs (alice, atlas, cms, lhcb)  a restricted section providing accounting per-user information, visible only by registered and authorized users

8 Main section Different sets of charts  Number of executed jobs  CPU Time (CPT) and Wall Clock Time (WCT) row and normalized, expressed in time (days) othe normalization is based on the KiloSpecInt2000 benchmark  Job efficiency in terms of (CPT / WCT) ratio Different keys for data aggregation and selection  Grid site  VO, group and role ogroups and roles can be granted to perform specific tasks within a VO  Certification Authority (CA) and Registration Authority (RA) othat released the x.509 personal certificate used by VO member to authenticate him/herself to the Grid  Job type (Grid or local) HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

Main section overview HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th

10 Main section charts Normalized CPT expressed in KSI2K-day per VO and per VO per day Job efficiency (CPT/WCT) per site per day and per VO per day Normalized CPT expressed in KSI2K-day per site and per site per day 10 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

VO / groups / roles menu VO Complete list of VOs with at least 1 job into the HLRmon DB VO groups For each VO, the VO groups with at least 1 job are listed VO roles For each VO group, the VO roles with at least 1 job are listed 11 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

VO groups / roles charts Job number per VO group Job number per VO group per day Job number per VO role Job number per VO role per day 12 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

CA / RA menu CA The CA list depends on the HLRmon server configuration. RA The IGI HLRmon server is configured to aggregate data per RA. For the INFN CA, the RA information is extracted from the L= field of the DN. 13 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

CA / RA charts Job number per CA Job number per CA per day Job number per RA Job number per RA per day 14 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

Storage accounting approach Storage accounting system integrated in DGAS under development Non standard schema has been used to generate storage URs  no standard schema exists  we join the Open Grid Forum (OGF) working group on the storage UR definition The information on the used space is collected from the BDII once a day  specific sensors can be developed to retrieve more accurate data omaking the system independent from the BDII Storage URs are stored on the HLR server as the computing URs HLRmon is able to retrieve storage information directly from the desired HLR HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th

Storage report HLRmon shows the daily disk consumption in terms of used space Data are aggregated per site, SE and Storage Area (SA) Used space will be compared to the total available space (at SA, SE and site level) The plotting of these data allowed to detect wrong values published in the BDII  sites have been notified in order to correct the values HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th

Storage charts Used space per site Used space per site per day Used space per SA Used per SA per day 17 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011

Outlook The storage accounting prototype will evolve in order to adopt a standard UR when available  we will follow this discussion into the OGF UR working group HLRmon will be made independent from the underlying accounting system  accounting services adopted in EGI are evolving to transport their data through ActiveMQ messaging system  HLRmon data collector will be improved to retrieve accounting data from a common interface, where EGI accounting services will publish their URs HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th

Conclusions HLRmon  is developed to meet requirements from different kind of users  is successfully deployed as accounting portal for IGI and other Grid projects/NGIs  provides a complete section on computing accounting reports  offers a fine granularity of data aggregation  is close to provide a complete storage accounting section  can be adopted to visualize data taken from other accounting systems Documents, instance list, installation guide, video tour, etc.: 19 HLRmon – WORLDCOMP GCA’11, Las Vegas, July 18th 2011