TS4.10 Comp Reports A new approach to Computing Availability/Reliability reports for EGI Progress Report C. Kanellopoulos GRNET 9/14/2018.

Slides:



Advertisements
Similar presentations
ONE STOP THE TOTAL SERVICE SOLUTION FOR REMOTE DEVICE MANAGMENT.
Advertisements

LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.
ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EG recent developments T. Ferrari/EGI.eu ADC Weekly Meeting 15/05/
HPDC 2007 / Grid Infrastructure Monitoring System Based on Nagios Grid Infrastructure Monitoring System Based on Nagios E. Imamagic, D. Dobrenic SRCE HPDC.
James Casey, CERN, IT-GT-TOM 1 st ROC LA Workshop, 6 th October 2010 Grid Infrastructure Monitoring.
1/22/08 RTR Project Presentation to TPTF RTR Project Michael Daskalantonakis & Brian Cook.
Module 8 : Configuration II Jong S. Bok
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Wojciech Lapka SAM Team CERN EGEE’09 Conference,
CERN IT Department CH-1211 Geneva 23 Switzerland t CF Computing Facilities Agile Infrastructure Monitoring CERN IT/CF.
HLRmon accounting portal DGAS (Distributed Grid Accounting System) sensors collect accounting information at site level. Site data are sent to site or.
System/SDWG Update Management Council Face-to-Face Flagstaff, AZ August 22-23, 2011 Sean Hardman.
XROOTD AND FEDERATED STORAGE MONITORING CURRENT STATUS AND ISSUES A.Petrosyan, D.Oleynik, J.Andreeva Creating federated data stores for the LHC CC-IN2P3,
HLRmon accounting portal The accounting layout A. Cristofori 1, E. Fattibene 1, L. Gaido 2, P. Veronesi 1 INFN-CNAF Bologna (Italy) 1, INFN-Torino Torino.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI TS8.10 A new approach to Computing Availability/Reliability reports for EGI.
SUM like functionality with WLCG-MON Ivan Dzhunov.
ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.
Prototype of new Site Usability interface Amol Wakankar March, /1/20111.
Probes Requirement Review OTAG-08 03/05/ Requirements that can be directly passed to EMI ● Changes to the MPI test (NGI_IT)
NGI France-Grilles: Infrastructure evolution H. Cordier.
Co-ordination & Harmonisation of Advanced e-Infrastructures for Research and Education Data Sharing Research Infrastructures Grant Agreement n
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Update on Service Availability Monitoring (SAM) Marian Babik, David Collados,
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI Services for Distributed e-Infrastructure Access Tiziana Ferrari on behalf.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Regional tools use cases overview Peter Solagna – EGI.eu On behalf of the.
Site notifications with SAM and Dashboards Marian Babik SDC/MI Team IT/SDC/MI 12 th June 2013 GDB.
TSA1.4 Infrastructure for Grid Management Tiziana Ferrari, EGI.eu EGI-InSPIRE – SA1 Kickoff Meeting1.
EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number Federated Cloud Update.
GOCDB Status and Plans David Meredith John Casson
Accounting Review Summary and action list from the (pre)GDB Julia Andreeva CERN-IT WLCG MB 19th April
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Operational Tools M2 Update James Casey.
Polish Infrastructure for Supporting Computational Science in the European Research Space EUROPEAN UNION Grid Resource Bazaar Platform for resource allocation.
DFR Downloader Theo Laughner, PE Presented at GPA User Forum August 5, 2015.
EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI EGI solution for high throughput data analysis Peter Solagna EGI.eu Operations.
Daniele Bonacorsi Andrea Sciabà
Gridpp37 – 31/08/2016 George Ryall David Meredith
Product Overview.
Job monitoring and accounting data visualization
The Operations Portal and the Grid Operations Interoperability
NGI and Site Nagios Monitoring
OVirt Data Warehouse 02/11/11 Yaniv Dary BI Software Engineer, Red Hat.
CyVerse Discovery Environment
GOCDB New Requirements
Overview – SOE PatchTT November 2015.
PL-Grid – an example of NGI support structure Marcin Radecki
POW MND section.
Pedro Andrade ACE Status Update Pedro Andrade
Overview – SOE PatchTT December 2013.
GOCDB Update 27/05/ Me: Working on GOCDB 3 days a week
Introduction to OAT presentations
Evolution of SAM in an enhanced model for monitoring the WLCG grid
Security Monitoring in a Nagios world
FTS Monitoring Ricardo Rocha
AppDB current status and proposed extensions
Experiment Dashboard overviw of the applications
Cloud Management Mechanisms
Advancements in Availability and Reliability computation Introduction and current status of the Comp Reports mini project C. Kanellopoulos GRNET.
Operations & Coordination Tools
Maite Barroso, SA1 activity leader CERN 27th January 2009
Monitoring Of XRootD Federation
Solutions for federated services management EGI
Monitoring of the infrastructure from the VO perspective
Cloud Management Mechanisms
Introduction OMB, T. Ferrari/EGI.eu 12/4/2018
EGI operations - news T. Ferrari/EGI.eu 12/9/2018.
SAP Value Assurance for SAP S/4HANA Implementation support for DVM Apps You want to be able to monitor your entire SAP landscape to understand the Data.
Operational Tools & Middleware Versions Monitoring
Operations Management Board January 29
Dynamicweb PIM General introduction Innovia 2018.
Analytics Plus Product Overview.
Core Activities re-assessment
Presentation transcript:

TS4.10 Comp Reports A new approach to Computing Availability/Reliability reports for EGI Progress Report C. Kanellopoulos GRNET 9/14/2018

Current Situation NGI Site A/R reports are delivered monthly Computations are performed on a centralized infrastructure Current implementation is more or less a closed solution EGI Operations cannot interact via a direct interface Re-computations etc are handled via GGUS (SLM unit) VO & NGI Core Services Reports are generated by the Ops Portal

New A/R reporting service Proposal New A/R reporting service Open source solution Include extensions for VO-wide metrics (in addition to service-wise, site-wise and NGI-wise) Direct interface for SLM Units and EGI Operations (via API) Computations performed under profiles Deliver/Query results via front-end module

Overview

Initial goal: Replicate current ACE functionality Demo at the EGI User Forum: Retrieve monitoring data from the Brokers Calculate A/R for Sites Calculate A/R for NGI Core Services & VOs (Lavoisier) Distribute A/R results through Lavoisier Perform re-calculcations

Consumer service has been developed Data Acquisition Consumer service has been developed Listens on a configurable set of message queues Initial message level filtering Supports multiple backends for storing the monitoring data Default backend is the filesystems to that it can be fed to Hadoop

Poem Retrieval Service A service has been developed that downloads the latest profiles (once per day) A POEM profile can change in time, but changes are not very often Need to be able to track history of the POEM profiles

Retrieve topology on a daily basis Topology Retrieval Retrieve topology on a daily basis Topology can change at any time multiple times per month ACE currently uses the current topology at the time of computation We will certainly need to keep topology information for the current and previous month (for re-calculation purposes) Do we need a longer period of retention?

Status Computation Engine Built on top of PIG Status of service endpoint is computed per day on 24h time slices (hourly status) Currently POEM profiles are used for aggregating metric results into status BUT engine is extendable to include other profiles at this stage as well Engine allows also for multiple status results per profile

A/R Computation Engine Built on top of HDFS & Hive (Hadoop SQL interface) WIP: Service Flavor and Site Availability based on specific topology retrieved from GOCDB Daily history of topology is to be kept so recalculations (i.e. for the previous month) will be performed based on the existing topology at that time

Computation Engine

Computation Engine Future Work: VO and NGI Core Services A/R Support of custom high level profiles

API Expose A/R Calculations via tha API Expose “Re-calculation” functionality via the API Work has not started yet

User Interface built on top of the Lavoisier engine Currently in operation providing VO and NGI Core Services A/R Reporting - Selection of the availability type (vo , ngi , sites) - Selection of the service group (CE , SE , TOP BDII ,ALL ) - Selection of a period then selection of the granularity(monthly , daily , hourly) . - Possibility to export in xls and pdf and different charts