WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche.

Slides:

Advertisements

Similar presentations

CACORE TOOLS FEATURES. caCORE SDK Features caCORE Workbench Plugin EA/ArgoUML Plug-in development Integrated support of semantic integration in the plugin.

Advertisements

Loupe /loop/ noun a magnifying glass used by jewelers to reveal flaws in gems. a logging and error management tool used by.NET teams to reveal flaws in.

Client/Server Grid applications to manage complex workflows Filippo Spiga* on behalf of CRAB development team * INFN Milano Bicocca (IT)

CERN IT Department CH-1211 Genève 23 Switzerland t Integrating Lemon Monitoring and Alarming System with the new CERN Agile Infrastructure.

LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.

CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.

Elasticsearch in Dashboard Data Management Applications David Tuckett IT/SDC 30 August 2013 (Appendix 11 November 2013)

CIS 375—Web App Dev II Microsoft’s.NET. 2 Introduction to.NET Steve Ballmer (January 2000): Steve Ballmer "Delivering an Internet-based platform of Next.

Monitoring Latency Sensitive Enterprise Applications on the Cloud Shankar Narayanan Ashiwan Sivakumar.

ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.

ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)

CERN IT Department CH-1211 Geneva 23 Switzerland t The Experiment Dashboard ISGC th April 2008 Pablo Saiz, Julia Andreeva, Benjamin.

Jolyon White GEC9, 4 th November 2010 OMF Aggregate Manager API.

Julia Andreeva. \ Monitoring of the job processing Analysis Production Real time and historical views Users Opera- tion teams Sites Data management monitoring.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services GS group meeting Monitoring and Dashboards section Activity.

Enabling Grids for E-sciencE Overview of System Analysis Working Group Julia Andreeva CERN, WLCG Collaboration Workshop, Monitoring BOF session 23 January.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES PhEDEx Monitoring Nicolò Magini CERN IT-ES-VOS For the PhEDEx.

OSG Area Coordinator’s Report: Workload Management April 20 th, 2011 Maxim Potekhin BNL

1 Geospatial and Business Intelligence Jean-Sébastien Turcotte Executive VP San Francisco - April 2007 Streamlining web mapping applications.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Overlook of Messaging.

CERN – European Organization for Nuclear Research Administrative Support - Internet Development Services CET and the quest for optimal implementation and.

And Tier 3 monitoring Tier 3 Ivan Kadochnikov LIT JINR

PanDA Monitor Development ATLAS S&C Workshop by V.Fine (BNL)

EGEE-III INFSO-RI Enabling Grids for E-sciencE Overview of STEP09 monitoring issues Julia Andreeva, IT/GS STEP09 Postmortem.

S. Shumilov – Zürich Analytical Visualization Framework - a visual data processing and knowledge discovery system Ivan Denisovich, Serge Shumilov Department.

Overview of DAQ at CERN experiments E.Radicioni, INFN MICE Daq and Controls Workshop.

EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Wojciech Lapka SAM Team CERN EGEE’09 Conference,

CERN IT Department CH-1211 Geneva 23 Switzerland t CF Computing Facilities Agile Infrastructure Monitoring CERN IT/CF.

EGEE-III INFSO-RI Enabling Grids for E-sciencE Ricardo Rocha CERN (IT/GS) EGEE’08, September 2008, Istanbul, TURKEY Experiment.

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks MSG - A messaging system for efficient and.

Xrootd Monitoring and Control Harsh Arora CERN. Setting Up Service  Monalisa Service  Monalisa Repository  Test Xrootd Server  ApMon Module.

INFSO-RI Enabling Grids for E-sciencE ARDA Experiment Dashboard Ricardo Rocha (ARDA – CERN) on behalf of the Dashboard Team.

XROOTD AND FEDERATED STORAGE MONITORING CURRENT STATUS AND ISSUES A.Petrosyan, D.Oleynik, J.Andreeva Creating federated data stores for the LHC CC-IN2P3,

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF Agile Infrastructure Monitoring HEPiX Spring th April.

Tier3 monitoring. Initial issues. Danila Oleynik. Artem Petrosyan. JINR.

Julia Andreeva on behalf of the MND section MND review.

Andrea Manzi CERN On behalf of the DPM team HEPiX Fall 2014 Workshop DPM performance tuning hints for HTTP/WebDAV and Xrootd 1 16/10/2014.

Conclusions on Monitoring CERN A. Read ADC Monitoring1.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI User-centric monitoring of the analysis and production activities within.

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Monitoring of the LHC Computing Activities Key Results from the Services.

Computing Facilities CERN IT Department CH-1211 Geneva 23 Switzerland t CF CF Monitoring: Lemon, LAS, SLS I.Fedorko(IT/CF) IT-Monitoring.

CERN IT Department CH-1211 Genève 23 Switzerland t CERN IT Monitoring and Data Analytics Pedro Andrade (IT-GT) Openlab Workshop on Data Analytics.

DDM FAX Dashboard status and future Luca Magnoni IT/SDC 2 nd June 2014.

MND review. Main directions of work  Development and support of the Experiment Dashboard Applications - Data management monitoring - Job processing monitoring.

Global ADC Job Monitoring Laura Sargsyan (YerPhI).

Import XRootD monitoring data from MonALISA Sergey Belov, JINR, Dubna DNG section meeting,

EGI-InSPIRE RI EGI-InSPIRE EGI-InSPIRE RI Data Management Highlights in TSA3.3 Services for HEP Fernando Barreiro Megino,

FTS monitoring work WLCG service reliability workshop November 2007 Alexander Uzhinskiy Andrey Nechaevskiy.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES Author etc Dashboard Latency monitoring (Update) Alexandre Beche.

Technology Drill Down: Windows Azure Platform Eric Nelson | ISV Application Architect | Microsoft UK |

CERN - IT Department CH-1211 Genève 23 Switzerland t Grid Reliability Pablo Saiz On behalf of the Dashboard team: J. Andreeva, C. Cirstoiu,

ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.

Streaming Analytics with Spark 1 Magnoni Luca IT-CM-MM 09/02/16EBI - CERN meeting.

Simulation Production System Science Advisory Committee Meeting UW-Madison March 1 st -2 nd 2007 Juan Carlos Díaz Vélez.

ConTZole Tomáš Kubeš, 2010 atlas-tz-monitoring.cern.ch An Interactive ATLAS Tier-0 Monitoring.

XRootD Monitoring Report A.Beche D.Giordano. Outlines  Talk 1: XRootD Monitoring Dashboard  Context  Dataflow and deployment model  Database: storage.

CERN IT Department CH-1211 Genève 23 Switzerland t Monitoring: Present and Future Pedro Andrade (CERN IT) 31 st August.

EMI is partially funded by the European Commission under Grant Agreement RI Future Proof Storage with DPM Oliver Keeble (on behalf of the CERN IT-GT-DMS.

WLCG Transfers monitoring EGI Technical Forum Madrid, 17 September 2013 Pablo Saiz on behalf of the Dashboard Team CERN IT/SDC.

IT Monitoring Service Status and Progress 1 Alberto AIMAR, IT-CM-MM.

Daniele Bonacorsi Andrea Sciabà

WLCG Transfers Dashboard

Monitoring Evolution and IPv6

Key Activities. MND sections

PVSS Evolution in Relation to Databases

POW MND section.

FTS Monitoring Ricardo Rocha

Monitoring Of XRootD Federation

Monitoring of the infrastructure from the VO perspective

BIOPAMA Data Management

Presentation transcript:

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche

Outlines  Mission  History of the data transfers monitoring tools  Generic monitoring workflow  New WLCG Transfers Dashboard  Summary 18 December 2013 Alexandre Beche – White Area 2

Mission 3 ATLAS FTS + XRootD ATLAS FTS + XRootD CMS FTS + XRootD CMS FTS + XRootD LHCb FTS LHCb FTS ALICE XRootD ALICE XRootD 18 December 2013 Alexandre Beche – White Area

4 ATLAS FTS + XRootD ATLAS FTS + XRootD CMS FTS + XRootD CMS FTS + XRootD LHCb FTS LHCb FTS ALICE XRootD ALICE XRootD Mission 18 December 2013 Alexandre Beche – White Area

5 ATLAS FTS + XRootD ATLAS FTS + XRootD CMS FTS + XRootD CMS FTS + XRootD LHCb FTS LHCb FTS ALICE XRootD ALICE XRootD Mission 18 December 2013 Alexandre Beche – White Area

History of data transfers monitoring tools 6 WEB API / UI WEB API / UI WLCG FTS WLCG Transfers Dashboard 18 December 2013 Alexandre Beche – White Area

History of data transfers monitoring tools 7 WEB API / UI WEB API / UI WLCG FAXAAAFTS WLCG Transfers Dashboard XRootD integration 18 December 2013 Alexandre Beche – White Area

History of data transfers monitoring tools 8 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXAAAFTS WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard 18 December 2013 Alexandre Beche – White Area

History of data transfers monitoring tools 9 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXEOSAAAEOSFTS WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard EOS integration 18 December 2013 Alexandre Beche – White Area

History of data transfers monitoring tools 10 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXEOSAAAEOSFTS ALICE WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard EOS integration ALICE integration 18 December 2013 Alexandre Beche – White Area

Generic monitoring workflow 11 AMQ Raw & Statistics Raw & Statistics Consumer WEB API WEB UI 18 December 2013 Alexandre Beche – White Area

From ActiveMQ to Oracle 12  Load balanced collector  Horizontal scaling Consumer AMQ DB Consumer 18 December 2013 Alexandre Beche – White Area

From ActiveMQ to Oracle 13  Load balanced collector  Horizontal scaling Consumer AMQ DB Stompclt* * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

From ActiveMQ to Oracle 14  Load balanced collector  Horizontal scaling Consumer AMQ DB Stompclt* Disk queue* * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

From ActiveMQ to Oracle 15  Load balanced collector  Horizontal scaling Consumer AMQ DB Stompclt* Disk queue* DB inserter  Customizable inserter  Data enhancement  Filtering * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

From ActiveMQ to Oracle 16  Modular architecture  Common building blocks (EPEL)  Reliable  Load balanced collector  Horizontal scaling Consumer AMQ Simplevisor* DB Stompclt* Disk queue* DB inserter  Customizable inserter  Data enhancement  Filtering * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

Raw data aggregation  Well optimized storage  Partitioned table  No global indexes (including the PK) December 2013 Alexandre Beche – White Area

Raw data aggregation  Well optimized storage  Partitioned table  No global indexes (including the PK)  Aggregation of unordered events  Stateless: Full re-computation of touched bins each time  Compute stats from raw data in 10 min bins  Aggregate 10 min stats in daily bins December 2013 Alexandre Beche – White Area

Aggregation methods 19 2pm3pm4pm5pm6pm7pm Transfers 18 December 2013 Alexandre Beche – White Area

Aggregation methods 20 2pm3pm4pm5pm6pm7pm Transfers Easy method 18 December 2013 Alexandre Beche – White Area

Aggregation methods 21 2pm3pm4pm5pm6pm7pm Transfers Easy method Transfers10021 Bytes December 2013 Alexandre Beche – White Area

Aggregation methods 22 2pm3pm4pm5pm6pm7pm Transfers10021 Bytes Transfers1 (1) 1 (0) 2 (0) 3 (2) 1 (1) Bytes8114 (9+6) 15 (1+9+5) 5 Easy method Transfers 18 December 2013 Alexandre Beche – White Area

Aggregation methods 23 2pm3pm4pm5pm6pm7pm Transfers10021 Bytes Transfers1 (1) 1 (0) 2 (0) 3 (2) 1 (1) Bytes8114 (9+6) 15 (1+9+5) 5 Easy method Transfers Adopted method Both method equivalent if: Small transfers or many transfers 18 December 2013 Alexandre Beche – White Area

24 Raw & Statistics Raw & Statistics DAO A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Generic 18 December 2013 Alexandre Beche – White Area

25 Raw & Statistics Raw & Statistics DAO Action A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Generic 18 December 2013 Alexandre Beche – White Area Run a parameterized query, Return the results in a dict

26 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Python dict to #format# converter Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area

27 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … Controller A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Router of url+mime To action + view Python dict to #format# converter Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area

28 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … Controller Configuration A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Router of url+mime To action + view Python dict to #format# converter List of actions Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area

29 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … Controller Configuration A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Router of url+mime To action + view Python dict to #format# converter List of actions Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area Django under investigation

Visualization layer Dashboard UI  Rich web single-page user interface  AJAX+JSON communication December 2013 Alexandre Beche – White Area

Visualization layer Dashboard UI  Rich web single-page user interface  AJAX+JSON communication  jQuery + popular plugin technology cocktail December 2013 Alexandre Beche – White Area

xbrowse Visualization layer Dashboard UI  Rich web single-page user interface  AJAX+JSON communication  jQuery + popular plugin technology cocktail  Modular code reuse with xbrowse framework  Lightweight in-house URL-driven MVC (<300 LOC)  Views shared with several other applications  Common look and feel December 2013 Alexandre Beche – White Area

History of data transfers monitoring tools 33 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXEOSAAAEOSFTS ALICE WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard EOS integration ALICE integration 18 December 2013 Alexandre Beche – White Area

WLCG Transfers Dashboard federated approach 34 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI FTS FAX AAA FAXEOSAAAEOSFTS ALICE WEB API WEB API ALICE No duplication anymore EOS could be included 18 December 2013 Alexandre Beche – White Area

WLCG Transfers Dashboard federated approach 35 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI FTS FAX AAA FAXEOSAAAEOSFTS ALICE WEB API WEB API ALICE WLCG Transfers Dashboard API / UI WLCG Transfers Dashboard API / UI No duplication anymore EOS could be included 18 December 2013 Alexandre Beche – White Area

Some plots 27 May 2013 Alexandre Beche 36

New WLCG Transfers Dashboard  Multi-threaded HTTP Call  Thread pool  Async request (std::future c++11) 37 Main thread Async call 1 Async call 2 Get results Aggregate Return 18 December 2013 Alexandre Beche – White Area

Not just a re-writing  Possible re-think of UI 38 WEB API / UI WEB API / UI WEB API/UI WEB API/UI FTS FAX EOSFTS 18 December 2013 Alexandre Beche – White Area

Not just a re-writing  Possible re-think of UI  VO specific UI 39 WEB API / UI WEB API / UI WEB API/UI WEB API/UI FTS FAX EOSFTS WEB API WEB API WEB API WEB API FTS FAX EOSFTS ATLAS Monitoring UI ATLAS Monitoring UI 18 December 2013 Alexandre Beche – White Area

Summary  More flexible architecture  Database-less  Based on common building blocks  UI code fully re-used 40 Thanks for your attention 18 December 2013 Alexandre Beche – White Area

Useful links  AAA Dashboard   FAX Dashboard:   FTS Dashboard:   WLG Transfers Dashboard:   Xbrowse framework:  November 2013 Alexandre Beche - ITTF