Presentation is loading. Please wait.

Presentation is loading. Please wait.

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche.

Published byDustin Byrd Modified over 8 years ago

Similar presentations

Presentation on theme: "WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche."— Presentation transcript:

1 WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche

2 Outlines  Mission  History of the data transfers monitoring tools  Generic monitoring workflow  New WLCG Transfers Dashboard  Summary 18 December 2013 Alexandre Beche – White Area 2

3 Mission 3 ATLAS FTS + XRootD ATLAS FTS + XRootD CMS FTS + XRootD CMS FTS + XRootD LHCb FTS LHCb FTS ALICE XRootD ALICE XRootD 18 December 2013 Alexandre Beche – White Area

4 4 ATLAS FTS + XRootD ATLAS FTS + XRootD CMS FTS + XRootD CMS FTS + XRootD LHCb FTS LHCb FTS ALICE XRootD ALICE XRootD Mission 18 December 2013 Alexandre Beche – White Area

5 5 ATLAS FTS + XRootD ATLAS FTS + XRootD CMS FTS + XRootD CMS FTS + XRootD LHCb FTS LHCb FTS ALICE XRootD ALICE XRootD Mission 18 December 2013 Alexandre Beche – White Area

6 History of data transfers monitoring tools 6 WEB API / UI WEB API / UI WLCG FTS WLCG Transfers Dashboard 18 December 2013 Alexandre Beche – White Area

7 History of data transfers monitoring tools 7 WEB API / UI WEB API / UI WLCG FAXAAAFTS WLCG Transfers Dashboard XRootD integration 18 December 2013 Alexandre Beche – White Area

8 History of data transfers monitoring tools 8 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXAAAFTS WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard 18 December 2013 Alexandre Beche – White Area

9 History of data transfers monitoring tools 9 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXEOSAAAEOSFTS WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard EOS integration 18 December 2013 Alexandre Beche – White Area

10 History of data transfers monitoring tools 10 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXEOSAAAEOSFTS ALICE WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard EOS integration ALICE integration 18 December 2013 Alexandre Beche – White Area

11 Generic monitoring workflow 11 AMQ Raw & Statistics Raw & Statistics Consumer WEB API WEB UI 18 December 2013 Alexandre Beche – White Area

12 From ActiveMQ to Oracle 12  Load balanced collector  Horizontal scaling Consumer AMQ DB Consumer 18 December 2013 Alexandre Beche – White Area

13 From ActiveMQ to Oracle 13  Load balanced collector  Horizontal scaling Consumer AMQ DB Stompclt* * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

14 From ActiveMQ to Oracle 14  Load balanced collector  Horizontal scaling Consumer AMQ DB Stompclt* Disk queue* * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

15 From ActiveMQ to Oracle 15  Load balanced collector  Horizontal scaling Consumer AMQ DB Stompclt* Disk queue* DB inserter  Customizable inserter  Data enhancement  Filtering * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

16 From ActiveMQ to Oracle 16  Modular architecture  Common building blocks (EPEL)  Reliable  Load balanced collector  Horizontal scaling Consumer AMQ Simplevisor* DB Stompclt* Disk queue* DB inserter  Customizable inserter  Data enhancement  Filtering * Developed by the CERN messaging team 18 December 2013 Alexandre Beche – White Area

17 Raw data aggregation  Well optimized storage  Partitioned table  No global indexes (including the PK) 17 18 December 2013 Alexandre Beche – White Area

18 Raw data aggregation  Well optimized storage  Partitioned table  No global indexes (including the PK)  Aggregation of unordered events  Stateless: Full re-computation of touched bins each time  Compute stats from raw data in 10 min bins  Aggregate 10 min stats in daily bins 18 18 December 2013 Alexandre Beche – White Area

19 Aggregation methods 19 2pm3pm4pm5pm6pm7pm Transfers 18 December 2013 Alexandre Beche – White Area

20 Aggregation methods 20 2pm3pm4pm5pm6pm7pm Transfers Easy method 18 December 2013 Alexandre Beche – White Area

21 Aggregation methods 21 2pm3pm4pm5pm6pm7pm Transfers Easy method Transfers10021 Bytes10001520 18 December 2013 Alexandre Beche – White Area

22 Aggregation methods 22 2pm3pm4pm5pm6pm7pm Transfers10021 Bytes10001520 Transfers1 (1) 1 (0) 2 (0) 3 (2) 1 (1) Bytes8114 (9+6) 15 (1+9+5) 5 Easy method Transfers 18 December 2013 Alexandre Beche – White Area

23 Aggregation methods 23 2pm3pm4pm5pm6pm7pm Transfers10021 Bytes10001520 Transfers1 (1) 1 (0) 2 (0) 3 (2) 1 (1) Bytes8114 (9+6) 15 (1+9+5) 5 Easy method Transfers Adopted method Both method equivalent if: Small transfers or many transfers 18 December 2013 Alexandre Beche – White Area

24 24 Raw & Statistics Raw & Statistics DAO A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Generic 18 December 2013 Alexandre Beche – White Area

25 25 Raw & Statistics Raw & Statistics DAO Action A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Generic 18 December 2013 Alexandre Beche – White Area Run a parameterized query, Return the results in a dict

26 26 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Python dict to #format# converter Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area

27 27 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … Controller A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Router of url+mime To action + view Python dict to #format# converter Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area

28 28 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … Controller Configuration A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Router of url+mime To action + view Python dict to #format# converter List of actions Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area

29 29 Raw & Statistics Raw & Statistics DAO Action Views Json, xml, … Views Json, xml, … Controller Configuration A well defined web API  Legacy dashboard framework:  Use the MVC design pattern (based on Struts) Application specific Collection of SQL queries Router of url+mime To action + view Python dict to #format# converter List of actions Run a parameterized query, Return the results in a dict Generic 18 December 2013 Alexandre Beche – White Area Django under investigation

30 Visualization layer Dashboard UI  Rich web single-page user interface  AJAX+JSON communication 30 18 December 2013 Alexandre Beche – White Area

31 Visualization layer Dashboard UI  Rich web single-page user interface  AJAX+JSON communication  jQuery + popular plugin technology cocktail 31 18 December 2013 Alexandre Beche – White Area

32 xbrowse Visualization layer Dashboard UI  Rich web single-page user interface  AJAX+JSON communication  jQuery + popular plugin technology cocktail  Modular code reuse with xbrowse framework  Lightweight in-house URL-driven MVC (<300 LOC)  Views shared with several other applications  Common look and feel 32 18 December 2013 Alexandre Beche – White Area

33 History of data transfers monitoring tools 33 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI WLCG FAX AAA FAXEOSAAAEOSFTS ALICE WLCG Transfers Dashboard XRootD integration XRootD Detailled Dashboard EOS integration ALICE integration 18 December 2013 Alexandre Beche – White Area

34 WLCG Transfers Dashboard federated approach 34 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI FTS FAX AAA FAXEOSAAAEOSFTS ALICE WEB API WEB API ALICE No duplication anymore EOS could be included 18 December 2013 Alexandre Beche – White Area

35 WLCG Transfers Dashboard federated approach 35 WEB API / UI WEB API / UI WEB API/UI WEB API/UI WEB API/UI WEB API/UI FTS FAX AAA FAXEOSAAAEOSFTS ALICE WEB API WEB API ALICE WLCG Transfers Dashboard API / UI WLCG Transfers Dashboard API / UI No duplication anymore EOS could be included 18 December 2013 Alexandre Beche – White Area

36 Some plots 27 May 2013 Alexandre Beche 36

37 New WLCG Transfers Dashboard  Multi-threaded HTTP Call  Thread pool  Async request (std::future c++11) 37 Main thread Async call 1 Async call 2 Get results Aggregate Return 18 December 2013 Alexandre Beche – White Area

38 Not just a re-writing  Possible re-think of UI 38 WEB API / UI WEB API / UI WEB API/UI WEB API/UI FTS FAX EOSFTS 18 December 2013 Alexandre Beche – White Area

39 Not just a re-writing  Possible re-think of UI  VO specific UI 39 WEB API / UI WEB API / UI WEB API/UI WEB API/UI FTS FAX EOSFTS WEB API WEB API WEB API WEB API FTS FAX EOSFTS ATLAS Monitoring UI ATLAS Monitoring UI 18 December 2013 Alexandre Beche – White Area

40 Summary  More flexible architecture  Database-less  Based on common building blocks  UI code fully re-used 40 Thanks for your attention 18 December 2013 Alexandre Beche – White Area

41 Useful links  AAA Dashboard  http://dashb-cms-xrootd-transfers.cern.ch http://dashb-cms-xrootd-transfers.cern.ch  FAX Dashboard:  http://dashb-atlas-xrootd-transfers.cern.ch http://dashb-atlas-xrootd-transfers.cern.ch  FTS Dashboard:  http://dashb-fts-transfers/ui/ http://dashb-fts-transfers/ui/  WLG Transfers Dashboard:  http://dashb-wlcg-transfers-new/ui/# http://dashb-wlcg-transfers-new/ui/#  Xbrowse framework:  https://twiki.cern.ch/twiki/bin/view/ArdaGrid/XbrowseFramework https://twiki.cern.ch/twiki/bin/view/ArdaGrid/XbrowseFramework 41 29 November 2013 Alexandre Beche - ITTF

Download ppt "WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers. Alexandre Beche."

Similar presentations

CACORE TOOLS FEATURES. caCORE SDK Features caCORE Workbench Plugin EA/ArgoUML Plug-in development Integrated support of semantic integration in the plugin.

CACORE TOOLS FEATURES. caCORE SDK Features caCORE Workbench Plugin EA/ArgoUML Plug-in development Integrated support of semantic integration in the plugin.
Loupe /loop/ noun a magnifying glass used by jewelers to reveal flaws in gems. a logging and error management tool used by.NET teams to reveal flaws in.

Loupe /loop/ noun a magnifying glass used by jewelers to reveal flaws in gems. a logging and error management tool used by.NET teams to reveal flaws in.
Client/Server Grid applications to manage complex workflows Filippo Spiga* on behalf of CRAB development team * INFN Milano Bicocca (IT)

Client/Server Grid applications to manage complex workflows Filippo Spiga* on behalf of CRAB development team * INFN Milano Bicocca (IT)
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Integrating Lemon Monitoring and Alarming System with the new CERN Agile Infrastructure.

CERN IT Department CH-1211 Genève 23 Switzerland t Integrating Lemon Monitoring and Alarming System with the new CERN Agile Infrastructure.
LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.

LHC Experiment Dashboard Main areas covered by the Experiment Dashboard: Data processing monitoring (job monitoring) Data transfer monitoring Site/service.
CERN - IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.

CERN - IT Department CH-1211 Genève 23 Switzerland t Monitoring the ATLAS Distributed Data Management System Ricardo Rocha (CERN) on behalf.
Elasticsearch in Dashboard Data Management Applications David Tuckett IT/SDC 30 August 2013 (Appendix 11 November 2013)

Elasticsearch in Dashboard Data Management Applications David Tuckett IT/SDC 30 August 2013 (Appendix 11 November 2013)
CIS 375—Web App Dev II Microsoft’s.NET. 2 Introduction to.NET Steve Ballmer (January 2000): Steve Ballmer Delivering an Internet-based platform of Next.

CIS 375—Web App Dev II Microsoft’s.NET. 2 Introduction to.NET Steve Ballmer (January 2000): Steve Ballmer "Delivering an Internet-based platform of Next.
Monitoring Latency Sensitive Enterprise Applications on the Cloud Shankar Narayanan Ashiwan Sivakumar.

Monitoring Latency Sensitive Enterprise Applications on the Cloud Shankar Narayanan Ashiwan Sivakumar.
ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, 17.07.12, JINR, Dubna.

ATLAS Off-Grid sites (Tier-3) monitoring A. Petrosyan on behalf of the ATLAS collaboration GRID’2012, , JINR, Dubna.
ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)

ATLAS DQ2 Deletion Service D.A. Oleynik, A.S. Petrosyan, V. Garonne, S. Campana (on behalf of the ATLAS Collaboration)
CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t The Experiment Dashboard ISGC 2008 9-11 th April 2008 Pablo Saiz, Julia Andreeva, Benjamin.

CERN IT Department CH-1211 Geneva 23 Switzerland t The Experiment Dashboard ISGC th April 2008 Pablo Saiz, Julia Andreeva, Benjamin.
Jolyon White GEC9, 4 th November 2010 OMF Aggregate Manager API.

Jolyon White GEC9, 4 th November 2010 OMF Aggregate Manager API.
Julia Andreeva. \ Monitoring of the job processing Analysis Production Real time and historical views Users Opera- tion teams Sites Data management monitoring.

Julia Andreeva. \ Monitoring of the job processing Analysis Production Real time and historical views Users Opera- tion teams Sites Data management monitoring.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services GS group meeting 07.03.08 Monitoring and Dashboards section Activity.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services GS group meeting Monitoring and Dashboards section Activity.
Enabling Grids for E-sciencE Overview of System Analysis Working Group Julia Andreeva CERN, WLCG Collaboration Workshop, Monitoring BOF session 23 January.

Enabling Grids for E-sciencE Overview of System Analysis Working Group Julia Andreeva CERN, WLCG Collaboration Workshop, Monitoring BOF session 23 January.
Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t DBES PhEDEx Monitoring Nicolò Magini CERN IT-ES-VOS For the PhEDEx.

Experiment Support CERN IT Department CH-1211 Geneva 23 Switzerland t DBES PhEDEx Monitoring Nicolò Magini CERN IT-ES-VOS For the PhEDEx.
OSG Area Coordinator’s Report: Workload Management April 20 th, 2011 Maxim Potekhin BNL 631-344-3621

OSG Area Coordinator’s Report: Workload Management April 20 th, 2011 Maxim Potekhin BNL
Www.korem.com 1 Geospatial and Business Intelligence Jean-Sébastien Turcotte Executive VP San Francisco - April 2007 Streamlining web mapping applications.

1 Geospatial and Business Intelligence Jean-Sébastien Turcotte Executive VP San Francisco - April 2007 Streamlining web mapping applications.
CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Internet Services Overlook of Messaging.

CERN IT Department CH-1211 Genève 23 Switzerland t Internet Services Overlook of Messaging.

Similar presentations

Ads by Google