KIT visit to Cascina , 22 March 2019 Virgo Computing Franco Carbognani KIT visit to Cascina , 22 March 2019

KIT visit to Cascina , 22 March 2019 Virgo Computing Reorg Virgo Offline and Online Computing management has been reorganized as one of the outcomes of the 2018 External Computing Committee (ECC ) recommendations This has also lead to the definition of the “AdVirgo Computing and Data Processing Infrastructure Work Breakdown Structure (WBS)” document AdVirgo Computing Model, Implementation Plan and Management Plan documents are being reviewed and updated as deliverables of WP10 KIT visit to Cascina , 22 March 2019

AdVirgo Data Flow schema Advanced Virgo Computing Model

AdVirgo Data Flow: The GW170814 case The signal arrives Data composed into frames Calibration of the data Veto, DQ flags production h(t) transfer Low-latency matched-filter pipelines Upload to GraceDB Data written into on-line storage Low-latency data quality Low-latency sky localization GCN Circular sent out Data written into Cascina Mass Storage Data transfer toward aLIGO and CCs KIT visit to Cascina , 22 March 2019

AdVirgo Computing Layers 17/10/2018 Advanced Virgo Computing Model

Data Processing Infrastructure Virgo Week Jan 2019 KIT visit to Cascina , 22 March 2019

Computing and Data Processing CDP WP8 Offline Computing Services WP9 Data Analysis Tools and Pipelines WP10 LVC Computing Strategy WP12 New Developments WP11 Public Data Open Science WP8.1 CVMFS support at CCs WP9.1 Continuous Waves (CW) WP10.1 Updated Virgo Computing Model WP11.1 O2 Data Release WP12.1 ML/DL strategies for GW detections WP8.2 Usage reporting in LVC accounting WP9.2 Compact Binary Coalescence (CBC) WP11.2 Open Data Science Workshop KIT visit to Cascina , 22 March 2019

Virgo Platform & Services (WP1) Expansion of Raw Data Circular Buffer Expanded to 700TB => ~ 6.3 months at 45 MB/s 1PB final expansion (~12 months) ongoing Online Storage buffer upgraded Condor farm, used for Noemi, Detchar and cWB low latency pipeline expanded to reach 160 cpus Cascina site link bandwidth increased from to 1Gb/s to 10Gb/s. Bulk data transfer to Tier1 CC Legacy solution, GFAL2/iRODS based running smoothly during ER14 LIGO/Virgo common web access via ligo.org authorization Completed for VIM, other relevant web pages will follow KIT visit to Cascina , 22 March 2019

Software Management (WP2) Supporting release cycles associated to Engineering Runs and Science Run A joint Ligo-Virgo Software Configuration Control Board (SCCB) is in place for the relevant Low Latency software Software stored on a centralized SVN archive in Cascina. Transition from SVN to Git ongoing. Currently mirroring relevant packages from Virgo SVN to git.ligo.org Setting up CVMFS for Virgo software distribution Joint Ligo-Virgo proposal “Sustainable software development and distribution with the conda package manager and the conda-forge” accepted and initial work has started KIT visit to Cascina , 22 March 2019

Online System Processes (WP3) KIT visit to Cascina , 22 March 2019

Data Quality & Detchar (WP4)  All analysis run at Cascina on EGO machines One dedicated machine for high CPU/memory analysis 5 virtual machines for online analysis Condor farm for the data quality reports (DQRs) Outputs uploaded to GraceDB to produce joint LIGO-Virgo DQRs Analysis of the past day run on a nightly basis Interactive machines: farmn & ctrl  Main input: raw data  Few TB of storage  DBs: segments, noise lines, etc.  Web server to display results and browse the associated reports KIT visit to Cascina , 22 March 2019

Low Latency data distribution (WP5) Activities and plans Transfer LIGO h(t) frames from LLH and LHO to EGO directly (by-passing the CIT node) Evaluating use of Kafka instead of Cm for intersite h(t) frame exchange (WP5.1) KIT visit to Cascina , 22 March 2019

Low Latency data distribution (WP5) Kafka evaluation We are evaluating the use of Kafka for the Cascina – aLIGO link as a replacement for the current Cm (Virgo specific) based solution. Kafka is a modern message-query which embed a smart fail-over mechanism and will be used by aLIGo for all other Low Latency links (including toward Kagra). Status: Setup parallel live data Kafka streams (CIT <-> Cascina) being compared with Cm based ones KIT visit to Cascina , 22 March 2019

Bulk Data Handling (WP7) The main problem We have been confronted with a strong diversity among Tier1 CC that has always prevented the development of a unified solution For the post O3 timeframe we are aiming to a unified solution Investigate and implement a unifying solution based on a EGI/OSG framework in coordination with Tier1 CCs Must provide a full Data Transfer and Access integrated solution for a full mesh topology among all AdVirgo/LIGO data endpoints KIT visit to Cascina , 22 March 2019

Offline-computing (WP8 – WP9) The main problem The majority of the LVC common pipelines: Run on HTCondor clusters fully devoted to LIGO Look for data and libraries in a local shared file system Pipelines are intrinsically uncompliant with GRID and it’s difficult to have them running at CNAF, CCIN2P3, Nikhef, …. For a better utilization of Virgo computing resources: Pilot projects to enable main CBC detection pipelines to run on Virgo resources directly, via European grid submission, or via OSG Exploration of DIRAC and Rucio for workflows Support of O2 CW analyses with additional computing power provided at CNAF and Nikhef/SURFSara clusters Setting up CVMFS areas at Virgo CCs for data KIT visit to Cascina , 22 March 2019

Offline-computing (WP10 – WP12) New draft of updated Virgo computing model (May 2019) Longer-term goal: integrated LVC computing model Open Data prep, formatting, publication, etc. Translation to local languages; mirror server in Europe Open Data Science Workshop in Paris (Apr 8-10 2019) Exploration and development of ML/DL pipelines 1st Conference on ML for GWs, Geophysics, Robotics and Control Systems (Jan 2019 at EGO) KIT visit to Cascina , 22 March 2019