Remote Online Farms Sander Klous 01 11 010 001 1101 1110 11001 01011 110110 001101 1111111 0111000 11101010 01001110 110111001 000101101 1111010001 0101111100.

Slides:

Advertisements

Similar presentations

Sander Klous on behalf of the ATLAS Collaboration Real-Time May /5/20101.

Advertisements

Copyright© 2000 OPNET Technologies, Inc. R.W. Dobinson, S. Haas, K. Korcyl, M.J. LeVine, J. Lokier, B. Martin, C. Meirosu, F. Saka, K. Vella Testing and.

Kostas KORDAS INFN – Frascati XI Bruno Touschek spring school, Frascati,19 May 2006 Higgs → 2e+2  O (1/hr) Higgs → 2e+2  O (1/hr) ~25 min bias events.

23/04/2008VLVnT08, Toulon, FR, April 2008, M. Stavrianakou, NESTOR-NOA 1 First thoughts for KM3Net on-shore data storage and distribution Facilities VLV.

The LHCb DAQ and Trigger Systems: recent updates Ricardo Graciani XXXIV International Meeting on Fundamental Physics.

Semester Copyright USM EEE442 Computer Networks The Data Link / Network Layer Functions: Switching En. Mohd Nazri Mahmud MPhil (Cambridge, UK)

Slide: 1 Richard Hughes-Jones T2UK, October 06 R. Hughes-Jones Manchester 1 Update on Remote Real-Time Computing Farms For ATLAS Trigger DAQ. Richard Hughes-Jones.

ACAT 2002, Moscow June 24-28thJ. Hernández. DESY-Zeuthen1 Offline Mass Data Processing using Online Computing Resources at HERA-B José Hernández DESY-Zeuthen.

CHEP03 - UCSD - March 24th-28th 2003 T. M. Steinbeck, V. Lindenstruth, H. Tilsner, for the Alice Collaboration Timm Morten Steinbeck, Computer Science.

IFIN-HH LHCB GRID Activities Eduard Pauna Radu Stoica.

1 The ATLAS Online High Level Trigger Framework: Experience reusing Offline Software Components in the ATLAS Trigger Werner Wiedenmann University of Wisconsin,

First year experience with the ATLAS online monitoring framework Alina Corso-Radu University of California Irvine on behalf of ATLAS TDAQ Collaboration.

Hall D Online Data Acquisition CEBAF provides us with a tremendous scientific opportunity for understanding one of the fundamental forces of nature. 75.

Virtual Organization Approach for Running HEP Applications in Grid Environment Łukasz Skitał 1, Łukasz Dutka 1, Renata Słota 2, Krzysztof Korcyl 3, Maciej.

JetWeb on the Grid Ben Waugh (UCL), GridPP6, What is JetWeb? How can JetWeb use the Grid? Progress report The Future Conclusions.

Worldwide event filter processing for calibration Calorimeter Calibration Workshop Sander Klous September 2006.

1 Chasing the Higgs boson with a worldwide distributed trigger system Sander Klous NIKHEF VENI proposal 2006.

CDF data production models 1 Data production models for the CDF experiment S. Hou for the CDF data production team.

Copyright © 2000 OPNET Technologies, Inc. Title – 1 Distributed Trigger System for the LHC experiments Krzysztof Korcyl ATLAS experiment laboratory H.

Calibration streams in the Event Filter. Status report Mainz, Thursday 13 October 2005 Sander Klous – NIKHEF On behalf of the EF calibration team: Martine.

CHEP'07 September D0 data reprocessing on OSG Authors Andrew Baranovski (Fermilab) for B. Abbot, M. Diesburg, G. Garzoglio, T. Kurca, P. Mhashilkar.

Developing & Managing A Large Linux Farm – The Brookhaven Experience CHEP2004 – Interlaken September 27, 2004 Tomasz Wlodek - BNL.

November SC06 Tampa F.Fanzago CRAB a user-friendly tool for CMS distributed analysis Federica Fanzago INFN-PADOVA for CRAB team.

EGEE is a project funded by the European Union under contract IST HEP Use Cases for Grid Computing J. A. Templon Undecided (NIKHEF) Grid Tutorial,

Management of the LHCb DAQ Network Guoming Liu * †, Niko Neufeld * * CERN, Switzerland † University of Ferrara, Italy.

Network Performance for ATLAS Real-Time Remote Computing Farm Study Alberta, CERN Cracow, Manchester, NBI MOTIVATION Several experiments, including ATLAS.

The ATLAS Trigger: High-Level Trigger Commissioning and Operation During Early Data Taking Ricardo Gonçalo, Royal Holloway University of London On behalf.

Prospects for the use of remote real time computing over long distances in the ATLAS Trigger/DAQ system R. W. Dobinson (CERN), J. Hansen (NBI), K. Korcyl.

Geneva – Kraków network measurements for the ATLAS Real-Time Remote Computing Farm Studies R. Hughes-Jones (Univ. of Manchester), K. Korcyl (IFJ-PAN),

Evolution of a High Performance Computing and Monitoring system onto the GRID for High Energy Experiments T.L. Hsieh, S. Hou, P.K. Teng Academia Sinica,

The KLOE computing environment Nuclear Science Symposium Portland, Oregon, USA 20 October 2003 M. Moulson – INFN/Frascati for the KLOE Collaboration.

2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna Lehmann - CERN EP/ATD The DataFlow of the ATLAS Trigger.

CHEP March 2003 Sarah Wheeler 1 Supervision of the ATLAS High Level Triggers Sarah Wheeler on behalf of the ATLAS Trigger/DAQ High Level Trigger.

Accessing PBeast and monitoring the L1 trigger Emily Thompson.

Xmas Meeting, Manchester, Dec 2006, R. Hughes-Jones Manchester 1 ATLAS TDAQ Networking, Remote Compute Farms & Evaluating SFOs Richard Hughes-Jones The.

Routing and Streaming in the HLT TDAQ Week Sander Klous Wednesday, May 17, 2006.

Interactive European Grid Environment for HEP Application with Real Time Requirements Lukasz Dutka 1, Krzysztof Korcyl 2, Krzysztof Zielinski 1,3, Jacek.

Experience with multi-threaded C++ applications in the ATLAS DataFlow Szymon Gadomski University of Bern, Switzerland and INP Cracow, Poland on behalf.

Kostas KORDAS INFN – Frascati 10th Topical Seminar on Innovative Particle & Radiation Detectors (IPRD06) Siena, 1-5 Oct The ATLAS Data Acquisition.

GridPP Meeting Jan 2003 R. Hughes-Jones Manchester ATLAS Trigger/DAQ Real-time use of the Grid Network Richard Hughes-Jones The University of Manchester.

The ATLAS DAQ System Online Configurations Database Service Challenge J. Almeida, M. Dobson, A. Kazarov, G. Lehmann-Miotto, J.E. Sloper, I. Soloviev and.

Software for the CMS Cosmic Challenge Giacomo BRUNO UCL, Louvain-la-Neuve, Belgium On behalf of the CMS Collaboration CHEP06, Mumbay, India February 16,

LHCbComputing Computing for the LHCb Upgrade. 2 LHCb Upgrade: goal and timescale m LHCb upgrade will be operational after LS2 (~2020) m Increase significantly.

INRNE's participation in LCG Elena Puncheva Preslav Konstantinov IT Department.

Monitoring for the ALICE O 2 Project 11 February 2016.

LECC2004 BostonMatthias Müller The final design of the ATLAS Trigger/DAQ Readout-Buffer Input (ROBIN) Device B. Gorini, M. Joos, J. Petersen, S. Stancu,

Joint Institute for Nuclear Research Synthesis of the simulation and monitoring processes for the data storage and big data processing development in physical.

Jianming Qian, UM/DØ Software & Computing Where we are now Where we want to go Overview Director’s Review, June 5, 2002.

ANDREA NEGRI, INFN PAVIA – NUCLEAR SCIENCE SYMPOSIUM – ROME 20th October

BaBar & Grid Eleonora Luppi for the BaBarGrid Group TB GRID Bologna 15 febbraio 2005.

Emanuele Leonardi PADME General Meeting - LNF January 2017

1 MANAGING THE DIGITAL INSTITUTION.

5/14/2018 The ATLAS Trigger and Data Acquisition Architecture & Status Benedetto Gorini CERN - Physics Department on behalf of the ATLAS TDAQ community.

U.S. ATLAS TDAQ FY06 M&O Planning

CMS High Level Trigger Configuration Management

Enrico Gamberini, Giovanna Lehmann Miotto, Roland Sipos

Controlling a large CPU farm using industrial tools

Introduction to Grid Technology

Operating the ATLAS Data-Flow System with the First LHC Collisions

Off-line & GRID Computing

ATLAS Canada Alberta Carleton McGill Montréal Simon Fraser Toronto

High Energy Physics Computing Coordination in Pakistan

ATLAS Canada Alberta Carleton McGill Montréal Simon Fraser Toronto

12/3/2018 The ATLAS Trigger and Data Acquisition Architecture & Status Benedetto Gorini CERN - Physics Department on behalf of the ATLAS TDAQ community.

Remote Online Farms TDAQ Sander Klous ACAT April

1/2/2019 The ATLAS Trigger and Data Acquisition Architecture & Status Benedetto Gorini CERN - Physics Department on behalf of the ATLAS TDAQ community.

Computing activities at Victoria

Presentation transcript:

Remote Online Farms Sander Klous H t W Z  0 On behalf of the Remote Online Farms Working Group ACAT April

2 Large Hadron Collider and ATLAS

3 Data processing nightmare There is no way to store all the info produced by ATLAS 40 million events per second x 1.5 MB/event = 60 TB per second In fact: % of the data is thrown away - So… The data processing nightmare is all about storage Unfortunately… No Rigorous multilevel trigger system - First level in hardware - Higher levels in software But what if your favorite channel is not in the %?

4 Online bottleneck So… The data processing nightmare is all about CPU LVL1 HLT Scarce CPU resources Well, that’s a problem…

5 Some are more equal than others… Detector calibration Physics selection Networking enables us to prioritize these activities So… The data processing nightmare is all about networking

6 Amsterdam NIKHEF/SARA Data Acquisition 40 MHz Level 1 Level 2 Accept 1 in 500 Accept 1 in 50 Accept 1 in 10 Level 3 Computing grid Network switch Economics In fact, it is about balance Maximize performance  Minimize costs

7 Gary Stix, editor of Scientific American Is it worth the effort? January 2001

8 This can be difficult… The basics ROB Data Collection Network L2PU EF Event Filter North Area ATLAS Detectors Level 1 Trigger Back End Network SFO Mass storage Bdlg. 513 Copenhagen Edmonton Krakow Manchester Amsterdam RF Remote Event Processing Farms RF Packet Switched (GEANT) Switch lightpath Local Farm The “Magni” Cluster ROB SFI

9 Input ExtPT Trash Output EFD B C D LVL2 RoiB L2sv L2pu Ros/RobinpRos DFM SFI (partially) build SFO Stream selection Stream n Stream 1Stream 2 Ath / CALid Ath / PESA PT From LVL1 Event LVL1 Info Add to StreamTag1 Stripping Ath / CalStr Ath / CALIB PT Add to RoutingTag Create RoutingTag Routing Duplicating Routing (Partial) Event LVL1 Info RoutingTag StreamTag 1 Event LVL1 Info RoutingTag StreamTag 2 Event LVL1 Info RoutingTag StreamTag 2 (Partial) Event LVL1 Info RoutingTag StreamTag (Partial) Event LVL1 Info RoutingTag StreamTag 1 Add to StreamTag2 Create StreamTag Remote Stream implementation

10 Grid/Proxy implementation SFI EFD Buffer PT Local PT Farm EFD Buffer PT Dispatcher Broker CE int.eu.grid HEP VO Infrastructure monitoring Application Monitoring Remote PTs Proxy PT Events HEP VO Database Worker Nodes UI Worker Nodes

11

12

13 Open issues Data management Software management Database access Authentication, Authorization and Accounting Performance and reliability Looking for PhD student… - If you are interested: Mail to

14 Conclusion Remote online farms are interesting - From a physics perspective - From a computer science perspective - From an organizational perspective Infrastructure is put in place - Many open questions - More news next year PhD candidates: mail to