1. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 1 Timm M. Steinbeck HLT Data Transport Framework

2. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 2 Outline – Overview Framework / Components – HLT Control System – Benchmarks & Tests ● Framework Communication Benchmark ● Framework Interface Benchmark ● Framework Sample Configurations – Summary & Conclusion

3. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 3 Overview of the Framework – Framework consisting of independant components – Communicating via publisher-subscriber interface ● Data driven architecture ● Components receive data from other components ● Process received data ● Forward own output data to next component – Named pipes used to exchange small messages and descriptors – Data is exchanged via shared memory Major design criteria ● Efficiency: CPU Cycles used for analysis ● Flexibility: Different configurations needed ● Fault-Tolerance: PCs are unreliable Subscriber Processing Publisher Subscriber Processing Publisher Shared Memory Data Named Pipe

4. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 4 Application Components Templates for application specific components Data Source Addressing & Handling Buffer Management Data Announcing Data Analysis & Processing Output Buffer Mgt. Output Data Announcing Accepting Input Data Input Data Addressing Data Sink Addressing & Writing Accepting Input Data Input Data Addressing ● Read data from source and publish it (Data Source Template) ● Accept data, process it, publish results (Analysis Template) ● Accept data and process it, e.g. storing (Data Sink Template)

5. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 5 Framework contains components to shape the flow of data in a cluster All components use publisher-subscriber interface Data Flow Components Subscriber Publisher 1.2; 2.2;... 1; 2;... Subscriber 1.1; 2.1;... Subscriber 1.3; 2.3;... Subscriber Publisher 1; 2;... Subscriber Publisher Subscriber 1; 2;... Publisher Subscriber Communication Node A Node B Network Merge parts of events Transparently connect components on different nodes (bridges) Split and rejoin a data stream (e.g. for load-balancing) (scatter/gather)

6. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 6 HLT Control System Purpose of the HLT Control System: Management of HLT Processes – Supervise processes – React to state changes ● E.g. errors, unexpected termination – Manage system startup ● Connect components only when they are ready ● Connect bridges between nodes when nodes are ready – Orchestration of HLT system

7. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 7 Requirements – Flexible ● HLT Configurations not yet defined ● Usable for other programs as well – Hierarchical ● >2000 processes not manageable by single supervisor instance ● Eases configurations – No single point of failure ● PC cluster nodes are unreliable ● No need for highly reliable, expensive, equipment – Should be able to run on the cluster nodes themselves, along the analysis processes.

8. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 8 Core TaskManager Architecture Configurati on Engine XML Configuratio n File Interface Library Controlled Programs Program State & Action Engine Embedded Python Interpreter TaskManager Program Interface Engine

9. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 9 Framework Communication Benchmark ● Network Throughput – GbE, InfiniBand w. TCP (IPoIB, SDP) IB w. uDAPL

10. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 10 Framework Communication Benchmark ● CPU Load per Network Throughput (sender+receiver)

11. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 11 Interface Benchmarks Benchmark of only Publisher-Subscriber Interface No shared memory access or anything else Benchmarks sensitive to L1 cache size (P4 family only 8 kB) Maximum rate achieved: >110 kHz (Dual Opteron) Time overhead f. event round-trip: < 18  s

12. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 12 2 Node Network Test ● 2 nodes connected by bridge components ● Dual 3.2 GHz Xeon 4 w. EM64T ● Gigabit Ethernet ● Sender node: 1 FilePublisher ● Receiver node: 1 EventRateSubscriber ● 4 kB Dummy Events ● Achieved Rate: – 16.7 kHz ERS FP

13. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 13 FP CF FP CF FP CF TR “Compact” HLT Configuration ● HLT Test Configuration using 5 nodes to process 1 slice ● 3 ClusterFinder nodes – 2 FilePublishers + 2 ClusterFinders per node (1 each per patch) ● 2 Tracker nodes – 1 Tracker process per node

14. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 14 “Compact” HLT Configuration Results ● Dual 800 MHz Pentium 3, Fast Ethernet – 870 Hz (runtime 142h) ● Dual 3.2 GHz Xeon 4 w. EM64T, Gigabit Ethernet – 2.9 kHz ● Using empty events FP CF FP CF FP CF TR

15. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 15 Local Scatterer Test ● Test on one node – 1 FilePublisher – 1 Scatterer – 2 DummyLoads ● Rates: – Dual 800 MHz Pentium 3 ● 6 kHz – Single 1.6 GHz Opteron 242 ● 9.8 kHz – Dual 1.6 GHz Opteron 242 ● 16.8 kHz FP S DL

16. Timm M. Steinbeck - Kirchhoff Institute of Physics - University Heidelberg 16 HLT Data Transport Framework Summary ● HLT Data Transport Framework has matured ● Already being used in projects, testbeams, commissionings, HLT data challenges ● Stable, 1 setup running for 1 month continuously ● Performance improved significantly – Enough for HLT already now ● Component approach already proved useful – Easy to create configurations for short tests – Flexible configuration changes, e.g. during testbeams, commissioning, data challenges