Plans for the renovation of the Post Mortem infrastructure TE-MPE-TM 82 15/09/2016 Matthias Pöschl – TE-MPE-MS

Matthias Pöschl – TE-MPE-MS Agenda Current Post Mortem Architecture Shortcomings Improvements New Post Mortem Storage New File Format Collaboration with CALS 2.0 Shared data and infrastructure New Post Mortem Architecture Conclusion 15/09/2016 Matthias Pöschl – TE-MPE-MS

Current Archtitecture 15/09/2016 Matthias Pöschl – TE-MPE-MS

Matthias Pöschl – TE-MPE-MS Shortcomings Direct user access to underlaying filesystem and file format Outdated data collection stack Manual load balancing Very limited horizontal scaling Unfit for future use-cases with strict time constraints 15/09/2016 Matthias Pöschl – TE-MPE-MS

Matthias Pöschl – TE-MPE-MS Improvements Update storage technology Allow for dynamic load balancing Update data collection stack and file format User access only through REST API 15/09/2016 Matthias Pöschl – TE-MPE-MS

New Post Mortem Storage Benchmark of Ceph, MongoDB and GlusterFS Use of real Post Mortem Data (all of 2015) Constraints: Technology has to handle drive and node failures, avoid data inconsistencies and degrade gracefully Three replicas of each object have to be stored Acknowledge of write only after all three copies have been written Adding more nodes should increase capacity and throughput by a known factor (“linear scaling”) 15/09/2016 Matthias Pöschl – TE-MPE-MS

New Post Mortem Storage GlusterFS performed best in read-only and write-only benchmarks, very closely followed by Ceph Ceph showed best mixed-workload performance: Technology Time to complete Objects/s Throughput NFS 26.3h 340 8,018 KB/s Ceph 20.4h 438 12,154 KB/s MongoDB 23.5h 381 8,973 KB/s GlusterFS 20.7h 432 10,187 KB/s Explain Testbench, Servers, HDD rpms, … 15/09/2016 Matthias Pöschl – TE-MPE-MS

New Post Mortem Storage Meeting with Dan van der Ster and Herve Rousseau from IT-ST-FDO to discuss Ceph IT has had good experiences with Ceph Biggest Ceph test so far with a ~30PB cluster IT is running the VMs on top of Ceph Not a single byte lost in +5 years of operation IT sees no problem in using it for Post Mortem IT is willing to provide support and assistance 15/09/2016 Matthias Pöschl – TE-MPE-MS

New Post Mortem Storage Test with test Ceph cluser provided by IT Explain variations in graph -> Filesize Benchmark Time to complete Objects/s Throughput Import 12.9h 345 8,135 KB/s Read 3.5h 1269 30,080 KB/s Read/Write 16.8h 795 12,552 KB/s 15/09/2016 Matthias Pöschl – TE-MPE-MS

Matthias Pöschl – TE-MPE-MS New file format Aspirations on using Apache Avro as file format for storage and to serve the users “Raw” RDA-data will still be stored for safety Avro offers many useful features Partial data retrieval (specific signals from a dump) Efficient and fast compression Self-describing schema Libraries for almost every programming language Allows direct conversion to JSON 15/09/2016 Matthias Pöschl – TE-MPE-MS

Collaboration with CALS 2.0 The Logging Service team is updating their whole infrastructure Oracle cluster will be replaced with Hadoop 15/09/2016 Matthias Pöschl – TE-MPE-MS

Collaboration with CALS 2.0 New CALS design will allow BigData queries “Show me the biggest deviations from the mean of a certain BLM in sector 67 in the year 2015 when dumping at 6.5 TeV” Idea: Feed the Logging Service the high resolution Post Mortem data to make this kind of queries even more valuable for the users 15/09/2016 Matthias Pöschl – TE-MPE-MS

Collaboration with CALS 2.0 Meetings with Chris Roderick, Jakub Wozniak and Marcin Sobieszek from BE-CO-DS to evaluate common use of technologies and services as well as details of data ingestion Result: CALS might also use Avro for their data ingestion Possible shared use of a Kafka cluster Shared data storage not (yet) possible 15/09/2016 Matthias Pöschl – TE-MPE-MS

New Post Mortem Architecture 15/09/2016 Matthias Pöschl – TE-MPE-MS

Matthias Pöschl – TE-MPE-MS Conclusion Ceph cluster for Post Mortem data seems feasible and suitable to tackle future use cases No common storage of CALS and PM (yet ) different timing constraints and data sizes different preferred storage technologies But: Data and infrastructure can be shared between both systems, providing a good trade-off for the users 15/09/2016 Matthias Pöschl – TE-MPE-MS

Matthias Pöschl – TE-MPE-MS Conclusion Avro allows efficient data storage, convenience for the users and easy integration with CALS 2.0 All data access through a REST API, serving uncompressed JSON compressed Avro 15/09/2016 Matthias Pöschl – TE-MPE-MS