<header> The Institute of Applied Astronomy of the Russian Academy of Sciences Operating experience of Data Transmission and Buffer System. Research of a disk subsystem performance. Ilya Bezrukov1, Andrey Vylegzhanin2,3, Alexandre Salnikov1, Vladislav Yakovlev1. 1. Institute of Applied Astronomy Russian Academy of Sciences 2. SPbD JSCC RAS - Branch of SRISA 3. Ioffe Physical Technical Institute Russian Academy of Sciences 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

Operating experience of Data Transmission and Registration System. The Institute of Applied Astronomy of the Russian Academy of Sciences <header> Operating experience of Data Transmission and Registration System. RT-13 since December 2015: About 2500 sessions; 3PB from each obs per year. 4Gbps 2Gbps 2 Gbps 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

DTRS: General Key features <header> The Institute of Applied Astronomy of the Russian Academy of Sciences DTRS: General Key features 1) registration of up to eight 2Gbps per channel data streams from Broadband Data Acquisition System (BRAS); 2) transfer of big volume of data to the CPC using 10Gbps links and simultaneous scientific data registration; 3) storing of up to 20TB of data at stations and up to 40TB of data at the CPC using DTRS disk subsystem. 200m optical link 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

Direct-attached storage - Dell PV M1220 The Institute of Applied Astronomy of the Russian Academy of Sciences <header> DTRS: hardware and software layout OS: FreeBSD 10.1; FS: ZFS (Zettabyte Filesystem); Netmap: a novel framework for fast packet I/O; Capture RAW Ethernet packets (jumbo frame); UDT for data transmission Server - Dell PE R720 CPU 2xIntel Xeon E5-2643 V2 @3.50 GHz RAM 256 Gb 1600 MHz PCI-e v3.0 1 x16; 6 x8 Internal HBAs (SAS2008) PERC H310 (LSI SAS 9211-8i) External HBAs (SAS2008) 2x6Gbps SAS HBA (LSI SAS 9200-8e) NIC 4 x Intel(R) 10G 2P X520 adapter Drives 2 x 2.5” NL-SAS (OS) Direct-attached storage - Dell PV M1220 HBAs, 6Gb/s SAS PERC H800 Drives 48 x 2.5” hot-pluggable SAS 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

<header> The Institute of Applied Astronomy of the Russian Academy of Sciences Conclusion The operating experience of the Data Transmission and Registration System (DTRS) in IAA RAS is successful. The DTRS successfully registers a data stream with a speed up to 16Gbps with simultaneous transmission to the Correlation and Processing Center. since mid-2016 six one-hour-long sessions are taking place daily (about 2500 total); the amount of scientific data is about 8TB per day or 3PB per year from one observatory; 2Gbps communication channels allow to finish data transfer within 3 hours after the beginning of each one-hour-long session; average observational cycle of data recording/antenna pointing is 10s/20s. Major flaw of current DTRS is relatively small amount of data storage. Therefore it isn’t possible to record and store day-long sessions. 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

Research of a disk subsystem performance. The Institute of Applied Astronomy of the Russian Academy of Sciences <header> Research of a disk subsystem performance. Results of testing disk performance for various observation cycles (3th International VLBI Technology Workshop 2014) Observational cycle recording/pointing, sec 1 channel write rate, MBps type of disks guiding the source 256 SAS 40/20 190 30/30 141 SAS/SATA 7.5/22.5 (VGOS) 65 Performance of disk subsystem of current DTRS (observational cycle 10/20, iostat) IOPS Read, Mbps Write, Mbps Latency, ms Read Write 247.652 605.728 30.927 74.471 29.849 245.496 612.837 30.658 75.376 31.225 246.185 611.352 30.743 75.116 30.077 243.465 620.567 30.406 76.383 38.028 244.342 610.057 30.513 75.042 37.861 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

Technique and test conditions <header> The Institute of Applied Astronomy of the Russian Academy of Sciences Technique and test conditions Test technique (the worst scenario of use): Record one scan during long time (1800 s); Simultaneous write and read; Parallel test (up to 8 threads); No buffering Input/Output operations. HDD types: Seagate ST600MM0006, 2.5” SAS 600 GB, 10К RPM, 2014; Seagate ST6000NM0095, 3.5” NL-SAS 6 TB, 7.2K RPM, 2017. Disk group types: 2 disks stripe(raid0); 4 disks stripe(raid0) Utilities: dd fio* 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

Results Test configuration: Max write speed test per disk group <header> The Institute of Applied Astronomy of the Russian Academy of Sciences Results Test configuration: Max write speed test per disk group Performance test at 256MBps (write) + 30MBps (read) Performance test at 512MBps (write) + 60MBps (read) Max write speed test per disk group with different configuration: Number of hard drives HDD type IOPS Latency, ms Avg write speed, MBps 2 SAS (2014) 2404 0,407 315,125 NL-SAS (2017) 3385 0,288 443,69 4 3425 0,286 448,95 5119 0,188 671,16 6th International VLBI Technology Workshop – Bologna 9-11 October 2017 <footer>

The Institute of Applied Astronomy of the Russian Academy of Sciences Results Performance test at 256MBps (write) + 30MBps (read) Number of hard drives HDD type IOPS Latency, ms Avg speed, MBps Read Write 2 SAS 233 1594 3,023 0,611 30,53 209 NL-SAS 240 2048 2,682 0,094 31,5 268 4 239 1726 3,38 0,596 31,2 226 0,151 Performance test at 512MBps (write) + 60MBps (read) Number of hard drives HDD type IOPS Latency, ms Avg speed, MBps Read Write 4 NL-SAS 480 4095 1,1 0,095 62 537 6th International VLBI Technology Workshop – Bologna 9-11 October 2017

Conclusions and future plans The Institute of Applied Astronomy of the Russian Academy of Sciences Conclusions and future plans Modern NL-SAS HDDs being combined into groups of two give a possibility to simultaneously record 2Gbps stream and to read stored data. Thus total number of hard drives in storage system may be reduced by half without significant performance drop and storage capacity therefore may be further increased. IAA RAS is going to upgrade DTRS disk subsystems at ‘Badary’ and ‘Zelenchukskaya’ this year. Each station will be able to record 16Gbps data streams. By installation of 20 NL-SAS 6TB hard drives storage capacity will be increased to 100TB. This upgrade will finally let us record day-long observations. In the future, if we have a necessity of recording 4Gbps data streams, the amount of hard drives will be increased to 40. 6th International VLBI Technology Workshop – Bologna 9-11 October 2017

Thank you for attention! The Institute of Applied Astronomy of the Russian Academy of Sciences Thank you for attention! 6th International VLBI Technology Workshop – Bologna 9-11 October 2017