1. <header>
The Institute of Applied Astronomy of the Russian Academy of Sciences
Operating experience of Data Transmission and Buffer System. Research of a disk subsystem performance.
Ilya Bezrukov1, Andrey Vylegzhanin2,3, Alexandre Salnikov1, Vladislav Yakovlev1.
1. Institute of Applied Astronomy Russian Academy of Sciences
2. SPbD JSCC RAS - Branch of SRISA
3. Ioffe Physical Technical Institute Russian Academy of Sciences
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

2. Operating experience of Data Transmission and Registration System.
The Institute of Applied Astronomy of the Russian Academy of Sciences
<header>
Operating experience of Data Transmission and Registration System.
RT-13 since December 2015:
About 2500 sessions;
3PB from each obs per year.
4Gbps
2Gbps
2 Gbps
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

3. DTRS: General Key features
<header>
The Institute of Applied Astronomy of the Russian Academy of Sciences
DTRS: General Key features
1) registration of up to eight 2Gbps per channel data streams from Broadband Data Acquisition System (BRAS);
2) transfer of big volume of data to the CPC using 10Gbps links and simultaneous scientific data registration;
3) storing of up to 20TB of data at stations and up to 40TB of data at the CPC using DTRS disk subsystem.
200m optical link
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

4. Direct-attached storage - Dell PV M1220
The Institute of Applied Astronomy of the Russian Academy of Sciences
<header>
DTRS: hardware and software layout
OS: FreeBSD 10.1;
FS: ZFS (Zettabyte Filesystem);
Netmap: a novel framework for fast packet I/O;
Capture RAW Ethernet packets (jumbo frame);
UDT for data transmission
Server - Dell PE R720
CPU
2xIntel Xeon E
RAM
256 Gb 1600 MHz
PCI-e v3.0
1 x16; 6 x8
Internal HBAs (SAS2008)
PERC H310 (LSI SAS i)
External HBAs
(SAS2008)
2x6Gbps SAS HBA (LSI SAS e)
NIC
4 x Intel(R) 10G 2P X520 adapter
Drives
2 x 2.5” NL-SAS (OS)
Direct-attached storage - Dell PV M1220
HBAs, 6Gb/s SAS
PERC H800
Drives
48 x 2.5” hot-pluggable SAS
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

5. <header>
The Institute of Applied Astronomy of the Russian Academy of Sciences
Conclusion
The operating experience of the Data Transmission and Registration System (DTRS) in IAA RAS is successful. The DTRS successfully registers a data stream with a speed up to 16Gbps with simultaneous transmission to the Correlation and Processing Center.
since mid-2016 six one-hour-long sessions are taking place daily (about 2500 total);
the amount of scientific data is about 8TB per day or 3PB per year from one observatory;
2Gbps communication channels allow to finish data transfer within 3 hours after the beginning of each one-hour-long session;
average observational cycle of data recording/antenna pointing is 10s/20s.
Major flaw of current DTRS is relatively small amount of data storage. Therefore it isn’t possible to record and store day-long sessions.
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

6. Research of a disk subsystem performance.
The Institute of Applied Astronomy of the Russian Academy of Sciences
<header>
Research of a disk subsystem performance.
Results of testing disk performance for various observation cycles (3th International VLBI Technology Workshop 2014)
Observational cycle recording/pointing, sec
1 channel write rate, MBps
type of disks
guiding the source
256
SAS
40/20
190
30/30
141
SAS/SATA
7.5/22.5 (VGOS) 65
Performance of disk subsystem of current DTRS (observational cycle 10/20, iostat)
IOPS
Read, Mbps
Write, Mbps
Latency, ms
Read
Write
30.927
74.471
29.849
30.658
75.376
31.225
30.743
75.116
30.077
30.406
76.383
38.028
30.513
75.042
37.861
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

7. Technique and test conditions
<header>
The Institute of Applied Astronomy of the Russian Academy of Sciences
Technique and test conditions
Test technique (the worst scenario of use):
Record one scan during long time (1800 s);
Simultaneous write and read;
Parallel test (up to 8 threads);
No buffering Input/Output operations.
HDD types:
Seagate ST600MM0006, 2.5” SAS 600 GB, 10К RPM, 2014;
Seagate ST6000NM0095, 3.5” NL-SAS 6 TB, 7.2K RPM, 2017.
Disk group types:
2 disks stripe(raid0);
4 disks stripe(raid0)
Utilities: dd
fio*
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

8. Results Test configuration: Max write speed test per disk group
<header>
The Institute of Applied Astronomy of the Russian Academy of Sciences
Results
Test configuration:
Max write speed test per disk group
Performance test at 256MBps (write) + 30MBps (read)
Performance test at 512MBps (write) + 60MBps (read)
Max write speed test per disk group with different configuration:
Number of hard drives
HDD type
IOPS
Latency, ms
Avg write speed, MBps 2
SAS (2014)
2404
0,407
315,125
NL-SAS (2017)
3385
0,288
443,69 4
3425
0,286
448,95
5119
0,188
671,16
6th International VLBI Technology Workshop – Bologna 9-11 October 2017
<footer>

9. The Institute of Applied Astronomy of the Russian Academy of Sciences
Results
Performance test at 256MBps (write) + 30MBps (read)
Number of hard drives
HDD type
IOPS
Latency, ms
Avg speed, MBps
Read
Write 2
SAS
233
1594
3,023
0,611
30,53
209
NL-SAS
240
2048
2,682
0,094
31,5
268 4
239
1726
3,38
0,596
31,2
226
0,151
Performance test at 512MBps (write) + 60MBps (read)
Number of hard drives
HDD type
IOPS
Latency, ms
Avg speed, MBps
Read
Write 4
NL-SAS
480
4095
1,1
0,095 62
537
6th International VLBI Technology Workshop – Bologna 9-11 October 2017

10. Conclusions and future plans
The Institute of Applied Astronomy of the Russian Academy of Sciences
Conclusions and future plans
Modern NL-SAS HDDs being combined into groups of two give a possibility to simultaneously record 2Gbps stream and to read stored data. Thus total number of hard drives in storage system may be reduced by half without significant performance drop and storage capacity therefore may be further increased.
IAA RAS is going to upgrade DTRS disk subsystems at ‘Badary’ and ‘Zelenchukskaya’ this year. Each station will be able to record 16Gbps data streams. By installation of 20 NL-SAS 6TB hard drives storage capacity will be increased to 100TB. This upgrade will finally let us record day-long observations.
In the future, if we have a necessity of recording 4Gbps data streams, the amount of hard drives will be increased to 40.
6th International VLBI Technology Workshop – Bologna 9-11 October 2017

11. Thank you for attention!
The Institute of Applied Astronomy of the Russian Academy of Sciences
Thank you for attention!
6th International VLBI Technology Workshop – Bologna 9-11 October 2017