Study on Prototype KVN high-speed Recorder Development using FirmOS Jae-Hwan Yeom, Se-Jin Oh, Duk-Gyoo-Roh, Dong-Kyu Jung, Chungsik Oh, Hyo-Ryoung Kim, Jae-Sik Shin, Ju-Yeon Hwang 6th IVTW, 09~11 October, 2017, @ INAF, Bologna, Italy

Contents Motivation Embedded Halcyon FirmOS Prototype KVN high-speed recorder Development and Experiments Future works

The East Asia VLBI Network Nanshan 26 m Tamna 21 m Miyun 50 m Yonsei 21 m Ulsan 21 m Gifu 11 m Usuda 64 m Tomakomai 11 m KJCC(Daejeon Corr.) Mizusawa 20 m Tianma 65 m Takahagi 32 m Sheshan 25 m Hitachi 32 m Kunming 40 m Kashima 34 m Ishigakijima 20 m Iriki 20 m Yamaguchi 32 m Ogasawara 20 m Tsukuba 32 m The East Asia VLBI Network (Image Credit: Reto Stöckli, NASA Earth Observatory) 6.7 GHz 8 GHz 22 GHz 43 GHz

KaVA (KVN and VERA Array) -future request- KYS KTN KUS VERA KVN/VERA, KaVA -1~2 Gbps normal Expected Obs speed in future 8Gbps obs (80TB/day,stn) 32Gbps obs (320TB/day,stn)

Korea-Japan Correlation Center(KJCC) Executive Board Daejeon Hardware Correlator DiFX Correlator on HPC

Recorders in the World(8~32Gbps) OCTADISK2 (VSREC@NAOJ) @Haystack @JIVE OCTADISK2 by Elecs Adopted Linux OS base, Specific own or Linux file system Resource Managements are needed Expensive(cost)

High-speed massive data processing FPGA(Field Programmable Gate Array) Base Secure the reliability and real-time of Parallel Processing Increase the development cost with log-scale in proportion to computation If computation is increased, it is difficult for synthesis, place & route. Require the specialty for design of PCB Ex) PCB design DSP or FPGA based Developing period 1~3years

OS based Workstation & Server Development Advantage Low developing cost compared with hardware Short developing period by freely program debug and modification Disadvantage According to OS scheduler and priority policy, it seems that the developing system will be unstable without securing the reliability. Developer could not fully use the system resource and performance during developing. I/O device access is complicated and very slow due to file system, virtual address, virtual memory, and kernel HAL(Hardware Abstraction Layer) structure.

Embedded FirmOS(Firmware+OS) architecture System Call Interface (Console Command) Hardware (RAM, Console, 1G/10G Eth, SATA) Process Management Memory Device Control Networking Storage Architecture dependent Code Manager Character Network Driver SATA Disk Concurrency/Multitasking Access Connectivity

FirmOS starting process Begin Finish ROM BIOS Initialize VGA Mode Switch Real Mode Protected Mode(32bit) IA-32e Mode(64bit) RAM AHCI(SATA) Copy 64bit Code SATA to RAM Copy 32bit Code SATA to RAM Bootloader GDT/TSS IDT Scheduler PIT(Timer) Keyboard PIC Enable Interrupt Serial Port Console

Recording process Input data from NIC is not pass Begin Initialize SATA NIC Record ON? No Packet UDP? Yes Overflow DRAM Block? Store to DRAM Transfer DRAM to SATA Disk Record OFF? Finish ISR (Interrupt Service Routine) Input data from NIC is not pass through CPU, directly recorded to HDD

Record data to HDD Write method Different according to OS or file system Generally outside to inside vice versa Adopt outside to inside writing method through tracks to increase the writing speed

Developing FirmOS, Halcyon (Specification) SATA 10ports (OS disk 1ea & Record disks) DDR4 RAM 32GBytes installed (Support Maximum 128GBytes) Intel Xeon multi-core Processor 10GbE NIC PCIe □ Target processor : Intel Xeon □ Target board ▪ Asrock X99 extreme11 ▪ DDR4 RAM 32GBytes ▪ Broadcom BCM57711 NIC 10GbE PCIe □ Operating System : FirmOS(like DOS) ▪ include scheduler ▪ support multi-core ▪ no filesystem ▪ DRAM/NIC/SATA control directly (without device driver) □ Build environment: gcc, nasm(boot code) □ can make full resources and performance ▪ recording speed 8.224Gbps(VDIF UDP) ▪ recordable capacity 90% of SATA HDD □ very cheap(Mainboard/CPU/RAM/NIC/Chassis) ▪ around $5,000 without HDDs

Experimental configuration @ Lab 2G x2ea 10GbE VSI UDP Noise + tone signal generator 4G Sampler VSI Divider & Fila10G File convert Linux server / SW correlation Halcyon KVN new recorder

1st Preliminary Recording Test @ KYS Halcyon Terminal SATA HDD x 8ea (8Gbps) □ Experiment code : a17094a(EAVN), 3C273 □ Recording time ▪ UT. 12:35:00 – 17:35:00 (5hours) ▪ 3.7TByte/hour at 8.224Gbps KVN Tamna 43GHz (Auto power spectrum) KVN Yonsei 43GHz Scan11 (15:05:00-) cross power phase Lag Scan16 (16:22:00-) DjROY Software Correlation

2nd KVN new recorder(Halcyon) test @ KYS Observing method : in 8 hours observing time, Halcyon KVN recorder was used for first 4 hours, then Mark6 recorder was used last 4 hours. After finished, the correlation result was compared between Halcyon and Mark6. Observation info *s17tj03a : 1/8Gbps simultaneous recording test May 24 08:00 UT - May 24 16:29 UT Sources : 3C273, 3C279, M87 Frequency : 22L 43L 86L 129L (Freq4x512MHz) Recorder : Mark5B(1G) / [(Halcyon) /Mark6(8G)] Recording Time : 28200 (sec) Disk Size : 28876.0 (GB) 22/43/86/129GHz simultaneous observing (Apply each IF 1/2/3/4) SNR comparison was done by performing Fringe fitting(30sec) as reference antenna of KYS

KYS-KTN baseline 1Gbps SNR (MK5) 8Gbps SNR (Halcyon + MK6) 3C273, 3C279, M87 Empty period according to recorder change 22GHz 43GHz 86GHz 129GHz MK5 recorder Halcyon recorder MK6 recorder ※ SNR comparison: 1Gbps(64 MHz BW/4IF) and 8Gbps(512MHz BW/4IF) Expected value of theoretical sensitivity increment (about 3 times) is well followed according to the bandwidth increase (8 times). The results of Halcyon are almost same as those of KYS Mark6

KYS-KUS baseline 1Gbps SNR (MK5) 8Gbps SNR (Halcyon + MK6) Empty period according to recorder change MK5 recorder Halcyon recorder MK6 recorder ※ SNR comparison: 1Gbps(64 MHz BW/4IF) and 8Gbps(512MHz BW/4IF) Expected value of theoretical sensitivity increment (about 3 times) is well followed according to the bandwidth increase (8 times). The results of Halcyon are almost same as those of KYS Mark6

Web-based control for remote operation

Chassis Assembly

Future works FUSE file system support Direct SW correlator support NVMe protocol support for SSD HDD User defined command support Various device support