1. Study on Prototype KVN high-speed Recorder Development using FirmOS Jae-Hwan Yeom, Se-Jin Oh, Duk-Gyoo-Roh, Dong-Kyu Jung, Chungsik Oh, Hyo-Ryoung Kim, Jae-Sik Shin, Ju-Yeon Hwang
6th IVTW, 09~11 October, INAF, Bologna, Italy

2. Contents
Motivation
Embedded Halcyon FirmOS
Prototype KVN high-speed recorder Development and Experiments
Future works

3. The East Asia VLBI Network
Nanshan 26 m
Tamna 21 m
Miyun 50 m
Yonsei 21 m
Ulsan 21 m
Gifu 11 m
Usuda 64 m
Tomakomai 11 m
KJCC(Daejeon Corr.)
Mizusawa 20 m
Tianma 65 m
Takahagi 32 m
Sheshan 25 m
Hitachi 32 m
Kunming 40 m
Kashima 34 m
Ishigakijima 20 m
Iriki 20 m
Yamaguchi 32 m
Ogasawara 20 m
Tsukuba 32 m
The East Asia VLBI Network
(Image Credit: Reto Stöckli, NASA Earth Observatory)
6.7 GHz
8 GHz
22 GHz
43 GHz

4. KaVA (KVN and VERA Array) -future request-
KYS
KTN
KUS
VERA
KVN/VERA, KaVA
-1~2 Gbps normal
Expected Obs speed in future
8Gbps obs (80TB/day,stn)
32Gbps obs (320TB/day,stn)

5. Korea-Japan Correlation Center(KJCC)
Executive Board
Daejeon Hardware Correlator
DiFX Correlator on HPC

6. Recorders in the World(8~32Gbps)
OCTADISK2
@Haystack
@JIVE
OCTADISK2 by Elecs
Adopted Linux OS base, Specific own or Linux file system
Resource Managements are needed
Expensive(cost)

7. High-speed massive data processing
FPGA(Field Programmable Gate Array) Base
Secure the reliability and real-time of Parallel Processing
Increase the development cost with log-scale in proportion to computation
If computation is increased, it is difficult for synthesis, place & route.
Require the specialty for design of PCB
Ex) PCB design
DSP or FPGA based
Developing period 1~3years

8. OS based Workstation & Server Development
Advantage
Low developing cost compared with hardware
Short developing period by freely program debug and modification
Disadvantage
According to OS scheduler and priority policy, it seems that the developing system will be unstable without securing the reliability.
Developer could not fully use the system resource and performance during developing.
I/O device access is complicated and very slow due to file system, virtual address, virtual memory, and kernel HAL(Hardware Abstraction Layer) structure.

9. Embedded FirmOS(Firmware+OS) architecture
System Call Interface
(Console Command)
Hardware
(RAM, Console, 1G/10G Eth, SATA)
Process
Management
Memory
Device
Control
Networking
Storage
Architecture dependent
Code
Manager
Character
Network
Driver
SATA
Disk
Concurrency/Multitasking
Access
Connectivity

10. FirmOS starting process
Begin
Finish
ROM BIOS
Initialize
VGA
Mode
Switch
Real Mode
Protected Mode(32bit)
IA-32e Mode(64bit)
RAM
AHCI(SATA)
Copy 64bit Code SATA to RAM
Copy 32bit Code SATA to RAM
Bootloader
GDT/TSS
IDT
Scheduler
PIT(Timer)
Keyboard
PIC
Enable
Interrupt
Serial Port
Console

11. Recording process Input data from NIC is not pass
Begin
Initialize
SATA
NIC
Record ON? No
Packet
UDP?
Yes
Overflow
DRAM Block?
Store to
DRAM
Transfer
DRAM to SATA Disk
Record OFF?
Finish
ISR
(Interrupt Service Routine)
Input data from NIC is not pass
through CPU, directly recorded
to HDD

12. Record data to HDD Write method
Different according to OS or file system
Generally outside to inside vice versa
Adopt outside to inside writing method through tracks to increase the writing speed

13. Developing FirmOS, Halcyon
(Specification)
SATA 10ports
(OS disk 1ea & Record disks)
DDR4 RAM 32GBytes installed
(Support Maximum 128GBytes)
Intel Xeon multi-core
Processor
10GbE NIC
PCIe
□ Target processor : Intel Xeon
□ Target board
▪ Asrock X99 extreme11
▪ DDR4 RAM 32GBytes
▪ Broadcom BCM57711 NIC 10GbE PCIe
□ Operating System : FirmOS(like DOS)
▪ include scheduler
▪ support multi-core
▪ no filesystem
▪ DRAM/NIC/SATA control directly
(without device driver)
□ Build environment: gcc, nasm(boot code)
□ can make full resources and performance
▪ recording speed 8.224Gbps(VDIF UDP)
▪ recordable capacity 90% of SATA HDD
□ very cheap(Mainboard/CPU/RAM/NIC/Chassis)
▪ around $5,000 without HDDs

14. Experimental configuration @ Lab
2G x2ea
10GbE
VSI
UDP
Noise + tone signal generator
4G Sampler
VSI Divider & Fila10G
File
convert
Linux server
/ SW correlation
Halcyon
KVN new recorder

15. 1st Preliminary Recording Test @ KYS
Halcyon Terminal
SATA HDD x 8ea
(8Gbps)
□ Experiment code : a17094a(EAVN), 3C273
□ Recording time
▪ UT. 12:35:00 – 17:35:00 (5hours)
▪ 3.7TByte/hour at 8.224Gbps
KVN Tamna 43GHz
(Auto power spectrum)
KVN Yonsei 43GHz
Scan11
(15:05:00-)
cross
power
phase
Lag
Scan16
(16:22:00-)
DjROY
Software Correlation

16. 2nd KVN new recorder(Halcyon) test @ KYS
Observing method : in 8 hours observing time, Halcyon KVN recorder was used for first 4 hours, then Mark6 recorder was used last 4 hours. After finished, the correlation result was compared between Halcyon and Mark6.
Observation info
*s17tj03a : 1/8Gbps simultaneous recording test
May 24 08:00 UT - May 24 16:29 UT
Sources : 3C273, 3C279, M87
Frequency : 22L 43L 86L 129L (Freq4x512MHz)
Recorder : Mark5B(1G) / [(Halcyon) /Mark6(8G)]
Recording Time : (sec)
Disk Size : (GB)
22/43/86/129GHz simultaneous observing (Apply each IF 1/2/3/4)
SNR comparison was done by performing Fringe fitting(30sec) as reference antenna of KYS

17. KYS-KTN baseline 1Gbps SNR (MK5) 8Gbps SNR (Halcyon + MK6)
3C273, 3C279, M87
Empty period according to recorder change
22GHz
43GHz
86GHz
129GHz
MK5 recorder
Halcyon recorder
MK6 recorder
※ SNR comparison: 1Gbps(64 MHz BW/4IF) and 8Gbps(512MHz BW/4IF) Expected value of theoretical sensitivity increment (about 3 times) is well followed according to
the bandwidth increase (8 times).
The results of Halcyon are almost same as those of KYS Mark6

18. KYS-KUS baseline 1Gbps SNR (MK5) 8Gbps SNR (Halcyon + MK6)
Empty period according to recorder change
MK5 recorder
Halcyon recorder
MK6 recorder
※ SNR comparison: 1Gbps(64 MHz BW/4IF) and 8Gbps(512MHz BW/4IF) Expected value of theoretical sensitivity increment (about 3 times) is well followed according to
the bandwidth increase (8 times).
The results of Halcyon are almost same as those of KYS Mark6

19. Web-based control for remote operation

20. Chassis Assembly

21. Future works FUSE file system support Direct SW correlator support
NVMe protocol support for SSD HDD
User defined command support
Various device support