VLBI Storage Manager (VSM) Matteo Stagni – Mauro Nanni 0

1.Attended antenna with local storage system (B) 2.Groups of antennas with a central storage system and a local correlator (C) 3.Unattended antenna with local storage system (D) 4.Antennas that use a remote storage (E or A ) Cloud ? 5.Antennas with poor network connection that use disk-pack Network and Storage in the future of the VLBI 1

Disk Pack - vs - Storage Systems Storage Systems:  Different hardware characteristics  Different capacity ( TBy) that change in time:  Presence of previous observation data  Space reserved for different observation programs  In a session for one antenna can be requested more storage systems  More antennas can store data on one storage system (low speed !)  Storage systems near the antennas or connected by high speed net Disk Pack:  Field System manage the Disk Packs substitution  Disk packs are empty and every Pack has a fixed capacity 2

VSM Storage Units Unit Box Capacity: TBy Network: 10 Gbit/sec File system: standard Writing speed : more than 4 Gbit/s Space on the unit: Used: data of previous observations Reserved : allocated for future obs. Free: can be allocated 3

Storage Units: Our Choice Supermicro Systems:  Motherboard: X8DTH-IF  CPU: 2 Intel XEON  RAM: 12 GB  RAID: 3WARE 9750DR  Disks: 24 + Os  Network: 10 Gbit  OS: Scientific Linux 6.2 Writing Raid 5 XFS Filesystem: Command dd : 1.2 GB/s Vlbistreamer : 6135 Mbit/s (raid 5 degraded: -10% in speed) 4

Write test on different filesystems Tests were run on a 12 disk RAID5 array with 22 TB space 5

Vlbi Storage Manager 6

The Vlbi Storage Manager (VSM) Our choices:  Hardware : commercial “Off The Shelf” components  Operating System: Linux  Standard file systems  Database Mysql  Shell Bash and PHP Different organizations can use different architectures and adopt the solution at a different pace The new storage device require a system for management of the data on disk:  management of disk space  reserving resources for the future sessions  overseeing the capture and registration  data transfer to the correlator 7

System Architecture 8

VSM general architecture VSM Network-DB: Session data: c/o the correlators or central organizations VSM Station-DB: Storage station data: c/o Radio observatory Storage stations get data of the sessions from the network DB 9

0 - Session sckedules are available 1 - Sckedules data are written in VSM-Network DB 2 – Station Manager get data in the VSM-StationDB 3 – Station Manager verify space (vsm_analize) 4 – Station Manager reserve the space (vsm_reserv) 5 – Station Manager create rec- sked (vsm_sched) 6 – Rec-Sked starts vlbi-stream at observation time 10

Network environment 11

This table is modeled after the present text file worked out in Bonn to coordinate european VLBI observations Vlbi antennas schedule 12

Each observation is registered In the antenna table 13

Each network station management host can be reached and its status monitored… 14

Network admin can conveniently update the vlbi observations through a FORM 15

System Architecture 16

Station environment 17

VSM Storage Station The “ VSM storage stations" are the structures that manage one or more "storage unit" under the supervision of the technical staff of the organization A storage station can manage one or more well connected antennas A VSM storage station is made by:  One or more storage units  The VSM Software  The VSM Database of space usage in the storage units  The VSM Database of the local experiments 18

The VSM System core concept is to manipulate the antennas’ scheduled and present data record on different MySQL databases. Dual implementation for the local admin management: - CLI (Command Line Interface) Bash - WEB-BASED Interface Php 19

Command set: Bash scriptAim vsm_atq.shCheck the at queue on machine vsm_clearReserv.shClear the antenna space reservation table vsm_storage_analyze.shCheck the space status for the machines assigned vsm_preSched.shCreate table filled with path and commands to submit vsm_storage_reserv.shReserve space for the antenna vsm_atqrm.shClear the jobs submitted to a machine vsm_sched.shSchedule the jobs on the machine(s) vsm_delete.shDelete sessions or experiments from machine filesystem CLI Interaction 20

WEB-BASED Interaction 21

A php replica of the bash command set… In detail: -Storage status -Reserve space -Schedule observations WEB-BASED Interaction 22

Through Storage status we get the space occupied on the filesystem by experiments on machines WEB-BASED Interaction 23

Reserve space has done its job and checked if there is enough room for the upcoming observations and now we’re ready to schedule them… WEB-BASED Interaction 24

The jobs are ready to be scheduled on the machines with the appropriate command at the selected time WEB-BASED Interaction 25

The jobs are now waiting on the machine in the at queue WEB-BASED Interaction 26

A primary PoP of the NREN at the IRA HQ building With Lambda DWDM device 27

Network for VLBI in Italy Antennanow Mc–32m10 Gb/s Nt–32m1 Gb/s10 Gb/s SRT-64m-1 Gb/s10 Gb/s IRA HQ 10 Gb/s 40 Gb/s 28

Radioastronomy network BO1 Oss. Medicina(BO) Oss. Noto(SR) Oss. S.Basilio(CA) IRA-INAF BO 3x10G 10G MI1 2x10G BO1 Oss. Medicina(BO) Oss. Noto(SR) Oss. S.Basilio(CA) IRA-INAF BO 3x10G 10G MI1 2x10G 29