Pablo Pinés León – FTEC 2016 Program BE5002 - Investigate redundant network filesystem technologies for CERN's Accelerator Control System servers Pablo Pinés León – FTEC 2016 Program

Introduction BE-CO-IN is responsible for the infrastructure of CERN’s accelerators control servers ACC-ADM team There is currently one dedicated NFS server for each of 4 subsystems Around 1500 being served in total An NFS server unavailability can affect many client machines Client machines cannot be rebooted Recovery can take significant effort (restart required) High Availability solution would be a clear improvement This infrastructure can be used for other services too

First steps Learning Ansible and git Evaluation of high-availability technologies E-groups management Benchmarking of physical and virtual systems (CPU/Storage/Network) Support on 2FA and deployment of servers on the Technical Network

investigation Previous effort with GlusterFS not completely successful CEPH suggested It was deemed not adequate due to its resources requirements Alternatives explored though online search (HA-NFS, Mars,…) Active/Active solution considered (GFS, OCFS2) Decision was taken to use Pacemaker and its associated stack Lots of documentation Support from big companies (Red Hat, Suse…)

Architecture Design Target OS is CentOS 7 Active/Passive on a 2 nodes setting 4 nodes configuration investigated Requires a newer software version for which no precompiled packages exist More complicated set up, specially in terms of connections 2 nodes was considered a “good enough” improvement Heartbeat deprecated in CentOS 7

Software Stack Virtual IP Active Sync NFS Master Slave Filesystem Ext4 DRBD Primary Secondary Resource Manager Pacemaker Messaging Corosync Hardware Node 1 Node 2

Test environments First on SLC6 machine on Openstack Network problems (multicast, virtual IP) Then VirtualBox on my Workstation First on SLC6, Python version too old Tried on Ubuntu, successful CentOS 7 successful too Currently, two Physical Machines (workstations) Floating Virtual IP Significant issues encountered

Cluster Status view

Virtual Personal Computers VPCs are VMs for developers in the accelerator servers on CERN’s Openstack BE, TE, EN developers ACC Operational support Support on the creation, configuration and troubleshooting Issues with new templates New ways to create VMs and meet users needs Phasing out old infrastructure Dealing with other groups Monitoring on DIAMON

NEXT STEPs Implement the solution on real servers Add more services Test dedicated connection for DRBD Add more services Continue the work on VPCs 3 screens configuration for CentOS 7 White Hat Challenge Egroups management and other ACC-ADM tasks

Any questions?