New CentOS7 Development Virtual Machines (CC7 VPCs) Luigi Gallerani –Pablo Pinés León BE-CO-IN 19 October 2017

Introduction Since Jan 2017, CentOS7 (CC7) is the operational platform in the CCC During 2016 we prepared a fully-functional BE-CO CentOS7 Development Virtual Machine (CC7 VPC) on Openstack 170 users are already using CC7 VPCs

Outline Today I will present: A short overview of the new Openstack platform which hosts the CC7 VPCs The status of deployment, statistics and plans The main differences in functionality/compatibility between SLC6 & CC7 Practical info on CC7, TigerVNC Connection, followed by a short demo Performance comparisons CC7 VPC as a pilot for the new monitoring system based on Collectd-Icinga2

General Architecture of Openstack Linux VPC Home client GPN Client NFS mount ssh nomachine TigerVNC lxplus7 TN (trusted) Ansible Linux console_vm E-group Wiki list Linux Virtual Machine SCL6 – CC7 465 cwe-513-vol* 293 Windows COSMOS Collectd-Icinga VPC Automation create/delete/assign 2 Openstack Projects (758 VPC) NetOps ITMon Wiki list Hardware Servers Local storage CEPH storage

Overview of Openstack VPCs in production (758) Openstack project BE ACC VPC Since 2014, 280 VPC - KVM+HyperV Openstack project BE ACC VPC 2016 Since 2016, 478 VPC - KVM 272 SLC6 machines 8 Windows Default config: 1-4 core Westmere/Sandy Bridge 45nm-32nm (Jan 2010) 4GB RAM 60-120 GB HDD Spining disk (90Mb/s) (one partition) 1Gb Ethernet cwe-513-vol(000450) 170 CC7 machines 23 SLC6 machines 285 Windows machines Default config: 4 core Haswell/Broadwell/Skylane 22-14nm, (2013-2015) 8GB RAM 40GB SSD + 80GB Ceph Storage root partition 250 + /opt/ exetension (120Mb/s) 1Gb Ethernet cwe-513-vol(500900) cwe-513-vmw* cwe-513-vow* New CC7 VPC

Overview of VPC Openstack in production (758) 293 Windows 295 SLC6 VPC 170 New CC7 VPC

Linux VPC evolution Openstack 295 SLC6 vol ~350 CC7 expected for LS2 OPENSTACK MIGRATION vmlvol 125 CC7 gap Openstack 170 CC7 vol HyperV CVI 250 SLC6 vml New monitoring SLC6CC7 HyperV CVI 60 SLC5 Ansible “FAST vpc” New os/ upgrade CC7 introduction CVI HyperV Phase out 2012 2013 2014 2015 2016 2017 2018 2019 2020 LS1 Openstack 2016 Openstack LS2 HyperV FAST HyperV

Openstack 2014 running mainly SLC6

NEW Openstack 2016 running CC7 + win

Distribution of Linux VPCs (465)

Distribution between SLC6 (295) and CC7 (170) 125 gap 170 CC7 New CC7 VPC

Comparison of SLC6 & CC7 SLC6 New CC7 Graphical connection NoMachine 3.5 - LU Aug 2011 TigerVNC - LU May 2017 Graphical Environment GNOME, different from CCC XFCE4 identical to CCC Session after AFS token expiration NX session expire and crash Infinite Session with k5reauth Stuck at shutdown issue Need Hard Reset (VPC-operator) Never experienced Multisession No Yes OpenGL JavaSwing JavaFX WinCC OA 3.15 No Partial No Yes Yes Yes

Compatibility of CC7 What can be (cross) compiled today on CC7? Input from Juan David Gonzalez Cobas BE-CO-HT Compatibility of CC7 What can be (cross) compiled today on CC7? As already pointed out, everything is there. CC7 compatibility with FESA/Fesa plugin? What about driver testing on CC7? Drivers for CC7 have been available since mid-2016, and have undergone several upgrades during the period from mid-2016 to last week, to adjust for the latest CC7 image provided by ACC-Adm. Extensive testing has been done, mostly on timing drivers. Drivers can also be generated using encore, which is the framework used by EGs to create their drivers.

Compatibility of CC7 Input from Juan David Gonzalez Cobas BE-CO-HT CentOS 7 in the front ends (aka L867) The CC7 operating system environment is available in FECs (L867). Booting L867 in a FEC is straightforward: run ./boot_set in cs-ccr-feop:/acc/sys/adm In FECs, the full CO stack is available in L867 versions device drivers, encore, FESA, CMW, timing, etc. up to LTIM Control software can be built for L867, standard development environment available all required Makefiles and frameworks Some OS and CO platform data: CentOS 7.4, real-time kernel 3.10.0-693.2.2.rt56.623.el7.x86_64 gcc/g++ v4.8.5 timdt 7.14 (no timlib-based timing anymore!) FESA v5.0 RDA3 v2.1.0

VPC-Support vpc-support@cern.ch 2015 2016 2017 Approximately 20 tickets/month for Linux vpc-support, as recorded in SNOW. (remaining 50% of VPCs are windows) To request a new CC7 VPC or get support: vpc-support@cern.ch SNOW ticket automatically generated Please specify special needs like /opt /local extension, GPN only, TN Trusted Documentation wikis.cern.ch/display/ACCADM/VPC+Virtual+Machines+BE-CO SNOW KB0005102 (search key VPC) VPC Centos7 CC7 Virtual Machine User Manual New CC7 VPC 2017 Guide List of Virtual Machine users updated daily

TigerVNC client Natively supported by CC7 (and also installed on lxplus7.cern.ch) VPC now running tigervnc-server v1.3 We will upgrade to the latest version,1.8, very soon You can already use the latest stable client version 1.8 Getting TigerVNC Client : Download and install latest stable client (1.8) for Windows, Mac, Linux from https://github.com/TigerVNC/tigervnc/releases Or…even easier: CERN Windows desktop: G:\Applications\TigerVNC\vncviewer\ CERN CC7 desktop already has it by default: vncviewer Ubuntu-Debian:  apt-get install tigervnc-viewer  (and launch with xtigervncviewer)

How to connect to CC7 VPC using TigerVNC From CERN: SSH to the machine ssh lgallera@cwe-513-vol686 The first time, set a TigerVNC password vncpasswd Launch TigerVNC as suggested k5reauth -x -i 3600 -f -- vncserver Copy and paste the screen number into the TigerVNC Client From home: SSH console from home: ssh -J lgallera@lxplus7.cern.ch cwe-513-vol686 Tiger VNC From home Tunnel ssh -f  -c blowfish-cbc -L 5901:cwe-513-vol686.cern.ch:5901 lgallera@lxplus7.cern.ch -N then connect to localhost:5901

Connecting from Operational Consoles (CCC) We have installed TigerVNC and other connection tools in the operational consoles Right click on the desktop Tools: TigerVNC Remmina: replaces NoMachine for connecting to SLC6VPCs NX Client: from SLC6 consoles to SLC6VPC

Useful commands and info List the active sessions vncserver –list Kill a session (i.e. session :2) vncserver –kill :2 Sessions logs are stored in .vnc/ ls .vnc/ (*.log *.pid) Check the log! We received multiple support requests caused by misconfiguration in the user space: Not using BASH as default AFS Home Quota is 100% Something wrong in custom .profile .bashrc or .config/

A short demo of TigerVNC Video* in DFS G:\Users\l\lgallera\Public\ TigerVNC2.mp4 TigerVNC connection demo TigerVNC3.mp4 Connection from Home, lxplus7 tunnel example * Plays best with VLC

CC7 Usability and Performance Usability Issues on SLC6 have all been solved on CC7 over TigerVNC : Eclipse, scrolling (SqlDev), rendering (java swing), fullscreen, rescaling, copy-paste,JavaFX, OpenGL …. all seems to works fine. Performance benchmark for new Openstack BE ACC VPC 2016 platform on CC7: Openstack Tuning with lot of IT expert interaction Y-Cruncher Time to compute 50 Million digits of Pi Intel LinPack Math operation Linux Kernel Compilation Phoronix Test Suite FESA compilation (thanks Jochen) Write disk speed (dd and hdparm) and Network speed iperf https://wikis.cern.ch/display/ACCADM/Benchmarks

Y-cruncher benchmark example How much time does it take to computer 50 million digits of Pi? Single Thread Multi-Thread http://www.numberworld.org/y-cruncher/ $ ./y-cruncher And you can see the result.txt 3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628 03482534211706798214808651328230664709384460955058223172535940812848111745028410270193852110555 964462294895493038196442881097566593344612847564823378678316527120190914564856692346034861045432664 8213393607260249141273724587006606315588174881520920962829254091715364367892590360011330530548820466521384146 9519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119 4912983367336244065664308602139494639522473719070217986094370277053921717629317675238467481846766940513200056 8127145263560827785771342757789609173637178721468440901224953430146549585371050792279689258923542019956112129021960864034418159813 62977477130996051870721134999999837297804995105973173281609631859502445945534690830264252230825334468503526193118817101000313783875 28865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201 9 ….

Y-cruncher 50M Pi digits (lower is better) DEFAULT VPC CONFIG Openstack SLC6 New Openstack Virtual CC7 Gain cwe-513-vol289 Westmere - 4 core Single: 75.518s Multi:21.431s Single: 25.92s Multi:7.001s x2.91 – x3.06 cwe-513-vol679 Broadwell - 4 core Physical machines Gain vs CC7 VPC cs-ccr-dev1 SLC6 G9 Xeon CPU E5-2630- 32 core Single: 30.08s Multi: 6.893 ~x0.83 – x1.02 cs-ccr-ctb06 CC7 (bob01 is similar) Xeon CPU E5-2637 - 16 core Single: 18.659 Multi: 8.629 ~x1.38 – x0.81 cwo-774-tstb CC7 i7  i7-4790 CPU @ 3.60GHz- -16 core Single: 17.551 Multi: 5.667 x1.47 - x1.23

CC7 VPC: pilot for the new monitoring system CC7 VPC used in a pilot for the new monitoring solution as part of the COSMOS project based on collectd-icinga2 All CC7 VPC have now the collectd_agent ansible role. Collectd records since 1 month ~140 different metrics for each VPC Data is collected by cs-ccr-mon1 collectd server Stored in RRD files Aggregated with Telegraph template into Graphite format, and stored to InfluxDB by IT DBOD Service Soon visible in graphana running on Docker Openshift https://acc-mon.web.cern.ch … preview in the next slide To be done: integration with Icinga2 for notification

Preview new collectd monitoring

Summary: New CC7 development VPCs CC7 VPCs are ready for development 170 CC7 VPCs are already in production the move from SLC6 to CC7 VPCs is ongoing and well planned We have seen the advantages of CC7 VPCs in terms of usability, performance and compatibility TigerVNC is the new way to connect to it, as demonstrated You can request a new CC7 VPC from vpc-support@cern.ch We are happy to receive your feedback in order to improve your development experience

New CentOS7 Development Virtual Machines (CC7 VPCs) Thank you for your attention…Questions? Luigi Gallerani BE-CO-IN 28 September 2017