1. New CentOS7 Development Virtual Machines (CC7 VPCs)
Luigi Gallerani –Pablo Pinés León BE-CO-IN 19 October 2017

2. Introduction
Since Jan 2017, CentOS7 (CC7) is the operational platform in the CCC
During 2016 we prepared a fully-functional BE-CO CentOS7 Development Virtual Machine (CC7 VPC) on Openstack
170 users are already using CC7 VPCs

3. Outline Today I will present:
A short overview of the new Openstack platform which hosts the CC7 VPCs
The status of deployment, statistics and plans
The main differences in functionality/compatibility between SLC6 & CC7
Practical info on CC7, TigerVNC Connection, followed by a short demo
Performance comparisons
CC7 VPC as a pilot for the new monitoring system based on Collectd-Icinga2

4. General Architecture of Openstack Linux VPC
Home client
GPN Client
NFS mount
ssh nomachine
TigerVNC
lxplus7
TN (trusted)
Ansible Linux
console_vm
E-group
Wiki list
Linux Virtual Machine
SCL6 – CC7
465 cwe-513-vol*
293 Windows
COSMOS Collectd-Icinga
VPC Automation create/delete/assign
2 Openstack Projects (758 VPC)
NetOps
ITMon
Wiki list
Hardware Servers
Local storage
CEPH storage

5. Overview of Openstack VPCs in production (758)
Openstack project BE ACC VPC
Since 2014, 280 VPC - KVM+HyperV
Openstack project BE ACC VPC 2016
Since 2016, 478 VPC - KVM
272 SLC6 machines Windows Default config:
1-4 core Westmere/Sandy Bridge 45nm-32nm (Jan 2010)
4GB RAM
GB HDD Spining disk (90Mb/s) (one partition)
1Gb Ethernet
cwe-513-vol(000450)
170 CC7 machines SLC6 machines Windows machines Default config:
4 core Haswell/Broadwell/Skylane nm, ( )
8GB RAM
40GB SSD + 80GB Ceph Storage root partition /opt/ exetension (120Mb/s)
1Gb Ethernet
cwe-513-vol(500900) cwe-513-vmw* cwe-513-vow*
New CC7 VPC

6. Overview of VPC Openstack in production (758)
293 Windows
295 SLC6 VPC
170 New CC7 VPC

7. Linux VPC evolution Openstack 295 SLC6 vol ~350 CC7 expected for LS2
OPENSTACK MIGRATION
vmlvol
125 CC7 gap
Openstack
170 CC7 vol
HyperV CVI
250 SLC6 vml
New monitoring
SLC6CC7
HyperV CVI
60 SLC5
Ansible
“FAST vpc”
New os/
upgrade
CC7 introduction
CVI HyperV Phase out
2012
2013
2014
2015
2016
2017
2018
2019
2020
LS1
Openstack 2016
Openstack
LS2
HyperV FAST
HyperV

8. Openstack 2014 running mainly SLC6

9. NEW Openstack 2016 running CC7 + win

10. Distribution of Linux VPCs (465)

11. Distribution between SLC6 (295) and CC7 (170)
125 gap
170 CC7
New CC7 VPC

12. Comparison of SLC6 & CC7 SLC6 New CC7 Graphical connection
NoMachine LU Aug 2011
TigerVNC - LU May 2017
Graphical Environment
GNOME, different from CCC
XFCE4 identical to CCC
Session after AFS token expiration
NX session expire and crash
Infinite Session with k5reauth
Stuck at shutdown issue
Need Hard Reset (VPC-operator)
Never experienced
Multisession No
Yes
OpenGL
JavaSwing JavaFX
WinCC OA 3.15 No
Partial No
Yes
Yes Yes

13. Compatibility of CC7 What can be (cross) compiled today on CC7?
Input from Juan David Gonzalez Cobas BE-CO-HT
Compatibility of CC7
What can be (cross) compiled today on CC7?
As already pointed out, everything is there.
CC7 compatibility with FESA/Fesa plugin? What about driver testing on CC7?
Drivers for CC7 have been available since mid-2016, and have undergone several upgrades during the period from mid-2016 to last week, to adjust for the latest CC7 image provided by ACC-Adm.
Extensive testing has been done, mostly on timing drivers.
Drivers can also be generated using encore, which is the framework used by EGs to create their drivers.

14. Compatibility of CC7 Input from Juan David Gonzalez Cobas BE-CO-HT
CentOS 7 in the front ends (aka L867)
The CC7 operating system environment is available in FECs (L867).
Booting L867 in a FEC is straightforward: run ./boot_set in cs-ccr-feop:/acc/sys/adm
In FECs, the full CO stack is available in L867 versions
device drivers, encore, FESA, CMW, timing, etc. up to LTIM
Control software can be built for L867, standard development environment available
all required Makefiles and frameworks
Some OS and CO platform data:
CentOS 7.4, real-time kernel rt el7.x86_64
gcc/g++ v4.8.5
timdt 7.14 (no timlib-based timing anymore!)
FESA v5.0
RDA3 v2.1.0

15. VPC-Support vpc-support@cern.ch 2015 2016 2017
Approximately 20 tickets/month for Linux vpc-support, as recorded in SNOW. (remaining 50% of VPCs are windows)
To request a new CC7 VPC or get support:
SNOW ticket automatically generated
Please specify special needs like /opt /local extension, GPN only, TN Trusted
Documentation
wikis.cern.ch/display/ACCADM/VPC+Virtual+Machines+BE-CO
SNOW KB (search key VPC) VPC Centos7 CC7 Virtual Machine User Manual
New CC7 VPC 2017 Guide
List of Virtual Machine users updated daily

16. TigerVNC client
Natively supported by CC7 (and also installed on lxplus7.cern.ch)
VPC now running tigervnc-server v1.3
We will upgrade to the latest version,1.8, very soon
You can already use the latest stable client version 1.8
Getting TigerVNC Client :
Download and install latest stable client (1.8) for Windows, Mac, Linux from
Or…even easier:
CERN Windows desktop: G:\Applications\TigerVNC\vncviewer\
CERN CC7 desktop already has it by default: vncviewer
Ubuntu-Debian:  apt-get install tigervnc-viewer  (and launch with xtigervncviewer)

17. How to connect to CC7 VPC using TigerVNC
From CERN:
SSH to the machine ssh
The first time, set a TigerVNC password vncpasswd
Launch TigerVNC as suggested k5reauth -x -i f -- vncserver
Copy and paste the screen number into the TigerVNC Client
From home:
SSH console from home: ssh -J cwe-513-vol686
Tiger VNC From home Tunnel ssh -f  -c blowfish-cbc -L then connect to localhost:5901

18. Connecting from Operational Consoles (CCC)
We have installed TigerVNC and other connection tools in the operational consoles
Right click on the desktop
Tools:
TigerVNC
Remmina: replaces NoMachine for connecting to SLC6VPCs
NX Client: from SLC6 consoles to SLC6VPC

19. Useful commands and info
List the active sessions
vncserver –list
Kill a session (i.e. session :2)
vncserver –kill :2
Sessions logs are stored in .vnc/
ls .vnc/ (*.log *.pid)
Check the log!
We received multiple support requests caused by misconfiguration in the user space:
Not using BASH as default
AFS Home Quota is 100%
Something wrong in custom .profile .bashrc or .config/

20. A short demo of TigerVNC
Video* in DFS G:\Users\l\lgallera\Public\
TigerVNC2.mp TigerVNC connection demo
TigerVNC3.mp Connection from Home, lxplus7 tunnel example
* Plays best with VLC

21. CC7 Usability and Performance
Usability Issues on SLC6 have all been solved on CC7 over TigerVNC : Eclipse, scrolling (SqlDev), rendering (java swing), fullscreen, rescaling, copy-paste,JavaFX, OpenGL …. all seems to works fine.
Performance benchmark for new Openstack BE ACC VPC 2016 platform on CC7: Openstack Tuning with lot of IT expert interaction
Y-Cruncher Time to compute 50 Million digits of Pi
Intel LinPack Math operation
Linux Kernel Compilation Phoronix Test Suite
FESA compilation (thanks Jochen)
Write disk speed (dd and hdparm) and Network speed iperf

22. Y-cruncher benchmark example
How much time does it take to computer 50 million digits of Pi?
Single Thread
Multi-Thread
$ ./y-cruncher
And you can see the result.txt ….

23. Y-cruncher 50M Pi digits (lower is better)
DEFAULT VPC CONFIG
Openstack SLC6
New Openstack Virtual CC7
Gain
cwe-513-vol289
Westmere - 4 core
Single: s Multi:21.431s
Single: 25.92s Multi:7.001s
x2.91 – x3.06
cwe-513-vol679
Broadwell - 4 core
Physical machines
Gain vs CC7 VPC
cs-ccr-dev1 SLC6
G9 Xeon CPU E core
Single: 30.08s Multi: 6.893
~x0.83 – x1.02
cs-ccr-ctb06 CC7 (bob01 is similar)
Xeon CPU E core
Single: Multi: 8.629
~x1.38 – x0.81
cwo-774-tstb CC7 i7  i GHz- -16 core
Single: Multi: 5.667
x x1.23

24. CC7 VPC: pilot for the new monitoring system
CC7 VPC used in a pilot for the new monitoring solution as part of the COSMOS project based on collectd-icinga2
All CC7 VPC have now the collectd_agent ansible role.
Collectd records since 1 month
~140 different metrics for each VPC
Data is collected by cs-ccr-mon1 collectd server
Stored in RRD files
Aggregated with Telegraph template into Graphite format, and stored to InfluxDB by IT DBOD Service
Soon visible in graphana running on Docker Openshift … preview in the next slide
To be done: integration with Icinga2 for notification

25. Preview new collectd monitoring

26. Summary: New CC7 development VPCs
CC7 VPCs are ready for development
170 CC7 VPCs are already in production the move from SLC6 to CC7 VPCs is ongoing and well planned
We have seen the advantages of CC7 VPCs in terms of usability, performance and compatibility
TigerVNC is the new way to connect to it, as demonstrated
You can request a new CC7 VPC from
We are happy to receive your feedback in order to improve your development experience

27. New CentOS7 Development Virtual Machines (CC7 VPCs)
Thank you for your attention…Questions?
Luigi Gallerani BE-CO-IN 28 September 2017