Virtualization News and Plans Luigi Gallerani BE-CO-IN TC 27 Nov 2014

Virtualization News and Plans 2 Outline Motivation: Improvement: Experience in VPC Consolidation VPC Limits identified by user feedback Phase out of IT HyperV Service  Openstack Goal: Design the new best solutions for ACC Priority on ACC Requirements/ Constraints Deep Evaluation of CERN OpenStack Alternatives investigations

Virtualization News and Plans 3 VPC Improvements experience 16 New VPC Servers (Total=29) 175 Machine migrated to x64bit Java7 and JWS Migration All 18 cerntsab* replaced with 32 Virtual Terminal Servers& Clusters Many bugs fixed and IT issue solved with strong IT collaboration Summer Student Project (Marina) on backend automation extract from Analysis, Strategy, Solutions & Future Plans L. Gallerani - TC Feb 2013 Lot of experience gained Confident we can offer more and better for the future

Virtualization News and Plans 4 User Feedback survey 2014 Many issue discovered and fixed thanks to your feedback Tech Student Project started from your feedback comments Use cases where VPC are not optimal have been clearly identified VPC user feedback survey page in our Wikis wikis.cern.ch/display/VM/VPC+Feedback+from+user+side+Feb+2014

Virtualization News and Plans 5 Optimization after feedback survey Fixes during Technical Student Project (Fotis) SQL Developer running on NX Automatic wrong user environment and unused machines detection Resource upgrade analysis and RAM CPU Network upgrade Perf tuning based on FESA class Compilation benchmark analysis slides from Fotis Liatsis September 2014 BE-CO-IN Section Meeting

Virtualization News and Plans 6 IT Hyper-V Service Phasing out BE-CO VPC are running on IT Hyper-V CVI service IT Hyper-V service has been frozen for all users except BE-CO Hyper-V Replacement is based on OpenStack TECHNOLOGY UPGRADE vmm.cern.ch home page – service is phased out

Virtualization News and Plans 7 Design ACC-Dev infrastructure for future Motivation: Improvement: Experience in VPC Consolidation VPC Limits identified by user feedback Phase out of IT HyperV Service  Openstack Goal: Find the best solutions for BE-CO: Priority on ACC Requirements/ Constraints Deep Evaluation of CERN OpenStack Alternatives investigations

Virtualization News and Plans 8 Priority on ACC Requirements My role is to find the optimal infrastructure solutions for the future considering: – Developers (Java, C++ Fesa, but also WinCC, Siemens, Labview, Mathlab, PVSS, Schneider Twido…) – Operational support (fast intervention and bug fix, CO, OP, BI, RF, but also experts of EN cryo, cooling, ventil…) – SysAdmin and DevTools support – Resources (Time, money, technology, performance…. ) – CERN IT solutions available and supported

Virtualization News and Plans 9 ACC Constraints No TN in offices No TN trust if machine not managed by BE-CO TN access = No Internet connection Only restricted access to our NFS Servers Only ACC users in the ACC infrastructure Limited number officially supported solutions Migrate dev infrastructure away from the TN

Virtualization News and Plans 10 Migrate Dev away from TN Document written by Vito Baggiolini (BE/CO), Alastair Bland (BE/CO), Uwe Epting (EN/CV), Luigi Gallerani (BE/CO), Timo Hakulinen (GS/ASE), Stefan Lüders (CSO), Stephen Page (BE/CO) With comments by Pierre Charrue (BE/CO), Stephen Jackson (BE/BI), Lars Jensen (BE/BI), Chris Roderick (BE/CO), Katarina Sigerud (BE/CO), Wojtek Sliwinski (BE/CO), Andy Butterworth (BE/RF), Jorg Wenninger, Kajetan Fuchsberger (BE/OP) Fully migrate the current development infrastructure away from the TN and fully decouple them; Extract from page 3 NEXT TC Proposal

Virtualization News and Plans 11 Design ACC-Dev infrastructure for future Motivation: Improvement Experience VPC Limits identified by user feedback Phase out of IT HyperV Service  Openstack Goal: Find the best solutions for BE-CO: Priority on BE-CO Requirements/Constraints Deep Evaluation of CERN OpenStack Alternatives investigations

Virtualization News and Plans 12 CERN Openstack Openstack is what IT is offering as replacement of current virtual machine infrastructure (HyperV now obsolete) Large portion of IT Computer Center is migrating from physical to OpenStack Virtual Slide from Presentation by Belmiro Moreira (CERN IT) More info at openstack.cern.ch

Virtualization News and Plans 13 CERN Openstack project scale Slide from Presentation by Thomas Oulevey (CERN IT ) More info at openstack.cern.ch ACC

Virtualization News and Plans 14 Scaling up in #cores, not speed HARDWARE of the FUTURE: More cores vs clock speed Huge RAM available Service oriented Easy and Cheap to virtualize many “slow” machines for general purpose use : - ideal for IT computer center - ideal for BE-CO Terminal Servers Performance oriented Hard to get high performance virtual machine for compilation (what our developers need)

Virtualization News and Plans 15 Investigating Open-stack for BE-CO Usability Test Performance analysis For BE-CO Migration plan of current 540 VPCs with low impact on users Request special config for ACC (8-Cores, 8GB RAM) The BE-ACC-VPC-TEST OpenStack project home page

Virtualization News and Plans 16 CPU benchmark: VPC vs Openstack YOUR BE-CO VPC OpenStack Preliminary results - benchmark comparison done by Fotis Liatsis Average Integer Float Prime Test Extended SSE Compression Performance oriented

Virtualization News and Plans 17 HDD benchmark: VPC vs OpenStak YOUR BE-CO VPC OpenStack Average Seq read Seq Write Random seek Performance oriented Preliminary results - benchmark comparison done by Fotis Liatsis

Virtualization News and Plans 18 Openstack not yet ready for BE-CO Dev benchmark comparison done by Fotis Liatsis between Windows BE-CO VPC (in red) vs Openstack (green& blue) Openstack today is not ready for BE-CO dev needs at the moment Performance issues for development We will lead acceptance tests before saying yes IT promise to put in place improved solutions Performance oriented

Virtualization News and Plans 19 Openstack is great for BE-CO TS BE-CO Openstack Virtual Terminal Servers for experts in cryo, vent, ele, en-ice… Pilot project driven by BE-CO-IN in collaboration with EN-ICE and IT-OS To provide better expert application terminal servers TN Trusted Slides from S. Bukowiec IT-OS, P. Golonka EN-ICE & L. Gallerani Terminal Server Cluster pilot project presentation service oriented

Virtualization News and Plans 20 Virtual Terminal Server Clusters ACCEPTED & RUNNING cerntsice cerntscryo cerntsel cerntscv slide from S. Bukowiec IT-OS, P. Golonka EN-ICE and L. Gallerani Terminal Server Cluster pilot project presentation (now in production and running) service oriented

Virtualization News and Plans 21 Advantages for BE-CO of the new Openstack clusters for experts terminal servers SERVICE ORIENTED CLUSTERS for many users Scale horizontally: service overloaded? more virtual servers added (or duplicated) to the clusters (parallel scaling) Upgrades without stopping service in the cluster HA: If a node goes down service stays up service oriented

Virtualization News and Plans 22 Design ACC-Dev infrastructure for future Motivation: Improvement Experience VPC Limits identified by user feedback Phase out of IT HyperV Service  Openstack Goal: Find the best solutions for BE-CO: Priority on BE-CO Requirements/Constraints Deep Evaluation of CERN OpenStack Alternatives investigations

Virtualization News and Plans 23 VPC Alternatives investigation Alternative to VPC page in our Wikis

Virtualization News and Plans 24 Possible Alternatives under analysis Openstack is not the unique solution we are evaluating for performance oriented development: Physical desktop PC in the GPN not TN Trusted – Nice Windows – Standard CERN Linux with mechanism to get secure NFS – BE-CO linux managed by us for GCC (man power?) Physical linux servers for high performance Remote X11 sessions with xRDP (no nx licence) Others: CernVM? VirtualBox? Lightweight virtualization? Performance oriented

Virtualization News and Plans 25 Desktop GPN Linux not TN Trusted ACC Eclipse with FESAPlugin running in GPN not TN trusted Developer can browse internet Standard CERN SLC6 in GPN Screenshot from physical GPN not TN Trusted desktop PC running ACC Eclipse with Fesa Plugin Tested by BE-BI Developer (M. Ferrari)

Virtualization News and Plans 26 Ways to mount NFS from GPN Different way to provide secure access to NFS, and different scenarios analysis SFTP / SSHFS PERFORMANCE TESTS? SSHFS Side Effect: 25% CPU taken for encryption during compilation (can be slower than VPC!) Secure controlled access to NFS via single gateway using SFTP and SSHFS from desktop PC BE-CO Linux only MORE on ACC services in GPN in the NEXT TC NFS4

Virtualization News and Plans 27 High perf physical linux with XRDP Windows native RDP connecting to BE-CO linux where XRDP linux server is running Windows native Remote Desktop Connecting to Linux VPC Could be used as solution for high performance compilation linux servers

Virtualization News and Plans 28 Timetable HyperV available for BE-CO until new satisfactory solution is found (max Dec 2015) Possible solutions Openstack tuning and evaluation (April 2015) Prototype GPN Desktop (April 2015) Prototype Linux server (April 2015) July 2015: Decision Working solutions in production Dec 2015

Virtualization News and Plans 29 Conclusions Ready for changes, motivations and competences – Lot of experience and competences gained in VPC Consolidation – User feedback requests, IT technology changes Analysis of Best solutions for BE-CO infrastructure illustrated – Priority on ACC Requirements/Constraints – Evaluation of new technology like OpenStack performance and service oriented, alternatives taken into account – Timetable We will move in 2015 only when satisfactory solutions are validated and accepted

Questions? Virtualization News and Plans Presentation available in DFS \\cern.ch\dfs\Users\l\lgallera\Public\ TC2014LuigiGallerani.pptx

Virtualization News and Plans 31 Multiple solutions consideration MOST OF DEV DONE IN THE TN TN TRUSTED MACHINE ONLY FOR FINAL VALIDATION TN TRUSTED MACHINE FOR FAST BUG FIX FOCUSING ON THE GPN SOLUTIONS examples: GCC / driver compilation?  BE-CO Linux managed by us, remote desktop to cernts for desktop applications. Clear statement what we support. Java Developer  Standard CERN Linux or Windows private machine with ACC Eclipse, remote desktop to cernts for desktop applications when use SLC on local machine. Nice machine are supported by IT ServiceNow with developer as main user LinuxServers  BE-CO linux fast remote development and support and bug fix

Virtualization News and Plans 33 Summer student Project VPC Automation (Marina) Automated Machine management VMM, LanDB, FeLab/Feop kickstart fully integrated and automated Fault machine detection and alert integrated with Diamon Automatic optimal resource allocation A slide from Marina Ricci – Section Meeting Presentation

Ongoing about Technical Network Luigi Gallerani BE-CO-IN 27 Nov 2014

Ongoing discussion Outline The keypoints from the vision documents Making the CERN Technical Network a Pure Network for “Operations” Steps already done from our group in this direction (CCDB, Testbed, emergency TS in the TN… other?) The network disconnection tests (motivation, issue solved, issue identified, impact on operations) The Micro TN Disconnection test model for the future with no impact on operation GPN-TN Routing and firewall control proposal, GPN-TN first traffic analysis results Virtual Dev Net via Routing and Firewall

Ongoing Discussion…. Documents keypoints

Ongoing Discussion…. Document proposal for changes

Ongoing Discussion…. Steps already done in this direction CMW Testbed migrated to the GPN Database and Controls Configuration Services available in the GPN Controls Configuration Services available now in the GPN - DEV at -> GPN accessible account used for our daily development and early adopters. - NEXT at -> GPN accessible test bed. A new account recently created by migrating existing TN next. - TEST at -> TN account, used for testing and preparation of data before moving to PRO. Also used formerly by some clients for integration testing (mistakenly). - NEXT at -> TN account, so-called test bed. Existing common Java APIs ConfigDB Directory Service can be configured to connect to any of these accounts. Now, from my perspective and a message which I would like to send is: - use DEV only with our agreement - this is an internal development database but fits well to early adoption (like we do currently with FESA), provided you are brave enough - use for test bed projects and system testing which does not require TN. - use for test bed projects which must be run in TN From this perspective purpose of existing is not clear and this account will be removed as some moment in future. From clients perspective the should be sufficient to carry on with any test which would require TN. Cheers, Lukasz

Ongoing Discussion…. Second layer authentication

Ongoing Discussion…. TN Disconnection tests For BE-CO Find hidden dependecies between TN and GPN in the control system Discover misconfigured IT services we daily trust

Ongoing Discussion…. Results and fixes after TN Disco test Emergency terminal servers

Ongoing Discussion…. Micro TN Disconnection test

Ongoing Discussion…. GPN/TN Routing traffic analysis

Ongoing Discussion…. Data size and graph rappresentation

Ongoing Discussion…. Some results

Ongoing Discussion…. Routing and Firewall rules

Ongoing Discussion…. Development Network

Ongoing discussion Conclusions We are discussing, proposing and already implementing many changes to improve design, quality, security and separation of the TN Making TN a pure network for operation, moving the development out Inveistigating stronger user authentication mechanism Going ahead with new “Micro TN disconnection test” model to discover and fix hidden dependencies Analyzing the traffic between GPN and TN routers using modern sophisiticated visual tools Define router and firewall rules to have full control of what we expose, and create a trusted network by these rules BE-CO is leading all the aspects of the