2. SOLARIS Tango Control System Status
Tango Meeting 2018
MICHAŁ OSTOJA-GAJEWSKI
National Synchrotron Radiation Center SOLARIS

3. National Synchrotron Radiation Center SOLARIS
Operation
Stored 500 mA at 1.5GeV energy, operating at 300
Working in 2 operators’ shifts (8:00-16:00, 14:00-22:00)
Beamlines
PEEM/XAS – commissioning / ready for users at
UARPES – commissioning / ready for users at
Diagnostic - construction started / X-ray ready at , visible light at
PHELIX – construction started, commissioning end of 2019
XMCD – construction started, commissioning end of 2020
Applying for funding of 4 new beamlines
CryoTEM
SOLARIS was chosen to host CryoTEM infrastructure
Users
First open call successful – total of 45 applications
42 applications submitted via DUO (Digital User Office) system
3 applications via CERIC-ERIC

4. Dealing with Meltdown and Spectre at SOLARIS, 1 of 2
2018 started with a shocking information on
critical vulnerabilities in modern processors
In December 2017, we upgraded VMware to 6.5, but this upgrade did not contain Meltdown/Spectre patches
In January 2018 – we applied patches to all servers running VMware ESXi
We have not observed any significant performance degradation

5. Dealing with Meltdown and Spectre at SOLARIS, 2 of 2
Guest OSes on VMs were not patched in the first
run, but are patched systematically when updating linux kernels
All workstations (control room, beamlines) were patched with new CentOS kernel
All Windows systems controlled by local domain were patched automatically
Great source of information :
Linux vulnerability checker : meltdown-checker

6. Control System @ SOLARIS
We have upgraded entire control system to Tango 9.2.5a and observed great improvement of system stability and performance
We observed that Tango is very sensitive to any DNS and Reverse DNS misconfiguration:
Lesson learned: if you see your client application switching to polling mode, be sure that the event system is not working properly and you should check your DNS configuration first.
We moved from Taurus 3.7 to 4.1 and it required a lot of extra effort to modify multiple applications for the new API.
Some apps still remain on Taurus 3, but are planned to be adapted for Taurus 4.x soon.
Taurus 4.x
Moving out from NFS distribution towards RPM packaging
Ansible for building control system workstation and VMs
Moving from Jenkins to Gitlab CI/CD
Deploying GitFlow
Moving from Redmine to YouTrack
Using Scrum , 2 weeks Sprints
Development environment – freedom, but prefered:
Jetbrains PyCharm, Clion, IntelliJ IDEA, YouTrack

7. Control System @ SOLARIS
Moving out from NFS distribution towards RPM packaging
Created RPMs for all software running on Control Room and
Beamlines workstations
Working on packaging of device servers software and configurations
Created and deployed Ansible playbooks for automatic building of control system workstations
Working on Ansible playbooks for automatic configuration of VMs and deployment of device servers software

8. Control System @ SOLARIS
Moving from Jenkins to Gitlab CI/CD and using Docker for builds
Deploying GitFlow as a standard branching model
Moving from Redmine to YouTrack for Agile/Scrum development
Development environment – freedom of choice, but preferred:
Jetbrains PyCharm,
CLion,
IntelliJ IDEA,
YouTrack
Challenge: automatic testing, mockups, automatic devops

9. Control System @ SOLARIS
Started using Valgrind (and Valkyrie) for debugging and memory analysis
Reviewed Python profilers: Yappi, CProfile, VMProf to identify ability to work in multithreaded environment
Very interested in the analysis of Tango kernel with special attention to communication and threading models
In process of identyfing suitable tools (profilers, debuggers) for analysis and visual representation of Tango runtime system

10. Control System @ SOLARIS
Deployed monitoring system based on :
ABRT,
Splunk
Monit See presentation by Michał Gajda
Contributed to Heating Unit Controller project (HUC)
Developed Tango GUI , Device Server and Archiving
Deployed HUC into production in SOLARIS
See presentation by Wojciech Kitka
Deployed HDB++ based on Cassandra:
3 clusters (Machine, BL04, BL05)
Each cluster consists of 2 nodes with replication factor 2
Archived attributes – total : 1800

11. SOLARIS @ Tango Community
Statement from SOLARIS
We are eager to contribute to development of Tango kernel or Tango related projects
Thank you