«The past experience with JCOP in ALICE» «JCOP Workshop 2015» 4-5 November 2015 Alexander Kurepin

The ALICE experiment at CERN Collaboration: 37 countries 154 institutes 1500 members The Detector: 10 000 tons 19 subdetectors built on different technologies 2 magnets The ALICE experiment is the one of the 4 big experiments on LHC In ALICE experiment involved about 1 thousand 5 hundred members from 157 institutes from 37 countries Detector represents facility with weight about 10 thousand ton with 19 subdetectors, one solenoid magnet and one dipole magnet The main goal of the experiment is studying havy ion collisions and research quark-gluon plasma. ALICE experiment also has proton-proton physics program. Main physics focus: Heavy ion physics at LHC energies Proton-proton physics

ALICE Detector Control System (DCS) Guaranties: 24/365 safe and stable operation of the experiment Partial detector control system are built by experts of the detector teams The Central DCS Team provides: Distributed control system based on commercial components Computing and network infrastructure Software framework and tools Implementation guidelines and operational rules Integration of detector systems Interfaces to external systems and LHC Training of operators and uninterrupted expert on call service First line support for software developers ALICE DCS provide 24 hours every day of the year safe and stable operation of the experiment Partial detector control system are built by experts of the detector teams The Central DCS Team provides: Distributed control system based on commercial components Computing and network infrastructure Software framework and tools Implementation guidelines and operational rules Integration of detector systems Interfaces to external systems and LHC Training of operators and uninterrupted expert on call service First line support for software developers

The ALICE DCS – Context and Architecture User Interface, External Systems Interface ALICE Framework CERN JCOP Framework SCADA System (WINCC OA) Devices with similar functionality are grouped into subsystems. About 100 different subsytems are implemented in ALICE. Central DCS project controls all experiment subsystems. For distributed control system ALICE DCS uses commercial SCADA system WinCC OA (open architecture) and other commercial products like OPC servers. To extend functionality in the project integrated JCOP framework. Together with ALICE and the detector experts panels this forms complete control project. Device Interface Layer (OPC or custom) Devices (Industrial and custom)

ALICE DCS Systems DETECTOR DETECTOR DETECTOR WCC OA WCC OA WCC OA For each detector we could have several projects connected to the detector common distributed network. The central systems connects to detector system and form one large distributed system sharing the data and the control. WCC OA WCC OA The central systems connects to detector system and form one large distributed system sharing the data and the control WCC OA WCC OA WCC OA WCC OA

JCOP framework in ALICE JCOP is an excellent example of successful common project at CERN Without JCOP we would have hard times to develop the systems ALICE would not be able to implement and run a DCS system as it is running now, or only at the expense of additional resources JCOP is an excellent example of successful common project at CERN Without JCOP we would have hard times to develop the systems ALICE would not be able to implement and run a DCS system as it is running now, or only at the expense of additional resources >As always happen with good products, people start to expect much more

ALICE challenges We all depend on WinCC OA and Siemens support has rather high threshold to start solving the problems Many components are very complex, but they lack documentation Sometimes extremely difficult to debug code without having powerful tool for analyse log files We often imagine a component differently from how it is seen by the developer, so possibility of customisation would be very helpful Every time we are doing upgrade we always have to find compromise between OS and components We can’t cay that we are running all the time without any problem. You will not believe. We all depend on WinCC OA and Siemens support has rather high threshold to start solving problems Many components are very complex, but they lack documentation. Often documentation are very basic. >It would be great to have: All documentation available, as for many our colleagues it is quite difficult to attend JCOP trainings Sometimes extremely difficult to debug code without having powerful tool for analyse log files. >Correlation between 2 log files, to find exact event in different sources. During operation period to see what command was send between lots of FSM logs. We would like to have mechanism to provide own experiment customisation and have them distributed in framework AES as an example of confusing filtering mechanism We would love to have the actual operating systems supported in time by all components. Usually there is always some showstopper and we need to go for compromise.

FSM in ALICE FSM is very powerful instrument but at a few occasions we have suffered from an loop error in the logic of a detector blocking the whole DCS One thing would be to implement a protection for loops Another thing would be to have a (light) procedure to check the logic of the FSM FSM is very powerful instrument but at a few occasions we have suffered from an loop error in the logic of a detector blocking the whole DCS One thing would be to implement a protection for loops Another thing would be to have a (light) procedure to check the logic of the FSM We once did such an exercise, but that was a rather heavy procedure. It would be worth to put some effort to revive this in a lighter form.

ALICE DCS ideas Organize ALICE specific training on JCOP tools In a collaboration of ALICE DCS team and EN/ICE Organise training/testing laboratory. DCS team could offer it’s lab and infrastructure and can host devices provided by others (EN/ICE) for specific tests Create ad-hoc working groups of experts from all experiments that can go into more technical details of a given subject and propose solutions/improvements and sharing information to the JCOP CB Before we start the upgrades we could make a list of all components including the OS, look at the roadmaps and set a goal in order to have all at the same level when we starts an upgrade We could offer some resources to look into FSM logic check JCOP trainings specific for ALICE experiment needs. In a collaboration of ALICE DCS team and EN/ICE Organise training/testing laboratory. DCS team could offer it’s lab and EN/ICE could provide some rare specific devices Create working group of experts and that can go into more technical details of a given subject and propose solutions/improvements and sharing information to the JCOP CB Before we start the upgrades we could make a list of all components including the OS, look at the roadmaps and set a goal in order to have all at the same level when. >Last time we missed the chance to go for the latest Windows server because of OPC. We could offer some resources to look into FSM logic check

Detector experience All ALICE detectors use JCOP framework in their projects Number of components depends on needs Very useful tool to prepare own FW component All ALICE detectors use JCOP framework in their projects Number of components depends on needs Very useful tool to prepare own FW component

Useful tools for detector experts For expensive devices to debug panels and scripts and teach newcomers it’s great to have powerful simulator We have one for CAEN devices during the training lessons, but it is difficult to find For expensive devices to debug panels and scripts and teach newcomers it’s great to have powerful simulator We have one for CAEN devices during the training lessons, but it is difficult to find

ALICE operational model We currently run with 4 shifters and on-call for all detectors and central systems (DCS on-call covered by central team) In the past we have been running with 6 shifters (HLT, DCS, DAQ/ECS, CTP, DQM, SL) We typically run the DCS shifts with a pool of ~100 different people to cover all shifts of a year We currently run with 4 shifters and on-call for all detectors and central systems (DCS on-call covered by central team) In the past we have been running with 6 shifters (HLT, DCS, DAQ/ECS, CTP, DQM, SL) We typically run the DCS shifts with a pool of ~100 different people to cover all shifts of a year

ALICE DCS operation The DCS is operated by a single operator In 2015 more than 150 operators were trained to assure the 24/7 operation Most of the operators did not have prior experience with the DCS We have developed dedicated panel that allows to perform routine actions with a single click (handshake, going safe at end of fill, going ready at start of fill etc.) The interface to the DCS is based on set of intuitive graphics interfaces From a single screen, the operator can access all controlled parameters The DCS is operated by a single operator In 2015 more than 150 operators were trained to assure the 24/7 operation Most of the operators did not have prior experience with the DCS We have developed dedicated panel that allows to perform routine actions with a single click (handshake, going safe at end of fill, going ready at start of fill etc.) The interface to the DCS is based on set of intuitive graphics interfaces From a single screen, the operator can access all controlled parameters

Shifts model For RUN_2 period we changed the shifts blocks from 6 shifts in the same slot, to 2-2-2 (MMAANN) Participating shifts give to institutes the ‘credits’ This steps allow us to fill the shift crew more efficient For RUN_2 period we changed the shifts blocks from 6 shifts in the same slot, to 2-2-2 (MMAANN) Participating shifts give to institutes the ‘credits’ This steps allow us to fill the shift crew more efficient

The ALICE O2 Project LHC will provide for RUN3 ~100x more Pb-Pb collisions compared to RUN2 (1010 – 4.1011 collisions/year) ALICE O2 Project merges online and offline into one large system TDR approved! Some O2 challenges: New detector readout 8400 optical links Data rate 1.1TB/s Data storage requirements: ~60PB/year LHC will provide for RUN3 ~100x more Pb-Pb collisions compared to RUN2 (1010 – 4.1011 collisions/year) ALICE O2 Project merges online and offline into one large system TDR approved! Some O2 challenges: New detector readout 8400 optical links Data rate 1.1TB/s Data storage requirements: ~60PB/year

DCS in the context of O2 DCS provides input to O2 ~100 000 conditions parameters are requested Data has to be injected into each data frame at 20ms rate DCS provides input to O2 ~100 000 conditions parameters are requested Data has to be injected into each data frame at 20ms rate

Conclusion Using JCOP components the DCS provides stable and uninterrupted services to the ALICE experiment since 2007 Modular and scalable design minimized the need for system modifications and coped with the evolution of hardware The more we are using JCOP tools the more support we will need Challenging O2 project requires full redesign of ALICE DCS data flow Using JCOP components the DCS provides stable and uninterrupted services to the ALICE experiment since 2007 Modular and scalable design minimized the need for system modifications and coped with the evolution of hardware The more we are using JCOP tools the more support we will need Challenging O2 project requires full redesign of ALICE DCS data flow