1. BE-CO LS1 review View from TE/ABT/EC
1st December Nicolas Magnin – TE/ABT/EC
On behalf of TE/ABT/EC software team, thanks to my colleagues for their inputs.

2. BE-CO LS1 review - View from TE/ABT/EC
OUTLINE
Controls Renovation Program
Redundant Power Supply Installation
New Hardware Installation
Linux Consoles and Windows TS
FESA3
Silecs / Ieplc
Logging
PostMortem
Summary
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

3. Controls Renovation Program
Controls renovation program went fine, good collaboration with BE/CO:
Replacement of RIO by MENA20
Replacement of old VME crates by ELMA
Replacement of ELONEX PC by KONTRON
Renaming of all SPS/PSB FECs (kdsba1 -> cfv-ba1-mkdtim)
Migration from SLC5 32-bit to SLC6 64-bit (Using Encore, worked fine)
Migration from FESA2 to FESA3 (More complicated, discussed later)
Replacement of cPCI CPU for SLC6 64-bit compatibility not performed.
FESA3 not working properly on mono-core CPU (background RT actions priority issue)
New CPU Concurrent Technologies PP B14 is under evaluation in lab since last week.
Some FECs were not renovated:
cfv-ua43-mkqatim: Still old CES RIO CPU
cfc-195-mkadhorn: Still SLC5 and FESA2
Nobody noticed… ABT realized this at the end of LS1. Renovation to be done during YETS.
Good coordination to prepare the renovation program BEFORE LS1, no follow-up DURING LS1. Better follow up of hardware/software renovation process with equipment groups during LS2 ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

4. Redundant Power Supply Installation
Installation of redundant power supplies in KISS computers for LBDS/TSDS:
The dual power supply module came very late!
Monitoring by DIAMON works fine.
Installation of redundant power supplies in ELMA computers for LBDS/TSDS not achieved:
We installed all the new ELMA crates, but with missing switch for redundant PSU detection.
We have to replace again all the redundant ELMA crates (LBDS critical FECs) ! Planned for YETS. Full revalidation of power distribution needed afterward = UPS tests.
Monitoring of redundant PSU by DIAMON and SIS is not implemented yet
FESA3 class CGMON_ELMA deployed but shows error due to second PSU not detected.
Also CGMON_WIENER, but no CGMON_KISS class to publish PSU state for SIS subscriptions.
Redundant PSU without monitoring is not redundancy !
=> Homogeneous approach for PSU state monitoring (CGMON classes for all PSU) ?
=> Take into account full redundant PSU support for future crate selection ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

5. New Hardware Installation
Most of new FEC installations were performed by ABT:
Transport of FECs to LHC / SPS
Installation & validation of configuration of FEC in racks
Network configuration debug & IT changes with help of BE/CO
Lots of time spend due to control configuration mistakes, or missing timing in rack, etc…
Needs a better integration of services for new hardware installation request :
Check presence of Power Supply (UPS? / power switches) in rack
Check presence of Timing in rack
Check presence of Network in rack
BE/CO to coordinate all these services ?
Availability of services, transport, installation and validation of configuration to be performed by BE/CO ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

6. Linux Consoles – Windows TS servers
We accepted to have all our computers on TN that run Windows to be re-installed with Linux SLC6.
- These computers are used by ABT experts and piquet during interventions, not Linux users!
We requested that the CCM BT-EXPERT is started automatically when ABT users perform login
- To have an easy way to start Java expert applications.
- To connect to our Windows TS computers, to use WinCC web applications / elogbook Java application.
But eventually, after all consoles were updated to Linux, BE/CO said that it is not possible
They do not want to touch user configurations…
RDP connection from Linux consoles to Windows TS servers with Load Balancer is not reliable.
- Many times it just does not work, very annoying for our experts and piquet ! (Still not understood)
FESA3 navigator not running on Windows
- This is an issue for us as most of our experts are connecting remotely (from home) to Windows TS servers.
=> Better support from BE/CO for Windows TS platform ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

7. BE-CO LS1 review - View from TE/ABT/EC
FESA3 - Development
Main issue: FESA3 v2 was not ready at the beginning of LS1 !
First working version was delivered just before first beam injections in LHC.
CCDB not ready either: Many problems with instance promotion process
Fields configuration lost during instance promotion (problems at LHC startup).
May times we had to completely delete deploy units and redeliver.
Release cycle too long (Needs to wait full tests and delivery of new FESA3 version for simple bug fix)
=> Usage of shared objects to ease the release of bug fixes to be evaluated ?
Huge investment in test/debug and implementation of work-arounds during FESA3 development
- Due to long release cycle, we often had to implement work arounds until the next release. - We invested more time in control software validation than upgrading our systems !
Discussions with equipment groups about functionalities implementation (timing/PM integration)
If any major software control upgrades are planned for LS2, be ready for test (~6 month ?) before LS.
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

8. FESA3 – Promotion / Rollback
Redeliver of FESA3 classes and deploy units for every release of FESA3 is very time consuming !
Promotion, compilation, validation on test bench and redeliver of FESA3 classes could be automated.
Expert responsible for deploy units promotion only.
BE/CO Test bench to integrate equipment groups software ? Deployment of BE/CO test bench in our lab if hardware is needed for tests ?
Rollback FESA3 instances: problem of persistence files !
We had sometimes to roll-back deploy units in operation: It worked fine at CCDB level, but persistency file not rolled-back = lost of operational settings !
At least roll-back of persistency file on FEC should be provided. Upload persisted data in CCDB ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

9. FESA3 – Non-backward compatible changes
API change: standard Status property in FESA3 is not compatible with FESA2 version. Field ‘errMsg’ was renamed ‘errorMessages’… Just because it looks ‘nicer’ !
Changes needed in Java expert applications ?
Reconfiguration of Logging process ?
Implementation of PM Data converters ?
Other consequences ?
Behavior change : Force field ‘acqStamp’ to current time if it was set to ‘0’ by the SET server action. Problem: I use acqStamp = ‘0’ to indicate ‘invalid data’ or ‘no data available’. Answer from FESA team: You can use ‘-1’ instead !
Changes in FESA3 classes
Changes in Java Expert applications
Avoid unnecessary non-backward compatible changes ! Evaluate the consequences of needed non-backward compatible changes, and anticipate update on impacted systems before release of change, discussion with equipment groups.
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

10. BE-CO LS1 review - View from TE/ABT/EC
IEPLC / SILECS
SILECS released, nice new management tools : It works fine.
Still same problems of disconnection/reconnection when PLC code is reloaded.
100% backward compatible API, but library/namespace/class names changed. Was it really needed ?
Modification and release of all FESA3 classes and ABT libraries to adapt these names = time consuming.
New features requested for IEPLC not available (data container with getter/setter).
Theses new features were implemented too late to be integrated in our code during LS1.
We were forced to switch to SILECS due to usage of new Siemens PLC (not supported by IEPLC)
This was not foreseen, as there is no PLC function blocks generated anymore (?). Silecs was not ready yet at this time…
Test and debug of IEPLC / SILECS done with ABT, using our PLC and PXI systems.
Usage of our test systems for IEPLC/SILECS debug = time consuming…
Is there a test bench for IEPLC / SILECS, with all supported PLC and PXI models ?
Better collaboration between BE/CO and EN/ICE needed ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

11. BE-CO LS1 review - View from TE/ABT/EC
Logging Service
The logging of all ABT variables was deactivated at the beginning of LS1, as planned ahead.
The logging of ABT systems was reactivated ‘on demand’ for commissioning needs.
Requested variables were logged properly for our commissioning phase.
BUT after LS1, many operational variables were not logged, because no explicit request from ABT.
Logging team had apparently no way to know which variables are ABT and should be enabled in operation.
It took us a lot of time, and many iterations, to have all operational variables logged properly.
Better identification of operational variables with responsible person/group, and force logging ?
Crash of Logging Measurement DB (During TS after LS1, but interesting nevertheless…)
We discovered with surprise that NO BACKUP of measurement DB was performed !
We lost calibration results at the end of TS, and had to perform again calibration runs during the week-end.
Introduction of redundancy and backup system for Measurement DB ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

12. Logging Service – SDDS files replacement
New logging API provided, Java expert application needed refactoring (as was planned)
New API worked fine, no much problems encountered.
Using Measurement/Logging DB to store properties in replacement of SDDS files on NFS :
Implementation using one logging variable per property field
It is possible to have a different ‘on change’ and filtering configuration for each variable.
Many problems with extraction of AcquiredProperty from logging:
Due to ‘on change’ or filtering configuration, fields were missing in properties stored in Measurement and Logging DB.
It took us time to understand the problem and remove all filters on AcquiredProperty variables.
Many LBDS calibration runs were unusable because of this problem.
Guaranty extracted property atomicity:
Check that filtering is disabled for all AcquiredProperty variables ?
Implement a cleverest filtering mechanism ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

13. BE-CO LS1 review - View from TE/ABT/EC
Java Applications
Upgrade of all BE/CO Java libraries for our Expert applications, not many problems encountered !
JAPC : RDA2 to RDA3 migration was transparent.
New Logging API integration was OK (But we had to ).
Migration to slf4j well prepared, worked fine using CommonBuild.
PM GUI / XPOC updates went fine.
Directory service description read from database introduced huge delays, was corrected quickly.
Etc…
No proper vertical integration, from FESA3 to WorkingSet (Knobs):
Features possible in FESA3 interface are not supported by higher level software (not visible in FESA3 design).
Various problems with JDataViewer were identified, but no fix so far.
Scaling problems with 3D data (images).
Zoom function do not work with inverted axis.
Better vertical integration from FESA to WorkingSet (Knobs) ?
Better follow up of JDataViewer problems, more resources on this important library ?
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

14. BE-CO LS1 review - View from TE/ABT/EC
Post Mortem
Integration problems with TE/MPE
Many releases of CMW / PM / FESA3 necessary (PM still makes use of RDA2)
First working version came very late (just before first injections)
Following these problems, integration of PM into FESA3 was started after LS1.
XPOC not available during sector tests and first injections.
Almost all XPOC analysis sessions failed because of missing data
XPOC not available during MKD and MKB waveform scan with beam.
We could not rely on PM (XPOC) to save the data and reload/analyse afterward.
Measurements based on screenshots!
BE/CO and TE/MPE must work together to avoid such problems in the future.
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

15. BE-CO LS1 review - View from TE/ABT/EC
Summary
BE/CO – TE/ABT collaboration during LS1 was good.
Nevertheless there is some space for improvements :
Hardware/Software upgrades not in phase with machine schedule -> Ready for tests ~6 months before LS.
BE/CO software development phase required a lot of resources from equipment groups.
Non backward compatible change consequences to be evaluated, discussed with equipment groups.
Long release cycle for bug fixes -> Consider using shared objects ?
Redeliver of FESA3 classes after every framework release is time consuming -> Automated deliver process ?
Rollback to previous FESA3 version not always possible (persistency file problem)
Etc..
Upgrades, modifications of API, new features should be discussed well in advance!
Now, renovated control framework seems stables for us, almost no releases since LHC startup ! We are quite happy with the new development tools, we appreciate the improvements provided.
We are looking forward to prepare LS2 with you !
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC

16. BE-CO LS1 review - View from TE/ABT/EC
01/12/2015
BE-CO LS1 review - View from TE/ABT/EC