Towards a better testability of the control system components S. Deghaye with help from M. Peryt and input from L. Burdzanowski, G. Kruk, W. Sliwinski, J. Lauener, J-C. Bau, A. Dworak, F. Hoguin, J. Wojniak, R. Gorbonosov, L. Cseppento

Introduction What is a testbed? Two cases in our context: A testbed is a platform for conducting rigorous, transparent, and replicable testing of scientific theories, computational tools and new technologies Two cases in our context: Sandbox No persistency except environment Release-less, cheap, throw-away tests or prototypes Full PRO-grade validation No risk to break PRO but very close to PRO We don’t mock any part under test Users expect pro-SLA Who are we targeting? Developers/Integrators of complete full-stack solutions from hardware to GUI. CO3 recommendation (EDMS) Docker could help for the sandbox

Current situation Integration tests in BE-CO Unit tests in BE-CO Control Testbed aka CTB set-up several years ago Main users are CMW & FESA Dedicated timing, all available platforms, isolated environment on the GPN Timing Testbed NXCALS FESA devices, old CCS test environment Uses the CMW test environment InCA had 2500 devices but they were unused => removed Unit tests in BE-CO The vast majority have unit tests (out of scope of this talk) We provide very little to our clients => Their feedback is… Devices on inca1-5

Common feedback from Equipment Groups Develop Release Deploy Import Configure Test Low-level development cycle Operational Timing ONLY Vertical testing not possible Long iteration time Impossible to be GPN-only Versioning too strict … Incompatible features (square peg into round hole) No DEV env => need to go to PRO => need to release + PRO timing only

Proposal Provide a complete infrastructure to facilitate the setup of testbed(s) Not one CTB but as many independent TBs as needed  no cross-talk Take into account the whole workflow; not just testing Transition to PRO Staging environment, instead of separate environment Developments to be integrated in individual service plans Incremental availability depending on needs (e.g. TE-MPE) This infrastructure and its maintenance comes at a cost Difficult to estimate

Proposal

Current situation SVN Hardware SW - Testbed SW - Production CBNG Test code Applic UT RBAC Auth. SRV CALS InCA/LSA LSA DB PRO CMW Dir SRV PRO version CCDB PRO CTR HW DUT CPU … FESA Process CTR HW DUT CPU … FESA Process CTR HW DUT CPU … FESA Process NFS Local dev

Proposal SVN Hardware SW - Testbed SW - Production CBNG Test code Applic UT RBAC Auth. SRV CALS InCA/LSA LSA DB LSA DB PRO CMW Dir SRV Dev version PRO version CCDB CCDB PRO CTR HW DUT CPU … FESA Process CTR HW DUT CPU … FESA Process CTR HW DUT CPU … FESA Process NFS GPN-only Timing (simulated)

Jenkins, GUI testers, Junit, etc… Automation SVN Hardware SW - Testbed SW - Production Context/Container CBNG Test code Applic UT RBAC Auth. SRV CALS InCA/LSA Jenkins, GUI testers, Junit, etc… LSA DB LSA DB PRO CMW Dir SRV Dev version PRO version CCDB CCDB PRO CTR HW DUT CPU … FESA Process CTR HW DUT CPU … FESA Process CTR HW DUT CPU … FESA Process NFS GPN-only Timing (simulated)

Control System missing features Run-time infrastructure is incomplete Integration environment does not exist for most components Dev release cycle for low-level CS PRO upgrades (2nd phase) Timing simulation is a must Tests must be replicable

Status & plans per service/component BE-CO Initiative CS-378 to collect WPs and track progress Initiative involves: CCS CMW FESA FEC environment InCA/LSA CALS Timing HW commissioning sequencer CBNG PM server

CCS Two environments for testing Known Issues: Sandbox use case: INT: integration Empty DB with dictionary entries only. For testbeds that need to be setup from scratch TESTING: Periodic, on-demand snapshot of PRO. No persistence For further details see: https://wikis.cern.ch/display/config/New+CCS+test+environment Known Issues: APEX not available Too many constraints from PRO (e.g. versioning) (FESA/CCS) Synchronisation to/from PRO Sandbox use case: Would a CCDB in Docker be worthwhile?

CMW RBAC authentication SRV & Directory SRV Test instances available on the GPN Connected to old ACCINT CCDB To connect to the new INT CCS would require a few days work Integration in INT to be improved Requires several flags (testbed users shouldn’t care)

FESA Dev release feature: Upgrade to PRO Build chain to be added + integration with Eclipse plug-in Allow re-release with build info e.g. 1.2.3-20170923034923 Cost: 0.5PM Upgrade to PRO Shouldn’t have to recompile – use what was tested! CODEINE is a must to avoid FESA-specific solution Cost:1.5PM

FEC environment Generation scripts to be re-written First step towards eradication of transfer.ref/new_dtab Quite risky, as it touches 30 year-old code Huge impact (2000+ FECs) As a lightweight first step, modify the existing scripts

InCA/LSA LSA Testbed DB to be put in place One DB for all testbeds, but more might be needed later, if too much cross-talk Ongoing LiquiBase effort should help LSA could be single instance InCA must be 1 instance per TB (Aquisition-ready event) Cost: 1PM Good opportunity to clean-up configuration Automatic insertion upon FESA class dev-release not obvious (details to be checked) Support effort hard to estimate Lack of tools Is 1 Testbed ≡ 1 Accelerator when it comes to support ?

CALS Hypothesis: NXCALS only Several environments already available but a PRO-level instance is required Hardware or OpenStack? Hadoop development environment provided by IT is pro-grade (limited space) Data policy to be discussed Cannot keep everything forever Connect to CCDB INT GPN w/o TN visibility Cost: <0.5PM

Timing Local Timing Central timing Not a problem LTIM PRO versions must be available as part of the infrastructure Central timing Operational timing is not sufficient Prevents reproducibility No control on what is played No possibility to validate corner cases Simulation/testbed must take into account the HW Cannot be FESA event simulation (lack of interrupts and pulses) How to simulate the GMT?

Timing simulation Simulated timing requires HW Courtesy J-C. Bau Timing simulation Simulated timing requires HW CTSYN – can it work w/o GPS? MTT to drive the GMT cable CTR for the distribution to apps (TIDE) Keep it simple to start with but… XML file required to configure the MTT (complex and support-intensive) Basic tool required to generate the file, adding a few checks (GUI not necessary) Can increase complexity later, if needed Cost: 3-4PM for the timing team

HWC sequencer Needs INT LSA DB Communication with acc-testing Deploy test instance of system server (MPE) Fwk to coord schedule the tests, auto analysis (10 y/o)

CBNG Currently either DEV (e.g. 1.2.3-20170831142300) or PRO release with version from product.xml Artifactory contains 3rd party, DEV and PRO releases. Deploy repository: ~pcrops Services on GPN Uses deploy tool (copera user) with --dev to access the dev deploy repository Future: promote DEV release to PRO

PM server FESA should send data to pm-test when in INT environment To be studied how FESA can redirect automatically w/o changing the FESA class design

Bamboo vs Jenkins Current testbed uses Bamboo A few teams already moved to Jenkins Jenkins appears to be more flexible and easier to script Jenkins is the market leader, and a product that is well-known by students Need to decide what we recommend Phasing-out Bamboo comes with a cost Bamboo’s Total Cost of Ownership

Network Icing on the cake: Add a separate network domain to further protect the testbed environment from the risks incurred from the GPN

SLA? What is expected? Proposal Remain within working hours: no out-of-hours service Testbed must be reasonably stable Intended to be PRO-grade, not Test Aim for a reaction time of a few hours At least look at the problem within a day next steps: - milestones (MPE, LS2)

Milestone plan

Milestone plan Phase 1 Phase 2 Phase 3 Separate environment on the GPN Versioning less strict than in PRO End of 2017, InCA/LSA end 2018 Phase 2 INT environment part of a pipeline => Promote from INT to PRO Phase 3 Simulated timing Hardware aspect (availability, redesign, etc.) to be considered Could be part of WhiteRabbit for timing ?