1. Designing a HEP Experiment Control System, Lessons to be Learned From 10 Years Evolution and Operation of the DELPHI Experiment. André Augustinus 8 February 2000

2. CHEP 2000 8 February 2000 2 Overview l The DELPHI experiment control system l The evolution of the DELPHI ECS l Observations and lessons learned from over 10 years of operation l Conclusions

3. CHEP 2000 8 February 2000 3 What is an ECS ? What is an Experiment Control System ? l All systems controlling or monitoring (parts of) an experiment l ‘Classical’ controls Ø Power supplies, temperatures l Data-flow controls Ø Start/stop of a run l Interface with external systems Ø Accelerator, safety system

4. CHEP 2000 8 February 2000 4 The DELPHI ECS What is the DELPHI Experiment Control System ? l Two main components: l Slow Controls system (SC) l Data Acquisition System (DAS) l Other components: l Trigger system l Communication with LEP accelerator l Data quality monitoring

5. CHEP 2000 8 February 2000 5 The DELPHI ECS (SC) Slow Controls system l Several 1000 channels l Power supplies, temperatures, pressures l Independent per sub-detector l G64/G96 crates (~100), PLC l M6809, M68340 processors l OS9, RPC/OSI, TCP/IP

6. CHEP 2000 8 February 2000 6 The DELPHI ECS (SC) The ECS is used to l Prepare sub-detectors for data taking l Report (and correct) anomalies l Integrate ancillary systems Ø Gas, magnet, safety l Store detector status on database(s) l Control & monitor the hardware

7. CHEP 2000 8 February 2000 7 The DELPHI ECS (DAS) Data Acquisition System: l Over 250 000 channels l 20 partitions + central partition l Can run independently l Fastbus (~180), OS9, TCP/IP The ECS is used to l Configure the readout l Initialise the partitions l Control and monitor the data flow

8. CHEP 2000 8 February 2000 8 The DELPHI ECS (Other) Trigger system l Decision and timing l Hierarchical: local and central ECS is used to l Configure and initialise the trigger system l Download look-up tables l Monitoring of counters

9. CHEP 2000 8 February 2000 9 The DELPHI ECS (Other) LEP communications l Bi-directional exchange of experiment and LEP machine parameters Ø Luminosity, background, machine settings Data quality monitoring l Several 100 histograms l Event processing farm

10. CHEP 2000 8 February 2000 10 The DELPHI ECS (Software) Two main packages: ¶ SMI++ Models the behaviour of a system as a Finite State Machine in a dedicated language (SML) l Objects Ø Associated: represent concrete entities Ø Abstract: behaviour defined in SML l Objects are in a definite state l Objects can receive action requests l Logically related objects are grouped in domains

11. CHEP 2000 8 February 2000 11 The DELPHI ECS (Software) l User interface Ø State of objects are presented to the operator Ø Commands are given by sending action requests · DIM l Publish/Subscribe paradigm l Universal data exchange package Ø Refer to previous talk by Clara Gaspar

12. CHEP 2000 8 February 2000 12 The Evolution of the ECS l Started taking data in 1989 l No ‘real’ ECS yet l Only a rudimentary version of SMI l Line mode interfaces l ‘Human’ synchronisation

13. CHEP 2000 8 February 2000 13 The Evolution of the ECS l Taking advantage of SMI (1990-1991) l Completion of SC and DAS domains l Centralised control of the experiment l Abstraction Ø Variety of hardware can be represented by one type of object Ø Logically related objects can be summarised in one single object l Uniformity across sub-detectors

14. CHEP 2000 8 February 2000 14 The Evolution of the ECS l Introduction of DIM (1993 onward) l Make use of user interfaces more flexible l Solve communication problems inside SMI l Evolved to a universal data exchange package in the experiment l DIM really started an integrated ECS : l Trigger, LEP communications became integrated part of ECS

15. CHEP 2000 8 February 2000 15 The Evolution of the ECS l Reengineering of SMI (1996) l Improve maintainability and portability l Using OO techniques l Smooth transition because of well defined interfaces (already in design phase) l Hardware and Software upgrades Ø New sub-detectors Ø New technologies Ø New versions of operating systems

16. CHEP 2000 8 February 2000 16 Automation in the ECS l Automation is unavoidable and imperative for efficient running of a complex experiment l Too complex for a non-expert operator l Gain in time, efficiency l Ensure consistency l Relatively easy to implement because of the use of SMI in all domains

17. CHEP 2000 8 February 2000 17 Automation in the ECS l Examples of automation l Trip recovery Ø Automatic reaction to trips l DAS auto-pilot Ø Automatic reaction to a variety of anomalies l SMI analyser Ø Analyse combination of SMI states l ‘Big Brother’ Ø Interconnection of various domains Ø ‘Hands-Free’ running of the experiment

18. CHEP 2000 8 February 2000 18 Future ECS l LEP experiments were probably the first generation that needed an ECS l One should take advantage of the expertise gained in running these big experiments

19. CHEP 2000 8 February 2000 19 Future ECS l Partitioning l Well thought out to allow stand-alone running (debugging, calibration) l Unhindered operation when part of the experiment is off l Central control l Small crew of operators l Well structured commands

20. CHEP 2000 8 February 2000 20 Future ECS l Uniformity across ECS subsystems l Will ease integration and automation l Will reduce maintenance efforts l Uniformity across sub-detectors l Common hardware and software Ø reduced costs and maintenance effort Ø reduced development efforts Ø easier operation l Use of commercial solutions

21. CHEP 2000 8 February 2000 21 Future ECS l Central support team l Strong central support is a great benefit l Provide guidelines and frameworks Ø enforce uniformity l Provide common solutions for common problems l Will ease maintenance over lifetime

22. CHEP 2000 8 February 2000 22 Future ECS l Flexibility l ECS is never ‘finished’ l Many changes will happen over the lifetime of an experiment, the ECS should cope with Ø Modification or addition of a sub-detector Ø Upgrades with new technology Ø Change of working points or operational procedures l Easy to modify or reconfigure Ø Good documentation

23. CHEP 2000 8 February 2000 23 Future ECS l Efficient day-to-day operation l Abstraction Ø Non-expert operators can run the experiment Ø Hide detailed information Ø Uniform representation and commands l Automation Ø Automatic error recovery Ø Automate ‘standard operations’ l Proper training and adequate documentation

24. CHEP 2000 8 February 2000 24 Summary l Designing an ECS Ø Strong central support Ø Partitioning Ø Uniformity Ø Flexibility Ø Abstraction Ø Automation