1. KLOE Slow Control Architecture
Fabrizio Murtas, INFN LNF
Paolo Valente, INFN Roma and INFN LNF
KLOE DAQ
KLOE
Slow Control
DAFNE
Control System
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

2. DAFNE f-factory Two interaction regions
Separate e+, e- rings to minimize beam-beam interactions,
510 MeV MeV
Beams cross at 12.5 mrad angle
Injection during data-taking
DAFNE parameters
Design
2002
Max. bunches
120 51
Lifetime (mins) 40
Bunch current (mA) 20
L, single bunch (cm-2s-1)
4.4 1030
1.5 1030
L, peak (cm-2s-1)
5.3 1032
0.75 1032
f per year (109) 15
0.9
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

3. KLOE experiment
General purpose detector : ~4 m diameter & ~4 m length.
Superconducting Coil of 6 KG.
Lead-Scintillating Fiber Calorimeter ADC & TDC Channels HV Channels LV Channels (Thresholds, Preamp, Pulse)
Helium Drift Chamber stereo cell geometry
Sense Wire HV Channels LV Channels (Threshold, Dead and Width Time, Preamps, Pulse)
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

4. Slow control functions - 1
The KLOE Slow Control is an hardware/software system to monitor and control:
High voltages used for detectors
Low voltages, thresholds and pulsing systems on FEE crates
Power supplies of all VME crates
Fast readout of VME scalers for monitoring of
Trigger rates
Background counters
Front End
Hardware
Slow Control
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

5. Slow control functions - 2
Some systems have to be integrated in the Slow control, due to different architecture and/or different hardware
Remote monitoring of DC gas system and its alarms
Remote monitoring of the KLOE magnet control system
Integration with data coming from DAFNE accelerator controls
Other
Control Systems
Front End
Hardware
Slow Control
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

6. Slow control functions - 3
With time more Slow Control functions were added:
Log of all Slow Control parameter to the KLOE DB and storage of main detector parameters directly in the events
Communication with Run Control for parameters setting at run initialization
Other
Control System
Front End
Hardware
Slow Control
events
DAQ,
Run Control,
Database
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

7. Slow control functions - 4
The monitoring of the slow control parameters is essential for the good running of the detector: a simple and efficient user interface has to be provided to the operators and experts
Other
Control System
Front End
Hardware
Slow Control
User
interface
events
DAQ,
Run Control,
Database
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

8. Used technologies
The Kloe Slow Control System has been realized using as much as possible common Standards :
VME Crate and CPU, as already used by the rest of DAQ System
CAENet Serial System to connect different crates controllers for HV, LV, and power supplies
TCP-IP for communication between main slow control and other control systems (Gas, Dafne & Magnet, using LabVIEW)
SQL for main slow control  database communication (Offline)
SNMP protocol for inter-processes communication
Web Server for information distribution
We will describe all the components of the system and its architecture
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

9. Slow control hardware DAFNE Control Front End Hardware Slow Control
User
interface
DAQ,
Run Control,
Database
Slow control hardware
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

10. CAENet interface VME Temperature/Gas Interlock Daisy chain Daisy chain
V288 CAENet
V288 CAENet
VME
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

11. HV 720 channels 5000 photomultipliers Drift chamber HV 3 SY527 crates
HV cards
6 wires
720 channels
Calorimeter HV
21 SY527 crates
5000 photomultipliers
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

12. Slow control levels Commands & monitoring Storage VME
Interface
Commands & monitoring
User level
Slow Control
machine
High level
process
High level
Low level
process
Storage
VME
CPU
Serial links
(CAENet protocol)
V288
Low level
Low voltages & PS
High voltages
Hardware level
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

13. Low level process Hardware Interface
5 Process level: manage commands, call level 4
4 Configuration files read/Log and statistics write
3 Detector parameters structure handling
2 CAENet slaves read/write operations
1 CAENet operations Send_command/Read_buffer
0 V288 CAENet Board addressing, VME read/write
Hardware
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

14. Slow_hvdc Slow_hvemc Slow_emc
5 main monitoring processes continously run on the low level machine (L2AX12 in the dedicated VME crate):
L2AX12
Slow_hvdc
Slow_dc
Slow_hvemc
Slow_emc
Slow_daq
CAENet
Chamber HV
Chamber LV
Calorimeter HV
Calorimeter LV
Crate PS
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

15. Logging and history Hardware
Low level process
Different kind of log and history files produced by each monitoring process:
Log file: standard output and errors from process
Error file: errors and failures from hardware level
Status file: current values of all monitored parameters
History file: values of relevant monitored parameters
Different logging time intervals:
15 seconds for fast changing variables (general status)
1 minute or 5 minutes for slow changing variables (thresholds, HV settings, gas parameters)
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

16. Inter-process communication
The Slow control implements the online template
It uses the KLOE message system:
Processes and their relevant variables are mapped in a SNMP1 tree
A private SNMP daemon runs in every node: cmdsrv (command server)
Variables are periodically updated, so monitoring tools can access them
Commands implemented as setting of SNMP variables
SNMP traps used for asynchronous notifications
1 Simple Network Management Protocol
cmdsrv
cmdsrv
UDP
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

17. Program interface
The interface of the Low level monitoring processes is realized using the message system of the KLOE DAQ
Commands can be sent and data can be received from any machine in the online farm
KLOESLOW
Program
High level program
Command
Low level process
cmdsrv
Output
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

18. Program interface Commands: Output Hardware
UPDATE
SET
GET
SWITCH
VERIFY
MANUAL
Low level process
INIT
START
PAUSE
RESUME
END
EXIT
Hardware
VME+CAENet
MONITORING
Log & History
Send commands is implemented by directly setting internal process variables
In the same way, Output receive is implemented by directly reading internal process variables
Variables are set/read by accessing shared memories (through the message system)
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

19. DAFNE
Control
Front End
Hardware
Slow Control
User
interface
DAQ,
Run Control,
Database
User interface
As user interface we have chosen a set of Web pages, displayed by a Web Server and produced by CGI programs
We use Apache Web server
CGI = Common Gateway Interface
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

20. User interface Client (Web browser) Web server Firewall protection
Internet
Client (Web browser)
CGI Interface
CGI program
Web server
HTTP protocol
DATA
Firewall
Firewall protection
Password protection
IP address protection
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

21. Slow control software - 1
Web pages
Web server
KLOESLOW
CGI programs
cmdsrv
Low level processes
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

22. User interface examples
DAFNE
Control
Front End
Hardware
Slow Control
User
interface
DAQ,
Run Control,
Database
User interface examples
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

23. Main web page Low level processes Main menu
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

24. Processes monitoring Calorimeter Drift chamber Detector Status Process
Comm. Status
Detector Status
Status
Started
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

25. Chamber HV It is possible to produce the map of all the channels
It is possible to locate and
switch on immediately
the channel in trip
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

26. Chamber HV 0.0 mA 0.1 mA 0.2 mA 0.3 mA 0.4 mA Dinamic map of the
current for all drift
chamber channels
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

27. HV control Process and HV status Switch on/off Change settings
Command output
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

28. Calorimeter HV Voltages of all the 5000 photomultipliers
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

29. HV control Process and HV status Switch on/off Change settings
Command output
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

30. Chamber LV Test pulse Threshold, dead time
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

31. LV control Process and LV status Set threshold Set pulsing and veto
Command output
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

32. Calorimeter LV Details
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

33. FEE crates L2 crates power supplies FEE crates power supplies
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

34. FEE crates Voltages & currents monitoring Fans & temperature
Switch on/off
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

35. Nano-amperometer “faster” Slow control device:
Nano-amperometer built and installed on 4 HV channels of the chamber, monitoring 4 x 12 wires in the very inner layers
Current to frequency converter (logarithmic)  scaler
Logarithmic accuracy (better at low current)
CAENet communication: max. readout speed 150 ms
Measured currents used to monitor machine background in the inner part of the chamber
Clear evidence of current peaks induced by DAFNE injections, in excellent agreement with all other background monitors
Monitoring process, integrated in the general framework
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

36. Nano-amperometer 4 HV channels, very sensitive to machine background
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

37. Currents in first layers
Chamber ch. 1 Current
[mAmpere]
1 day
e+ and e- currents
[Ampere]
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

38. Communication with other control systems
Front End
Hardware
Slow Control
User
interface
DAQ,
Run Control,
Database
Communication with other control systems
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

39. Other monitored systems: Gas
Serial lines
(flow meters,
analyzer)
Digital I/O
(valves, pump)
Analog I/O
(temperatures,
pressures)
Gas PC
A sophisticated gas system was realized in order to run the drift chamber in closed loop as well as in flushing mode
Gas analysis for isobutan content, oxygen and water content measurement
Realized in NI LabVIEW:
need to integrate in the general Slow control framework!
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

40. Gas system General status Pressures, flow, contaminants
Alarms, errors and log
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

41. Other monitored systems: Magnet
Serial lines
(power supply,
cryogenics controls)
Digital I/O
(valves for cryogenics)
Analog I/O
(sensors)
Magnet PC
Superconducting coil of the KLOE magnet from OXFORD Instruments
Cryogenics and power supply full custom control and monitoring
Realized in NI LabVIEW:
need to integrate in the general Slow control framework!
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

42. Other monitored systems: solution
Hardware
Control program
Gas PC
Hardware
TCP/IP socket
Slow Control
Control program
Data servers
Magnet PC
TCP/IP socket
Spy data
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

43. Other systems: DAFNE DMA Low level VME GUI Hardware device
Optical links
High level CPU
User interface tasks
VME CPU RAM
Dumper
Storer
Distributed, RAM,
Real-time database
DMA DB
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

44. DAFNE controls
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

45. DAFNE/KLOE common DB DAFNE info KLOE info Dumper KLOE Storer
Slow Control
Storer
NFS cross-mount
Slow
Fast
Slow
Fast
User task
User task
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

46. Other system: offline other data sources:
Collection of data from other monitoring system (online and offline farms)
Collection of data from monitoring programs, PHYSMON, running online reconstruction on the experiment data
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

47. Other systems
For each “system” external to the Slow control, but monitored by it, a dedicated DATA SERVER runs on the high level machine, in order to continously collect data:
From TCP-IP process
From the DAQ message system
From access to data storage on disk
2 Listen for TCP connection or Read status
1 Configuration files read/Log and statistics write
0 TCP/IP socket routines / Read routines
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

48. Slow control software - 2
Web pages
Web server
KLOESLOW
CGI programs
Data servers
Low level processes
Other
systems
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

49. Data collection & synchronization
Slow_emc calorimeter LV
Slow_hvemc calorimeter HV
Slow_dc chamber LV
Global.fast:
refresh every 15’’
Slow_hvdc chamber HV
Slow_daq power supplies
GASserver chamber gas
Global.slow:
refresh every 60’’
MAGserver magnet …
PHYSMON event online reconstruction
TRGMON trigger monitor
DAFNE database
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

50. Slow control software - 3
Web pages
Web server
KLOESLOW
Global
CGI programs
Data servers
Low level processes
Other
systems
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

51. Other functions… Fast-Ethernet
The slow control high level CPU is also the server for the diskless L2 VME CPU’s (OS download and NIS)
KLOESLOW
Fast-Ethernet
L2AX01
...
L2AX10 +
L2AX12
L2AX13
+ 2 spares
10 DAQ chains
Slow control
Trigger
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

52. Slow control – DAQ communication
DAFNE
Control
Front End
Hardware
Slow Control
User
interface
DAQ,
Run Control,
Database
Slow control – DAQ communication
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

53. Slow control – DAQ communication
At run initialization: set thresholds & channel veto for chamber, calorimeter from Run control to Slow control
To have a quick run initialization, a single INIT command is sent to a intermediate process, Slow-RC
Slow-RC distributes the appropriate commands to low level slow processes
Commands through KLOE message system (cmdsrv)
KRUNC
Run Control
cmdsrv
cmdsrv
Low level processes
Slow-RC
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

54. Slow control – DAQ communication
ONLINE FARM
builder
builder
cmdsrv
KRUNC
cmdsrv
Run Control
During the run, the Slow-RC process sends all the relevant detector parameters to the Run Control (every 15’’)
Detector parameters are then stored by the builders in the event data structures
cmdsrv
cmdsrv
Low level processes
Slow-RC
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

55. Slow control software - 4
ONLINE FARM
KRUNC
Run Control
builder
builder DB
Web pages
Web server
KLOESLOW
Global
CGI programs
Data servers
Low level processes
Other
systems
Slow-RC
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

56. Architecture - 1 Architecture - 1 Detector Operator User Interface
Hardware
Low level
software
High level
Architecture - 1
Architecture - 1
Operator
Fire, Gas Leak,
Water
Hardware Alarms
Slow Control
Detector data
L2 filesystem
DAFNE
Gas
Magnet
LV,HV,PS
serial
VME,
CAENet L2
Detector
VME
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

57. Architecture - 2 Operator User Interface Slow Control Online farm
Hardware
Low level
software
High level
Architecture - 2
Operator
Slow Control
Online farm
Run Control
Detector data
L2 filesystem
Run Condition
+ Software
Event data DB
LV,HV,PS L2
Run DB
Calibration DB
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

58. User interface examples
DAFNE
Control
Front End
Hardware
Slow Control
User
interface
DAQ,
Run Control,
Database
User interface examples
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

59. Main web page Low level processes Data server processes Main menu
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

60. Processes monitoring Calorimeter Drift chamber Detector Status Process
Comm. Status
Detector Status
Status
Started
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

61. KLOE & DAFNE summary General KLOE status General DAFNE status
Trigger rates
from Event
reconstruction
from
Slow control
Luminosity estimate
Beam position and E
Chamber noise
Calorimeter noise
Calorimeter timing
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

62. Background monitor
Fast feedback from noise counts in the calorimeter endcaps
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

63. Other monitoring
Simple Web interface to run database for run lists queries for
Raw files
Reconstructed files (streams)
Simple Web monitoring of reconstruction jobs running in the offline farm and of tape library status
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

64. Reconstruction monitoring
List of
reconstruction
jobs
List of CPU’s
and their status
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

65. DB web interface List of runs and general Information DB
Links to reconstruction
information in DB
Cross section counters
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

66. DB web interface Reconstructed files: general information
and links to detail
Reconstructed
parameters
summary
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

67. Log and History
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

68. User interface: ROOT presenter
ONLINE FARM
presenter
presenter
KLOESLOW
Global
Presenter:
all quantities in the global database can be histogrammed ONLINE in the last 1-24 hours or
HISTORY charts spanning long periods
Based on ROOT libraries from CERN.
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

69. ROOT presenter Menu & draw classes General draw class Read DB classes
ROOT libraries
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

70. ROOT presenter The presenter can produce charts with the trends of:
Slow control variables showing the status of the detector, such as LV, HV, gas parameters
DAFNE accelerator parameters (such as the currents or lifetimes)
Offline quantities, such as noise estimate, luminosity estimate, background rates, beam boost and energy, etc.
Very useful for luminosity/background optimization
Important for global monitoring of the status of the detector
Automatic update of the time charts
Help by Zhou Yongzhao
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

71. ROOT presenter e- and e+ currents from accelerator controls
Luminosity estimate
from large angle
Bhabha events
Trigger rate
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

72. ROOT presenter
Very easy graphical interface with ROOT characteristics (zoom, etc.)
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

73. Running KLOE in automatic
runs:
Run control, Slow control,
Histogram monitoring
2 shifters
2004 run:
Slow control-Run control integration,
Automatic run start-pause-stop, automatic reset of DAQ errors.
Only 1 shifter for general monitoring
Alarms largely remotized, call experts when needed ?
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

74. KLOE remote control Automatic data taking in Run Control
of hardware alarms
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

75. Alarms
There are watchdog processes, checking the status of all monitoring processes both in the high level machine (KLOESLOW, running the data servers and the Web server) & in the L2 machine (L2AX12):
generate alarms
restarting automatically missing monitoring tasks
For critical errors, such as problems on the chamber gas systems,
SMS alerts
alerts
GSM
AT modem commands
serial link
Watchdog
Detector data
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

76. Slow control software - 5
ONLINE FARM
KRUNC
Run Control
builder
builder DB
presenter
presenter
Web pages
Web server
KLOESLOW
Global
Watchdog
CGI programs
Data servers
Low level processes
Other
systems
Slow-RC
Watchdog
L2AX12
Hardware
VME+CAENet
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

77. Architecture - 3 Architecture - 3 Detector Operator User Interface
Hardware
Low level
software
High level
Architecture - 3
Architecture - 3
Operator
GSM
WebCam
Fire, Gas Leak,
Water
Hardware Alarms
Slow Control
Alarms
Online farm
Run Control
Detector data
L2 filesystem
Run Condition
+ Software
Event data
Gas
Magnet DB
LV,HV,PS
serial
VME,
CAENet L2
Run DB
Calibration DB
Detector
VME
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京

78. 谢谢 !
KLOE Slow Control Architecture – Fabrizio Murtas, Paolo Valente - 北京