Karol Buńkowski Michał Pietrusiński University of Warsaw RPC PAC trigger software CMS L1 Trigger Online Software Review, 5 February 2009

RPC PACT online software schema CMS L1 Trigger Online Software Review, 5 February 2009 Karol Buńkowski, Warsaw University DB service RPC FM RPCT TS Cell TC access XDAQ Trigger and Sorter Crates DCC/CC S crate … To RCMS Top FM To TS Central Cell … LBox’es SC access XDAQ … FEB FECFEC VME FECFEC CCU rings DCC access XDAQ DCC access XDAQ LBox access XDAQ LBox access XDAQ LBox access XDAQ LBox access XDAQ … DCCDCC I2C rings … VME Monitoring and test manager JAVA Monitoring and test manager JAVA Config DB TS Subcell … TStore Conditio n DB TS Subcell Dashed = planned

Hardware Access XDAQ Hardware Access (HA) XDAQs –The only applications with direct access to the hardware via the VME interface, –One application per each Trigger Crate (12TC+1SC); XdaqTestBench class, –One application per each CCU ring (= 6 LBoxes = 90 LBs), in total 18 applications; XdaqLBoxAccess class, Functionalities: –Hardware setup: Retrieve hardware structure from DB and build objects corresponding to devices, Retrieve configuration data (including firmware version ID) for the given config key from the DB, Check if correct firmware is loaded, reload if necessary, Check the hardware status (TTCrx, QPLL), reset if necessary, Check if correct config data are applied (ID of config data version is written in a dedicated registers), if not – reconfigure. –Hardware monitoring, –Access point to the hardware for other applications via the SOAP (e.g. FEB config XDAQ, JAVA Test Manager). Both low level functions (register read/write) and high level (massive configuration of FEBs or diagnostic modules) available via SOAP, as well as info about hardware structure. 5 computers used to control the hardware, on them the HA XDAQ are running. The XDAQs are running as services, but started manually (what is need at the moment for debuging). Starting through the CMS Trigger-Process Controller can be configured at any time. CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

HA XDAQ GUI - TC CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University In standalone mode: available config keys from DB. If controlled by TS – the key is received from TS Monitoring control Configure options Hardware Status monitorables Monitoring histograms

HA XDAQ GUI – Link System CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University Firmware upload Monitoring: Chambers rate

Trigger Supervisor Present RPC TS architecture: RPC TS central cell  13 TS subcells (one for each TC and SC)  13 standalone TC/SC HA XDAQs Each subcell communicates always with one XDAQ to allow parallel configuration using TS framework. Functionalities of the central cell: –MTCCIIConfiguration operation to configure the system: very simple procedure: pass configuration command with configuration key to HA XDAQs via the subcells. –Monitoring: collects, analyses and displays the monitoring data obtained as flash-lists and monitoring charts (png files) from the HA XDAQs All cells are started by CMS Trigger-Process Controller CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Databases CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University Configuration DB (~ 60 tables) Hardware structure  Chambers structure: location (no geometry info), strips, FEBs, connections,  Electronic hardware components (crates, boards, chips) and their relations and connections,  Assignment of hardware partition for given HA XDAQ,  List of disabled devices  Configuration data: config key, register values, firmware ID, name of the script for FPGA loading. Fully implemented and used Condition DB –Not implemented yet –Planed to put the monitoring flash list to the DB with the TStore –Not clear what to do with the monitoring data from the histograms, which are currently stored in the ROOT files.

DB Service JAVA, Created in the similar technology as the Function Manager. Tomcat (simple JAVA www server) is used to deploy DB Service as a web application. Access to DB Service is provided by SOAP using Apache AXIS library. Implemented custom serializers/deserializers for Axis so that integration with XDAQ is very simple. Access to Oracle through JDBC and Hibernate. Java package (rpct.db.domain) with object representation of all database entities: hardware structure and configuration data. This package is also used by Central Test Manager (JAVA), what allows fast and organized access to the configuration database. CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Central (Monitoring and) Test Manager JAVA software, which communicates with the HA XDAQs via SOAP (Axis), direct access to configuration DB using rpct.db.domain package (JDBC and Hibernate). Architecture: –Objects corresponding to the devices are crated based on the DB; interconnections are also represented, –Second set of objects corresponding to the devices and diagnostic modules are crated based on the info from the HA XDAQs, provide the hardware control via SAOP and HA XDAQs, –Mapping between above object into HW-DB objects, –TestManager for performing test with diagnostic readouts and test pulse generators, –EventBuilder for analysis of data from diagnostic readouts (base of the interconnection test), –Mapping of the config data from the DB to the JAVA objects – used for putting the config data to the DB. Allows to execute complex processes from one application for whole system (e.g. configuration of the diagnostic modules), Main applications: –Interconnection tests, –Algorithm implementation tests, –Finding hardware configuration parameters (e.g. synchronization of the transmissions), the parameters are directly uploaded to the config DB, –dedicated applications for hardware debugging and testing (easy to write, fast development), –Planned to use for complex monitoring tasks, which required to talk to many different HA XDAQ (e.g. LB- TB transmission monitoring). Currently run as command line applications but first successful attempts has been made to deploy them as a web service what allowed to execute the tests by the Trigger Supervisor, Benefits from JAVA easiness of software development and debugging (Eclipse graphic debugger) and advanced libraries for DB, XML, etc. CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Internal Interface Access – JAVA web application CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University Functionalities Application for hardware debugging devices tree: crates, boards, chips, registers Reading and writing of each register Architecture Tomcat as Java WWW server (servlet container) Axis with custom serializers/deserializers to make communication with XDAQ's easy Takes info about hardware from HA XDAQs by SOAP register read and write via XDAQ JavaServer Faces web framework to build rich internet applications (e.g. we use powerfull AJAX TreeTable)

Hardware monitoring Monitoring is performed directly by the HA XDAQs Monitoring of the hardware status– status monitorables: –Device status (e.g. TTCrx ready, QPLL locked) – simple check, if not ok– ERROR is issued, –Transmission error counters: WARNIG or ERROR is issued depending on the error rate, The process in XDAQ reads repetitively the values of selected registers: –separate thread with timer, sampling rate: every few seconds. One board is read per one timer tick, in this way the HA XDAQ is not blocked for long time (important in case of the Link System, where many board in one HA XDAQ). –The results are printed on the XDAQ web page, and are put to the flash list. –Flash list contains OK status for each board or WARNING or ERROR with description (e.g. rate of the error counters). Monitoring of the trigger data (non-event data) – statistic monitorables (counters and histograms analyzing the data stream): Separate thread with timer: –the diagnostic modules are configured, –the diagnostic modules are started for 10 sec. –the data from the diagnostic modules are readout and analyzed, the plots (accumulated histograms, last histograms, history plots) are created with use of ROOT and displayed by the XDAQ web page, –The “raw” data of every iteration are saved in the root file, so that other analysis can be performed offline. CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

TS snapshot I – flashlist display CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University Overall status of each TC Click here to get list of monitorables TS central cell collects, analyses and displays the monitoring data obtained as flash-lists and monitoring charts (png files) from the HA XDAQs

TS snapshot II - Output rate history CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Propagation of configuration data to offline CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University Kind of infoCMSSW objectTransfer mechanism Disabled devicesL1RPCHwConfigPopCon Cabling mapRPCEMapPopCon PAC patternsL1RPCConfigL1 O2O Logical coneL1RPCConeBuilderascii file + on the fly (will be handled by O2O) Cabling map (from chamber strips to PAC inputs) – mature PopCon application, reads several OMDS tables and fills out RPCEMap object. Frequent usage is not required – manual operation when needed is fully sufficient (agreed with DB group), no upgrades foreseen. PAC patterns - Up to recently a dedicated PopCon application has been used – will be superseded, now integrated with L1 O2O infrastructure. Full sets of patterns are predefined well before the start of run and given in form of xml files. Preloading jobs (O2O – based) to transfer xml files to DB and associate payloads to configuration keys - exist and have been tested. Logical cones (PAC inputs definition) – constructed dynamically from ascii files. Creation of a dedicated DB object which contains those parameters is in progress. Plan is to have it tested before the deadline for CMSSW_31X. Should be integrated with L1 O2O. From config DB

Development of missing items, consolidation plans and priorities HA XDAQ: –in principle complete – almost all major functionalities implemented, but some polishing and debugging is needed. –If manpower available we would like to convert the HA XDAQs to TS sub-cells, –Upgrade to XDAQ7. TS: –TS central cell almost ready. In practice TS was not used for configuration due to the need of running the opto synchronization software procedure as a part of config process (delay of ~15 mins). Configuration so far has been done by hand with use of HA XDAQs. With new firmware that performs automatic optical links synchronization, we can start configuring via TS. Short term task: –Test the integration with the HA XDAQs, –Integrate LBOX XDAQs with TS for configuring - make dedicated TS sub-cell for them, –Integrate LBOX XDAQs for monitoring – straightforward at TS level, LBOX XDAQs must send their own flashlists with hardware statuses, –Upgrade to TS 1.6 Medium term tasks : –Make use of sentinel error messages and alarms Other tasks: –Improve FEB configuration (speed up) – create the software and test the firmware of improved I2C controller –Finish implementation of the software and firmware features for automatic LB reloading and reconfiguration, CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Software repository and documentation Organization and management of the software repository: –Currently CVS repository on the server in Warsaw –We want to move the CERN repository, but it needs some effort… Status of documentation: –TWiki page with users manuals (for experts - Warsaw server, for others – CMS TWiki), –Documentation of sources – only some comments in not straightforward sections of codes, CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Manpower Main developer of the online software - Michał Pietrusiński. However presently working with us part-time (15%), cannot spend much time on CMS. Karol Buńkowski: HA XDAQ, JAVA, Mikołaj Ćwiok: LinkBoxAccess XDAQ Michał Szleper: O2O CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University

Questions and answers Which programming languages are you using? : C++ and JAVA Which external frameworks/middleware/packages are you using for the online software? : Axis, Hibernate, Maven, Tomcat, Can you make a build of all the online software in one step? : C++ in one step, a few steps for JAVA, Do you have automatic tests for the Configuration operation? : No Do you have a test to assure the hardware functionality in one step? : Hardware setup and monitoring provides (almost) full tests of hardware functionality, How are you implementing the Interconnection Tests? Is it a one step process? : JAVA Central Test Manager; a few standalone applications for a few types of tests, Are you using the HAL library to access to the hardware? : No Are you accessing to the Linux drivers directly? : Yes Are you using C++ namespaces? : Yes rpct Do you have a hierarchy of exceptions inherited from CellException to describe all the possible malfunctions of the software? : No Do you have cout or cerr in any part of the online subsystem code? : some debug (which should be cleaned) What is the worst case configuration duration? : a few minutes in case of the firmware loading What is the worst case duration of a control panel event? A few seconds (in XDAQ standalone mode) Are you using global variables in the subsystem online software? : No How much memory is leaked in each sequence of: create configuration operation, configure, destroy operation? : Not tested recently, not problems experienced CMS L1 Trigger Online Software Review, 5 February 2009Karol Buńkowski, Warsaw University