Stefan KoestnerTutorial – User GuideSeptember 2007 TELL1 - CONTROL Guidelines for Users & Developers

Stefan KoestnerTutorial – User GuideSeptember 2007 Contents: framework components (fwCcpc, fwHw, fwTell1) configuration of Tell1 boards finite state machines design issues - special requirements user specific - what you still have to do

Stefan KoestnerTutorial – User GuideSeptember 2007 fwCcpc – in a nutshell: interface between PVSS and CCPC using DIM and a generic server basic test panels & PVSS library doxygen documentation and tutorial under: online/ecs/PVSS_CCPC/default.html

Stefan KoestnerTutorial – User GuideSeptember 2007 fwCcpc – in a nutshell: communication via Ccpc DP each Tell1 has its own DP and each bus has its own DP element Note: use fwHw for accessing registers and not this DP! (model your hardware)

Stefan KoestnerTutorial – User GuideSeptember 2007 fwHw – in a nutshell: a tool (panels & library) for modeling hardware as PVSS DPs and creating recipes. generic: SPECS & CCPC

Stefan KoestnerTutorial – User GuideSeptember 2007 fwHw – in a nutshell: each register has its own DP and DIM srvcs/cmnds associated subscription : - register settings transferred to ccpc and stored there in list. - create DIM server cmnds/srvcs. (establishes communication between server & PVSS) doxygen documentation and tutorial under: online/ecs/FwHw/default.html

Stefan KoestnerTutorial – User GuideSeptember 2007 Recipes – in a nutshell: a recipe is a collection of registers, whose values can be stored in the configuration DB - from where they can be applied to hardware at any time. Panels and framework functions to create and operate on recipes are provided. documentation under: online/ecs/FwHw/default.html

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 – in a nutshell: datapoint representations of Tell1 types (including detector specific parts) standard recipes of all recipe types (including detector specific part) To be imported from hardware: myTell1_type panels for device units (stopped counting ) – adopting automatically to Tell1 type control unit with user interface finite state machines

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 – in a nutshell: use fwHw tool to instantiate your Tell1 boards (instantiate datapoints) use DEN (device editor/navigator) to associate device unit with datapoint create your FSM tree screenshot of Device Editor/Navigator

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 control unit: pof – file: stores the firmware of FPGAs cfg – file: server can access EPFL C-code to configure the boards each board has its own ccpc

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 control unit: Commonly modified configurations can be sent to enabled boards Send down commands to enabled boards. (configure uses recipes e.g. PHYSICS\Configure

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: A tab register is provided that allows you to access the common part of each Tell1 board. A tab for specific part is foreseen. The panel configures automatically according to board type. Some checks are performed. Watch the info buttons and tooltips for documentation. And many registers to be monitored … (a few hundred)

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: in fact a lot of panels … We provide tool-tips and info buttons for help! And perhaps a dedicated tutorial on device unit panels.

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: This is where you can put your own panels!

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: We are not providing the monitoring of each sub-detector specific parameters. (but a few…)

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: We allow for spying on the data of the MEP buffer. The structure of the MEP buffer differs – we do not provide parsers for each sub-detector.

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: Some examples for parsing the MEP buffer…

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Tell1 device unit: We divide between 3 sub-recipes. Each sub-recipe has its own panel. You can export/import recipes or simply read them directly from hardware.

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 : Define your specific recipe type. Create the appropriate panels. specific recipes:

Stefan KoestnerTutorial – User GuideSeptember 2007 fwTell1 – conclusion: for commissioning you can just use the common part BUT: start developing the specific part NOW! NOTE: the specific part is related to your firmware. Get in touch with EPFL Lausanne before you start! You might profit from already existing things. doxygen documentation and tutorial under: online/ecs/PVSS_TELL1/default.html

Stefan KoestnerTutorial – User GuideSeptember 2007 Ctrl managers – in a nutshell: fwCcpcScript.ctl: connects to running servers, automatic register subscription (crash/restart) fwTell1FSM.ctl: needed to stop monitoring of registers when switching panels in DUs fwTell1DisableArchives: disables the archive bit of registers gain speed for configuration fwFSMServer

Stefan KoestnerTutorial – User GuideSeptember 2007 Configuration of many boards: trials and errors benchmarking results

Stefan KoestnerTutorial – User GuideSeptember 2007 Configuration of many boards: A first impression of scaling: a snapshot from the talk of June 2007: 50 boards ~ 6 minutes

Stefan KoestnerTutorial – User GuideSeptember 2007 first steps … black: old - no changes applied yellow/yellow: disabling archiving (but 35% CPU load from panels) l pink: disabling archiving & 1 ctrl manager per DU (still 35% CPU load from panels) green: disabling archiving (no CPU load from panels) red: disabling archiving & 1 ctrl manager per DU (no CPU load from panels) blue: reference when applying just specific part of recipes (disabled & 1 ctrl/DU) 50 boards ~ 3.5 minutes disabling archiving significant additional ctrl managers in FSM significant under high load panels consuming much CPU?

Stefan KoestnerTutorial – User GuideSeptember 2007 same as before – two (lower) lines added … black: checks on serverside - comparison of applied values with hardware values read back cyan: common part of recipes (16 registers of 32 bit) 50 boards ~ 2.5 minutes changes on server significant but still too small complete recipe of type ST: 236 registers ~80 kB grouping and packing together memory blocks would help, but not so easy to implement in FPGAs

Stefan KoestnerTutorial – User GuideSeptember 2007 so where does the time go … most of the time consumed by DIM, Event and Data manager in current design each register is sending a DIM command

Stefan KoestnerTutorial – User GuideSeptember 2007 parallelizing helps … … but not yet enough!

Stefan KoestnerTutorial – User GuideSeptember 2007 new approach – without changing the hardware representation … getApplyRecipes() for fast Tell1 configuration: retrieve recipes from configuration DB pack them together in one buffer send them as one buffer/DIM command decode it on server

Stefan KoestnerTutorial – User GuideSeptember 2007 \0& …… C&C& … The buffer - array of unsigned char: # registers register-names size of registers data checks buffer corruption … command character

Stefan KoestnerTutorial – User GuideSeptember 2007 \0& … &C& on the server side: parsing through buffer get number of registers get strings, compare them with list fill array of pointers to list of registers (all hardware info stored) note: subscription is necessary! check for corruption fill array of register sizes check for corruption execute commands with appropriate data final check

Stefan KoestnerTutorial – User GuideSeptember 2007 the results for getApplyRecipes(): red: one DIM command per register (old design) – but improved framework blue: sending one single buffer containing all registers (new design) grey: applying.cfg file 1.5 sec/board 1 sec/board no slope 50 boards < 1 minute most time gets lost when retrieving the recipes from DB storing the whole buffer? I am optimistic that we could come closer to the.cfg file!

Stefan KoestnerTutorial – User GuideSeptember 2007 how to create recipes without typing several thousand parameters: get image from current board settings reads registers from hardware and saves them as a recipe (you can upload the.cfg file before) import recipe from other hardware takes the recipe previously defined for another hardware. (you can use myTell1_type as default recipe)

Stefan KoestnerTutorial – User GuideSeptember 2007 Conlcusion: creating recipes is not a big deal (tools are available) downloading recipes to hardware is already fast & sufficient we still support the interface to the.cfg files (original C-code) Note: some parameters are provided by Gaudi jobs. interface under development

Stefan KoestnerTutorial – User GuideSeptember 2007 Finite State Machines: towards a fully automatized expert system …

Stefan KoestnerTutorial – User GuideSeptember 2007 Tell1 boards belong to the DAQ domain. Predefined transitions and states: FSM – in a nutshell: Recipes are applied under Configure – transition from NOT_READY to READY. Checks are done to verify the status of the board. If they fail status moves to ERROR. If connection to ccpc is lost, it moves to UNKNOWN.

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – in a nutshell: functions to be called by FSM are provided in fwTell1.ctl if needed you can provide a specific function to be called by the FSM (e.g. for calibration) fwTell1_Muon_Cmnd(string domain, string device, string command, string parameter) Please read Integration Guidelines: and FSM Guidelines:

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – from a higher point of view: OTA DCS OTA HV OTA DAI OTA DAQ INFDCSHVDAIDAQL0TFCHLTLHC OTA LV OTA TEMP OTA FEE OTA TELL1 OTA CRATES OTA COOLING OTAOTC OT Templates available: Please use them as they are Contact Clara for integration issues! Storage ECS

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – Run Control: Run Control is like a Matrix, can exclude/include: –by SubDetector part –Or by Domain Standalone run, need to allocate: TFC HLT Storage Monitoring Allocate uses the Partition Settings TFC, HLT, Storage, etc. will be ready soon.

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – Run Sequences: 1 - Configure/RUN_TYPE=CALIB_TIMMING 1 - Configure/RUN_TYPE=CALIB_TIMMING/N_STEPS= READY 3 – Start 4 – RUNNING 5 - Start/N_TRIGGERS= RUNNING 7 - TFC Paused (after N_TRIGGERS) 8 - Step/RUN_TYPE =CALIB_TIMMING/STEP_NR=1 9 - RUNNING 10 - Continue/N_TRIGGERS=10000 OTAOTA_DAQ TFC N Steps OTA DAQ OTA TFC OTA FEE OTA TELL1 OTA OTA Dev1 OTA DevN OTA Dev1 OTA Tell1N

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – Run Sequences: The Activity can be saved, contains: –Nr Triggers –Nr Steps –Nr Sub-Farms, etc. It corresponds to the RUN_TYPE that gets propagated down and used to form a recipe name by all devices.

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – Run Sequences: Configure Command –As usual propagates down so that each device can apply a recipe (or more) based on: –RUN_TYPE (and N_STEPS) Step Command –Again propagates down, should be accepted in state RUNNING (and return RUNNING) Each device should change only the necessary values based on: –RUN_TYPE and STEP_NR

Stefan KoestnerTutorial – User GuideSeptember 2007 FSM – Conventions: Recipe Names: –Guideline: /, ex.: »PHYSICS/Configure, PHYSICS/Start »CALIBRATION/Configure, etc. Necessary because a Configurator Object will preload recipes from DB to cache

Stefan KoestnerTutorial – User GuideSeptember 2007 started with a generic/altruistic server more and more intelligence moved to server side few comments on design:

Stefan KoestnerTutorial – User GuideSeptember 2007 clear separation between hardware and control issues – a smarter server: server keeps track of all the information about the register (e.g. type, address,…) and stores it in a list server treats each register in the appropriate manner (e.g. Fifo vs. standard blockwrite or different I2C derivates) Once the register is defined the control system (PVSS) knows nothing about the hidden details and just writes to or reads from a register less and more readable code & better performance (interpreted language) server-side PVSS-side Intelligence related to hardwareIntelligence related to ECS

Stefan KoestnerTutorial – User GuideSeptember 2007 how to keep the design open: a commonOperation part of the Ccpc DP allows to send a string to the server. Anything can be done. Identifier and parameters are separated by comas. a user specific register type which offers many parameters to be filled in inside the fwHw tool. Any type of register can be modeled. the new recipe block executed via configurationOperation on the Ccpc DP is in fact a set of instructions, which in addition can be stored in the Configuration DB. \0& … &C& commands

Stefan KoestnerTutorial – User GuideSeptember 2007 What is left for the sub-detectors? in fact not much … Specific part has to be inserted into datapoint type The panels related to the specific part: - MEP buffer spy - Monitoring of specific registers/values - Recipe related things specific functions to be called by FSM (e.g. for calibration) preferably packed as a library (obey to naming conventions) Scripts inside device units might have to be extended … but still a bit!