1. Embedded PCs for electronics-control in LHCb Flavio Fontanelli, Giuseppe Mini, Mario Sannino, INFN Genoa Beat Jost, Niko Neufeld CERN, PH

2. Niko Neufeld CERN, PH 2 Traditional board-control Board Crate Controller 3, 6 or 9U crate with backplane bus (VME, CompactPCI) and Crate Controller VME Extender e.g. USB VME library Ethernet LAN (NFS) Server

3. Niko Neufeld CERN, PH 3 Problems with Crate-based Board Control Scalability / ThroughputRobustness Board Crate Controller Access to all boards blocked!  Each board shares the bus (bandwidth) and resources in Controller  Need crate-controller + crate for a single board

4. Niko Neufeld CERN, PH 4 Microcontroller for Board-control Accessed via cheap ubiquitous LAN infrastructure (Ethernet) Scales well (you pay what you need) Nice for table setups (Debugging!) (photo: LHCb Optical Pattern Generator card by M. Mücke)

5. Niko Neufeld CERN, PH 5 An example: the AMD ELAN 520 32-bit, 133 MHz, i486 based / 33 MHz PCI, Lots of useful features: hardware watch-dog, high-resolution timers, GPIOs Low-power (1.4 W typical, 2 W max), no active cooling necessary Enormous software base of i386 (GNU/Linux, MS-Windows)

6. Niko Neufeld CERN, PH 6 A Microcontroller is not (yet) a board-controller Add a “few” things (Ethernet, USB, Flash ) to make an Embedded PC from a Microcontroller Use a commercially available device SM520PC from Digital Logic AG Switzerland, http://www.digitallogic.ch http://www.digitallogic.ch Designed around standardized SM480 bus (whole family up to PIII available)

7. Niko Neufeld CERN, PH 7 An embedded PC is still not a board-controller Traditional PC interfaces: PCI, ISA, parallel port, USB  not very suitable for chip-control For controlling, configuring and monitoring FPGAs and ASICS need rather I 2 C, JTAG and a high-speed, “simple”, long distance parallel bus Need some small adapter or “glue-”logic The LHCb glue-card has an I2C / FPGA controller + a fast local bus generated from a PLX 9030 all controlled by an FPGA. (For details see: F. Fontanelli et al. PO2.062)

8. Niko Neufeld CERN, PH 8 Example: the LHCb Common Readout Board TELL1 Ethernet to CCPC SM520PC a.k.a. “Creditcard PC” / CCPC) Glue-Card Power In the final LHCb experiment ~ 400 boards of ~ 15 types In the current phase, ~ 10 setups in labs all over Europe, all using centrally maintained software base at CERN

9. Niko Neufeld CERN, PH 9 Capabilities and Performances 64 MB Ram, 2 MB Flash, 133 MHz CPU 4 x I 2 C, 3 x JTAG, PLX 9030 bus, 8 GPIOs, busswitch to isolate all signal lines of the SM520PC/Gluecard (manually or automatically when reset-line of SM520PC is pulled low), attached via 100 MBit Ethernet Running Linux and standard software we achieve: > 20 MB/s throughput on 9030 bus (32-bit read) from memory and FPGA registers, > 1.5 MBit/s on JTAG Program configuration devices and FPGAs via JTAG: ~ 4 min for an Altera EPC16 (16 Mbit, including erase and verify)

10. Niko Neufeld CERN, PH 10 Software Based on Scientific Linux, a freely available Linux distribution made by Fermilab and CERN, fully compatible with Redhat Enterprise Linux (https://www.scientificlinux.org)https://www.scientificlinux.org Scientific Linux 4.0 Custom device drivers User Code Carrier board adapter library FPGA ProgJTAGI2CGluecardPLX localbus Usercode in C, C++, Java… Hardware access libraries in C Carrierboard adapter transparently handles carrier- board differences Low level C libraries for generic resources (GPIOs, busswitch) Device-driver with direct memory mapping for fast access Special small footprint kernel diskless boot

11. Niko Neufeld CERN, PH 11 Software (Boot) After a power-cycle or a reset the busswitch is open and the CCPC disconnected from the carrier-board Diskless CCPCs boot using Etherboot via BOOTP and TFTP Kernel configures networking via DHCP Kernel mounts the shared root-filesystem read-only and a small number of files read-write using a modified version of RedHat’s diskless package The glue-card FPGA is boot-strapped via the PC parallel port  access to the carrier-board is now possible The carrier-board access is initialized and the busswitch is opened User applications can use the board! (e. g. a DIM-server as described in: R. Jacobsson PO1.045-6)

12. Niko Neufeld CERN, PH 12 Software (Server) NFS Server can be (practically) any Linux/Unix box  base distribution via tar-ball or rpm Need in addition tftp and dhcp server CCPC users are created on server with a home directory shared with the CCPCs Customised Makefiles allow building software for CCPCs on the server, using the compiler-chain and header-files of the CCPC distribution CCPC NFS filesystem tree is automatically updated from central CERN repository (using yum)

13. Niko Neufeld CERN, PH 13 Deployment in LHCb One server PC (“Control- PC”) per ~ 20 CCPCs Sub-detectors do not share Control-PCs (“partitioning” in the wider sense) CCPCs are on a private LAN, only connected to the Control-PC (security, robustness, broadcast domain) Global integration via the ECS: PVSS is running o the Control-PCs CCPC ECS Control PC PVSS Private LAN

14. Niko Neufeld CERN, PH 14 Conclusions PCs and Linux demonstrated to be a very good choice: moderate porting effort, re-use most of standard HEP Linux base Central software repository and automated distribution of packages allows keeping disconnected sites up-to-date Deployment on strictly private LANs for greater security and robustness

15. Backups

16. Niko Neufeld CERN, PH 16 A word about “Real-time” Do you need it for board-control? Really? Standard Linux has no hard real-time support (i.e. guaranteed maximum time to handle an interrupt/syscall). It has however support for soft or maybe better quasi real-time (i.e. processes can be strictly prioritised) In LHCb we have not (yet) found an application for even soft real-time. Polling critical registers if necessary gives a very good response time. There exist of course a host of true real-time OS or Linux variants (RTLinux), which would run fine on the Elan520