EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 pciGeneral PCI Device Support of EPICS which is used in TPS Control System Presented by: Jenny Chen TPS Control Group June 14, 2011

2 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Outline Goals of the Device Support – pciGeneral Rules of the Kernel Driver for PCI Cards Details of the Kernel Driver Programs Details of the Device Support Programs Performance Tests Access TLS control system via EPICS

3 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Goals of the pciGeneral Programs Manipulate as many PCI cards as possible Manipulate the relevant resources Handle interrupts Convert byte order (big to little endian) if needed Keep maintenance programs to a minimum

4 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Manipulate as Many PCI Cards as Possible Generalized kernel driver + pciGeneral + asynDriver(EPICS), all PCI cards we have were tested successfully The tested PCI cards: cPCI-7452, 24DSI32, cPCI-EVG-300, cPCI-EVR-300, TCP201 + DAC8402, TCP201 + DAC8415, TCP201 + ADC8417 Properties of PCI cards: – address space: IO-PORT or MEMORY –DMA –byte order: little-endian or big-endian –daughter boards One kernel driver may handle several cards of the same model Special kernel driver (without accessing hardware): creates shared memory – may exchange data between processes

5 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Manipulate the Relevant Resources Extract 3 regions of the address spaces of a PCI card Fully access the address spaces plx -- the chip sets of PCI interface, ex: plx9000 series, the mostly accessed registers are interrupt control and status card – includes the main control/status registers of the PCI cards which are mostly for configuration data – separated from the main registers (card), ex: DMA, transient data or processed data (may be done by the kernel driver or a running process)

6 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Handle Interrupts APs call read() to wait for the interrupts instead of using ioctl() or signal(), which is easy to program APs may directly access interrupt registers (plx), ex: reset the registers when interrupts are not handled properly Kernel drivers serve enable or disable interrupts that are requested by APs (ioctl)

7 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Keep Maintenance Programs to a Minimum Keep the software simple and small Minimize the maintenance of IOC software One kernel driver program per card + 2 device programs –Ex: kernel driver: pci_adl7452.c for cPCI-7452 –EPICS device support programs: pciGeneralDrv.c, pciGeneralLinux.c

8 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Rules of Kernel Driver Implementation Create char device file in /dev, file name ex: pciCardName- x, x: pci-slot, ex: pciadl7452-1, pciadl ioctl() -- get the card information, enable/disable the interrupt, setup/start/abort DMA, special service for the card mmap() -- map the PCI address spaces into AP's read() -- block AP to wait for an interrupt

9 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Details of a Kernel Driver Program Interface to a PCI card Create a char device interface file Serve char device access requests -- open/close/ioctl/mmap/read

10 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Interface to PCI card Create char device interface file #define MY_VENDOR_ID0x144A #define MY_DEVICE_ID0x7452 #define SUB_VENDOR_IDPCI_ANY_ID #define SUB_DEVICE_IDPCI_ANY_ID #define MYCARD_NAME "pci7452" static struct pci_device_id MY_DRIVER_PCI_TABLE[2] __devinitdata = { {MY_VENDOR_ID, MY_DEVICE_ID, SUB_VENDOR_ID, SUB_DEVICE_ID, 0, 0, 0}, {0, 0, 0, 0, 0, 0, 0} }; Declaration static struct file_operations mypci_fops = {.owner = THIS_MODULE,.open = mypci_open,.read = mypci_read,.ioctl = mypci_ioctl,.mmap = mypci_mmap,.release = mypci_release, };

11 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Interface to PCI card Create char device interface file Probe callback - alloc_chrdev_regionallocate char device resource, assign the char device name (should be unique) - cdev_jnit - cdev_add - class_createcreate a class in directory /sys/class, ex: /sys/class/pci device_createcreate a device in /sys/class/device, ex: /sys/class/pci7452/pci to insert/remove kernel module dynamic – according to a rule listed in /etc/udev/rules.d, the udev daemon sends uevent to OS to create interface file: /dev/pciadl (kernel version: linux or higher) Create device interface file: Request to access PCI address spaces - pci_request_regionsrequest pci resources and associate with the device file: pci pci_resource_startget pci address spaces information - pci_resource_end - pci_resource_len - pci_set_masterto be the dma master Initialize a waiting queue for interrupt request and variables - init_waitqueue_head, atomic_set - kzalloc

12 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Interface to PCI card Create char device interface file Remove callback - device_destroydestroy created device structure, ex: delete /sys/class/pci7452/pci class_destroydestroy created class structure, ex: delete /sys/class/pci cdev_deldelete cdev structure the char device interface file will be removed then Delete device interface file: Release PCI resources - pci_release_regionsrelease pci resources - pci_disable_devicerelease pci access Free allocated memory - kfree

13 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Provide char device interface: open/close/ioctl/mmap/read - openallow AP to access the char-device, call request_irq - closerelease to access, call free_irq - ioctlserve command: IOCTL_CARD_INFO, IOCTL_PLX_INFO, IOCTL_DATA_INFO, IOCTL_INT_ENABLE, IOCTL_INT_DISABLE, IOCTL_DMA_SETUP, IOCTL_DMA_START, IOCTL_DMA_ABORT, IOCTL_INITIAL, IOCTL_COMMAND transfer PCI address information to AP, process interrupt requests, process DMA requests, special service request to the kernel driver - mmapmap PCI address (memory type only) to user process address space: card, plx and data - readserve interrupt request: block AP, wake up AP and copy interrupt count to AP Serve the char device access requests

14 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Details of the Device Support Programs for EPICS They are based on asynDriver Programs: –Header: pciGeneral.h –pciGeneralDrv.c – the interface to EPICS, create records, register interrupt, parse arguments,... –pciGeneralLinux.c -- the interface to PCI cards, open/close device file, mmap pci address spaces, access the io-port/memory and create interrupt threads (call read() to wait)

15 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 EPICS Record types of pciGeneral bi/bo/mbbi/mbbo ai/ao/longin/longout Waveform with DTYP: asynInt32ArrayIn/Out, asynFloat32ArrayIn/asynFloat32ArrayOut asynFloat64ArrayIn/asynFloat64ArrayOut

16 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 DTYP: asynUInt32Digital for bi/bo/mbbi/mbbo. bi/mbbi command location command: input/input16location: plx/card/data. bo/mbbo command location command: output/output16/write/write16location: plx/card/data (output/output16 – get readback first). bo (special) command location actionState command: intEnable/intDisable/initial/dmaSetup/dmaStart/dmaAbort location: card actionState: 0/1/-1 (0/1:process when state is 0/1, -1:process every time) INP/OUT Field of EPICS Record record(bi,"diBank0Ch0") { field(SCAN, “I/O Intr”) field(DTYP,"asynUInt32Digital") input card") field(ZNAM,"low") field(ONAM,"high") } record(bo,"doBank0Ch1") { field(DTYP,"asynUInt32Digital") output card") field(ZNAM,"low") field(ONAM,"high") } appendix

17 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Db example record(mbbi,"erClockControlSts") { field(DTYP,"asynUInt32Digital") input card") field(DESC, "Clock Control Status") } record(mbbo,"setIntMaskBank0") { field(DTYP,"asynUInt32Digital") write card") } record(bo,"enableInt7452") { field(DTYP,"asynUInt32Digital") intEnable card 1") field(ZNAM,"null") field(ONAM,"enable") field(DOL, "clearIntBank0") field(OMSL, "1") field(HIGH, “0.1”) field(FLNK, "enableIntBank0") } st.cmd: pciGeneralDrvInit("tcp201-0","/dev/pcitcp201-0-IP") ) pciGeneralDrvInit("adc8417-0","/dev/pcitcp201-0-IPB") ) pciGeneralDrvInit("pci7452-1","/dev/pci7452-1")) pciGeneralDrvInit("er230-2","/dev/er230-2")) appendix

18 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 DTYP: asynInt32 for ai/ao/longin/longout. ai/longin command location extractMask command: input/input16, location: plx/card/data, extractMask: extract several bits of 32/16 bits. longin (special) intCount card. command location extractMask command: output/output16/write/write16/ioctl, location: plx/card/data extractMask: extract/update several bits of 32/16 bits INP/OUT Field of EPICS Record record(longin,"diBank0") { field(SCAN,"I/O Intr") field(DTYP,"asynInt32") input card 0xffffffff") } record(longout,"erPrescaler0Set") { field(DTYP,"asynInt32") output card 0x0000ffff") } record(longin,"intCountEg230") { field(SCAN,"I/O Intr") field(DTYP,"asynInt32") intCount card") } code Ex: request kernel driver service: Ioctl card code parameter Code may be the pv value record(longout,"$(IOC)-adc8417-$(S):ioctlCmd") { field(DTYP,"asynInt32") ioctl card 0 $(HNUM)") field(DOL, "$(IOC)-adc8417-$(S):dataProc") field(OMSL, "1") field(FLNK, "$(IOC)-adc8417-$(S):ch0WfProc") } appendix

19 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 DTYP: asynInt32ArrayIn/asynInt32ArrayOut for waveform. waveform input command location extractMask increaseAddrOffset command: inArray/inArray16, location: card/data. waveform output command location extractMask increaseAddrOffset command: outArray/outArray16, location: card/data IncreaseAddrOffset depend on the data layout: typeAch0ch0ch0...one data block for one channel (addrOffset:2/4) ch1ch1ch1... typeBch0ch1ch2...one data block for N channels (addrOffset:2N/4N) INP/OUT Field Format of EPICS Record record(waveform, "dac8402ch0Wf") { field(DTYP,"asynInt32ArrayIn") inArray16 data 0xffff 2") field(NELM, "2048") field(FTVL, "LONG") field(PINI, “YES”) } record(waveform, "dac8402ch0WfOut") { field(DTYP, "asynInt32ArrayOut") outArray16 data 0xffff 2") field(NELM, "2048") field(FTVL, "LONG") field(FLNK, "dac8402ch0Wf") } appendix

20 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 DTYP: asynFloat32ArrayIn/asynFloat32ArrayOut for waveform command location extractMask increaseAddrOffset factor offset input waveform command: inArray/inArray16, location: card/data output waveform command: outArray/outArray16, location: card/data IncreaseAddrOffset: depend on the data layout INP/OUT Field Format of EPICS Record record(waveform, "dac8402ch0WfE") { field(DTYP,"asynFloat32ArrayIn") inArray16F data 0xffff e-4 -10") field(NELM, "2048") field(FTVL, "FLOAT") } Refer to: appendix

21 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Performance Tests int_value = *((int *)mapped_ptr ; vs. ioctl(fd, IOCTL_READTEST, &int_value) ; - Memory access reduces about 25% CPU time compared to ioctl - Interrupt service test – EPICS client: get the 5ms interrupt counter by LabCA (matlab) ADLINK cPCI-6910: CPU:dual-core Intel Xeon 2GHz, cPCI:64bit/66MHz Jitters:. OS scheduler. Dmc controller + network

22 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Interrupt Service Test - Interrupt test: the response time of a 5ms interrupt triggered by a network process (measured by an AP) Shared memory kernel driver char dev file: /dev/dmc Data receiver (from DMC4000) Ethernet as field bus DMC4000 controller send data UDP at every 5ms Prepare for insertion devices control: the field compensation by setting correction coils (setting follows gap/phase) needs to be done as fast as possible. Wake up (read) Ethernet kernel driver (socket interface UDP) open/mmap ioctl Control Network EPICS ioc EPICS Client LabCA open/mmap

23 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Interrupt Test with cPCI-7452 DIO Card - Hardware interrupt test: connect a TTL signal to a DI channel, the state change would trigger the PCI interrupt, the figures show the response time of 100Hz and 200Hz signal; the reason of some missing changes was unknown but it was probably not caused by the OS scheduler (maybe the interrupts were not delivered) cPCI-7452 kernel driver char dev file: /dev/pci Test Program open/mmap/read Test condition: 1. hi/low signal connect to ch0 2. testing loop: test process priority adjusted cPCI-7452 DI Card Wake up Interrupt 200Hz/100Hz ADLINK cPCI-6910: CPU:dual-core Intel Xeon 2GHz, cPCI:64bit/66MHz

24 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 EPICS ioc and LabCA Monitor Test - update test: the ioc updates data at every 5ms, the interrupts trigger the ioc to update the dmc4000 and cPCI-7452 DIO channels Shared memory kernel driver char dev file: /dev/dmc Receive data sent by DMC4000 Private Ethernet DMC4000 controller send data UDP at every 5ms Wake up Ethernet kernel driver open mmap Ethernet EPICS ioc EPICS Client LabCA cPCI-7452 DI Card Interrupt Figure 1: ioc CPU load: 8%(top/Irix), db record: about 40 Figure 2: ioc CPU load: 20%(top/Irix), db record: about 170 Note: in figure 2, 128 DI records scaned but not in figure 1 kernel driver char dev file: /dev/pci Wake up 100Hz ADLINK cPCI-6910: CPU:dual-core Intel Xeon 2GHz, cPCI:64bit/66MHz

25 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Access Existing TLS Control System via EPICS  Modify pciGeneral to tlsControl, because the data byte order of TLS control system is not standard  Start up a softIOC(+tlsControl) on a TLS console as a gateway  A simple char device driver (kernel) allocates memory to store DDB which is updated by TLS ILCs and serves file operations: open/read/ioctl/mmap  TLS DDB-receiver flushes the DDBs into the memory which is also mapped by the softIOC, then two control systems access the DDB simultaneously  With the kernel driver running, the softIOC may be waked up by the interrupt which is triggered by the DDB receiver in order to update data synchronously  To control TLS devices from EPICS, streamDevice is used, a server waits for the string commands (name+value) then converts it (access SDB of TLS) to the format that ILC accepts

26 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 TLS Network TLS ILCs TLS control software on console EPICS IOC Shared memory kernel driver char dev file: /dev/ilc-x Setting server for EPICS StreamDevice TCP socket interface TLS setting functions TPS Network File interface mmap() ioctl() File interface mmap() read() TLS SDB/DDB The Conjunction for TPS and TLS Control System 10Hz DDB On-demand Setting Wake up DDB Upload Setting server TLS ILC control system Run EPICS ioc on an ILC of TLS control system TLS ILC kernel driver char dev file TLS Network EPICS IOC TCP socket Access any device of TLS control system via EPICS TLS Console Setting server (ca-lib) for ILC IO hardware

27 EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Summary Efficiency Flexibility Maintenance

EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011 Acknowledgement Marty Kraimer Thank you for your attention!