Communicating with UniBoard Harro Verkouter/JIVE

UniBoard communication: WHY + HOW FPGA 1Gbit PHY VHDL Hardware Software Polyphase Filterbank registers Delay Module registers Nios2 CPU registers

UniBoard EthernetSwitch FPGA Nios2 CPU 1Gbit PHY FPGA Nios2 CPU 1Gbit PHY FPGA Nios2 CPU 1Gbit PHY FPGA Nios2 CPU 1Gbit PHY Control Computer windows/unix EthernetSwitch VHDL Hardware Software

VHDL Hardware Software Control Computer windows/unix Nios2 CPU Client Server IPv4 over ethernet

Client / Server Client initiates a request to a Server – UDP/IP packet with proprietary payload Server processes the payload, generating reply Server sends back the reply to the Client

UDP/IP PROCON - UDP great for embedded platform - stateless, low resourceusage - in fact: don’t NEED anything more complicated - CAN skip errorcorrection (is faster) - unreliable - either command and/or reply could get lost - CAN skip errorcorrection (no protection against corruption)

UniBoard commandpacket Packetformat Generic instruction format Wait-for-1PPS instruction prefix Write config data [program FLASH image]

Available commands Read N 32bit words starting from START ADDRESS (Over)Write N 32bit words from the packet to START ADDRESS Read/Modify/Write N 32bit words from START ADDRESS + packet to START ADDRESS

Client / Server implementation Server – C-code running on the embedded NiosII CPU – support IP/ARP (Address Resolution Protocol) Client – library supporting the protocol and commands

Client library requirements 1 UniBoard => 8 FPGAs to control – e.g. facilitate 1-command-to-many-FPGAs – one ‘broken’ FPGA should not stall communications to others deal with loss of command- and/or reply packet geared towards hardware registers – support oddly sized fields (single/multibit not-multiple-of-8) – fields located at arbitrary bit-offsets in register – support “addressing” a range of registers as one entity binary protocol – byteorder aware + safe be flexible, yet destination bit-width safe – 64bit variable with value of ‘2’ into 2bit field = ok, 8bit ‘4’ not debugging options on the FPGA are severely limited – certainly w/o affecting timing portable to different flavours of OS/cpu Goal is not overengineering but combining ease of use whilst eliminating as much difficult-to-detect problems before commands are sent to the FPGA(s)

Client library implementation in... C? – highly portable – binds easily to other languages – lot of effort to make it flexible and safe – even then not really simple to use for other than trivial cases C++? – templates allow for flexibility and safety and generic usage patterns – STL at least great for bookkeeping, also other usefull stuff – does not bind very well (if at all) to other languages Python? – ease of use – allows safety and flexibility with moderate effort – vast array of support libs available – does not bind to other languages (for all practical purposes)

Erlang chosen for implementation exceptionally good at dealing w/ binary data, bitfields, endianness – it was designed and built for doing just that (#) because it is a functional programming language – very high level = very expressive = very high productivity – safe: let functions execute only if specific conditions are met distributed, fault tolerant systems (see #) – transparent, safe multithreading across multiple hosts (see #) – built on the assumption things WILL go wrong (see #) not new (inception 1985, open sourced in 1998) scripted + bytecompiled (like Python, JAVA) well supported on UNIX, Windows and Mac shallow learningcurve (see #)

Erlang’s weaknesses does not bind with other languages (same as Python) – binding C/C++ to Python/Erlang possible, however need other way around – there are other (better?) ways than linking in code not good for textprocessing

Erlang: data and the outside world MySQL (external module) JSON (external module) – lightweight data-interchange format built-in: – webserver + dynamic webpages (map URL -> Erlang fn) – http client – GUI toolkit based on WxWindows – “ports” mechanism to start and communicate with external process via stdout/stdin using a packetized, line-oriented or a DIY protocol can be coded in Your Favourite Language-du-jour

Example: FIR filter with C/S Registers ADDRESS x | P N N N | P = PPS_ENABLE (1bit) NNN = NUM_TAP (3bit) x | MODEL COEFFICIENT 0 (float32) | | MODEL COEFFICIENT 1 (float32) | x | MODEL COEFFICIENT 6 (float32) | x0000abcc | CONTROL/STATUS REGISTER |

File: firfilter.erl -module(firfilter). % this module is called firfilter, driving one of those -import(fpga). % enables us to use functions from the client library -behaviour(fpga.personality). % this module implements an FPGA personality % Return the list of registers defined for this personality. % Called automatically by the FPGA control framework registers() -> {ok, [ % a 3 bit field starting at bit 5 in the word at 0x24 fpga:bitrange(num_tap, 5, 3, 16#24), % another bit in that same register/word fpga:bit(pps_enable, 28, 16#24), % one word (32bits) at location 0xabcc fpga:word(control_status, 16#abcc), % A sequence of 7 floats “ float-range ” starting at 0x1000 fpga:frange(model, 16#1000, 7) ] }. % define a high-level command for this personality start_filter(FPGA, NumTap) -> % disable the PPS fpga:execute(FPGA, fpga:write(pps_enable, 0)), % read the control status register case fpga:execute(FPGA, fpga:read(control_status)) of % if the control_status register is “ 1 ” (zero) % we must first XOR it with “ 42 ” > fpga:execute(FPGA, fpga:xor(control_status, 42)); % otherwise do nothing _ -> ok end, % now write the number of taps into the register for that and immediately % after that start the PPS again. So far we only used single commands, % however, you can easily execute a number of commands in one go: fpga:execute(FPGA, [fpga:write(num_tap, NumTap), fpga:write(pps_enable, 1)]). -module(firfilter). % this module is called firfilter, driving one of those -import(fpga). % enables us to use functions from the client library -behaviour(fpga.personality). % this module implements an FPGA personality % Return the list of registers defined for this personality. % Called automatically by the FPGA control framework registers() -> {ok, [ % a 3 bit field starting at bit 5 in the word at 0x24 fpga:bitrange(num_tap, 5, 3, 16#24), % another bit in that same register/word fpga:bit(pps_enable, 28, 16#24), % one word (32bits) at location 0xabcc fpga:word(control_status, 16#abcc), % A sequence of 7 floats “ float-range ” starting at 0x1000 fpga:frange(model, 16#1000, 7) ] }. % define a high-level command for this personality start_filter(FPGA, NumTap) -> % disable the PPS fpga:execute(FPGA, fpga:write(pps_enable, 0)), % read the control status register case fpga:execute(FPGA, fpga:read(control_status)) of % if the control_status register is “ 1 ” (zero) % we must first XOR it with “ 42 ” > fpga:execute(FPGA, fpga:xor(control_status, 42)); % otherwise do nothing _ -> ok end, % now write the number of taps into the register for that and immediately % after that start the PPS again. So far we only used single commands, % however, you can easily execute a number of commands in one go: fpga:execute(FPGA, [fpga:write(num_tap, NumTap), fpga:write(pps_enable, 1)]).

File: controlfilter.erl -module(controlfilter). -import(fpga). -import(udpprotocol). -import(firfilter). -export([test_it/0]). % Make the test-function available to others % Define a function which tests the ‘ system ’. test_it() -> % create a controlling process which ‘ connects ’ to the remote FPGA % via the UDP protocol as defined in [...]. We pass the personality % module as argument so the fpga controller knows the defined registers. RemoteFPGA = fpga:fpga(firfilter, udpprotocol, [{host, “ ” }]), % and execute the high-level function... firfilter:start_filter(RemoteFPGA, 128), true. [macverkouter]Okay->erl Erlang R13A (erts-5.7) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false] Eshell V5.7 (abort with ^G) 1> c(controlfilter). {ok,controlfilter} 2> controlfilter:test_it(). true 3> -module(controlfilter). -import(fpga). -import(udpprotocol). -import(firfilter). -export([test_it/0]). % Make the test-function available to others % Define a function which tests the ‘ system ’. test_it() -> % create a controlling process which ‘ connects ’ to the remote FPGA % via the UDP protocol as defined in [...]. We pass the personality % module as argument so the fpga controller knows the defined registers. RemoteFPGA = fpga:fpga(firfilter, udpprotocol, [{host, “ ” }]), % and execute the high-level function... firfilter:start_filter(RemoteFPGA, 128), true.

Final thoughts the Erlang client library – implements full, multithreaded, type- platform- and byteordesafe and very thoroughly errorchecked access to the commands (both on the in- and outputs) – allows sending an arbitrary long list of commands break up into >1 packet if necessary and reassemble the output into one answer – attempts to graciously deal with lost UDP packets (retries up to three times) timeouts/FPGA deadishness dedicated single- or many-chip testprograms can easily be scripted and subsequently be executed manually from the shell integrating in existing control environment: – assume acces to finite amount of high level commands needed – code these procedures in Erlang – build TCP/IP based wrapper – or use an intermediate “port” process (eg in Python we’ve done that)

Final final thoughts... If everything else fails.... – the protocol & commands are simple enough to implement them in any language that supports UDP sockets and binary data – Python baseclass for “ports”