1. V. Filimonov, T. Hemperek, F. Hügging, H. Krüger, N. Wermes
USBpix Upgrade
ATLAB Annual Assembly 18th February 2013
V. Filimonov, T. Hemperek, F. Hügging, H. Krüger, N. Wermes

2. Outline Research project overview Migrating from FX2 to FX3
18 February '13
Outline
Research project overview
Migrating from FX2 to FX3
Firmware development
During this talk I will give you an overview of the USBpix Upgrade project, compare USB 2.0 microcontroller from the previous MultiIO Board with USB 3.0 microcontroller of new FX3 device and tell you about steps that were made in the firmware development for the FX3 device and FPGA.
USBpix Upgrade

3. Research project overview
18 February '13
Research project overview
Pixel detector electronics for ATLAS Upgrade
Readout ASICs (FE-I4)
Test environment (USBpix)
HL-LHC outer layers stave concept:
- 1.2 m long stave
chip-modules per stave
16 on the top, 16 on the bottom
My current project is connected with pixel detector electronics for ATLAS Upgrade, specifically with test environment for readout ASICs. In my case these ASICs are FE-I4 readout chips. They will be used in the 4-chip modules placed on the outer layer pixel staves.
USBpix Upgrade

4. Research project overview
18 February '13
Research project overview
USBpix Test System (Test system hardware)
Currently I’m working on the upgrade of the USBpix test system. USBpix was developed as a small and light weighting test system for ATLAS FE-I3 pixel readout chips. Now it provides the functionality which is needed for a full characterization of FE-I4 pixel readout chips. Here you can see one variant of USBpix system. The USBpix hardware is built up in a modular way. It consists of three different PCB boards:
S3 Multi IO Board
Adapter Card
Single Chip Card
(CLICK) The S3 Multi IO Board is the central control unit of the test system containing all of the programmable hardware parts. It includes a free programmable Xilinx Spartan3 FPGA, SRAM Memory, USB2.0 Interface and an 8051 microcontroller with I2C and SPI functionality.
The Adapter Card provides the necessary voltage for the Single Chip Card and holds level shifters, power supplies and diagnostic circuits.
The Single Chip Card holds the FE-I4 chip.
USBpix system for FE-I4 single chip read-out
USBpix Upgrade

5. Research project overview
18 February '13
Research project overview
Upgrade Plans for USBpix
Resources on the MultiIO board are coming to their limits
2 MB SRAM
FPGA resources (Spartan 3, XC3S1000)
Data bandwidth (USB 2.0, ~15 Mbyte/s)
→ Simultaneous r/o of four chips and support of new chip generations with higher data rates becomes challenging
Requirements:
– Downward compatible to USBpix adapter cards and SW
KEL connector
USB interface for low-level software compatibility
– larger FPGA and memory
– USB 2.0 → USB 3.0 – Possible data bandwidth (~ 150 Mbyte/s)
Unfortunately, resources on the MultiIO board are coming to their limits. That’s why it’s hard to r/o few chips simultaneously and to support new chip generations with higher data rates. That’s why it’s necessary to upgrade the MultiIO board.
Upgraded MultiIO board should be downward compatible to USBpix adapter cards and SW, should have larger FPGA (half the power consumption of previous Spartan families, and faster) and memory and should have USB 3.0 interface.
USBpix Upgrade

6. Research project overview
18 February '13
Research project overview
Upgrade Plans for USBpix
Commercial board (Linera FMU3-S6 Series FPGA Module)
Custom made carrier PCB
65mm x 50mm board size
USB3 Connectivity, including software drivers and APIs
Xilinx Spartan6 LX45 application FPGA (Available also with LX150)
1Gbit DDR3 SDRAM (Optional 2 or 4 Gbit)
MicroSD card slot, accessible from FPGA
Very easy to integrate: Single 5V supply
Programmable via USB3.0 port, JTAG or the on-board flash
2 100-pin high speed board-to-board connectors to mount on a host-board
150 user I/O signals. 72 differential pairs
8 User LEDs on-board
Pin-compatible with FM-S651 series
On-board, programmable clock generator with 4 output clocks, with available SSC
FMU3-S6 series FPGA module
One of the possible upgrade options is to use a commercial board with a custom made carrier PCB. In our case this commercial board is a board from LINERA company that holds both FX3 and FPGA. Here you can see the main characteristics of the board such as USB 3.0 connectivity, Spartan6 FPGA and 1Gbit SDRAM.
USBpix Upgrade

7. Research project overview
18 February '13
Research project overview
FX3 Firmware
FX3 DVK Board
USB 3.0 interface
Firmware development
FX3 device should be compatible with the software for the 8051 based USB 2.0 microcontroller
FX3 device
One of my tasks is connected with the Cypress FX3 development board that includes USB 3.0 interface as well as a number of other interfaces.
I have to develop a firmware for the FX3 device so that it would be compatible with the software for the 8051 based USB 2.0 microcontroller, that is currently used in the MultiIO board.
After this FX3 device can be used in the upgraded MultiIO board to implement USB 3.0 functionality.
USBpix Upgrade

8. Migrating from FX2 to FX3 Architectural Differences
18 February '13
Migrating from FX2 to FX3
Architectural Differences
Here you can see architectural differences between FX2 and FX3 devices. FX3 has many advantages, but the most important in our application are the following (CLICK): 100 MHz, 32 bit GPIF II interface in comparison with 48 MHz 16 bit GPIF interface of FX2 device and USB 3.0 functionality in comparison with USB 2.0 functionality of FX2 device.
USBpix Upgrade

9. Firmware development FX3 DVK Board Block Diagram
18 February '13
Firmware development
Here you can see the FX3 DVK Board Block Diagram. This board is used for the firmware development for the FX3 device. Firmware development for FX3 includes implementing (CLICK) USB, I2C, SPI and GPIF functionality.
FX3 DVK Board Block Diagram
USBpix Upgrade

10. Firmware development USB 3.0 Interface
18 February '13
Firmware development
USB 3.0 Interface
4 endpoints for different transfer types were configured
USB enumeration descriptors were configured
DMA Manual channels were created between USB endpoints and CPU sockets
FX3 is correctly recognized as SiUSB device on Windows 32 and 64-bit
During the implementation of USB interface 4 endpoints for different transfer types were configured as well as USB enumeration descriptors. DMA Manual channels were created between USB endpoints and CPU sockets in order to send data to the host and receive data from it. FX3 device now is correctly recognized as SiUSB device on Windows 32 and 64-bit systems.
(Ep 2 out, EP 6 in – Block transfers
EP1 in/out – control transfers, slow peripherals)
USB Device Manager interface
USBpix Upgrade

11. Firmware development Low performance peripherals (LPP) I2C EEPROM
18 February '13
Firmware development
Low performance peripherals (LPP)
I2C EEPROM
Configuration of the interface parameters (100 KHz bit rate, register transfer mode, no bus timeout, no DMA timeout)
I2C transfer (preamble: the slave address, the direction of the transfer, the address inside the slave; wait for ACK)
SPI SPI Flash
Configuration of the interface parameters (8 MHz SPI clock, 8 bits word length, MSB first data shift mode, idles high clock polarity (CPOL = 1), clock phase (slave samples at active-idle edge (CPHA = 1)), active low polarity of SSN line, slave select using FW)
SPI transfer (location: command, page address; set SSN line, erase sector, wait for status)
After implementing I2C and SPI functionality it’s possible to write and read data from EEPROM and SPI Flash using the software for the FX2 device.
For I2C interface it was necessary to configure the interface parameters such as bit rate, transfer mode and timeouts as well as to realize the data transfer that is to configure the preamble and other interface specific features.
For SPI interface it was necessary to configure interface parameters such as SPI clock, word length, data shift mode, clock polarity, clock phase, SSN line’s parameters as well as to realize the data transfer that is to configure the location and other interface specific features.
USBpix Upgrade

12. 18 February '13
Firmware development
FX3 – FPGA connectivity GPIF II (General Programmable Interface)
Functions as master or slave
Provides 256 firmware programmable states
Supports 8 bit, 16 bit, and 32 bit parallel data bus
Enables interface frequencies up to 100 MHz
Supports 14 configurable control pins when 32 bit data bus is used. All control pins can be either input/output or bidirectional
Supports 16 configurable control pins when 16 or 8 data bus is used. All control pins can be either input/output or bidirectional
FX3 - FPGA connectivity is realized with the GPIF interface. Here you can see the main characteristics of GPIF interface. It can function as master or slave; it provides 256 firmware programmable states; supports 8 bit, 16 bit, and 32 bit parallel data bus and enables interface frequencies up to 100 MHz.
USBpix Upgrade

13. Firmware development GPIF II Designer
18 February '13
Firmware development
GPIF II Designer
Define the interface in the form of a state machine diagram
The electrical interfacing details should be defined using the Interface definition tab before entering the state machine using the state machine canvas tab
To configure the GPIF port of FX3 device to connect to the FPGA a software tool called GPIF designer is used. It allows to define the interface in the form of a state machine diagram as well as to define the electrical interfacing details.
USBpix Upgrade

14. Firmware development GPIF II Designer
18 February '13
Firmware development
GPIF II Designer
In our case electrical interfacing details look like this: we are using I2C and SPI, FX3 is a master in this case, communication type is synchronous, internal clock is used with positive clock edge, data bus width is 16 bits so far. Some control signals are also used. All signals are currently set to active high.
USBpix Upgrade

15. Firmware development GPIF II Designer
18 February '13
Firmware development
GPIF II Designer
The interface itself is defined in the form of a state machine diagram. Here you can see 2 disjoint state machines: for writing and for reading. I will explain more in detail the operation of the writing state machine. First of all after application is started and GPIF interface is initialized the state machine goes to the IDLE_WR state where counters are initialized. After this depending on the type of the request from the host it switches to reading state machine or continues execution of writing state machine. The first state in writing state machine is IDLE_WR. It will go to the Start Of Frame state if the data in the corresponding DMA socket is available. If it is it goes to the Start Of Frame state and drives corresponding gpios that signal FPGA about the start of frame. FPGA answers with acknowledge. The next state is for driving data to the databus. Here the data counter is used based on the length of the data that should be written to FPGA. WAIT state is necessary if in some moment during the transition there is no valid data in the socket. After data is driven to the databus the data counter hit condition becomes true and state machine goes to the End Of Frame state and waits for the acknowledge from the FPGA. Here another counter is used to implement a time-out.
I have to mention that all data transfers with GPIF interface are done with the help of DMA Factory without CPU intervention.
USBpix Upgrade

16. Firmware development IDLE SOF WR_DATA EOF IDLE
18 February '13
Firmware development
IDLE
SOF
WR_DATA
EOF
IDLE
Here is what we have on the scope during writing to the FPGA. (CLICK) All the states are passed according to the state machine. IDLE, SOF, WR_DATA, EOF and again IDLE.
USBpix Upgrade

17. Firmware development GPIF II Designer FMU3-S6 series FPGA module
18 February '13
Firmware development
GPIF II Designer
FMU3-S6 series FPGA module
Now it is not longer possible to develop a firmware for FX3 device without response from the FPGA. One of the possible options to have an FX3-FPGA connectivity is to connect FX3 development board to the GigaBee Header Baseboard that holds an FPGA.
But we decided to use another option (CLICK) – board from LINERA company that holds both FX3 and FPGA.
GigaBee Header Baseboard with TE0600 module and cables to connect to FX3
USBpix Upgrade

18. Firmware development FX3 device: FPGA:
18 February '13
Firmware development
FX3 device:
Can be programmed with software for the FX3 development kit (USB 3.0)
FPGA:
Can be programmed with JTAG interface (self-made connector + USB-JTAG Programming Cable)
FX3 device on this board can be programmed with the software provided for FX3 development kit through USB. FPGA can be programmed with JTAG interface through the self-made connector and USB-JTAG Programming Cable. It’s enough to start testing.
FMU3-S6 connected to the power supply, to the host through USB 3.0 interface and to the USB-JTAG Programming cable through the self-made connector.
USBpix Upgrade

19. Firmware development Response from the FPGA (Timing)
18 February '13
Firmware development
Response from the FPGA (Timing)
In order to write a correct response from the FPGA it is necessary to take into account such GPIF timing parameters as data and control signals to clock setup time, to clock hold time, propagation delays and some others.
USBpix Upgrade

20. Firmware development GPIF II Designer (Timing simulation)
18 February '13
Firmware development
GPIF II Designer (Timing simulation)
Some of the timing parameters are also reflected in the GPIF designer timing simulation. This is a simulation of a writing state machine. Here it’s possible to see not only data and control signals but also DMA and counter flags.
USBpix Upgrade

21. 18 February '13
Firmware development
Response from the FPGA to the reading access (FX3 delay)
Response from the FPGA to the writing access (FX3 delay)
Writing and reading responses have been written for FPGA taking into account GPIF timing parameters. So that FPGA can distinguish the start of frame and give certain response for FX3. Here are ISim simulations for reading and writing responses.
USBpix Upgrade

22. 18 February '13
Firmware development
Response from the FPGA to the reading access (FX3 + FPGA delay)
Response from the FPGA to the writing access (FX3 + FPGA delay)
And here you can see ISim simulations where FPGA delays have also been taken into account. After the delays’ analysis was finished the firmware for the FX3 and FPGA was adapted to delays.
USBpix Upgrade

23. Firmware development ChipScope waveform (Writing to the FPGA)
18 February '13
Firmware development
ChipScope waveform (Writing to the FPGA)
ChipScope core was inserted in the FPGA firmware in order to monitor real data and control signals from FX3 and FPGA. Here you can see ChipScope waveform that was captured during writing access to the FPGA. In order to have a correct waveform it was necessary to modify FX3 and FPGA firmware.
USBpix Upgrade

24. Conclusion FX3 Firmware USB 3.0 GPIF II I2C EEPROM SPI SPI Flash
18 February '13
Conclusion
FX3 Firmware
USB 3.0
GPIF II
Writing to the FPGA
Reading from the FPGA
4 endpoints for different transfer types were configured
USB enumeration descriptors were configured
DMA Manual channels were created between USB endpoints and CPU sockets
Correctly recognized as SiUSB device on Windows 32 and 64-bit
I2C EEPROM
SPI SPI Flash
In conclusion I have to say that during the firmware development for the FX3 device USB 3.0 interface was implemented. FX3 device now is correctly recognized as SiUSB device on Windows 32 and 64-bit systems.
After implementing I2C and SPI functionality it’s possible to write and read data from EEPROM and SPI Flash using the software for the FX2 device.
GPIF interface was also implemented to connect to FPGA.
USBpix Upgrade

25. Conclusion Next steps FPGA firmware (started)
18 February '13
Conclusion
Next steps
FPGA firmware (started)
FX3 firmware (should be modified every time FPGA timing changes)
PCB design of the new MultiIO Board
Now it’s necessary to continue developing a firmware for the FPGA. FX3 firmware should be modified every time FPGA timing changes.
After all the firmware is written the PCB design of the new MultiIO Board should be made.
USBpix Upgrade

26. 18 February '13
Contact Information
Viacheslav Filimonov
USBpix Upgrade

27. EZ USB FX3 System Diagram
18 February '13
Backup
EZ USB FX3 System Diagram
SPI Flash 4 Mbit
EEPROM 64kB
USBpix Upgrade

28. EZ USB FX3 System Diagram
18 February '13
Backup
EZ USB FX3 System Diagram
USBpix Upgrade

29. 18 February '13
Backup
(Currently 8 MHz)
USBpix Upgrade

30. 18 February '13
Backup
USBpix Upgrade

31. 18 February '13
Backup
Programming View of FX3
USBpix Upgrade

32. 18 February '13
Backup
DMA Mechanism
System Memory
USBpix Upgrade

33. 18 February '13
Backup
DMA Components
USBpix Upgrade

34. 18 February '13
Backup
USB Comparison
USBpix Upgrade

35. 18 February '13
Backup
Changes in USB 3.0
SuperSpeed — New higher signaling rate of 5Gbps (625MB/sec)
Dual-bus architecture — Low-Speed, Full-Speed, and High-Speed bus plus SuperSpeed bus
Asynchronous instead of polled traffic flow
Dual-simplex simultaneous bi-directional data flow for SuperSpeed instead of half-duplex unidirectional data flow
Support for streaming
Fast Sync-N-Go technology
Support for higher power
Better power management
USBpix Upgrade

36. Backup USB 3.0 Pin Description Electrical Interface
18 February '13
Backup
USB 3.0 Pin Description
Electrical Interface
USB 3.0’s pinout is different from that of USB 2.0. Besides the VBUS, D-, D+, and GND pins required for 2.0, 3.0 has five additional pins – two differential pairs plus one ground (GND_DRAIN). The two differential pairs are for SuperSpeed data transfer, supporting dual simplex SuperSpeed signaling. The added GND_DRAIN pin is for drain wire termination, managing signal integrity, and EMI performance.
For high throughput and biwire transfer, USB 3.0 defines nine pins to enhance this feature. These pins contain four pins for USB 2.0 and five additional pins for 3.0. Table 1 shows the description of these nine pins.
USBpix Upgrade