1. Fast A/D sampler FINAL presentation
Presented By: Tal Goihman, Irit Kaufman
Instructor: Mony Orbach
Winter 2012

2. Goals
Project Goal:
Design and implement an A/D system using Xilinx Virtex6 development board for sampling at highest possible rate.
Sample to virtex6 development board DDR memory.
Transfer the sampled data to PC memory through PCIe and save the data to disk.

3. PC H/W Block Diagram ML605 development board FMC125 Fast A/D FMC Conn
PCI-E Connector
FMC Conn
Virtex6 FPGA
ML605 development board
DDR3
FMC125 Fast A/D
A/D IC PC

4. ML605 development board

5. FMC125 Fast A/D

6. A/D Sampler
The FMC125 is a Quad-Channel ADC that provides four 8-bit ADC channels enabling simultaneous sampling of 1, 2, or 4 5 , 2.5 , 1.25Gsps respectively.
Problems:
4DSP provides free of charge a reference design only for 1.25Gsps. A reference design for 5Gsps priced at 2300 EU.

7. 4DSP reference design

8. Design Block Diagram “wormhole” AXI4 Implemented In XPS
MicroBlaze
DDR3
Memory Controller (MIG)
AXI Master
UART
ADC Control
FMC125
Aggregator
CDMA
PCI Express
AXI Slave
Mailbox
Timer
In ISE project Navigator
“wormhole”
AXI4-lite
AXI4

9. Block Diagram: Main Data Channel
FMC125
Aggregator
AXI Master
DDR3
Memory Controller (MIG)
AXI Slave
FMC125 delivers data using bit lanes of a proprietary bus running at 125Mhz into the aggregator.
Aggregator unites and synchronizes the different channels into a bit bus.
AXI Master sends the data over a 256-bit wide AXI bus running at 200Mhz to the AXI Slave interface of the Memory Controller.
Memory controller handles Writes to DDR3.
FIFO’s and H/W Flow control in every component’s input (and some components output) throughout the channel to achieve highest possible bandwidth.
Throughput is measured using a timer from the start of the write operation until assertion of a write done signal from the AXI master
16 samples per clock per channel
Total 16*4*125Mhz=8GB/s
Aggregator synchronizes samples so that all channel’s sample 0 will come together
Aggregator works at 125Mhz, has a 4bit channel enable input
AXI master throughpuf of 256bit/8*200Mhz=6.25GB/s

10. AXI Bus background
Xilinx has adopted AXI bus, which is a standard bus protocol from ARM used in modern ARM SoC.
Characteristics
Memory mapped, 32-bit addresses
Write Address, write data, write response, read address, read data
Variable width, clock & burst length over a single bus.

11. AXI write example

12. Memory & Memory controller
512MB DDR3 400Mhz (800MT/s)
Theoretical bandwidth of MT/s * 64bit / 8 = 6.25GB/s
Main channel matched to this theoretical bandwidth (200Mhz * 256bit /8 = 6.25GB/s)
Memory & controller isn’t perfect, has a utilization factor.
We achieved 5.13GB/s Throughput (82% utilization!)

13. Design Block Diagram “wormhole” AXI4 Implemented In XPS
MicroBlaze
DDR3
Memory Controller (MIG)
AXI Master
UART
ADC Control
FMC125
Aggregator
CDMA
PCI Express
AXI Slave
Mailbox
Timer
In ISE project Navigator
“wormhole”
AXI4-lite
AXI4

14. Block Diagram: memory to PCIe
After the data is sampled to memory the system transfers the first chunk to the Host PC’s DMA buffer.
The subsequent chunks are transferred upon receiving a command from the Host PC.
Data is read from memory and transferred to PCIe by the DMA engine
Throughput was matched to PCIe
Transferred in 16MB chunks (the chosen DMA buffer size)
Address translation occurs in the PCIe core
Talk about address translation and the issues we had with it

15. Block Diagram: PCIe Xilinx Available solutions include:
A PCIe integrated block with support for up to x8 gen2. Hard to work with:
Need to know inner workings of PCIe
Need to implement several proprietary interfaces with many rules and signals
A wrapper for memory mapped AXI with support for up to x4 gen1 / x2 gen 2
Connects to a standard AXI4 bus
Chosen in our design due to high ROI
We chose to use the wrapper due to high ROI
x4 gen1 configuration, up to 1GB/s
One 64bit AXI BAR for sample data transfer with Configurable address translation, one PCIe BAR for mailbox communication.

16. Block Diagram: MicroBlaze
MicroBlaze is a soft-core processor by Xilinx.
Runs firmware written in C from a dedicated 64KB BRAM.
Firmware communicates with Host PC’s software through a PCIe mailbox
Supports a predefined set of commands to accomplish the functional use case and provide debug capabilities
Initializes and configures the entire system per to the Host PC’s instructions.
Provide visual status and information to the user through UART massages.

17. Block Diagram: ADC ctrl
ADC ctrl is a custom core deigned to enable MicroBlaze to communicate with other custom cores
Developed using the Create peripharel wizrd in XPS and implements mamory mapped register access through AXI4-lite
Provides the following capabilities:
Translate the wormhole transactions to / from the FMC125 core.
Provide AXI Master with burst count, write start, test signals
Provides the aggregator with enabled ADC channels

18. Software Written in C# .NET 4 in VS2012, GUI in WPF
Enables simple interaction with the system
User adjustable burst size,
burst count, active channels,
output file location and
external trigger mode
compatible with Windows
7/8 x64.

19. Software: Jungo WinDriver
Easy driver creation
Simple interface for DMA buffer allocation, read and write operations
Slightly problematic from a .NET environment since it lacks proper documentation, and the wizard generates a non working solution
Reference for .NET and DMA was found in a bundled example for PLX chips

20. Test Environment
Verification of sampled data on DDR3 is accomplished by firmware on MicroBlaze comparing read data with predetermined written pattern
FMC125 incoming data is observable through ChipScope.
Bandwidth was calculated by measuring the time to write a chunk of data using a Timer.
Data transfer from memory to PCIe is validated through observing the memory on system and comparing with the data received from PCIe manually.

21. Questions? Thanks

22. Backup
Documantaion