2. HS/DSL Project Yael GrossmanArik Krantz Implementation and Synthesis of a 3-Port PCI- Express Switch Supervisor: Mony Orbach

3. Agenda  Project Goals  Project Specifications  PCI vs. PCI-Express  Switched Environment  PCI-Express Layers  Top Level Design Blocks  Local/Global Mechanisms  “A Day in the Life of a Packet”  Logic Blocks Walkthrough  Problems And Potential Pitfalls  Completed Tasks  Second Semester Plan

4. Project Goals  Familiarize ourselves with all aspects of hardware design:  Definition of soft and hard cores  Interaction with existing cores  Software and hardware integration  Simulation  Synthesis  Working with state of the art design tools such as EDK v8.1  Design a 3-Port PCI-Express switch  Learn the PCI-E protocol  Create block-diagram flow and design of the switch  Implement and test the switch we designed Part A Part B

5. Project Specifications  Design a 3 port PCI-Express switch.  Design must support high transfer rates – 2.5Gbit/sec, full-duplex, per port.  Support simultaneous connections on all ports.  Support complex real-time algorithms for:  Packet routing between ports  Error checking (CRC) and retransmission in case of error  Packet arbitration and prioritization according to type (data / control)  Implement the switch on a Memec FF1152 design board using a Virtex-II Pro FPGA.

6. PCI Bus  PCI - Peripheral Component Interconnect.  The standard bus, used to connect peripheral devices (hard-drive, CDROM/ DVD player) to a computer motherboard.  Typical clock frequency of 33Mhz with a throughput of 1.056Gbit/sec. Problems:  Only one device can use the bus at any given time!  Limited throughput with scalability issues.

7. PCI Bus vs. PCI-Express  The new state of the art computer I/O interconnect technology.  A serial point-to-point (P2P) technology.  Layered design, similar to the successful OSI model used in computer networks.  Answers all of the specified shortcomings of the standard PCI bus: Many devices can interconnect simultaneously. Highly scalable design - adding devices does not affect the throughput of existing ones. Initial throughput of 2.5Gbit/sec, easily extendable by using additional lanes.

8. Switched Environment  The PCI-E fabric is comprised of endpoints and switches, connected in a tree-like structure, where switches are internal nodes and endpoints are leaves.  The role of switches is to route and forward data packets between endpoints, while ensuring data integrity and quality of service.  Each link is defined exclusively between two devices, replacing the shared-bus technology.

9. Layered Structure  Transaction Layer: Creates data packets obtained from the device during a data send flow and restores the data from the packets during a data receive flow.  Data Link Layer: Ensures data integrity during packet transmission and reception on each Link.  Physical Layer: Performs the actual, physical process of sending and receiving packets on the electrical lines. Each layer interacts solely with the parallel layer on the peer device

10. Packet Structure Transaction Layer Data Link Layer Physical Layer Transaction Layer Packet Data Link Layer Packet

11.  Multi-Gigabit Transceiver (MGT) Handles high speed transmission and reception of packets..  Receive Block RAM (BRAM) Stores received packets on arrival to the switch.  CRC Check Checks if the packet arrived without errors.  ACK/NACK Generator Creates a DLL packet which indicates to the sending device whether the packet arrived intact.  ACK/NACK Queue (ACKQ) Stores the generated DLLPs while they await sending. Top Level Design

12.  Routing Block Decides on which outgoing port to forward the packet, according to lookup tables it contains.  Replay Buffer Contains all the packets that have been sent on the outgoing queue and have not yet received an ACK.  Send Queue 1 and 2 (SQ) Contain packets routed to current port from the other two switch ports.  Arbiter Decides which packet to transmit on the outgoing lane. Top Level Design

13.  Send Queue 1 and 2 (SQ1, SQ2) Contain packets routed to current port from the other two switch ports. We have one buffer for each queue in order to prevent simultaneous memory write.  ACK/NACK Queue (ACKQ) Used to store DLLPs for each received TLP while they await sending.  Replay Buffer(RB) Contains all the packets that have been sent on the outgoing queue and have not yet received an ACK.  Send/Receive Pipelines Includes logic for incoming and outgoing packets.  Routing Mechanism Includes an independent copy of the routing table.  Multi Gigabit Tranceiver(MGT) Allows sending and receiving packets via Rocket-I/O  Arbitration Mechanism Chooses next packet to send via MGT Local/Global Mechanisms  Packet Storage in Common Memory Space Enables fast packet forwarding between ports Local Global

14. A Day in the Life of a Packet TLP ACK TLP ? ?

15. CRC Logic Block TLP Handler DLLP Handler Continue

16. Return

17. Check if NACK needs to be sent. If so, set NACK_SCHEDULED ACK/ NACK Send Logic Route Packet and Enqueue

18. Return

22. RB Logic

23. Return

24. Send Pipeline

25. Potential Traps and Pitfalls Memory/Logic constraints  Need to make sure required logic can fit on development board  Make sure we have enough BRAM space for all required buffers Speed and concurrency  Need to make sure speed can match PCI-E speed specs. Potential issue with extensive bus access.  Potential synchronization issues in a multiple endpoint environment, under heavy load.

26. Completed Tasks Detailed design  Data flow diagram (transmitter/ receiver) for ports  Queue and buffer structure and size  Adaptation of design to board requirements System block design  Completed high and low-level block design  User core design in progress Work with design tools  Learned VHDL  Learned to work with EDK 8.1  Created custom projects and user cores

27. Second Semester Plan Complete coding of project 4 weeks  Block implementation and design in VHDL  Creating test benches and simulation  Adaptation of VHDL code to user core format Debugging and integration 4 weeks  Synthesis  Simulation  Testing  Fix bugs.. (if we have any, of course.. )