Design Slides for RAD Port to techX http://www. arl. wustl Design Slides for RAD Port to techX http://www.arl.wustl.edu/arl/projects/techX/ppt/RAD_Design_Port_Slides.ppt some slides take from: http://www.arl.wustl.edu/arl/projects/techX/ppt/virtualRouter.ppt John DeHart jdd@arl.wustl.edu http://www.arl.wustl.edu/arl

Virtual Networking – Basic Concepts thru channel substrate link virtual link substrate router virtual router substrate links may run over Ethernet, IP, MPLS, . . . virtual end-system

Virtual Router and Substrate Virtual routers are constructed from Virtual Interfaces (VI) that connect to virtual links numbered sequentially from 0 for each VR have specified bandwidth may have “internal VLs” connecting VRs in same substrate router Virtual Packet Processors (VPP) numbered sequentially from 0 terminate VIs (one VPP may terminate multiple VIs) one of several types IXP 2850 type (full system of cluster of size 1, 2, 4, 8 MEs) V2Pro P100 (full chip or half chip) Virtual Switch (VS) has numbered ports that connect to VPPs ports have specified bandwidth (input, output may differ) Control software maps virtual routers onto physical resources provided by substrate.

Substrate Virtual routers are interconnected via Virtual Links Virtual Links exists within Substrate Links Substrate Links are terminated in the Substrate of a Substrate Router A Substrate Router may contain 0 or more Virtual Routers A substrate link may contain Virtual Links that connect to several different Virtual Routers The end of any particular virtual link is connected to one and only one Virtual Router

Packet Formats External packet format (Ethernet) Internal format Ethernet type field identifies packet as IP packet or VNET packet VNET packets carry Virtual Link Identifier (VLI) and length field IP packets treated like VNET packets Internal format Virtual Router (VR) Virtual Destination (VDST) format field (3 bits) VI, VPP, VPP pair, VPP range priority (2 bits) destination (2B - VI, VPP, VPP pair, VPP range) Physical Destination (PDST) Format field (3 bits) port, port pair, port range destination (2B - port, port pair, port range) Header Error Check (HEC) detects hw faults Lookup tables use VR+VDST to obtain PDST. VNET IP SRC adr VLI DST adr LEN CRC payload Padding TYPE SRC adr DST adr CRC IP packet Padding TYPE VR payload HEC VDST PDST

Lookup Tables Line card mapping on input. Line card mapping on output for VNET type, map VLI to (VR, VDST, PDST) use VR=0 to distinguish “thru channels” “thru channels” are ones ones that do not terminate here. for IP type, map VLI=0 to (VR, VDST, PDST) rate limit each outgoing stream as configured Line card mapping on output map (VR, VDST) to (output interface, Ethernet type, VLI) rate limit each outgoing stream PPC substrate mappings. map (VR, VDST) to (local interface, PDST) rate limit VI streams from PPs Switch elements mapping map VR to rate limit for outgoing stream per VR queues at each output link use PDST for forwarding decisions

Control Packets We should support a general mechanism for getting control information to the VRs. Our IPv4 VR will get them as ATM cells embedded in packets The CCP right now operates on ATM cells so it will be easiest to maintain that and have the IN module strip the packet away from the embedded ATM Cell How does the packet get to the VR? How does the VR recognize something as control vs. data to be routed? Proposal: VI=0 Coming in from PPC?

Packet Passing Through techX System VLI Including Switch Out Port VI VR … VR, VDST, PDST Phys Itf, Substrate Link, VLI LC Switch IPP Switch OPP Switch LC

Packet Passing Through techX System VR VDST HEC PDST Phys Loc of VPP VR,VI LC Switch IPP Switch OPP Switch LC

Packet Passing Through techX System VR VDST HEC PDST Substrate identifies VR based on physical Links/signalrs that pkt Came to it on. Including Switch Out Port VPP VR … VR, VDST, PDST VR+VPP LC Switch IPP Switch OPP Switch LC

Packet Passing Through techX System VR VDST HEC PDST Phys Loc of VPP VR,VPP LC Switch IPP Switch OPP Switch LC

Packet Passing Through techX System VR VDST HEC PDST Including Switch Out Port VI VR … VR, VDST, PDST VR,VI LC Switch IPP Switch OPP Switch LC

Packet Passing Through techX System VR VDST HEC PDST VLI Phys Itf VI,VR LC Switch IPP Switch OPP Switch LC

Packet Passing Through techX System VLI LC Switch IPP Switch OPP Switch LC

IPv4 Virtual PP PP Card Substrate: Identifies packet as for a VR on this card Sends packet to PP/H VR VDST HEC PDST Phys Loc of VPP VI LC Switch IPP Switch OPP Switch LC

IPv4 Virtual Packet Processor SRAM SRAM TCAM RSQ QM IN CRL from virtual links OUT to virtual links PSM SDRAM Plugin Subsystem Arriving packets received from multiple virtual links. Classification & Route Lookup (CRL) implements route lookup and classification, using VLI+IP header fields – target 20 M lookups/s. Packets stored off-chip by PSM, Queue Manager (QM) schedules packets for outgoing VLs – target 40 M enqueues/s. Resequencer (RSQ) needed for medium, large systems.

IPv4 Virtual Packet Processor SRAM TCAM SRAM OUT IN CRL QM from virtual links to virtual links PSM SDRAM Plugin Subsystem Design Changes in Target System: Change where plugin subsystem connects Have IN store packet payload to PSM/SDRAM Have OUT retrieve packet payload from PSM/SDRAM No RSQ in this design. It must reside in Switch Fabric or at its interface. Also need to add CCP in to design

IPv4 Virtual Packet Processor Control Cell Processor (CCP) SRAM TCAM SRAM OUT IN CRL QM from virtual links to virtual links PSM SDRAM Plugin Subsystem

Gigabit Ethernet Line Card GE SUNI GE FLASH GE SUNI GE Substrate V2Pro P100 (?) 4 GE SUNI GE Power GE SUNI Power GE

techX: Packet Processor Card Power SDRAM QDR 2Mx18b QDR QDR SDRAM QDR TCAM Power PP/H V2Pro P100 40625 (160 pins) 512MB or 1GB SDRAM 5 Substrate V2Pro P100 FLASH 10 102.5 G SDRAM PP/H V2Pro P100 SDRAM 5 Power 40625 (160 pins) TCAM Power QDR QDR QDR QDR

Gigabit Ethernet Line Card VNET Ethernet Frame Ethernet Frame SRC adr VLI DST adr LEN CRC payload Padding TYPE GE SUNI SRC adr VLI DST adr LEN CRC payload Padding VNET VR payload HEC VDST PDST GE FLASH GE SUNI GE Substrate V2Pro P100 (?) 4 4 GE SUNI GE Power GE SUNI Power GE

techX: Packet Processor Card Power 10 5 Substrate V2Pro P100 SDRAM PP/H V2Pro P100 QDR 2Mx18b PP/H V2Pro P100 FLASH 40625 (160 pins) 102.5 G 512MB or 1GB TCAM VR payload HEC VDST PDST VR payload HEC VDST PDST

IPv4 Virtual Packet Processor SRAM TCAM SRAM OUT IN CRL QM PSM SDRAM Plugin Subsystem

IPv4 Virtual Packet Processor SRAM TCAM SRAM OUT IN CRL QM PSM SDRAM Plugin Subsystem

IPv4 Virtual Packet Processor SRAM SRAM OUT IN CRL QM PSM SDRAM

IPv4 Virtual Packet Processor IN CRL QM OUT SRAM SDRAM TCAM PSM RSQ Plugin Subsystem

IPv4 Virtual Packet Processor SRAM SRAM TCAM RSQ QM IN CRL OUT PSM SDRAM Plugin Subsystem

IPv4 Virtual Packet Processor SRAM SRAM QM IN CRL OUT PSM SDRAM

IPv4 Virtual Packet Processor SRAM SRAM IN CRL QM OUT PSM SDRAM

IPv4 Virtual Packet Processor SRAM SRAM VR payload HEC VDST PDST VR payload HEC VDST PDST IN CRL QM Packet Headers/References: Do we want one uniform header to go through Or specific header for each interface Packet Header/Reference Packet Payload OUT PSM SDRAM

IN  CRL ISAR  CARL: 4 words of 64 bits each Flags(16b) IVIN(5b) OVIN(5b) PPN(3b) QID(10b) PktPtr(20b) Saddr(32b) Daddr(32b) Sport(16b) Dport(16b) Protocol(8b) MTP(8b) TotalLength(11b) IpOptionsWord1(32b) (used for LFS) IpOptionsWord2(32b) (used for LFS) VR payload HEC VDST PDST IN CRL PSM SDRAM

IN  CRL IN  CRL: Flags(16b) PktPtr(20b) Saddr(32b) Daddr(32b) IVIN(5b)  Input VI OVIN(5b) PPN(3b) QID(10b) PktPtr(20b) Saddr(32b) Daddr(32b) Sport(16b) Dport(16b) Protocol(8b) MTP(8b) TotalLength(11b) IpOptionsWord1(32b) (used for LFS) IpOptionsWord2(32b) (used for LFS) WHAT ELSE DO WE NEED for CRL? IN CRL SDRAM PSM VR payload HEC VDST PDST

CRL  QM CARL  QMGR: 3 words of 32 bits each Flags(16b), Mb(1b) CopyCnt(2b) QID(10b) PPN(3b) Ovin(5b) PktPtr(20b) LFS Rate2(8b) LFS Rate1(8b) TotalLength(11b) CRL QM SRAM

CRL  QM CRL  QM Plugin Bit Initial Copy bit Final Copy bit Drop bit CopyCnt(2b) for monitoring? Queue Set (Vitrual Interface) (3b) QID(10b) : size? PktPtr(20b) TotalLength(11b) WHAT ELSE DO WE NEED for QM? CRL QM SRAM

QM  OUT QMGR  OSAR: 4 words of 32 bits each Flags(16b), LFS Rate2(8b) LFS Rate1(8b) QID(10b) Mb(1b) PktPtr(20b) QueueLength(24b) SRAM QM OUT

QM  OUT QM  OUT Plugin Bit Free Space Bit (i.e. free PSM memory) Single Chunk Bit PluginFlowIndex (QID) (10b) PktPtr(20b) Virtual Interface # WHAT ELSE DO WE NEED for OUT? SRAM QM OUT

Flags from NSP System Flags (8 bits) Internal Flags (8 bits) DP: Drop Packet RC: ReClassify packet CARL should do its thing (possibly again) NM: No Match CARL found no match. Send packet somewhere for handling EX: Exception packet Some kind of exception found, like IP options. Send packet somewhere for handling HO: Header Only Special header only processing for SPC HR: Header-only Return FM: From LC/SW Directional bit for where packet came from: Line Card or Switch TO: To LC/SW Directional bit for where packet is going to: Line Card or Switch Internal Flags (8 bits) DG: Datagram packet Only CARL lookup result was route entry, put in datagram queue SB: SPC bound packet SR: SPC return packet IC: Initial Copy This is the first copy of the packet (may also be last and only copy) (CARL  QM) FC: Final Copy This is the last copy of the packet (may also be first and only copy) (CARL  QM) LP: LFS option present SC: Single chunk packet (why was this needed?) spare

Flags for techX System Flags (8 bits) Internal Flags (8 bits) DP: Drop Packet RC: ReClassify packet NM: No Match EX: Exception packet How do we want to handle packets that need special processing or that don’t match any route or filter? HO: Header Only HR: Header-only Return FM: From LC/SW TO: To LC/SW Internal Flags (8 bits) DG: Datagram packet SB: SPC bound packet SR: SPC return packet IC: Initial Copy: do we need this? FC: Final Copy Tells QM/OUT when packet payload can be released from PSM. LP: LFS option present SC: Single chunk packet (why was this needed?) Spare Others ??

IN  CRL IN  CRL: Flags(16b) Input VI PktPtr(20b) Saddr(32b) Daddr(32b) Sport(16b) Dport(16b) Protocol(8b) TotalLength(11b) IN CRL SDRAM PSM VR payload HEC VDST PDST

CRL  QM CRL  QM Flags(16b) CopyCnt(2b) for monitoring? QID(10b) : size? Ovin(5b)  VDST? PktPtr(20b) TotalLength(11b) CRL QM SRAM

QM  OUT QM  OUT Flags(16b) OVIN (5b)  VDST? PktPtr(20b) SRAM QM OUT

Changes ISAR interface – remove some header fields FIFOs change due to bram differences Virtex: 4Kbits (up to 16 bits wide) V2Pro: 16Kbits (up to 32 bits wide) Block rams for CARL fifos will need to be regenerated SRAM interface for CARL: Asynchronous 125MHz internal rate 200 MHz DDR external rate May need to redo the clock latency for SRAM access Width issue New accesses are 72 bits wide Old access are 36 bits wide

Changes for QM SRAM interface: Asynchronous 125MHz internal rate 200 MHz DDR external rate QDR vs. SDR May need to redo the clock latency for SRAM access Width issue New accesses are 72 bits wide Old access are 36 bits wide Support for multiple Virtual Links (max=8) Each Virtual Link has: A set of N queues (N=256 or 512) 64 Datagram N-64 Reserved One set of Plugin Subsystem queues: 256 or 512 queues. Clean up the intermix of SPC and LC queue handling