Washington WASHINGTON UNIVERSITY IN ST LOUIS Overview of the APIC Pacer

2 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM APIC Pacing: General Stuff Pacing is for Transmit Channels only Cells are NOT Paced out onto the wire –Not Exactly Pacing is done on the PCI bus Pacing is not a Guarantee, it is just a Restriction Pacing Calculations include the ATM headers –But not the APIC header More APIC Info: –

3 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM APIC Pacing: General Stuff Two pacer controls: –Global Pacing APIC Pacing Parameter register (Global, 0x208) –Per VC Pacing TX Channel Pacing Parameter Register (TX, 0x500XX68) –XX is the Channel ID Three types of Channels: –Low Delay (Highest Priority) –Paced –Best Effort (Lowest Priority) All channels are paced by the Global Pacing Paced Channels also use Per VC Pacing

4 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM APIC Data Transfers APIC pulls data from memory across the PCI bus in Batches of cells. –The number of cells in a Batch is controlled by a register The Pacer identifies when it is time to transmit data and which connection should transmit Pacer “wakes up” every 14 PCI Bus clock ticks –checks to see if it is time to transmit Controlled by the Global APIC Pacing Parameter (APP) –If it is time to transmit, it takes the first connection off the previously sorted list of keys and transmits its data. A lot of gory details about keys and heap storage of connections is not going to be included here. Read Rex’s documentation and/or read the VHDL if you want that level of detail

5 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Global Pacing Parameter Pacing parameters are 24 bits –16 bits of Integer –8 bits of fractional part Global Apic Pacing Parameter (APP) (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = (14 * ClockEstimate * LinkRateMbps) [Items in formula explained on next slide]

6 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Explanation of Expression (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = (14 * ClockEstimate * LinkRateMbps) 256 : shifts left by 8 bits to set “decimal point” BatchSz: How many cells per transfer 53*8: Translate cells/second into bits/second 8192, InternalClockMhz (85MHz), ClockEstimate –APIC counts how many of its internal 85MHz clock ticks take place during the time it takes for 8192 PCI bus clock ticks. This value is the ClockEstimate. –PCI Bus Clock Rate in MHz = (8192 * 85)/ClockEstimate 14: # of PCI Bus Ticks in a Pacer Period LinkRateMbps: Our target rate [Example on next 2 slides]

7 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Example: Units in the APP Formula (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = (14 * ClockEstimate * LinkRateMbps) (256 * Cells * Bytes/Cell * Bits/Byte * 8192 * M/sec) APP = (14 * 1 * MBits/sec)

8 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Example: APP for 1Gb/s Link Rate (256 * BatchSz * 53 * 8 * 8192 *InteralClockMhz) APP = (14 * ClockEstimate * LinkRateMbps) BatchSz=8 53*8: Translate cells/second into bits/second InternalClockMhz = 85MHz ClockEstimate = (typical value) LinkRateMbps: 1000 (1000 Mb/s == 1Gb/s) (256 * 8 * 53 * 8 * 8192 * 85) APP = = (14 * * 1000) APP = 2061 = 0x80D

9 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Example: APP for 1Gb/s Link Rate APP = 2061 = 0x80D This means that every 14*8 = 112 PCI Bus clock ticks the APIC will be able to pull 8 Cells worth of data across the PCI Bus. (8 Cells)/(112 * 30ns) = (3392 bits)/(3360ns) ~= 1Gb/s

10 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Per VC Pacing Per VC Pacing Parameter –What portion of the full link rate can be used –e.g. an integer value of 2 means that this channel can use half the link rate If you change the Pacing Paramter for a VC: –It takes affect the next time the OLD pacing expires!!! Conceptually like this: This Tx Channel is Ready to Transmit BATCH Cells Count to 14 Count to APP Count to TX Channel Pacing Parameter 33 MHz PCI Bus Clock

11 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Per VC Pacing XXXXXX time Expired connections X oldExpirationTim + vcPacingParameter  newExpirationTime current pacedTime vcPacingParameter ~ 10 One APIC Pacing Period

12 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM pacedTime pacedTime is incremented every global pacing cycle in which a non-LowDelay connection wins contention Example with two connections: –(L) Low Delay at 1/24 th of the global rate –(P) Paced at 1/6 th of the global rate ( ) L P PPPPPPPPPPPPPP LLL

13 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM pacedTime (continued) PPPPPPPPPPPPPPP LLLL We might expect the Paced channel to miss its exact turn and fire on the next global pacing interval but keeps it next expiration on the (0,6,12,18,…) boundaries. But…

14 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM pacedTime (continued) PPPPP P P P P P P P P P P LLLL Actual rate for Paced connection: –(GlobalRate) * (3*(1/6) + 1*(1/7))/4 – (GlobalRate) * (.1607) For a Global Rate of 24Mb/s (DQ test example) –24 *.1607 = “Real” time pacedTime t+

15 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Per VC Pacing If you change the Pacing Paramter for a VC: –It takes affect the next time the OLD pacing expires!!! –A Resume of the VC will cause an immediate change to the new Pacing parameter This artifact causes some strange behavior for DQ where there are large changes in the Per-VC pacing parameters. –Going from a low rate to a high rate can be delayed by several DQ periods. –This can cause the results to appear flawed. See the next slide for an illustration.

16 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Per VC Pacing X time X Just Transmitted Next Transmission by oldPacing Paramter Right after, Want to Change Pacing so it will Transmit every other Pacing Interval XXXX These will Transmit This will not transmit

17 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Example of a Pacing Oddity Suppose we have a channel on which we are sending single cell packets at a rate of 2 cells every pacing period for that channel and the BATCH size is 1 cell so that the channel should only send 1 cell during each pacing period. DDDDDDD You would expect the connection to build up a backlog, but it doesn’t……

18 Washington WASHINGTON UNIVERSITY IN ST LOUIS John DeHart- 12/6/2015 3:38 PM Example of a Pacing Oddity (con’t) Turns out the Driver does a RESUME each time it puts data in an empty transmit queue to restart it. A RESUME causes the ExpireTime to be set to the current PacedTime. This causes the channel to be expired at the very next Pacer Period. Thus the channel transmits at twice its expected rate DDDDDDD RRRRRRR TTTTTTT