1. Planet Lab Memory Map
David M. Zar Applied Research Laboratory Computer Science and Engineering Department

2. Revision History 8/xx/06 (DZ): 8/29/06 (JDD): Created
Updated with more details on counters

3. Planet Lab Memory Map The following SRAM is used in every design:
Buffer Descriptors in Channel 1 (8 MB): 0x
RX PACKET COUNTERS (debug) in Channel 3 (640 bytes)
TX PACKET COUNTERS (debug) in Channel 3 (640 bytes)
Scratch Memory usage:
TO_XSCALE_RING: 0x2800 – 0x2bff
FROM_XSCALE_RING: 0x2c00 - 0x2cff
PS_TO_QM_RING_1: 0x3000 – 0x33ff
PS_TO_QM_RING_2: 0x x37ff
QM_TO_TX_RING_1: 0x3800 – 0x3bff
QM_TO_TX_RING_2: 0x3c00 – 0x3fff
TX1 - PACKET_TX_AVAIL_TBUF_ELE_SCR_ADDR: SCRATCH_BASE + 0x1f00

4. Substrate Decap (IPv4) Memory Map
Initialization data
Enet MAC Addr MR blade (48b)
Dynamic Data
VLAN-to-CodeOption index table
2^12 = 4096 VLANs
4b Code Option field
1LW for minimal cycles (16KB req’d: SRAM)
or, 4b for minimum memory (2KB req’d: SRAM, Scratch, LM)
Counters
2 LWs total for packets received & sent
4 LWs total for Ethernet VLAN validation
6 LWs per VLAN for UDP/IP validation
using power-of-2 sizes saves cycles; multiply becomes shift
now 32B per VLAN x 4096 VLANs = 128KB SRAM req’d

5. Parse (IPv4) Memory Map Initialization data Dynamic Data: counters
none; only compile-time constants and ring data used
Dynamic Data: counters
2 LWs total for packets received and sent
9 LWs per VLAN for UDP/IP validation
using power-of-2 sizes saves cycles; multiply becomes shift
now 64B per VLAN x 4096 VLANs = 256KB SRAM req’d

6. Lookup (IPv4) Memory Map
Initialization data
Dynamic Data

7. HF(IPv4) Memory Map Initialization data (88b + #GP*88b)
Enet Addr MR blade (48b)
IP Addr MR blade (32b)
#GPs (8b)
Table[#GP]
Enet Addr (8b)
Upper 40 bits are same as Enet Addr MR Blade above
IP Addr (8b)
Upper 24 bits are same as IP Addr MR Blade above
Local Delivery
UDP dest (16b)
QID (20b)
Exception
Dynamic Data (None)

8. QM (IPv4/LCI/LCE) Memory Map
Constants
#Q – 128K (total for all QM)
QM_NUM_QUEUES
QD_BASE_ADDR 0x000000
QD_SIZE 0x200000
QPARAMS_BASE_ADDR 0x200000
QPARAMS_SIZE 0x200000
QSCHED_BASE_ADDR 0x400000
QSCHED_SIZE 0x140000
PORT_RATES_BASE_ADDR 0x540000
PORT_RATES_SIZE 0x28
Initialization data (none)
Dynamic Data (SRAM)
Port Rates (5 * 32b)
Per Queue
Queue Parameters (per queue)
Discard Threshold (32b)
Quantum (32b)
Qlen (32b)
Rsvd (32b)
Queue Descriptor (128b)
Scheduling Structure/Freelist (SRAM) (12W/5Q)

9. KeyExtractor (LC) Memory Map
Ingress
Initialization data (none)
Dynamic Data (none)
Egress

10. Lookup (LC) Memory Map Ingress Egress Initialization
Dynamic Data (none)
Egress
Initialization (none)

11. HF (LC) Memory Map Egress Initialization Dynamic Data (none)
Ingress
Initialization
Enet Addr (48b)
Dynamic Data (none)
Egress
Initialization (none)

12. Port Splitter (LC) Memory Map
Ingress
Initialization (none)
Dynamic Data (none)
Egress

13. Counters Two sets of counters maintained for each successful lookup:
Pre-Queue: Pkt counter and Byte counter
Post-Queue: Pkt counter and Byte counter
Each counter is 32-bits
Organization of Counters:
Addr of a pre-q pckt ctr: BASE+(4 * 4 * Index) + (0*4)
Addr of a pre-q byte ctr: BASE+(4 * 4 * Index) + (1*4)
Addr of a post-q pckt ctr: BASE+(4 * 4 * Index) + (2*4)
Addr of a post-q byte ctr: BASE+(4 * 4 * Index) + (3*4)
IPV4_LU_COUNTERS_SRAM_INIT_BASE
Counter Index 0
Pre-Queue Pkt Ctr
Pre-Queue Byte Ctr
Post-Queue Pkt Ctr
Post-Queue Byte Ctr
Counter Index 1
Pre-Queue Pkt Ctr
Pre-Queue Byte Ctr
Post-Queue Pkt Ctr
Post-Queue Byte Ctr
. . .
Counter Index N
Pre-Queue Pkt Ctr
Pre-Queue Byte Ctr
Post-Queue Pkt Ctr
Post-Queue Byte Ctr

14. LC Ingress SRAM Channel 3
0x000000
0x000000
PreQ/PostQ Cntrs
0x00FFFF
MI Counters
0x010000
0x08FFFF
0x100000
Unallocated
0x090000
HF Init
0x090007
0x090008
Counter Error Cnts
0x090027
0x200000
Unallocated
0x090028
Unallocated
0x0FFFFF
0x300000
Unallocated
0x400000
Unallocated
0x700000
Unallocated
0x500000
Unallocated
0x7FFDFF
Rx Pkt Counters
(turned off for Perf.)
0x7FFE00
0x600000
Unallocated
0x7FFE9F
Tx Pkt Counters
(turned off for Perf.)
0x7FFEA0
0x700000
0x7FFF3F
Unallocated
0x7FFF40
0x7FFFFF

15. LC Ingress SRAM Channel 2
0x000000
Buffer Descriptors
(0x38000 * 32B)
( * 32B)
(7MB)
0x100000
0x200000
0x300000
0x400000
0x500000
0x600000
0x700000
Queue Desc Array
(65536 * 16B = 1MB)

16. LC Ingress SRAM Channel 1
0x000000
Q Params
(65536 * 16B)
0x100000
QM1 Sched
(13109 * 44B= 0x8CD1C)
0x18CD1C
QM2 Sched
(13109 * 44B= 0x8CD1C)
0x219A38
QM1 Freelist
(13109 * 4B = 0xCCD4)
0x22670C
QM2 Freelist
(13109 * 4B = 0xCCD4)
0x2333E0
QM1 & QM2 Port Rates
(10 * 4B)
0x233418
Unallocated
0x600000
Unallocated
0x7FFFFF

17. LC Egress SRAM Channel 3 0x000000 PreQ/PostQ Cntrs MI Counters
0x00FFFF
MI Counters
0x010000
0x08FFFF
0x100000
Unallocated
0x090000
HF Init
0x09003F
0x090040
Counter Error Cnts
0x09006F
0x200000
Unallocated
0x090070
Unallocated
0x0FFFFF
0x300000
Unallocated
0x400000
Unallocated
0x700000
Unallocated
0x500000
Unallocated
0x7FFDFF
Rx Pkt Counters
(turned off for Perf.)
0x7FFE00
0x600000
Unallocated
0x7FFE9F
Tx Pkt Counters
(turned off for Perf.)
0x7FFEA0
0x700000
0x7FFF3F
Unallocated
0x7FFF40
0x7FFFFF

18. LC Egress SRAM Channel 2 0x000000 0x100000 Buffer Descriptors
(7MB)
0x100000
0x200000
0x300000
0x400000
0x500000
0x600000
0x700000
Queue Desc Array
(65536 * 16B = 1MB)

19. LC Egress SRAM Channel 1 0x000000 Q Params (65536 * 16B) 0x100000
QM1 Sched
(13109 * 44B= 0x8CD1C)
0x18CD1C
QM2 Sched
(13109 * 44B= 0x8CD1C)
0x219A38
QM1 Freelist
(13109 * 4B = 0xCCD4)
0x22670C
QM2 Freelist
(13109 * 4B = 0xCCD4)
0x2333E0
QM1 & QM2 Port Rates
(10 * 4B)
0x233418
Unallocated
0x600000
Unallocated
0x7FFFFF

20. NPE/MR SRAM Channel 3 0x000000 SD Init HF Init 0x100000 Unallocated
0x000FFF
0x001000
Rx Pkt Counters
(turned off for Perf.)
0x200000
Unallocated
0x00109F
0x0010A0
Tx Pkt Counters
(turned off for Perf.)
0x300000
Unallocated
0x00113f
0x001140
SD Counters
0x001157
0x400000
Unallocated
0x001158
SD VLAN Code Opt Tbl
0x009157
0x009158
PreQ and PostQ Ctrs
0x500000
Unallocated
0x019157
0x019158
Error Ctrs
0x019177
0x600000
Unallocated
0x019178
Slice Specific Mem
0x099177
0x099178
IPv4/I3 specific data
0x700000
Unallocated
0x099A30
Unallocated
0x0FFFFF

21. NPE/MR SRAM Channel 2 0x000000 0x100000 Buffer Descriptors
(7MB)
0x100000
0x200000
0x300000
0x400000
0x500000
0x600000
0x700000
Queue Desc Array
(65536 * 16B = 1MB)

22. NPE/MR SRAM Channel 1 0x000000 Q Params (65536 * 16B) 0x100000
QM1 Sched
(13109 * 44B= 0x8CD1C)
0x18CD1C
QM2 Sched
(13109 * 44B= 0x8CD1C)
0x219A38
QM1 Freelist
(13109 * 4B = 0xCCD4)
0x22670C
QM2 Freelist
(13109 * 4B = 0xCCD4)
0x2333E0
QM1 & QM2 Port Rates
(10 * 4B)
0x233418
Unallocated
0x600000
Unallocated
0x7FFFFF

23. Scratch Memory (16 KB) 0x0000 0x1F00 0x1F04 TO_SCALE_RING … 0x2400
PACKET_TX_AVAIL_TBUF_ELE_SCR_ADDR
TO_SCALE_RING
FROM_XSCALE_RING
COUNTER_RING
PS_TO_QM_RING_1
PS_TO_QM_RING_2
QM_TO_TX_RING_1
QM_TO_TX_RING_2
0x1F00
0x1F04 …
0x2400
0x2800
0x2C00
0x3000
0x3400
0x3800
0x3c00
0x3FFC

24. dl_system
#define BUF_SRAM_BASE SRAM_CHANNEL_1_BASE_ADDRESS
#define NUM_IN_PORTS 10 ; number of input ports
#define NUM_OUT_PORTS 10 ; number of output ports
#define_eval PACKET_COUNTERS_SRAM_BASE SRAM_CHANNEL_3_BASE_ADDRESS
#define RX_PACKET_COUNTERS_SRAM_SIZE 16 * NUM_IN_PORTS
#define_eval PACKET_TX_COUNTER_BASE PACKET_COUNTERS_SRAM_BASE +
RX_PACKET_COUNTERS_SRAM_SIZE
#define PACKET_TX_COUNTER_SIZE 16 * NUM_OUT_PORTS
#define PS_TO_QM_RING_
#define PS_TO_QM_RING_1_SIZE
#define PS_TO_QM_RING_1_BASE 0x3000
#define PS_TO_QM_RING_
#define PS_TO_QM_RING_2_SIZE
#define PS_TO_QM_RING_2_BASE PS_TO_QM_RING_1_BASE + 4 * PS_TO_QM_RING_1_SIZE
#define QM_TO_TX_RING_
#define QM_TO_TX_RING_1_SIZE
#define QM_TO_TX_RING_1_BASE PS_TO_QM_RING_2_BASE + 4 * PS_TO_QM_RING_2_SIZE
#define QM_TO_TX_RING_
#define QM_TO_TX_RING_2_SIZE
#define QM_TO_TX_RING_2_BASE QM_TO_TX_RING_1_BASE + 4 * QM_TO_TX_RING_1_SIZE
Um, shouldn’t this be “4” for the four bytes per queue entry? And… Should the rings be 256 or 512?

25. Extra Slides
The rest of the slides are old or here for extra info.

26. SRAM Channel 1 (8 MB) – 0x40000000 0x000000 0x100000 0x219A38 0x2333E0
Queue Parameters
- Each is 4, 32-bit words (16 bytes) So room for 64K
Queue Scheduling Structure (*2)
- Each is 11, 32-bit words (44 bytes) Allocated 13109
Queue Free List (*2)
- Each is 1, 32-bit words (4 bytes) Allocated (same as # of QSS)
Port Rates Each is 1, 32-bit words (4 bytes) Ten ports
0x100000
0x219A38
0x2333E0
0x233408 .
0x7FFFFC

27. SRAM Channel 2 (8 MB) – 0x80000000 BUF_SRAM_BASE - Buffer Descriptors
- Each is 8 32-bit words (32 bytes)
- So room for 224K
Queue Descriptors Each is 4 32-bit words (16 bytes)
- So room for 64K
0x6FFFFF
0x7FFFFC

28. IPv4 SRAM Channel 3 (8 MB) – 0xC0000000
Packet/Byte Counters (4 words * 16K)
- PRE_Q_PKT_CNT PRE_Q_BYTE_CNT POST_Q_PKT_CNT POST_Q_BYTE_CNT
MI Counters (512K)
Initialization Data
0x010000
0x090000
0x190000
0x7FFFFC

29. IPv4 SRAM Channel 3 (8 MB) – 0xC0000000
Per Block Initialization Memory
RX: Base = 0x000000, Size = 0
SD: Base = 0x000000, Size = 8
PR: Base = 0x000008, Size = 0
LK: Base = 0x000008, Size = 0
HF: Base = 0x000008, Size = 8
PS: Base = 0x000010, Size = 0
QM: Base = 0x000010, Size = 0
TX: Base = 0x000010, Size = 0
0x000010
0x001000

30. IPv4 SRAM Channel 3 (8 MB) – 0xC0000000
Per Block Dynamic Memory
RX: Base = 0x001000, Size = 0x280
TX: Base = 0x001280, Size = 0x280
SD: Base = 0x001500, Size = 0x24018
PR: Base = 0x025018, Size = 0x40008
IPv4 POST_Lookup Counters (PreQ, PostQ):
Base: 0x025018, Size 0x
0x074520
0x100000

31. IPv4 SRAM Channel 3 (8 MB) – 0xC0000000
BUF_SRAM_BASE
- Buffer Descriptors
- Each is 8 32-bit words (32 bytes)
- (0x – 0x100000)/32B =
- So room for Buffer Descriptors
0x100000
0x7FFFFC

32. LC SRAM Channel 3 (8 MB) – 0xC0000000
PACKET_COUNTERS_SRAM_BASE
- 16 words per port, 16 ports (1024 bytes, total)
- Used in RX for debugging
PACKET_TX_COUNTER_BASE
Per Lookup Result Counters (pre-Q and post-Q)
- 16 MR, 256 indices, 4 32-bit counters per index = 64KB (See next slide for ctr details)
Per MI Counters
- 64K * 2 * 4 = 512KB
0x000400
0x000800
0x010800
0x090800
0x100000
0x7FFFFC