DPDK: Prevention & Detection of DP/CP memory corruption Quick Intro : Sys App Eng Manager at Intel, SivaPrasad is part of my team Network Processor based Dataplane packet processing application background, Now transitioning to the DPDK based DP packet processing on the X86 platforms. Recently ran in to a DP memory corruption issue that took us 4 weeks to debug 9 March 2018 Amol Patel, Sivaprasad Tumalla

Background Most of the custom packet-processing hardware i.e. Network Processors provides mechanisms to, Separate Dataplane and Controlplane memory Protect and detect the Dataplane memory corruption/configuration issues We are Proposing something similar for the DPDK applications running on the GPP (General Purpose Processor)

Background The DPDK applications are running on the multi-threaded model Main process spawns worker-threads for the Dataplane processing This is a flat memory architecture wherein Control-thread and DP worker- thread’s memory is part of the same address-space, it would be difficult to isolate the issue in case of Dataplane/Controlplane data structure corruption We need ways to prevent/detect the Controlplane/Dataplane memory corruption. Shared Libraries Shared Libraries DPDK primary process memory map Primary ProcessMemory Hugepage mmap Thread - n Stack-n Huge Page Segment 1 Thread - 2 Stack-2 Mem-zone area 1 Thread - 1 Stack-1 Huge Page Segment 2 Heap Mem-zone area 2 Huge Page Segment 3 text

Ways to Prevent and Detect DP/CP Memory corruption Prevention Control-plane memory (stack) protection Dataplane memory protection Controlplane and Dataplane memory access control Detection Dataplane Memory Corruption Detection DPDK Mem-zone Dump

Prevention Controlplane stack protection Dataplane memory protection Controlplane and Dataplane memory access control

Controlplane memory Protection : Existing memory model Worker-threads are the Linux threads Stack memory for the worker- threads is allocated in primary- threads’ Address space Thread level MMU protection does NOT exist All the worker-threads of the primary-process has access to stack memory of all the other threads, one worker thread can corrupt the other worker thread’s stack Shared Libraries Primary ProcessMemory Hugepage mmap Thread - n Stack-n Huge Page Segment 1 Thread - 2 Stack-2 Mem-zone area 1 Thread - 1 Stack-1 Huge Page Segment 2 Heap Mem-zone area 2 Huge Page Segment 3 text

Worker thread Stack Protection: Proposed Memory Model Mechanism: Stack of the worker-thread to be allocated on mem-zone Assign the allocated mem-zone memory to the worker-thread stack address, int thread_attr_setstack(pthread_attr_t *attr void *stackaddr, size_t stacksize); Advantages: Stricter control on any possible out-of-bound memory access for the stack variables Localizing the stack corruption effect to the thread Ability to dump the thread’s stack contents by dumping the mem-zone Shared Libraries Primary Process Hugepage mmap Worker Thread-n Stack-n Huge Page Segment 1 WorkerThread - 2 Stack-2 Mem-zone area 1 Worker Thread- 1 Stack-1 Huge Page Segment 2 Heap Mem-zone area 2 Huge Page Segment 3 text

DP Memory Protection: Existing DPDK Dataplane Memory Layout Mem-zone 1 Huge Page Segment 1 DP table Mechanism: Actual tables are allocated using the rte_lmp_create/rte_hash_create which internally invokes the rte_dpdk callocs() to allocate memory from the Hugepage. Disadvantages Memory allocated for the different Dataplane tables might be scattered in different memory-segments/huge pages No direct way to check for out-of-bound write for the Dataplane tables There is no way to dump the contents of all the Dataplane tables and counters in one go Mem-zone 2 Huge Page Segment 2 Mem-zone 2 RTE_CONFIG Shared Libraries Huge Page Segment 3 Stack Thread - n Stack Thread - 2 Ability to Dump the all the Dataplane tables in one go is important for troubleshooting the packet drop issues. Huge Page Segment 4 Stack Thread - 1 malloc, calloc, zmalloc regions DP tables DP caches DP counters Huge Page Segment 4 Heap - n Other tables Huge Page Segment 5 text

rte_malloc, rte_calloc DP Memory Protection: Proposed DPDK DP Memory Allocation using memory zones DP-Mem-zone text Stack Thread - n Heap - n Shared Libraries Stack Thread - 2 rte_malloc, rte_calloc DP table2 DP counters DP table1 Guard-band Huge Page Segment 1 Mechanism One mem-zone reserved for all the dataplane tables, data-structures, all the DP tables are placed on this this mem-zone by a user-defined wrappers Advantages Dataplane memory separation DPDK memory zone dump APIs supported to dump all the DP tables. Ease the troubleshooting the memory and configuration issues Guard-band allows checking of any out-of-bound access Disadvantages Additional wrappers for allocating the Dataplane tables (LPM, Hash, counters) on the mem-zones DP-Mem-zone Huge Page Segment 2 Huge Page Segment 3 Dumping the tables to find the reason for the packet drop Huge Page Segment 4 Stack Thread - 1 Huge Page Segment 4 Huge Page Segment 5

CP / DP Memory Access Control Worker thread to access the dataplane-table memzones in Read- only mode (mmap call supports the RO access, DPDK would have to add this support). In the worker threads, all the dataplane pointers MUST be accessed with the “constant” primitive to avoid any accidental corruption Using “restrict” qualifier for all Read- Only function argument Dataplane code not to change any dataplane entries except the fastpath counters Control plane tracking the #entries for the DP tables and returning error if it’s reached MAX (this way we can avoid any over-stepping to the next DP table)

Detection Detection (Debugging / troubleshooting) Dataplane Memory Corruption Detection DPDK Mem-zone Dump

Dataplane Memory Corruption Detection Single DP memzone for the dataplane objects (forwarding tables, counters, structure etc.), each DP reserves few extra-rows for every DP table (Max + N) which serves as the guard-band Monitoring the guard-bands b/w the individual DP tables to keep check on any out of bound access Use DPDK utilities to dump the memzones to dump the DP tables, counters for debugging Application test-framework to check any out-of-bound access for DP tables by monitoring the memzone

4 Stage Memzone Dumping Process DPDKProcInfo –m : Dump Memzone physical and Virtual address cat /proc/<pid>/maps : Dump process’ virtual memory map  Huge_page mapping Indentify the Memzone  Hugepage mapping by comparing the virtual-address hexdump /mnt/huge/mntrte_N : Dump the Memzone content

Detection – DPDK Memzone Dump : dpdkprocinfo –m

cat /proc/<pid>/maps

Troubleshooting – DPDK Memzone Dump : hexdump /mnt/huge/rtemap_

Questions?