Valgrind, the anti-Alzheimer pill for your memory problems Philippe Waroquiers FOSDEM 2017 valgrind devroom

Talk content Discuss/demo a new functionality, that provides an easier way to visualise memory usage and other types of data. If time permits: Discuss/demo memory pool and leak heuristics

How much memory, allocated where ? Massif : records the evolution of the memory By taking snapshots regularly Some snapshots are detailed, showing the allocation stack traces The peak snapshot is also detailed In 3.13 (SVN), new ' tree view' memory feature --xtree-memory=none|allocs|full to produce a memory report at the end of execution Can also be produced on demand, using vgdb

Demo : Who allocated how much memory ? valgrind --tool=massif --xtree-memory=full \ soffice.bin --invisible \ --convert-to pdf fosdem2017_optim_xt_hg.odp # If specifying a .ms extension, produces a Massif format: valgrind --tool=massif --xtree-memory=full \ --xtree-memory-file=xtmemory.ms.%p \ mfg # Default xtmemory.kcg.%p produces a callgrind format: valgrind --tool=massif --xtree-memory=full \ mfg

Demo : ms_print output

Demo : massif-visualizer

Demo : kcachegrind xtree memory full

Demo : massif-visualize xtree memory full

Massif format and --xtree-memory=full valgrind --tool=massif --xtree-memory=full \ --xtree-memory-file=xtmemory.ms.%p \ soffice.bin --invisible \ --convert-to pdf fosdem2017_optim_xt_hg.odp Massif format : Not designed for huge set of data Not designed for more than 1 'data kind' => the above generates a 6.3 Gb file => very (too?) heavy e.g. for massif-visualizer => for xtree, kcachegrind format is better

Xtree and memcheck/helgrind Like with Massif, --xtree-memory=allocs|full will produce a report at the end of execution To produce report during execution: vgdb xtmemory <filename> Supported by massif/memcheck/helgrind. Can be done even when --xtree-memory=none => it will produce a “=allocs report”

How to best visualise memory ? massif+ms_print : not that easy for big apps massif+massif-visualizer : nice evolution graph, digging in stack traces not that easy massif-visualizer+--xtree-memory-file=full : not really appropriate kcachegrind+--xtree-memory-file=full : easy to use, but file can become huge Maybe kcachegrind and callgrind format should be expanded to present evolution of data ?

Xtree and memcheck leak valgrind --tool=memcheck --xtree-leak=yes \ soffice.bin --invisible \ --convert-to pdf fosdem2017_optim_xt_hg.odp --xtree-leak=yes : final leak report as xtree report is automatically a “full” leak report report file name controlled by --xtree-leak-file=xtleak.kcg.%p vgdb monitor command 'leak_check' can also produce xtree leak reports Note: only the callgrind kcg format is supported for xtree leak reports

Xtree for syscalls experimental, not (yet?) in SVN valgrind --tool=massif --xtree-syscall=yes \ soffice.bin --invisible \ --convert-to pdf fosdem2017_optim_xt_hg.odp --xtree-syscall=yes currently keeps track of Nr of syscalls Nr of failed syscalls Syscall time (micro seconds) Bytes read Bytes written

kcachegrind xtree syscalls

kcachegrind cycles Cycles in a callgraph are created by Recursive calls, e.g. fnA calls fnB that calls fnA “Superposition” of 2 non recursive stack traces main calls fnA that calls fnB main calls fnB that calls fnA Cycles and kcachegrind visualisation Inclusive costs cannot be added kcachegrind represents all the functions of a cycle by a “Cycle x” function Complex cycles not easy to analyse/grasp

Xtree and kcachegrind cycles Xtree stacktraces : Code addresses are translated to function names Function names are displayed by kcachegrind What if there is no symbol for a code address ? Currently, xtree uses a “UnknownFn???” Fn name Consequence: many (artificial) cycles => some more ideas and work needed here Maybe heuristic to guess a function start address ? Maybe use callgrind-like callstack follow-up logic ?

Xtree is work in progress ... In 3.13 SVN, xtree provides already an easy way to visualise memory usage and leaks Some further work : Xtree for system calls Better handling of unknown function names Optionally keeping the leaf function (malloc/free/...) Allow to visualise evolution of data Kcachegrind improvements : virtual top function, show exact callstack, ... … Your ideas/suggestions here …

Questions about xtree?

Valgrind and address info Address information based on Knowledge of malloc/free calls Debug info of variables, when using --read-var-info=yes Mempool information ... Valgrind can give information about an address Complementary to (sometimes better) than gdb

mempool (most) valgrind tools need to understand memory allocation => such tools are replacing malloc/free Some applications have specific allocators E.g. for small blocks fast allocation + release all blocks in one operation How to “valgrind” these specific allocators ? Either a compile time option to use malloc/free Or describe these specific allocators using 'client requests' (documented in User manual and valgrind.h)

Memcheck leak search heuristics “Possibly lost” : block only found via “inner pointer” (pointing inside the block) Such inner pointers is a common pattern e.g. in C++ library implementation Memcheck has heuristics to detect some patterns : C++ stdstring, C++ multiple inheritance, C++ array of objects length64 (sqlite) Note : these heuristics depends on the ABI

Questions?