Improve Embedded System Stability and Performance through Memory Analysis Tools Bill Graham, Product Line Manager Development Tools November 14, 2006

2 out of 48 All content copyright of QNX Software Systems Co. Agenda 1. Introduction 2. Memory Analysis Primer 3. Memory-related Error Detection 4. Memory Leak Detection 5. Memory Profiling 6. Memory Optimization 7. QNX Momentics Development Suite 8. Q&A

Introduction – The role of an OS

4 out of 48 All content copyright of QNX Software Systems Co. Memory-related Challenges  Memory leaks, errors and other crash-causing problems can have catastrophic effects on other processes and threads ► Undetected memory leaks can lead to system crashes ► Sub-optimal memory allocation can degrade performance ► Excessive memory usage can inflate costs ► Memory errors can cause failures in the field

5 out of 48 All content copyright of QNX Software Systems Co. The Role of OS Architecture  First and last line of defense is a robust operating system architecture  Memory protection and microkernel architecture can protect the OS and applications from each other

6 out of 48 All content copyright of QNX Software Systems Co. Microkernel Architecture Realtime Executive (VxWorks, proprietary) >No MMU and no protection >Applications, drivers, etc. are all in kernel space Monolithic Kernel (XP, Unix, Linux, etc.) >MMU with partial protection >Applications are protected True Microkernel (QNX ® Neutrino ® RTOS ) >MMU with full protection >Applications, drivers, and protocols are all protected Device DriversTCP/IP StackFile System Application Kernel space Device DriversTCP/IP Stack File System Application Kernel Device DriversTCP/IP StackFile System Application Kernel space User Space  System-wide corruption

7 out of 48 All content copyright of QNX Software Systems Co. Microkernel Architecture Realtime Executive (VxWorks, proprietary) >No MMU and no protection >Applications, drivers, etc. are all in kernel space Monolithic Kernel (XP, Unix, Linux, etc.) >MMU with partial protection >Applications are protected True Microkernel (QNX ® Neutrino ® RTOS ) >MMU with full protection >Applications, drivers, and protocols are all protected Device DriversTCP/IP StackFile System Application Kernel space Device DriversTCP/IP Stack File System Application Kernel Device DriversTCP/IP StackFile System Application Kernel space User Space System-wide corruption    Contained (re-startable) System-wide corruption

8 out of 48 All content copyright of QNX Software Systems Co. Microkernel Architecture Realtime Executive (VxWorks, proprietary) >No MMU and no protection >Applications, drivers, etc. are all in kernel space Monolithic Kernel (XP, Unix, Linux, etc.) >MMU with partial protection >Applications are protected True Microkernel (QNX ® Neutrino ® RTOS ) >MMU with full protection >Applications, drivers, and protocols are all protected Device DriversTCP/IP StackFile System Application Kernel space Device DriversTCP/IP Stack File System Application Kernel Device DriversTCP/IP StackFile System Application Kernel space User Space System-wide corruption    Contained (re-startable)   System-wide corruption Contained (re-startable)

9 out of 48 All content copyright of QNX Software Systems Co. Microkernel Architecture File System Protocol Stack Microkernel Application Microkernel the only trusted component Audio Driver Graphics Driver Applications and drivers:  Processes that plug into a message bus  Reside in memory-protected address space  Cannot corrupt other software components  Can be started, stopped, and upgraded dynamically  Critical for survivability Message Bus …

Memory Analysis Primer

11 out of 48 All content copyright of QNX Software Systems Co. Target Management  Embedded systems need to be monitored, debugged and controlled remotely ► Much of the complexity and difficulty with embedded development is limited insight into the operation of the target system

12 out of 48 All content copyright of QNX Software Systems Co. Run-time Memory Analysis malloc() free() strcpy() … Debug Library Application  Memory analysis works through trapping memory-related API calls

13 out of 48 All content copyright of QNX Software Systems Co. Memory Analysis  Observe ► See time history of all allocations and deallocations ► Catch run-time errors ► Monitor memory leaks  Correct ► Back trace to source line of code – gives you direct place to fix the problem ► Remove leaks to increase stability  Profile ► See history of allocations and deallocations, peak usage, average usage, overhead  Optimize ► Match allocation distribution to memory allocation algorithm ► Reduce peak usage to save RAM cost

Memory-related Error Detection

15 out of 48 All content copyright of QNX Software Systems Co. Runtime Memory Error Detection  Certain C library calls (which cause system crashes) are intercepted ► Perform sanity check on parameters  e.g. strcpy(), memcpy()  Memory allocation functions are intercepted and recorded ► Perform a sanity check on pointer values  e.g. malloc(), calloc(), free(), etc  If these methods don’t trap the error and the application crashes, a core file is created ► Use for postmortem debugging

16 out of 48 All content copyright of QNX Software Systems Co. Run-time Memory Analysis malloc() strcpy() free() Debug Library str = (char *) malloc(10); strcpy(str,"AbCdEfGhIjK"); strcpy(mystring,str); free(str); Intercepted CallsApplication

17 out of 48 All content copyright of QNX Software Systems Co. Report Errors Error Back trace Source view

18 out of 48 All content copyright of QNX Software Systems Co. Postmortem Debugging Core file session Source view Memory/Variables Outline

Memory Leak Detection

20 out of 48 All content copyright of QNX Software Systems Co. Memory Leaks  Memory leaks are cases of missing deallocations for allocations  Memory allocations that are never freed are tagged as memory leaks  Rapid leaks are usually found early enough but slow leaks can lead to system instability and failure in the field  Examples: ► 128 byte leak every 500ms for 1 year = 7.7 MB ► 128 byte leak every 20s for 1 year = 197 KB ► Devices expected to run for very long periods in the field will eventually fail inexplicably

21 out of 48 All content copyright of QNX Software Systems Co. Memory Leaks Allocation/ deallocation details Distribution: time-order relationship Time selection

22 out of 48 All content copyright of QNX Software Systems Co. Memory Leaks – Details  Allocations/deallocations can be readily traced to code  Leaks are detected when there is no pointer reference to allocated memory  Potential memory leaks can still exist as outstanding traces Details with back trace

23 out of 48 All content copyright of QNX Software Systems Co. Memory Leaks – Details Details with back trace Source view

Memory Profiling

25 out of 48 All content copyright of QNX Software Systems Co. Memory Profiling  Memory usage can be observed over time ► Peak memory usage ► Distribution of allocation size ► Time / memory usage relationship ► Allocation algorithm overhead  Visual reference for memory usage can give more meaningful data ► Optimization based on observation

26 out of 48 All content copyright of QNX Software Systems Co. Memory Profiling Usage: allocated, free, overhead Usage: time- order relationships Time selection Time playback

27 out of 48 All content copyright of QNX Software Systems Co. Statistics Types of calls and number of requests Allocation size distribution

28 out of 48 All content copyright of QNX Software Systems Co. Allocation Size Distribution Histogram Distribution: time relationship Time selection Time playback

Memory Optimization

30 out of 48 All content copyright of QNX Software Systems Co. Memory Optimization  Two benefits of optimization work ► Reduce memory usage or address peak demands ► Ensure allocation algorithm matches usage pattern  Optimization is fed by observations from memory analysis  Results can be evaluated through same tools

31 out of 48 All content copyright of QNX Software Systems Co. Application Scenario  Program that creates a linked list of large data items ► List is used for a period of time then all elements are deleted  Program that used copy and forward for data packets ► There is new or malloc() for each copy  Program that has many class constructions ► Classes with large payload maybe used wastefully throughout program ► Developers may not understand or know about memory payload

32 out of 48 All content copyright of QNX Software Systems Co. Peak Usage & Memory Load Observe total usage and overhead over time Observe spikes in allocation/deallocation Observe trend in overhead

33 out of 48 All content copyright of QNX Software Systems Co. Peak Usage & Memory Load  Here is an example of a program does large free() of data structure ► Investigation would reveal: was this necessary? Was this a copy and forward scenario?

34 out of 48 All content copyright of QNX Software Systems Co. Memory Allocation Schemes  Most schemes break up available heap space into fixed block sizes  Allocation requests are distributed among the blocks  114 bytes = 24 bytes + 48 bytes + 48 bytes ( 120 bytes) ► Overhead for this allocation – 6 bytes (roughly 5%) bytes Overhead – extra memory used per allocation

35 out of 48 All content copyright of QNX Software Systems Co. Memory Allocation Distribution Observe pattern of allocations Observe time trends in usage

36 out of 48 All content copyright of QNX Software Systems Co. Memory Allocation Bands Observe pattern of allocations versus malloc() band implementation Observe time trends in band usage

37 out of 48 All content copyright of QNX Software Systems Co. Allocation Band Mismatch  Distribution of allocation size versus memory allocation block size (band) can be compared  Mismatch between band and allocation distribution can increase overhead  Memory allocation scheme can be tuned on a per- application basis

QNX ® Momentics ® Development Suite

39 out of 48 All content copyright of QNX Software Systems Co. QNX ® Momentics ® at a Glance

40 out of 48 All content copyright of QNX Software Systems Co. Built on the Eclipse Platform

41 out of 48 All content copyright of QNX Software Systems Co. Summary  Memory analysis is a key function in ensuring the quality of embedded systems  Developer productivity is enhanced when easy-to-use visual tools are provided  QNX Momentics Development Suite provides the tools needed to observe, correct, profile and optimize memory usage

Thank you! Bill Graham, Product Line Manager Development Tools Questions and Answers