4 December 2001 SEESCOASEESCOA STWW - Programma Debugging of Real-Time Embedded Systems: Experiences from SEESCOA Michiel Ronsse RUG-ELIS

SEESCOASEESCOA Problem description  embedded systems have become increasingly sophisticated  debugging becomes a major problem  types of bugs  computational errors  synchronisation errors  performance errors  testing and debugging is responsible for a huge part of the development time   definitive need for appropriate debugging tools

SEESCOASEESCOA Debugging embedded systems  two test/debug phases  during the development  during the actual use  an embedded system  typically runs continuously  interactive debugging?  can have hard and soft real-time constraints  Heisenbugs  parallel and/or distributed systems: absence of a global clock  uses lots of input  repeatability?  cyclic debugging?  hardware related problems: code in ROM, limited amount of RAM  Solution: hardware add-ons for on-chip debugging techniques

SEESCOASEESCOA On-chip debugging techniques (I)  Logic Analyser  ROM monitor  ROM emulator  In-Circuit Emulator  Background Debug Mode  JTAG

SEESCOASEESCOA ualmost all contemporary embedded processors have add-ons for debugging uas embedded processors get faster, these add-ons get closer to (or in) the processor uthese add-ons take up valuable chip area (up to 10%) uthese add-ons are also available in the consumer produsts, making on-site testing possible u  hardware manufacturers believe in design for debugability On-chip debugging techniques (II)

SEESCOASEESCOA Embedded software uNowadays, software engineering methods designed for `business’ applications are also used for embedded systems: wUse of `higher’ languages (C, C++, Java) wUse of reusable components wMultithreaded applications w... uDebugging/maintaining these complex applications becomes a difficult task Design for Debugability

SEESCOASEESCOA Debugging software uMost important notion: wRepeatability  cyclic debugging wObservability  what is going on? uThese are two problems for embedded systems: wNon-determinism present in most embedded systems wLow observability (real-time constraints,...)

SEESCOASEESCOA Causes of non-determinism  Sequential programs: input (keyboard, disk, network, certain system calls (e.g. gettimeofday (), …) uParallel programs: race conditions: wtwo threads waccessing the same shared variable (memory location) win an unsynchronised way wand at least one thread modifies the variable

SEESCOASEESCOA Execution Replay uGoal: make repeated equivalent re-exections possible uMethod: two phases wRecord phase: record all non-deterministic events during an execution in a trace file wReplay phase: use trace file to produce the same execution uQuestion: what & where to trace?

SEESCOASEESCOA Example code #include unsigned global=5; thread2(){ global=global+6; } thread3(){ global=global+7; } main(){ pthread_t t2,t3; pthread_create(&t2, NULL, thread1, NULL); pthread_create(&t3, NULL, thread2, NULL); pthread_join(t2, NULL); pthread_join(t3, NULL); printf(“global=%d\n”, global); }

SEESCOASEESCOA Possible executions L(5) global=12 global=18global=11 L(5) L(11) S(11) S(12) S(11) S(12) S(11) S(18) A A A A A A

SEESCOASEESCOA Example code II #include unsigned global=5; thread2(){lock(); global=global+6; unlock();} thread3(){lock(); global=global+7; unlock();} main(){ pthread_t t2,t3; pthread_create(&t2, NULL, thread1, NULL); pthread_create(&t3, NULL, thread2, NULL); pthread_join(t2, NULL); pthread_join(t3, NULL); printf(“global=%d\n”, global); }

SEESCOASEESCOA Possible executions II global=18 L(5) L(11) S(11) S(18) A A global=18 L(5) L(11) S(11) S(18) A A

SEESCOASEESCOA Race conditions uThree types of conflicts: wload/store wstore/load wstore/store uTwo types: wsynchronisation races:  doesn’t allow the use of cyclic debugging techniques  is not a bug, is desired non-determinism wdata races:  doesn’t allow the use of cyclic debugging techniques  is a bug, is undesired non-determinism

SEESCOASEESCOA Tracing all memory operations Introduces an intolerable overhead, both in time and space Due to out-of-order stores, the order in which threads see events can differ Will replay both synchronization and data races.

SEESCOASEESCOA Tracing synchronization operations The only events that are assured to be seen in the same order are synchronisation operations Synchronization operations are a subset of all memory operations => lower overhead Will replay only synchronization races. Will fail if data races occur. Solution Record: trace synchronization operations Replay: check for data races (once)

SEESCOASEESCOA Record phase Main: 1,2,3,4,5,13,14,... T2: 3,6,7,8,9,... T3: 4,5,6,8,10,11,12,...

SEESCOASEESCOA Replay phase Main: 1,2,3,4,5,13,14,... T2: 3,6,7,8,9,... T3: 4,5,6,8,10,11,12,...

SEESCOASEESCOA Implementation uRecPlay for Solaris (SPARC) and Linux (x86) uUses JiTI for dynamic instrumentation uRecord overhead: 1.6%

SEESCOASEESCOA Instrumenting Java classes

SEESCOASEESCOA The RecPlay system for Java

SEESCOASEESCOA Current Situation uRecord/replay for synchronization operations works uTo be added: wRace detection wTracing of input