1. Efficient Diagnostic Tracing For Wireless Sensor Networks Vinaitheerthan Sundaram Patrick Eugster Xiangyu Zhang ACM SenSys 2010

2. Motivation  Wireless Sensor Network (WSN) deployments  Great Duck Island, MacroScope, Volcano, SensorScope, LOFAR, VigilNet, ExScal, PermaSense  Deployment lessons  Murphy loves WSN deployments!  Low data yield reported (2% to 70%) [Beutel et al.]*  Several deployment failures went unexplained  Post-deployment diagnosis is important but challenging 2 * “Deployment Techniques for Sensor Networks", J Beutel, K Römer, M Ringwald, M woehrle

3. Our Approach: Efficient Diagnostic Tracing  We record control-flow path  Advantages  Many faults manifest as abnormal control-flow  split-phase faults, initialization faults, finite-state machine faults  Basic block level accuracy of preemption information  data races  Effective compression of repetitive computation 3

4. Roadmap  Motivation  Efficient diagnostic tracing  Trace concurrency  Trace control-flow  Run-time compression  Implementation  Evaluation  Related work  Conclusions 4

5. Tracing Concurrency in TinyOS 5  Events (“async”) are interrupts that drive the execution  Tasks run when there are no events  Tasks cannot preempt other tasks or events Time E1E1 E2E2 E1E1 T1T1 T1T1 E1E1 Interrupt level E 1 E 2 E 1 T 1 E 1 T 1 Trace EUnit* EUnit ID Event* $ | ε ID Tid | Eid Event Eid Event* $ | ε Nested! E 1 E 2 $ $ T 1 E 1 $ $

6. Roadmap  Motivation  Efficient diagnostic tracing  Trace concurrency  Trace control-flow  intra-procedural  inter-procedural  Run-time compression  Implementation  Evaluation  Related work  Conclusions 6

7. Tracing Control-Flow: Intra-Procedural 7  Encode n acyclic control-flow paths with an integer in [0, n-1] Entry A: if (p1) B: s1 D: if(p2) Exit C: s2 E: s3 id += 1 id = 0 Output(id) id += 2 Foo() { A. if (p1) B. s1; else C. s2; D. if(p2) E. s3; } Path ABDE ABD ACDE ACD id 0 1 2 3 Foo()

8. Code Instrumentation 8  Ball-Larus (BL) algorithm [Micro ’96]  Optimal encoding and uses minimal instrumentation Entry A: if (p1) B: s1 D: if(p2) Exit C: s2 E: s3 [0,3] [2,3] [0,1] [1] Each node, v, is annotated with the number of paths to the exit Num(v) = ∑ child(i) Num(i) 4 4 2 2 2 1 1 [0,0] Divide encoding space at each node Foo()

9. +1 +2 Code Instrumentation 9  BL algorithm [2] At forks, annotate each edge with the sum of paths contributed by preceding edges Entry A: if (p1) B: s1 D: if(p2) Exit C: s2 E: s3 [0,3] [2,3] [0,1] 4 4 2 2 2 1 1 [0,0] +1 +2 +0 +1 +2 +6 +2 +0 v w x y 2 4 2 8 [0,7] [0,1] [2,5] [6,7] [1] Each node, v, is annotated with the number of paths to the exit Num(v) = ∑ child(i) Num(i) Entry A: if (p1) B: s1 D: if(p2) Exit C: s2 E: s3 Foo()

10. E 1 E 2 2 $ 1 $ T 1 E 1 0 $ 0 $ Tracing Control-Flow: Intra-Procedural 10 Intra-procedural trace Trace EUnit* EUnit ID (Event|Path)* $ | ε ID Tid | Eid Path Pid Event Eid Event* $ | ε Time E1E1 E2E2 E1E1 T1T1 T1T1 E1E1 Interrupt level Entry A: if (p1) B: s1 D: if(p2) Exit C: s2 E: s3 id += 1 Output(label) id = 0 Output(id) Output(‘$’) id += 2 Trace Foo 2 $ Foo()

11. Roadmap  Motivation  Efficient diagnostic tracing  Trace concurrency  Trace control-flow  intra-procedural  inter-procedural  Run-time compression  Implementation  Evaluation  Related work  Conclusions 11

12. Tracing Control-Flow: Inter-Procedural 12  Allow function calls inside events and tasks D E 1 () A B: F() C E G 2 2 1 1 2 1 + 1 F() I J K L M 2 2 1 1 1 + 1  Functions are nested  Treat functions as events or tasks E 1 F 0 $ 1 $ E1FInter- Procedural Path 000 101 012 113

13. Tracing Control-Flow: Inter-Procedural 13 D C E G 1 E 1 () A B: F() 4 4 2 1 1 F() I J K L M +2 2 4 2 2 4  Encode callee’s control-flow within the caller’s control-flow path +1  Inline F() inside E 1 () E 1 2 $  Inlining is expensive E 1 F 0 $ 1 $

14. Inter-Procedural Path: Context-Sensitivity 14 F() I J K L M +1 D E 1 () A B: F() C E G 2 4 1 4 2 1 1 4 2 2 4 +2  Edge increment inside function F depends on the call site 1 2 2 1 1 2 2 1 T 1 () A B: F() C Call Edge

15. Inter-Procedural Summary Analysis 15 n - number of paths in F() x - number of paths to exit after the call site of F() p - path taken in F() at run-time. Note, p ε [0,n-1] …… x Exit S1 …… n Exit Entry F() n*xn*x +p*x+p*x Entry n*xn*x [1] Compute BL individually [2] Adjust at call site  Annotate node with n*x (each of the n paths in the callee can be trailed by one of the x paths in the caller)  Annotate edge with +p*x (skip (p*x -1)inter-procedural paths preceding p) [3] Recompute BL from call site to entry E 1 () [0, n*x-1] x x

16. Inter-Procedural Summary Analysis 16 F() I J K L M 1 n = 2 2 1 1 +1 E 1 () A B: F() n*x=4 4 +1 D C E G 1 x=2 1 1 +(p*2) n - number of paths in F() x - number of paths to exit after the call site of F() p - path taken in F() at run-time inter-procedural path AB 0 CDG AB 0 CEG AB 1 CDG AB 1 CEG id 0 1 2 3 F() path IJKM IJLM id 0 1

17. Tracing Control-Flow: Inter-Procedural 17 Inter-procedural trace Time E1E1 E2E2 E1E1 T1T1 T1T1 E1E1 Interrupt level E 1 E 2 4 $ 3 $ T 1 E 1 4 $ 3 $

18. Roadmap  Motivation  Efficient diagnostic tracing  Trace concurrency  Trace control-flow  Run-time compression  Implementation  Evaluation  Related work  Conclusions 18

19. Run-time Trace Compression 19  WSNs repeat the sequence of tasks or events at run-time  Pattern replacement  Find two most frequent patterns by profiling offline  During online, replace patterns with symbols recursively  Run-length encoding  Loop compression  Run of symbols

20. Roadmap  Motivation  Efficient diagnostic tracing  Implementation  Evaluation  Related work  Conclusions 20

21. Implementation 21  TinyTracer - Automatically instruments the program and generates traces at run-time into flash (radio)  Multiple granularity  Component – SurgeM  Function – SurgeM__Timer__Fired  Tools  CIL, TinyOS 1.x, nesC1.3

22. Roadmap  Motivation  Efficient diagnostic tracing  Implementation  Evaluation  Effectiveness  case studies of common faults  Efficiency  overhead measurements  Related work  Conclusions 22

23. Case Studies of Common Faults 23  Initialization faults  EEPROM component in TinyOS 1.x (previously unknown)  Split-phase faults  High-level data race in LEACH implementation  Failure handling in DeferredPowerManagerM  Task queue overrun  PermaSense [Keller et al., SenSys’09]  CC1000 radio deadlock  State machine implementation faults  VoltageM, EEPROM

24.  Notation  true path = 1, false path = 0  Normal input (N)  E 1 0 $ E 2 0 $ T 1 1 $  Abnormal input (A)  E 1 0 $ E 2 1 $ T 1 0 $  Trace  NNNNNNNNNNANNN  From trace, localize failure to the path taken inside T 1 Hypothetical Bug Case Study 24 State = 0; /*{0, 1} */ Event E 1 () { state = 0; post T1; } Event E 2 (input) { if(input > threshold) state = 1; … } Task T 1 (){ if(state == 0) sendToBase(); else /* bug */ } Failure  Comparison with function call tracing  Normal operation – E 1 E 2 T 1  Abnormal operation – E 1 E 2 T 1

25. Overhead Measurements 25  Metrics  Overheads - energy, RAM, and code size  Trace size  Benchmarks  Standard TinyOS programs  Blink – blinks red LED every second  Oscilloscope – samples light every 1/8 th second  Surge – samples light every 2 seconds  CountToLedsAndRfm – displays and broadcasts counter every 1/4 th second  LRX (Golden Gate bridge monitoring)  A component to send large packets reliably  Test program sends large messages every 2 seconds  About 1000 lines of nesC code  Ran benchmarks for 30 minutes

26. Energy Overhead for Surge 26 Overhead mainly caused by flash Inter-procedural tracing overhead is about 1.3 % for 4 components Inter-procedural tracing consumes less energy than function-call tracing None JoulesJoules 46% 72% 6.4% 2.4% 1.3% 0.9%

27. Memory Overheads 27 Data StructureRAM (in bytes) Flash Pages (16)256 Circular Buffer (2)384 Miscellaneous300 Program memory overhead RAM overhead

28. Trace Size 28  Ran benchmarks for 30 minutes  Shows trace can be compressed well BenchmarksCompressed trace size (bytes/second) Compression ratio (uncompressed/ compressed) Blink 0.221.71 Oscilloscope 2.711.99 Surge 0.314.11 CountToLedsRfm 2.04.625 LRX 51.71.74

29. Roadmap  Motivation  Efficient diagnostic tracing  Implementation  Evaluation  Related work  Conclusions 29

30. Related Work 30  Network faults  Sympathy [Ramanathan et al., SenSys’05], PAD [Liu et al., SenSys ‘08], SNMS [Tolle and Culler, EWSN’05]  Logging  NodeMD [Krunic et al., MobiSys ‘07], LIS [Shea et al., DATE’10], Dustminer [Khan et al., SenSys’08]  Visibility  Marionette [Whitehouse et al., SenSys’06], Clairvoyant [ Yang et al., SenSys’07], Hermes [Kothari et al., IPSN’08]

31. Conclusions 31  We showed the feasibility of program tracing in WSNs  Our contributions  Novel context-free grammar execution encoding  Efficient inter-procedural control-flow path recording  Effective trace compression scheme  Looking ahead  Better compression techniques  Distributed tracing schemes  Variable value tracing

32. Thank you 32 Q & A