1. Log20: Fully Automated Optimal Placement of Log Printing Statements under Specified Overhead Threshold
Xu Zhao*, Kirk Rodrigues*, Yu Luo*, Michael Stumm*, Ding Yuan*, Yuanyuan Zhou†
University of Toronto*
University of California, San Diego †

2. Logs are critical if (current_value > max) {
LOG.info("Update max value to: " + max);
max = current_value; }
Logs are often the only information available in post-mortem debugging
Developers rely on logs to reconstruct execution paths 2

3. Where to log?
Find the most informative placement of log printing statements
Under a specified overhead threshold, e.g. no more than 4 log messages 1 2 3 4 5 6 7 8 9 10 11
int max3(uint a[]) {
if (a.length != 3)
return ERROR;
uint max = 0, i = 0;
while (i < 3) {
if (a[i] > max)
max = a[i];
++i; }
return max; 3

4. Writing log printing code is hard
As a deadline approaches, developers focus on functionality code
Developers need to anticipate the failure
Hard to find the balance between informativeness and overhead 4

5. Total log modifying revisions
Revision history of log printing statements
Studied three distributed systems
42% existent logging statements are modified at least once
Change rate of logging code is 1.8x higher than other code [Yuan et al. ICSE’12]
System
Total logging stmt.
% Modified
Total log modifying revisions
Hadoop
9125
42%
12158
HBase
4644
14039
ZooKeeper
1094
41%
2624
Total
14863
28821 5

6. Our contributions
A metric, entropy, to measure the informativeness of logging statement
An algorithm to find the most informative logging placement under budget
A system, Log20, that allows developers to find the right balance 6

7. Disambiguating paths with log output
Placement
Disambiguating paths with log output
<6, 8>
Placement P1 P2 P3 P4 P5 1 2 3 4 5
int max3(uint a[]) {
if (a.length != 3)
return ERROR;
uint max = 0, i = 0;
while (i < 3) {
Input []
[0 0 0]
[0 1 0]
[1 2 3]
[3 2 1] 3 2 2 2 2 2 5 5 5 5 ⑥ 6 ⑥ 6 ⑥ 6 7 ⑥ 6
LOG.debug("Loop start"); 7 ⑧ 8 ⑧ 8 ⑧ 8 ⑧ 8 6 7
if (a[i] > max)
max = a[i]; 5 5 5 5 ⑥ 6 ⑥ 6 ⑥ 6 ⑥ 6 7 7 8 9 10 11
++i; }
return max;
LOG.debug("Loop end"); ⑧ 8 ⑧ 8 ⑧ 8 ⑧ 8 5 5 5 5 ⑥ 6 ⑥ 6 ⑥ 6 ⑥ 6 7 ⑧ 8 ⑧ 8 ⑧ 8 ⑧ 8 5 5 5 5 10 10 10 10
Log output []
[686868]
[686868]
[686868]
[686868] 7

8. Disambiguating paths with log output
Placement
Disambiguating paths with log output
<6, 8>
Placement 1 2 3 4 5
int max3(uint a[]) {
if (a.length != 3)
return ERROR;
uint max = 0, i = 0;
while (i < 3) { P1 P2 P3 P4 P5 3 2 2 2 2 2 5 5 5 5 ⑥ ⑥ ⑥ ⑥ 7 7 ⑧ ⑧ ⑧ ⑧
LOG.debug("Loop start"); 5 5 5 5 ⑥ 6 7
if (a[i] > max)
max = a[i]; ⑥ ⑥ 7 7 ⑥ ⑧ ⑧ ⑧ ⑧ 5 5 5 5
LOG.debug("Loop end"); ⑥ ⑥ ⑥ ⑥ 7 8 9 10 11
++i; }
return max; ⑧ ⑧ ⑧ ⑧ 5 5 5 5 10 10 10 10
Log output []
[686868]
[686868]
[686868]
[686868]
The path partition of <6, 8> is {P1}, {P2, P3, P4, P5} 8

9. Comparing log placement strategies
Placement A P1 P2 P3 P4 P5
Placement B 9

10. Comparing log placement strategies
Placement A P1 P2 P3 P4 P5
Placement B 10

11. 𝐻 𝑋 =− 𝑥∈𝑋 𝑝 𝑥 log 2 𝑝 𝑥 Entropy 𝐻 𝑋 : the metric of informativeness
𝑋: all program execution paths observed
𝑥 : an execution path in 𝑋
𝑝 𝑥 : the probability of 𝑥 occurring in 𝑋 11

12. Calculate the entropy 𝐻=0 2 𝐻 =1 𝐻=2.32 2 𝐻 =5 𝐻=0.65 2 𝐻 =1.5 P1 P2
Probability
20% 5% 2% 1%
90% 0%
100%
20%
20%
20%
20% 1 2 3 4 5 6 7 8 9 10 11
int max3(uint a[]) {
if (a.length != 3)
return ERROR;
uint max = 0, i = 0;
while (i < 3) {
if(a[i] > max)
max = a[i];
++i; }
return max;
Input []
[0 0 0]
[0 1 0]
[1 2 3]
[3 2 1] 2 3 2 2 2 2 5 5 5 5 6 6 6 6 7 7 8 8 8 8 5 5 5 5 6 6 6 7 7 6 8 8 8 8 5 5 5 5 6 6 6 6 7 8 8 8 8 5 5 5 5 10 10 10 10
𝐻=0
2 𝐻 =1
𝐻=2.32
2 𝐻 =5
𝐻=0.65
2 𝐻 =1.5 12

13. Calculate the entropy of placement
𝐻 =−50%×log50%−50%×log50%=1.0 P1 P2 P1 P2 P3 P5 P4 2 3 2 2 2 2
𝐻 7 =−50%×log50%−50%×log50%=1.0 P3 P5 5 5 5 5 6 6 6 6 ⑦ ⑦ 8 8 8 8 5 5 5 5
𝐻 =−100%×log 1.0=0.0 P4 6 6 6 ⑦ 6 ⑦ 8 8 8 8 5 5 5 5
𝐻 [7] =40%×𝐻 %×𝐻 %×𝐻 =0.8 6 6 6 6 ⑦ 8 8 8 8 5 5 5 5 10 10 10 10
Log output
[ ]
[ ]
[7]
[7]
[777]
2 𝐻([7]) =1.74 13 P1

14. Estimate the overhead of placement
𝑂 =0.0 2 3 2 2 2 2 5 5 5 5
𝑂 7 =1.0 6 6 6 6 ⑦ ⑦ 8 8 8 8 5 5 5 5
𝑂 =3.0 6 6 6 ⑦ 6 ⑦ 8 8 8 8 5 5 5 5
𝑂 [7] =40%×𝑂 %×𝑂 %×𝑂 =1.0 (logs per path) 6 6 6 6 ⑦ 8 8 8 8 5 5 5 5 10 10 10 10
Log output
[ ]
[ ]
[7]
[7]
[777] 14

15. Entropy-overhead tradeoff
Threshold = 0.5
Log20 generated placement:
<3> P1 P2 P3 P4 P5 2 2 2 2 2 ③ 5 5 5 5 6 6 6 6 7 7 8 8 8 8 2
Entropy = 5 5 5 5 6 6 6 7 7 6
0.2
Overhead = 8 8 8 8 5 5 5 5 6 6 6 6 7 8 8 8 8 5 5 5 5 10 10 10 10
Log output
[3] [] [] [] [] 15

16. Entropy-overhead tradeoff
Threshold = 1
Log20 generated placement:
<7> P1 P2 P3 P5 P4 2 3 2 2 2 2 5 5 5 5 6 6 6 6 ⑦ ⑦ 8 8 8 8
0.8
Entropy = 5 5 5 5 6 6 6 ⑦ 6 ⑦ 8 8 8 1
Overhead = 8 5 5 5 5 6 6 6 6 ⑦ 8 8 8 8 5 5 5 5 10 10 10 10
Log output
[ ]
[ ]
[7]
[7]
[777] 16

17. Entropy-overhead tradeoff
Threshold = 4
<6, 7>
Log20 generated placement: P1 P2 P3 P4 P5 3 2 2 2 2 2
Entropy<6,7> = 5 5 5 5 ⑥ ⑥ ⑥ ⑥ ⑦ ⑦
3.4
Overhead<6,7> = 8 8 8 8 5 5 5 5 ⑥ ⑥ ⑥ ⑥ ⑦ ⑦ 8 8 8 8
Log20 will never generate <6, 8> 5 5 5 5 ⑥ ⑥ ⑥ ⑥ ⑦ 2
Entropy<6,8> = 8 8 8 8 5 5 5 5
3.6
Overhead<6,8> = 10 10 10 10
Log output []
[666]
[6676]
[676767]
[6766] 17

18. Collecting execution path profiles
High performance tracing library
Sample production workload to collect path profile
Logging placement is optimized for a particular workload pattern
Every invocation to the library takes 27ns
Manually provided request starting and ending points
Log all functions not covered by the traces 18

19. Implementation Dynamic programming algorithm for log placement
Because a brute-force placement algorithm has complexity O(2N)
Logging variable values 1 2 3 4 5
LOG.info("Request start");
if (host == "local") {
LOG.info("Running on localhost"); … } 1 2 3 4
LOG.info("Request start. localhost = " + (host=="local"));
if (host == "local") { … } 19

20. Evaluation Understand trade-off between entropy reduction and overhead
Compare with developers’ manual placement
Help debug real world failures
Compare against Ball-Larus path profiling [Ball & Larus MICRO’96]
Evaluate how well Log20 approximates optimal placement
Performance of tracing libraries
Checkout the paper! 20

21. Evaluation Understand trade-off between informativeness and overhead
For HDFS, Log20 uses 90% less logs to achieve the same entropy
Similar result for HBase, YARN, and ZooKeeper
Entropy = 3.8
Log20 overhead = 0.2
Existing log overhead = 3.7 21

22. Usefulness in debugging
We study 42 HDFS failures, compared with manual logging 3 4 24
Failures debuggable using logs
Existing log printing statements are helpful
Log20 is helpful 22

23. Usefulness in debugging
HDFS-4328: freezes in shutdown because of large block scanning
Log20 helps by logging the following two basic blocks:
private boolean isLongRead() {
if((endOffset - initialOffset) > LONG_READ)
return true;
return false; }
Block Receiver Thread
Log20.log();
Waiting
while (…) { try { wait (block_receiver_thread); }
catch (InterruptedException ignored) { }
Block Scanner Thread
Log20.log(); 23

24. Limitations Obsolete verbosity level
Need time to adapt to the workload change 24

25. Related work Where to log What to log
Ball-Larus profiling has to consider all possible paths [Ball & Larus. MICRO’96]
Errlog only automates error log placement [Yuan et al. OSDI’12]
What to log
LogEnhancer collects variable values automatically [Yuan et al. ASPLOS’11] 25

26. Conclusion
Log20: automated logging statement placement under specified overhead
Guided by information theory: entropy P1 P2 P3 P4 P5 2 5 10
[777]
[7] []
<7>
Placement 3 8 6 ⑦
0.8
Entropy = P3 P4 P5 [] P1
[3] 2 P2
<3>
Placement ③ 5 10 7 8 6
Entropy = 26

27. “The Practice of Programming” by Brian Kernighan and Rob Pike
“As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.”
“The Practice of Programming” by Brian Kernighan and Rob Pike