1. Lecture XI: Logging and Monitoring CS 4593 Cloud-Oriented Big Data and Software Engineering

2. 2 Logging and Monitoring  Basic and Logging frameworks  Logging and monitoring for web services

3. Basic Tracing Logging Auditing

4. First Logging: System.out.println Easy Quick No dependencies Redirect to file or null in Production Log everything or nothing System.out.println(“Start method1”);

5. Example Start method1 Before retrieve information from DB After retrieve information from DB Updating information Before save information in the DB After save information in the DB Finish method1 Start method2 Before retrieve information from DB Error retrieving information from DB: NullPointerException

6. Next evolution: Logging Libraries Log4j, Java Logging… log.info(“Start method1”); Provide Time of the message The class, the method, the line … Print easily the stacktrace Log to files, databases, system log mechanisms

7. Example 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

8. Example 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

9. Example 06:05:56 07.11.2011 INFO ( Class1.java:211 ) Start method2 06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

10. Example 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 INFO (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

11. Logging levels (I) Log4j Trace Debug Info Warning Error Fatal Java Logging Finest Finer Fine Config Info Warning Severe

12. Example 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 INFO (Class1.java:211) Parameter ‘age’ is not an Integer 06:05:56 07.11.2011 INFO (Class1.java:211) Finish method2 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 DEBUG (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

13. Example 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 INFO (Class1.java:211) Parameter ‘age’ is not an Integer 06:05:56 07.11.2011 INFO (Class1.java:211) Finish method2 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 DEBUG (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

14. Example 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 INFO (Class1.java:211) Parameter ‘age’ is not an Integer 06:05:56 07.11.2011 INFO (Class1.java:211) Finish method2 06:05:56 07.11.2011 INFO (Class1.java:211) Start method2 06:05:56 07.11.2011 DEBUG (Class1.java:220) Before retrieve information from DB 06:05:56 07.11.2011 ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

15. Where to log? Text files (Plain text, XML,…) Binary files Database Syslog (Windows & Unix/Linux) Network (TCP & UDP)

16. Where to log? Text files (Plain text, XML,…) Binary files Database Syslog (Windows & Unix/Linux) Network (TCP & UDP) Always to a Text file (+ something else)?

17. Performance "hidden" cost of parameter construction logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i]));

18. Performance "hidden" cost of parameter construction logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i])); Disable logging with options if (logger.isDebugEnabled()) { logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i])); }

19. Multiple users (I) Different users working simultaneously Log messages mixed for different users Difficult to trace user’s actions

20. Multiple users (I) Different users working simultaneously Log messages mixed for different users Difficult to trace user’s actions Log User ID

21. Multiple users - Example 06:05:55 07.11.2011 INFO (Class1.java:41) User=bfernan; message=‘Start method1’; 06:05:55 07.11.2011 DEBUG (Class1.java:44) message=‘Before retrieve information from DB’; 06:05:55 07.11.2011 DEBUG (Class1.java:46) message=‘After retrieve information from DB’; 06:05:55 07.11.2011 INFO (Class1.java:50) message=‘Updating information’; 06:05:55 07.11.2011 DEBUG (Class1.java:94) message=‘Before save information in the DB’; 06:05:56 07.11.2011 DEBUG (Class1.java:96) message=‘After save information in the DB’; 06:05:56 07.11.2011 INFO (Class1.java:100) User=bfernan; message=‘Finish method1’; 06:05:56 07.11.2011 INFO (Class1.java:211) User=xxxxxxx; message=‘Start method2’; 06:05:56 07.11.2011 DEBUG (Class1.java:220) message=‘Before retrieve information from DB’; 06:05:56 07.11.2011 ERROR (Class1.java:243) message=‘Error retrieving information from DB: NullPointerException’;

22. Multiple users (II) Mapped Diagnostic Contexts (MDC) public void serviceMethod1(String userName,…) { String userName = getLoggedUserName(); MDC.put( “userName”, userName); } Configuration …PatternLayout=%T User=%X{userName};message=‘%m’;%n

23. Multiple users (III) public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest req = (HttpServletRequest) request; Principal principal = req.getUserPrincipal(); // Please note that we could have also used a cookie to // retrieve the user name if (principal != null) { String username = principal.getName(); MDC.put(USER_KEY, username); } try { chain.doFilter(request, response); } finally { MDC.remove(USER_KEY); } }

24. Multiple users (III) public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest req = (HttpServletRequest) request; Principal principal = req.getUserPrincipal(); // Please note that we could have also used a cookie to // retrieve the user name if (principal != null) { String username = principal.getName(); MDC.put(USER_KEY, username); } try { chain.doFilter(request, response); } finally { MDC.remove(USER_KEY); } }

25. Multiple users - Example 06:05:55 07.11.2011 INFO (Class1.java:41) User=bfernan; message=‘Start method1’; 06:05:55 07.11.2011 DEBUG (Class1.java:44) User=bfernan; message=‘Before retrieve information from DB’; 06:05:55 07.11.2011 DEBUG (Class1.java:46) User=bfernan; message=‘After retrieve information from DB’; 06:05:55 07.11.2011 INFO (Class1.java:50) User=bfernan; message=‘Updating information’; 06:05:55 07.11.2011 DEBUG (Class1.java:94) User=bfernan; message=‘Before save information in the DB’; 06:05:56 07.11.2011 DEBUG (Class1.java:96) User=bfernan; message=‘After retrieve information in the DB’; 06:05:56 07.11.2011 INFO (Class1.java:100) User=bfernan; message=‘Finish method1’; 06:05:56 07.11.2011 INFO (Class1.java:211) User=xxxxxxx; message=‘Start method2’; 06:05:56 07.11.2011 DEBUG (Class1.java:220) User=xxxxxxx; message=‘Before retrieve information from DB’; 06:05:56 07.11.2011 ERROR (Class1.java:243) User=xxxxxxx; message=‘Error retrieving information from DB: NullPointerException’ ;

26. Logging for Web Services What to log? Where the logs go to?

27. What to log? Platform Level Time and Memory Stamp and profiling tools Database access Queries and size of result Network incoming and outgoing traffic Requests and size of packets System software usages Parameters

28. What to log? Third Party Services Time stamp Parameters Size of responses

29. Where the logs go to? Log files Using virtual machine Databases Using data store services by the platform Web services Using a web service to show the logs

30. Summary Basic concepts Auditing, logging, and tracing Logging levels Multi-user logging Logging of web services