Lecture XI: Logging and Monitoring CS 4593 Cloud-Oriented Big Data and Software Engineering

2 Logging and Monitoring  Basic and Logging frameworks  Logging and monitoring for web services

Basic Tracing Logging Auditing

First Logging: System.out.println Easy Quick No dependencies Redirect to file or null in Production Log everything or nothing System.out.println(“Start method1”);

Example Start method1 Before retrieve information from DB After retrieve information from DB Updating information Before save information in the DB After save information in the DB Finish method1 Start method2 Before retrieve information from DB Error retrieving information from DB: NullPointerException

Next evolution: Logging Libraries Log4j, Java Logging… log.info(“Start method1”); Provide Time of the message The class, the method, the line … Print easily the stacktrace Log to files, databases, system log mechanisms

Example 06:05: INFO (Class1.java:211) Start method2 06:05: INFO (Class1.java:220) Before retrieve information from DB 06:05: INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Example 06:05: INFO (Class1.java:211) Start method2 06:05: INFO (Class1.java:220) Before retrieve information from DB 06:05: INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Example 06:05: INFO ( Class1.java:211 ) Start method2 06:05: INFO (Class1.java:220) Before retrieve information from DB 06:05: INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Example 06:05: INFO (Class1.java:211) Start method2 06:05: INFO (Class1.java:220) Before retrieve information from DB 06:05: INFO (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Logging levels (I) Log4j Trace Debug Info Warning Error Fatal Java Logging Finest Finer Fine Config Info Warning Severe

Example 06:05: INFO (Class1.java:211) Start method2 06:05: INFO (Class1.java:211) Parameter ‘age’ is not an Integer 06:05: INFO (Class1.java:211) Finish method2 06:05: INFO (Class1.java:211) Start method2 06:05: DEBUG (Class1.java:220) Before retrieve information from DB 06:05: ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Example 06:05: INFO (Class1.java:211) Start method2 06:05: INFO (Class1.java:211) Parameter ‘age’ is not an Integer 06:05: INFO (Class1.java:211) Finish method2 06:05: INFO (Class1.java:211) Start method2 06:05: DEBUG (Class1.java:220) Before retrieve information from DB 06:05: ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Example 06:05: INFO (Class1.java:211) Start method2 06:05: INFO (Class1.java:211) Parameter ‘age’ is not an Integer 06:05: INFO (Class1.java:211) Finish method2 06:05: INFO (Class1.java:211) Start method2 06:05: DEBUG (Class1.java:220) Before retrieve information from DB 06:05: ERROR (Class1.java:243) Error retrieving information from DB: NullPointerException java.lang.NullPointerException at pkg.Class1.method2(Class1.java:225) at pkg.Class1.methodN(Class1.java:458) at pkg2.ClassZ.methodUpdate(ClassZ.java:458) at … at java.lang.Thread.run(Thread.java:595)

Where to log? Text files (Plain text, XML,…) Binary files Database Syslog (Windows & Unix/Linux) Network (TCP & UDP)

Where to log? Text files (Plain text, XML,…) Binary files Database Syslog (Windows & Unix/Linux) Network (TCP & UDP) Always to a Text file (+ something else)?

Performance "hidden" cost of parameter construction logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i]));

Performance "hidden" cost of parameter construction logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i])); Disable logging with options if (logger.isDebugEnabled()) { logger.debug("Entry number: " + i + " is " + String.valueOf(entry[i])); }

Multiple users (I) Different users working simultaneously Log messages mixed for different users Difficult to trace user’s actions

Multiple users (I) Different users working simultaneously Log messages mixed for different users Difficult to trace user’s actions Log User ID

Multiple users - Example 06:05: INFO (Class1.java:41) User=bfernan; message=‘Start method1’; 06:05: DEBUG (Class1.java:44) message=‘Before retrieve information from DB’; 06:05: DEBUG (Class1.java:46) message=‘After retrieve information from DB’; 06:05: INFO (Class1.java:50) message=‘Updating information’; 06:05: DEBUG (Class1.java:94) message=‘Before save information in the DB’; 06:05: DEBUG (Class1.java:96) message=‘After save information in the DB’; 06:05: INFO (Class1.java:100) User=bfernan; message=‘Finish method1’; 06:05: INFO (Class1.java:211) User=xxxxxxx; message=‘Start method2’; 06:05: DEBUG (Class1.java:220) message=‘Before retrieve information from DB’; 06:05: ERROR (Class1.java:243) message=‘Error retrieving information from DB: NullPointerException’;

Multiple users (II) Mapped Diagnostic Contexts (MDC) public void serviceMethod1(String userName,…) { String userName = getLoggedUserName(); MDC.put( “userName”, userName); } Configuration …PatternLayout=%T User=%X{userName};message=‘%m’;%n

Multiple users (III) public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest req = (HttpServletRequest) request; Principal principal = req.getUserPrincipal(); // Please note that we could have also used a cookie to // retrieve the user name if (principal != null) { String username = principal.getName(); MDC.put(USER_KEY, username); } try { chain.doFilter(request, response); } finally { MDC.remove(USER_KEY); } }

Multiple users (III) public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest req = (HttpServletRequest) request; Principal principal = req.getUserPrincipal(); // Please note that we could have also used a cookie to // retrieve the user name if (principal != null) { String username = principal.getName(); MDC.put(USER_KEY, username); } try { chain.doFilter(request, response); } finally { MDC.remove(USER_KEY); } }

Multiple users - Example 06:05: INFO (Class1.java:41) User=bfernan; message=‘Start method1’; 06:05: DEBUG (Class1.java:44) User=bfernan; message=‘Before retrieve information from DB’; 06:05: DEBUG (Class1.java:46) User=bfernan; message=‘After retrieve information from DB’; 06:05: INFO (Class1.java:50) User=bfernan; message=‘Updating information’; 06:05: DEBUG (Class1.java:94) User=bfernan; message=‘Before save information in the DB’; 06:05: DEBUG (Class1.java:96) User=bfernan; message=‘After retrieve information in the DB’; 06:05: INFO (Class1.java:100) User=bfernan; message=‘Finish method1’; 06:05: INFO (Class1.java:211) User=xxxxxxx; message=‘Start method2’; 06:05: DEBUG (Class1.java:220) User=xxxxxxx; message=‘Before retrieve information from DB’; 06:05: ERROR (Class1.java:243) User=xxxxxxx; message=‘Error retrieving information from DB: NullPointerException’ ;

Logging for Web Services What to log? Where the logs go to?

What to log? Platform Level Time and Memory Stamp and profiling tools Database access Queries and size of result Network incoming and outgoing traffic Requests and size of packets System software usages Parameters

What to log? Third Party Services Time stamp Parameters Size of responses

Where the logs go to? Log files Using virtual machine Databases Using data store services by the platform Web services Using a web service to show the logs

Summary Basic concepts Auditing, logging, and tracing Logging levels Multi-user logging Logging of web services