1. Manage Tomcat in production environments INFN-CNAF 13/11/09 Michele Orrù Network&Security Lead @ IntegratingWeb 1

2. Outline Virtual Hosting Motivation Two-tiers with Apache Different JVMs Performance Motivation Testing with Jmeter Connectors, Heap, GC and APR Monitoring Motivation JMX and Hyperic 2 Outline

3. Virtual Hosting 3 Virtual Hosting: motivation Name-based virtual hosting: multiple FQDN share the same IP address Cheap solution Hundreds of domains hosted on the same machine IP-based virtual hosting can become a nightmare Consolidate web application monitoring

4. Virtual Hosting 4 Virtual Hosting: Implementation New directory structure:

5. Tomcat configuration 5 Virtual Hosting: Implementation <Host name="www.integratingweb.com" debug="0" appBase="/home/tomcat/sites/integratingweb.com/webapps" unpackWARs="true”> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="/home/tomcat/sites/integratingweb.com/logs" prefix="integratingweb.com.access." suffix=".log" pattern="common"/>... In conf/server.xml:

6. Tomcat configuration 6 Virtual Hosting: Implementation # Set the root logger for Tomcat # log4j.rootLogger=INFO, TomcatINFO, TomcatERROR # Send all INFO messages and above to a file # log4j.appender.TomcatINFO=org.apache.log4j.FileAppender log4j.appender.TomcatINFO.File=/home/tomcat/logs/virtual_host_catalina_.log # Use the simple layout log4j.appender.TomcatINFO.layout=org.apache.log4j.SimpleLayout # Send all ERROR messages and above to the console # log4j.appender.TomcatERROR=org.apache.log4j.ConsoleAppender log4j.appender.TomcatERROR.Target=System.out log4j.appender.TomcatERROR.layout=org.apache.log4j.PatternLayout log4j.appender.TomcatERROR.layout.ConversionPattern=%p: %m: %d{ABSOLUTE} %n log4j.appender.TomcatERROR.Threshold=ERROR # Define a log for the integratingweb.com host # log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[www.integratingweb.com]=INFO,integratingweb # Log to a file log4j.appender.integratingweb=org.apache.log4j.FileAppender log4j.appender.integratingweb.File=/home/tomcat/sites/integratingweb.com/logs/integratingweb.com.log # Use the simple layout log4j.appender.integratingweb.layout=org.apache.log4j.SimpleLayout # repeat last 4 lines for every virtual host In conf/logging.properties:

7. Virtual Hosting 7 Virtual Hosting: Two-tiers with Apache

8. Apache configuration 8 Virtual Hosting: Two-tiers with Apache ServerAdmin webmaster@barbarabottini.com ServerName www.barbarabottini.com ServerAlias www.barbarabottini.com ProxyRequests Off ProxyPreserveHost On Order deny,allow Allow from all ProxyPass / ajp://localhost:8009/ ErrorLog logs/www.barbarabottini.com-error_log CustomLog logs/www.barbarabottini.com-access_log common Be sure that the following module is loaded in httpd.conf: LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

9. Per-Virtual Host JVM 9 Virtual Hosting: per-virtual host JVM Why? Security: if one application crashes, consumes resources, is being hacked, the problem is confined in his dedicated JVM without affecting others Large-scale Tomcat hosting: pay for JVM Heap size Per-virtual host JVMs must be supported by good hardware (multi-core, lot of RAM)

10. Per-Virtual Host JVM 10 Virtual Hosting: per-virtual host JVM Tomcat depends on two environment variables: CATALINA_HOME: to find internal classes and libraries CATALINA_BASE: configuration-specific files, webapps, logs. If not defined, default to CATALINA_HOME What we do here is to create N directories (one per virtual host) that will be registered as CATALINA_BASE(s) for our virtual hosts. Every CATALINA_BASE will have her own server.xml, with specific Connectors and Contexts

11. Per-Virtual Host JVM 11 Virtual Hosting: motivation

12. Per-Virtual Host JVM 12 Virtual Hosting: motivation Startup script for Tomcat mass-virtual hosting #!/bin/bash CATALINA_HOME="/home/tomcat" SITE_ROOT="/home/jvms" SITES=`ls ${SITE_ROOT}` for host in ${SITES} do export JAVA_OPTS CATALINA_BASE=${SITE_ROOT}/${host}/catalina echo “Starting server: ${host} with CATALINA_BASE=${CATALINA_BASE}” export CATALINA_BASE ${CATALINA_HOME}/bin/startup.sh done

13. Performance 13 Performance: motivation User’s perspective: how fast the content is served to the client We want to measure: – Response time – Throughput – Number of users that the application should support

14. Apache Jmeter 14 Performance: Testing with Jmeter Let see a demo…

15. APR 15 Performance: APR Apache Portable Runtime can be used to improve scalability, stability and performance Uses Java Native Interface (JNI) wrappers for APR (libtcnative) Powerful when using Tomcat as a webserver (OS syscalls such as sendfile(), epoll(), native process handling, OpenSSL instead of Java SSL) without Apache $tar xvzf bin/tomcat-native-1.1.16-src.tar.gz $cd bin/tomcat-native-1.1.16-src/jni && ant $cd bin/tomcat-native-1.1.16-src/jni/native $./configure --with-apr=/usr/local/apache2/bin/apr-1-config $make #make install $export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/apr/lib

16. APR 16 Performance: APR Restarting Tomcat, having defined Connectors such as: The following output will be present in the logs: Nov 12, 2009 1:40:04 PM org.apache.coyote.http11.Http11AprProtocol init INFO: Initializing Coyote HTTP/1.1 on http-9080 Nov 12, 2009 1:40:04 PM org.apache.coyote.ajp.AjpAprProtocol init INFO: Initializing Coyote AJP/1.3 on ajp-9099 Nov 12, 2009 1:40:04 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 723 ms Nov 12, 2009 1:40:04 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Nov 12, 2009 1:40:04 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.20 Nov 12, 2009 1:40:19 PM org.apache.coyote.http11.Http11AprProtocol start INFO: Starting Coyote HTTP/1.1 on http-9080 Nov 12, 2009 1:40:19 PM org.apache.coyote.ajp.AjpAprProtocol start INFO: Starting Coyote AJP/1.3 on ajp-9099 Nov 12, 2009 1:40:19 PM org.apache.catalina.startup.Catalina start INFO: Server startup in 14982 ms

17. Heap and Garbage Collection 17 Performance: Heap and GC Some optimizations for the JVM: – -XX:+AggressiveOpts (compiler optimizations, default on Sun jdk>1.5u6) – -Xms256m (minimum Heap size): note that really big values cause the GC to run with bigger delay until the heap minimum value is full – -Xmx512m (maximum Heap space): prevent java.lang.OutOfMemory – -XX:MaxPermSize=256m (Permament Generation): if your application loads many classes (hundreds, thousands) then increase the value in a way that Xms>permSize<Xmx – -server: the JIT compiler is optimized for application overall performance instead of small footprint and fast startup

18. Heap and Garbage Collection 18 Performance: Heap and GC Some considerations – If you think your system will be in high load, set the Xms and Xmx heap sizes to the same value: the Garbage Collection execution can became more predictable and run faster – If you receive OutOfMemory errors such as “unable to create a new thread”, then you can decrease the amount of memory of the stack per-thread (-Xss512k for instance) – If you receive errors regarding “too many open files”, then increase the number of them in the OS: #cat /proc/sys/fs/file-max 75000 #sysctl -w fs.file-max=371304 – Use a 64-bit OS if you need more than 2Gb of Heap space for your JVM: this is the max value in 32bit OS – Use Sun JDK 1.6u17 (current): differences of performance from 1.5 to 1.6 are in terms of 20/25%

19. Monitoring 19 Monitoring: motivation You need to know: – the status and the health of your applications – how many resources and bandwidth the application X is consuming (in a way to adjust the payment to your customer ) – the monthly availability of your services, to ensure SLA – that en email just arrived to your mailbox at 04:00 AM saying that Tomcat is down

20. Monitoring: Hyperic HQ 20 Monitoring: Hyperic HQ Hyperic HQ is a SpringSource (now division of VMware) JEE application for monitoring of web applications, common services and network devices Server-Agent logic: agents deployed to N machine where services that need monitoring are running Sigar API (System Information Gatherer), engineered by the main developer of mod_perl Doug MacEachern Auto-discovery of services Tens of pre-configured templates for common services

21. Monitoring: Hyperic HQ 21 Monitoring: Hyperic HQ

22. Monitoring: Hyperic HQ Sigar 22 Monitoring: Hyperic HQ Sigar Usage example (with PTQL - Process Table Query Language) We want to list Java processes that runs ofbiz.jar, owned by the user deadfish. java -jar./agent-4.1.2/bundles/agent-4.1.2-1053/pdk/lib/sigar.jar sigar> ps State.Name.eq=java,Args.*.ct=ofbiz.jar,CredName.User.eq=deadfish 4989deadfishNov8948M329M3.9MS26:28java:ofbiz.jar sigar> pargs 4989 pid=4989 exe=??? cwd=??? 0=>/usr/java/jdk1.6.0_10/bin/java<= 1=>-Xms256M<= 2=>-Xmx512M<= 3=>-Dofbiz.admin.port=10700<= 4=>-Dofbiz.admin.key=so3du5kasd5ln<= 5=>-Denv.ofbiz.home=/home/deadfish<= 6=>-XX:MaxPermSize=256m<= 7=>-server<= 8=>-Dcom.sun.management.jmxremote<= 9=>-Dcom.sun.management.jmxremote.port=10701<= 10=>-Dcom.sun.management.jmxremote.ssl=false<= 11=>-Dcom.sun.management.jmxremote.authenticate=false<= 12=>-jar<= 13=>ofbiz.jar<=

23. Monitoring: Hyperic and JMX 23 Monitoring: Hyperic and JMX The JMX technology provides a simple, standard way of managing resources such as applications, devices, and services (Sun JDK> 5.0) The resources inside the JVM are instrumented by Java objects called Managed Beans (MBeans) that act as JMX agents With JMX is possible to monitor and control remotely (even via SSL) the resources used by your applications To enable JMx on Tomcat, add the following in bin/catalina.sh: [ $1 != "stop" ] && JAVA_OPTS="-Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.port=6969 \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false $JAVA_OPTS" export JAVA_OPTS

24. Monitoring: Hyperic and JMX 24 Monitoring: Hyperic and JMX Let see a demo…

25. Questions? 25 Thanks from IntegratingWeb