1. Profiling Applications David Rubio Forsythe Solutions Group, Inc. drubio@forsythe.com

2. 2 Overall Performance Analysis Approach Concentrate on business critical application(s) Understand thread states ○ Blocked (Sleeping) – on what? ○ Running – what code? Use appropriate tools ○ truss, strace ○ snoop, tcpdump ○ pfiles, lsof ○ prstat –Lmn20 1 ○ DTrace Latency Profiles ○ I/O ○ Network ○ Memory ○ Functions in application

3. 3 Profiling Application Execution Useful DTrace one-liner to capture what code is being executed: # dtrace -n 'profile-301 /arg1/ {@[execname, ustack()] = count()} tick-20s {trunc(@,16); exit(0)}' dtrace: description 'profile-301 ' matched 2 probes … java libjvm.so`__1cJMarkSweepOIsAliveClosureLdo_object_b6MpnHoopDesc__i_+0x8 libjvm.so`__1cLPSMarkSweepQinvoke_no_policy6Fpii_v_+0x6b4 libjvm.so`__1cLPSMarkSweepGinvoke6Fpii_v_+0xa4 libjvm.so`__1cVVM_ParallelGCSystemGCEdoit6M_v_+0xf8 libjvm.so`__1cMVM_OperationIevaluate6M_v_+0x80 libjvm.so`__1cIVMThreadDrun6M_v_+0x6e0 libjvm.so`__1cG_start6Fpv_0_+0x208 libc.so.1`_lwp_start 209 ns-slapd libdb.so.2`__memp_stat_hash+0x54 libdb.so.2`__memp_trickle_pp+0x144 libback-ldbm.so`trickle_threadmain+0xb8 libnspr4.so`_pt_root+0xc8 libc.so.1`_lwp_start 368

4. 4 Profiling Application Latency #!/usr/sbin/dtrace -s /* * Usage: time-funcs.d -p pid * time-funcs.d -c 'application args' */ pid$target:::entry { self->ts[self->depth++] = timestamp; } pid$target:::return /self->ts[--self->depth] != 0/ { @[probefunc ] = quantize(timestamp - self->ts[self->depth]); self->ts[self->depth] = 0; } tick-30s { exit(0); } END { trunc(@, 20);}

5. 5 Partial Output from D Script select value ------------- Distribution ------------- count 33554432 | 0 67108864 |@@@@@@@@@@@@@@@@@@@@@@ 294 134217728 |@@@@@@@@@ 118 268435456 | 0 536870912 |@@@@@@@@@ 116 1073741824 | 0 pthread_cond_timedwait value ------------- Distribution ------------- count 524288 | 0 1048576 | 1 2097152 | 0 4194304 | 0 8388608 | 1 16777216 | 2 33554432 |@@ 8 67108864 |@@@@@@@ 32 …