1. 1 J2EE: Black Box in the Oracle World Don Bergal, Confio Software August 17, 2006

2. Objective: Answer a Question  “Why does a DBA need visibility upstream in the application in order to manage the database?”

3. An alternate title for today:  “How to be a better DBA by watching something other than the Oracle database”

4. Agenda J2EE Black Box in an Oracle World a)The Black Box Application Problem b)The DBA Hot Seat c)System Statistics versus Wait Time Diagnostics d)A Better J2EE End-To-End Performance Solution e)Best Practices Response Time Diagnostics Methodology f)Q & A

5. The Black Box Application Problem

6. J2EE: Black Box in Oracle World How can you manage here? If you cannot see here! Web customer Internal user Batch job Billing Financial System Web store Black Box: Applications and End Users

7. See the applications that impact database performance Web customer Internal user Batch job Billing Financial System Web store DatabaseApp ServersWeb Users

8. Result: Disconnect Between IT Groups? DBA’s Databases Database Management Database Development SQL Development SQL Tuning & Optimization Tools Ratio Based Monitoring & Tuning J2EE Web & App Tiers Java Development Client Web/Applet/Application JSP/Servlet Development EJB Container Development JDBC Development Uses Desktop J2EE Profilers No Visibility into the Database Tier No Visibility into the J2EE Tier

9. Effect: Black Boxes leave DBA defenseless  ‘Silo-based, Single System’ tools only shows database system perspective (even though they are blamed for problem in any system). Can’t tell when to hand-off problem to another group (Dev/SysAdmin/etc…) Blamed for all layers, but can only see the database Black Box

10. The DBA Hot Seat

11. Finger Pointing Happens When You Can’t See the Problem What’s wrong with your *&!@$# database? It’s a database issue! Java/Web Developer Management DBA Guilty until proven innocent It’s a configuration issue! Software Vendor It’s a database issue! Network Engineers DBA Hot Seat

12. Do Customers See the Problem Before You? Java Developer DBA Not my code… What performance problems ?!? SLA Violation User Wait Time Warning! Not my Network, Servers, Load Balancer, Firewall… System Admin

13. Hot Seat comes from UNCLEAR Ownership Need evidence to show where the problem exists, avoid “Finger Pointing” to the database It’s an Application issue!It’s a Database issue! Java App Owner DBA

14. 14 DBADBA Networking Developer System Admin Performance Suspects Line Up…

15. Lack of Visibility is Source of Problem J2EE layer App Layer Oracle eBusiness Oracle 9iAS Peoplesoft Siebel JD Edwards SAP Custom apps Oracle databaseWeb Server Web page request Apache WebLogic WebSphere IIS Service Levels Measured Here… …Blame assigned Here Typical Multi-Tiered J2EE System Leaves the DBA Blind

16. Why Does the DBA Need J2EE Visibility? You can’t control what you can’t see You can’t show/demonstrate it’s not a DB Issue You can’t manage your database if you are blind to the source of the requests

17. Traditional Performance Tools versus Wait-Time Diagnostics

18. Current Tools – Never Cross the Silo Boundaries  Tune inside each “Silo” to improve end-end performance  “System Health” or ratio-based approach to performance… Statistics…% Utilization…Event Counts…  No direct relationship to End-User experience or SLA  Weak linkage between tiers limits collaboration CPU Memory Sessions DatabaseJ2EE ServerWeb Server CPU Heap Size Threads CPU Reads Processes

19. Standard J2EE App Performance Monitoring: How Does This Help Identify User Response? Idle threads Number of requests Throughput Waiting Requests Memory (Heap) Usage

20. J2EE Profilers: High Load, Test Data Only

21. OS Level System Monitoring (rstatd/perfmon)

22. Network System Monitoring

23. Traditional Tools vs. Wait-Time Methodology  Ratio-Base traditional tools show “Oracle system health” but no link to end user service  Use Wait Time to correlate user requests to SQLs and performance impact

24. A Better J2EE End-To-End Performance Solution

25. Requirements for Oracle & J2EE Monitoring  Use Oracle Wait-Time Based Monitoring  Visibility across Java and Oracle layers  Lightweight JVM Method Tracker  Agnostic JVM server support  Fast installation and data collection techniques  Cost Effective  EASE OF USE – MAKE IT OBVIOUS!

26. Wait-Time Based Performance Analysis  Tune by determining where client request time is spent  Determine where Oracle resources are spent during the execution of SQL execution  Emerging best-practice for database tuning “You can’t tell how long something took by counting how many times it happened. … If you’re only measuring event counts, then you’re not measuring what the users care about.” — Optimizing Oracle Performance, O’Reilly Press  Oracle is building up Wait-based tuning capabilities with each database release…10g has over 800 wait-time based performance counters

27. Oracle Wait Interface Book Free book at www.confio.com

28. A Wait-Time J2EE Performance Methodology: 3 Key Requirements 1. Follow every transaction individually 2. Identify every “step” along the path 3. Measure the “wait-time” at each step Result: Minimize time to resolution and division of work by clearly identifying which step is performing poorly.

29. Alternative Monitoring Architectures

30. Byte Code Instrumentation (BCI) ● Best of breed Java monitoring technique ● Production Monitoring – not synthetic transactions ● Instrumentation inserted into selected compiled Java methods by choosing specific class files ● Measure start/stop to execute each Method ● Calibration: Adjust load to < 1% load by controlling level of instrumentation ● Auto Throttling, Low risk

31. Best Practices Wait Time Diagnostics Methodology (Day in the life on the performance team)

32. 4 Steps for J2EE to Oracle Visibility 1. Capture Every User Request (end-to-end transaction) 2. Capture Every Step Inside Each Tier 3. Measure Response Time of Each Step 4. Identify Possible Causes Based on Wait-Times Since every step is visible, you can always see the problem

33. 4 Steps for J2EE to Oracle Visibility 1. Capture Every User Request (end-to-end transaction) 2. Capture Every Step Inside Each Tier 3. Measure Response Time of Each Step 4. Identify Possible Causes Based on Wait-Times Since every step is visible, you can always see the problem

34. 1. Capture Every User Request Identify the slowest performing end-to-end Requests Slowest performing can be: Longest average request-response time Largest accumulated Wait-Time in application (across all sessions)

35. 1. Capture Every User Request PlaceOrder.jsp Request has most serious delays

36. 2. Capture Every Step Inside Each Layer Identify the slowest steps within the slowest Request

37. Key Requirement: Correlate Application Response to SQL Response Times Question: Which SQL statements are causing Wait Time for my application?

38. 2. Capture Every Step, Identify Slowest UpdateOrder Total (SQL) is slowest in PlaceOrder.jsp (Most Wait-Time accumulated)

39. 3. Measure Response Time of Each Step Focus on steps that consume the most time

40. Detailed Response Time for the Critical Step Update Order Total ‘Undo Segment Extension’ Wait-Event is bottleneck

41. 4. Possible Causes Based on Wait Times Diagnose the step with the greatest Wait Time

42. 4. Possible Causes Based on Wait Times Execution path and time spent on each ‘Hot Spot”

43. Identify Chain of Calls URL Request -> Method Call –> SQL Allocate Wait Time to each Method or SQL Call Profiler data Without the System Impact

44. Recall 3 Key Requirements to Provide the Analysis 2. Identify Every step – Java method or Wait-Event 3. Measure Wait time for Every Step 1. Identify individual Request

45. Customer Example  Retail transaction system  Online media fulfillment  Customer requests running slowly, unknown reason  Ignite for Oracle and Ignite for Java used to examine situation

46. Wait Time for “Authorize Transaction” Identifies Problem Steps

47. Associates Request Wait-Time with Single Procedure  Same query multiple times in single procedure  Same row retrieved 7 times  Change to select once, cut Wait-Time to less than half

48. Summary: Look For…  Continuous monitoring 24/7 – not session/period specific  Production data – not synthetic “test” transactions  Correlate Application & Oracle – where most of the time is spent  Individual transactions, not system-wide statistics

49. Visibility for the DBA  DBA has a need to see the J2EE Application  Solve database problems by understanding the other systems  Look for Wait Time – that’s where the action is  Achieve collaboration between Developers, Architects and DBA’s  DBA can be the leader, not the one to take the blame!

50. About Confio Software  Wait-Time based performance tools for continuous, low impact monitoring  Ignite for Oracle – Deep, detailed visibility into SQLs and Wait-Events  Ignite for Java – End-to-End Application through Oracle visibility, correlates Java and SQL Wait-Time  Ignite for SQL Server – Coming soon, customer testing Free Trial at www.confio.comwww.confio.com

51. About the Author  Don Bergal, Confio Software  Responsible for product management, among other things…  18 years software/telecom industry experience donbergal@confio.comdonbergal@confio.com

52. Q & A