1. © 2012 Progress Software Corporation. All rights reserved. 1 Click to edit Master title style Remove Performance Bottlenecks in Your SQL Connections - Data Connectivity Performance Best Practices Steve Pittman, Architect OpenEdge SQL PUG Challenge Americas – 2012 -

2. © 2012 Progress Software Corporation. All rights reserved. 2 Disclaimer The future features described in this presentation are under consideration by Progress Software and are not commitments for future products, technologies, or services. Any forward looking statements represent our estimates only as of today and should not be relied upon as representing our estimates as of any subsequent date. While we may elect to update forward looking statements at some point in the future, we specifically disclaim any obligation to do so, even if our estimates change.

3. © 2012 Progress Software Corporation. All rights reserved. 3 Agenda  Where did all that data come from?  What can we do?  A systematic approach to performance  An OpenEdge perspective  Questions?

4. © 2012 Progress Software Corporation. All rights reserved. 4 A Brief Audience Survey  # using ODBC –Tools – Crystal?  # using JDBC –Tools – DbVis?  # using Java  Java appservers –WebLogic?

5. © 2012 Progress Software Corporation. All rights reserved. 5 Humans are information machines  Humans are now storing over 988 Exabytes of information (at the end of 2010)  15 petabytes of information generated every day  Scientific, Engineering, Financial, Social – all areas of our lives continue to generate and store the world’s data at an increasing rate.

6. © 2012 Progress Software Corporation. All rights reserved. 6 Making this data useable  Performance trends in 2010-11 were centered around making this data more quickly accessible (and more useful).  Sorting, cataloging, querying, inserting…all these are required for making the data mean something – and it has to be fast!

7. © 2012 Progress Software Corporation. All rights reserved. 7 Performance trends in 2012  Virtualization made efficiency a way of life  Concurrency will gain steam in 2012 –Research on multi-core has been building –New tools by chip makers will help –Additional Language constructs and compiler directives will come –Java application development frameworks –Application servers – WebLogic, JBoss  Larger buffers and algorithm changes will take place to fully utilize 64 bit processors.

8. © 2012 Progress Software Corporation. All rights reserved. 8 How did we get here? 1982 2012 The advance of information technology (hardware and software) gives you a dizzying array of choices on how to store your data…. …and not everyone has the same opinion. Multi-tenant db

9. © 2012 Progress Software Corporation. All rights reserved. 9 A typical Enterprise App Architecture Database Buffers, I/O, etc.. Network efficiency Database Settings Application UI Business Logic & Data Access Code Driver

10. © 2012 Progress Software Corporation. All rights reserved. 10 Time is of the essence  Large amounts of data require long processing times – small percentage gains result in large time savings  Performance tuning steps remain the same

11. © 2012 Progress Software Corporation. All rights reserved. 11 So, what can we do to keep up? Just follow the PANDDa.

12. © 2012 Progress Software Corporation. All rights reserved. 12 Where Performance Bottlenecks Lie P.A.N.D.Da P latform A pplication Code N etwork D river D atabase Which element will give the most payback?

13. © 2012 Progress Software Corporation. All rights reserved. 13 Platform  Consider native calls into the Platform and whether you can do them more efficiently.  Varying client platform performance levels –Windows –Linux: 32bit, 64bit –Unix –Scale up, scale out  What are your optimization options? Are tools available? Ex. Intel Parallel Studio Consider Resource Usage when the Platform is Virtualized! Consider Resource Usage when the Platform is Virtualized!

14. © 2012 Progress Software Corporation. All rights reserved. 14 Platform – Virtualization On a machine that has much less available Memory and/or CPU, this graph will look different.

15. © 2012 Progress Software Corporation. All rights reserved. 15 Memory Usage Scalability – from the Lab Application #1 Application #2 Application #3 Application #4 Application #5 Still consumes less Memory!!

16. © 2012 Progress Software Corporation. All rights reserved. 16 Virtualization Update  Updated Performance Lab to scale Virtual machines to measure VM Impact. VM1VM2VM3TotalComparison DDCompDDCompDDCompDDCompVM1VM2VM3Total 13559.72514.73848.12575.24334230511741.87394.941.60%49.40%88.00%58.80% 25974.440296002.34155.970063162.718982.711347.648.30%44.40%121.50%67.30% 37778.94807.68276.34846.48117.63631.424172.813285.461.80%70.80%123.50%82.00% 49645.55329.69809.851918593.5369928048.814219.681.00%89.00%132.30%97.30% 510866.25567.911031.75678.69163.53832.231061.415078.795.20%94.30%139.10%106.00% 611921.44426.112037.24493.311259.74146.835218.313066.2169.30%167.90%171.50%169.50% 712798.35636.212986.4631212401.24451.738185.916399.9127.10%105.70%178.60%132.80% 813718.46317.813851.15820.313224.63259.740794.115397.8117.10%138.00%305.70%164.90% 914429.76230.114601.7648914135.63905.44316716624.5131.60%125.00%262.00%159.70% 1015228.76294.115316.7627714648.63329.34519415900.4142.00%144.00%340.00%184.20% The Competition needs 3 virtual machines to achieve the same performance as 1 of our virtual machines! + + =

17. © 2012 Progress Software Corporation. All rights reserved. 17 6277rps 15229rps 14649rps 15317rps Virtualized Optimizations give you options! Increase throughput Machine Consolidation 6294rps 3329rps Total = 15900 rps Total = 45194 rps Add Progress | DataDirect

18. © 2012 Progress Software Corporation. All rights reserved. 18 Application Code & Infrastructure  Faster communication –Caching & Object Reuse (avoiding comm!)  Increase Parallelism –Threading, Concurrency –Network async streams –Not for basic reporting!  Minimize Slow operations –Disk I/O, blocking operations  Re-architect when ? –Reference architectures  Tools – latest versions It’s like the Tower of Babel; we have great infrastructure… but we still can’t communicate

19. © 2012 Progress Software Corporation. All rights reserved. 19 Network Physical Location Security is expensive How many Hops? At what speed? Co-locate, Same country, City, etc..

20. © 2012 Progress Software Corporation. All rights reserved. 20 The Network has much to tune…

21. © 2012 Progress Software Corporation. All rights reserved. 21 The Driver is the network

22. © 2012 Progress Software Corporation. All rights reserved. 22 Database  Utilize OpenEdge Tools if available –E.g., promon statement cache options  Data Layout –Tune your SQL queries –Eliminate unnecessary calls and scans –Save MetaData  Memory tuning –Buffer sizes, pre-fetch, etc.  Disk Usage –Striping, storing more data in memory  Logging and health monitors – look in the OpenEdge documentation

23. © 2012 Progress Software Corporation. All rights reserved. 23 OpenEdge sql techniques – better connectivity  Quick ordering – “SELECT TOP”  Minimize round trips –Stored procedures  Capturing preferred query form –Views  Managing variable sized data –dbtool for column width –Network buffer memory impact –Rows per network message  Managing concurrency effects –Lockwait timeout

24. © 2012 Progress Software Corporation. All rights reserved. 24 SELECT TOP – Order and Time db App TOP ordering Optimized ordering ORDER BY Sorting machine

25. © 2012 Progress Software Corporation. All rights reserved. 25 Push network traffic to server Java Stored Procedure – logic on the server App Stored Proc SQL db

26. © 2012 Progress Software Corporation. All rights reserved. 26 Pushing best query elements into sql views  Views can be a toolbox of nicely working query components –Pre-fab join relationships for naturally related tables The right tables with the right join predicates The right WHERE clause predicates to choose correct data Avoid those queries missing a vital WHERE predicate  Views can be a storehouse of totally built queries –End users pick the view vs. end users built query from scratch  Accelerate those ad-hoc reports with views as building blocks  Views can provide security –Expose only “need to know” tables, columns –Screen out data which must be private or secured away  Simple to setup and use –CREATE VIEW myview AS –SELECT * FROM myview WHERE myview.region = ‘NW’

27. © 2012 Progress Software Corporation. All rights reserved. 27 Failing transactions – the problem 12345,Smith Trucking, Westford,MA,02423,….. Customer table UPDATE customer SET city = ‘Northboro’ WHERE custnum = 12345 SELECT * FROM customer WHERE custnum BETWEEN 10000 AND 20000 lockWaitTimeout expires

28. © 2012 Progress Software Corporation. All rights reserved. 28 Failing transactions – some help  Global lockwait timeout –Environment variable PROSQL_LOCKWAIT_TIMEOUT export PROSQL_LOCKWAIT_TIMEOUT=30 Significant shortcomings  JDBC Connection property –“lockWaitTimeout” –As of 10.2B04 String url = "jdbc:datadirect:openedge://localhost:9878;" + "databaseName=mys2k;lockWaitTimeout=60"; Connection conn = DriverManager.getConnection(url, ….. );  FUTURE: ODBC Connection property –“lockWaitTimeout”, same format –Target delivery in OE 11.2 –In ODBC Data Admin “Advanced” Tab

29. © 2012 Progress Software Corporation. All rights reserved. 29  Power of the ABL –ABL: size unlimited – TEXT –Sql: size limited – VARCHAR (61)  Result: “over-stuffed” columns OE sql error: “Column XXX has value exceeding it’s max length”  Server side fixes –Data Dictionary Options - SQL Properties - Adjust Field Width –OE SQL as DBA ALTER TABLE ALTER COL SET PRO_SQL_WIDTH ….. –dbtool High performance, multi-threaded db maintenance Menu choice: SQL Width Scan w/Fix Option Online schema updater Coping with column sizes

30. © 2012 Progress Software Corporation. All rights reserved. 30  Client side fixes –Find column which has size problem from error text “Column Name in table PUB.Customer has value exceeding its max length or precision.” –Change sql statement column reference to “substring()” SELECT Name, city, state, country FROM pub.customer …… SELECT SUBSTR( Name, 1, 84) AS ‘Name’, city, state, country, …… –OR, truncate over-stuffed value to column defined size SELECT SUBSTR( Name, 1) AS ‘Name’, city, state, country, …… Coping with column sizes - more

31. © 2012 Progress Software Corporation. All rights reserved. 31 32K * 100 cols * 50 rows = 160M Coping with column sizes – maxing out App db Table2 result set rows Table1 result set rows Table3 result set rows Query network buffers

32. © 2012 Progress Software Corporation. All rights reserved. 32 New Challenges & Research in 2012  More 2012 Enterprise Performance Research –Caching, Concurrency, Cloud deployment  Mobile devices –3g to 4g to wiMax –Hardware upgrades coming fast  Services still the norm – what’s next? –SaaS –PaaS  SQL vs NoSQL “Cloud”

33. © 2012 Progress Software Corporation. All rights reserved. 33 In Summary  Humans continue to create copious amounts of data at breakneck speed. This data must be cataloged, sorted, and queried – the latest improvements in data access middleware will ensure this remains scalable for your applications in the enterprise!  R&D trends in 2011 give way to actionable development changes in 2012. Follow the P.A.N.D.Da model to keep your application up to date!

34. © 2012 Progress Software Corporation. All rights reserved. 34 For More Information  Progress|DataDirect Website: www.datadirect.comwww.datadirect.com  More than a Driver: http://web.datadirect.com/products/feature-overview/data-connectivity/anatomy/index.html http://web.datadirect.com/products/feature-overview/data-connectivity/anatomy/index.html  Progress|DataDirect Performance Data: http://web.datadirect.com/products/feature-overview/data-connectivity/performance.html http://web.datadirect.com/products/feature-overview/data-connectivity/performance.html  Progress|DataDirect Virtualization: http://web.datadirect.com/solutions/virtualization/virtualization.html http://web.datadirect.com/solutions/virtualization/virtualization.html  Documentation: http://www.datadirect.com/support/product_info/proddoc_product/index.ssp http://www.datadirect.com/support/product_info/proddoc_product/index.ssp Contact Us: http://web.datadirect.com/contact-us/request_information.html http://web.datadirect.com/contact-us/request_information.html NA Tel Number: +1-800-876-3101 Intl Tel Number: +44 (0) 1753-218 930