1. Application Performance Tuning @ KBC Dirk Beauson

2. 2 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Application Performance Tuning Organization @ KBC  New Challenges  Current optimizations  Conclusions

3. 3 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

4. 4 Why Application Performance Tuning @ KBC  RECOGNIZE ?  DB2 online -Poor Response Time -Timeouts/Deadlocks  Long Running DB2 jobs  Locking problems

5. 5 Why Application Performance Tuning @ KBC  80% of the performance problems are related to inefficient application and/or database implementation !! Building CostCost DesignImplementation Problems can cost up to 50 times more to fix during implementation than in design !!!

6. 6 Why Application Performance Tuning @ KBC  Performance Players  Applications (Design/Build)  Databases  DB2 subsystem  z/OS  Network  Performance Factors  Workload  Throughput  Resources  Contention

7. 7 Why Application Performance Tuning @ KBC  Using more CPU  results in using more MSU  Can result in higher 4 hour rolling average  Common Performance Issues @ KBC  Index design/usage  SQL (please know your tables/indexes and explain)  Application Design  Lock waits  Database design  So we want to run applications fast and as cheap as possible

8. 8 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

9. 9 EXPLAIN Tools Used  Simply Explain and PLAN_TABLE  Need to look @ extra explain tables !!! (Study DB2 11)  Data Studio (visual explain)  SQL Explorer  Catalog Manager (to view Current Access Path) COST*RATE QB PL MIX QTYPE METH ACC MTCH IX TBNAME IXNAME 11.083908 1 1 0 SELECT 0 N 2 N CBTXXX22 CBX7XXX 1.118052 1 2 0 SELECT 1 I 1 N CBTXXX03 CBX1XXX 1.118566 1 3 0 SELECT 1 I 2 N CBTXXX02 CBX3XXX 1.293676 1 4 0 SELECT 3 0 N Access table CBTXXX22 using index CBX4XXX22 (1 COL) Join using nested loop join with Access table CBTXXX03 using index CBX1XXX03 (1 COL) Join using nested loop join with Access table CBTXXX02 using index CBX1XXX02 (1 COL) Composite table is sorted for ORDER BY clause,

10. 10 EVAL reporting tool  Everyday we report bad BMP’s & MPP’s @ DEV  List under construction :  ABEND : ABENDs per BMP / DAY > x (dependent on USER/U0240/pseudo/system abends) ABENDs per MPP / DAY > y (dependent on USER/U0240/pseudo/system abends)  LOCKING : DEADLOCKS & TIMEOUTS per BMP / DAY > x DEADLOCKS & TIMEOUTS per MPP / DAY > x LOCK ESCALATIONS  COMMIT FREQ : COMMIT FREQ in BMP > x sec and updates ELP time MPP > x SEC and updates Online Transaction running under IMS Batch running under IMS

11. 11 EVAL reporting tool  Number of SQL CALL : Number of SQL CALLS in BMP > x Number of SQL CALLS in MPP > x  Number of GETPAGES per PLAN : Number of GETPAGES per BMP > x Number of GETPAGES per MPP > x  Number of GETPAGES per PACKAGE/function : Number of GETPAGES per PACKAGE in BMP > x Number of GETPAGES per PACKAGE in MPP > x  Number of GETPAGES per STMT : Number of GETPAGES per PLAN per STMT in BMP > x Number of GETPAGES per PLAN per STMT in MPP > x

12. 12 EVAL reporting tool  Long running SQL STMT in MPP : SQL STMT > x SEC DB2 CPU

13. 13 Performance Monitor Tools  Mainview  Locking problems Date / Time Lockout Victim Victim Victim Victim Victim ----------------- Type Plan AuthID Connect CType Corr ID 22MAY-11:36:52.54 DEADLOCK PLAN1 USER001 IKC1 MPP 0037PLAN1 22MAY-11:29:13.49 TIMEOUT PLAN2 USER002 ITC2 MPP 0012PLAN2 Blocker Blocker Blocker Holder Plan Connect Corr ID AuthID PLAN1 IKC2 0111PLAN1 USER011 DISTSERV SERVER db2jcc_appli USER012

14. 14 Performance Analyse Tools  ITSV &EXTA-DB2-tables (SMF info in files & DB2-tables) day-1 :  Performance information about every execution (MPP, BMP) PLANNAME CORRID BEGINTIME DB2AUTH ELPTIME --------++------------++--------------------------++--------++------------ PLAN1 0012PLAN1 2015-05-21-09.22.55.309341 USER1 0.077 PLAN1 0008PLAN1 2015-05-21-09.23.01.482621 USER1 0.227 CPUTIME CPUTIMEDB2 EWAITIO EWAITLAL -------++------------++------------++------------ 0.011 0.002 0.060 0.000 0.023 0.005 0.167 0.000 BPGETPAGE BPSYNCRD BPPREFET BPDPF ---------++-----------++-----------++----------- 37 17 0 0 93 44 0 0 SELECT OPEN FETCH UPDATE INSERT DELETE ------++-----------++-----------++-----------++-----------++----------- 8 1 3 0 0 0 18 2 2 0 1 0 …

15. 15 Performance Analyse Tools  RapitWeb  Interface on ITSV data & extra info -KBC made Reporting on performance information -Graphical presentation -Also known as

16. 16 Performance Analyse Tools  ITSV (files and derived DB2 tables) :  RapitWeb -A lot of system information for the system guys -Also lots of information to help DEV

17. 17 Performance Analyse Tools  ITSV (files and derived DB2 tables) :  RapitWeb -IMS info -DB2 info TRX1

18. 18 Performance Analyse Tools  ITSV (files and derived DB2 tables) :  RapitWeb  TOPPERS (overall, domain, subsystem, …) -Toppers Online -Toppers Batch MPP1 MPP2 MPP3 BMP1 BMP2 BMP3

19. 19 Performance Analyse Tools  Apptune  Performance problems -PLAN -PACKAGE -STMT -OBJECT

20. 20 Performance Analyse Tools  DWH from Apptune  PLAN  PACKAGE  STMT  OBJECT  STMT/OBJECT  Daily and monthly base -Daily, 2 hour interval (16 days) -Daily (15 months) -Monthly (15 months)

21. 21 Compare Access paths & UOC  Changing access paths  Test -> Acceptance : -A list is published of changing access paths LIMITATION : only if queryno doesn’t change  Acceptance -> Production : -Same list  No follow up because list is not complete -Search for better (100 %) solution. TS SCAN instead of IX access Wrong registration

22. 22 DB2 Object Follow Up Tools  Daily reporting :  Object not RW state  70% or more full -Only TS/PT  Not recoverable TS  …  Weekly process :  Runstats of selected objects -Own selection <> DSNACCOX  IC  REORG of selected objects -Own selection <> DSNACCOX

23. 23 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

24. 24 Reactive Application Performance Tuning  Triggers :

25. 25 Reactive : INCIDENTS/PROBLEMS  CASE :  REBIND query in function XXXXX is performing badly in acceptance  QUERY QUERY

26. 26 Reactive : INCIDENTS/PROBLEMS XXXXX – C200 op ACC: BAD Access table CBTXXX22 using index CBX4XXX22 (1 COL) Join using nested loop join with Access table CBTXXX03 using index CBX1XXX03 (1 COL) Join using nested loop join with Access table CBTXXX02 using index CBX1XXX02 (1 COL) Composite table is sorted for ORDER BY clause, XXXXX – C200 op PRO: GOOD Access table CBTXXX22 scan index CBX7XXX22 (2 COLS) Join using nested loop join with Access table CBTXXX03 using index CBX1XXX03 (1 COL) Join using nested loop join with Access table CBTXXX02 using index CBX3XXX02 (2 COLS) Composite table is sorted for ORDER BY clause,

27. 27 Reactive : INCIDENTS/PROBLEMS  IX information QUALIFIER: TABLE=CBTXXX22 CmIndex Name Column Seq Num O Lth DatTyp ----v----1----v----2----v----3----v----4----v----5----v----6----v----7----v---- CBX1XXX22 OVK_NR 1 8 A 12 CHAR CBX2XXX22 LOG_BSD_REG_DS 1 1 A 10 TIMESTMP CBX2XXX22 VLG_NR 2 2 A 2 SMALLINT CBX3XXX22 OSP_REK_NR 1 10 A 34 CHAR CBX4XXX22 OLOG_BSD_REG_DS 1 3 A 10 TIMESTMP CBX5XXX22 OVK_NR 1 8 A 12 CHAR CBX5XXX22 LOG_BSD_REG_DS 2 1 A 10 TIMESTMP CBX5XXX22 OSP_REK_NR 3 10 A 34 CHAR CBX6XXX22 OSP_REK_NR 1 10 A 34 CHAR CBX6XXX22 LOG_BSD_REG_DS 2 1 A 10 TIMESTMP CBX6XXX22 OVK_NR 3 8 A 12 CHAR CBX7XXX22 OVK_NR 1 8 A 12 CHAR CBX7XXX22 OLOG_BSD_REG_DS 2 3 A 10 TIMESTMP QUALIFIER: TABLE=CBTXXX22 C Index Name Table Name UT Cl Col Keys ----v----1----v----2----v----3----v----4----v----5----v----6----v----7----v--- CBX1XXX22 CBTXXX22 D2 NN 1 79K CBX2XXX22 CBTXXX22 P2 YY 2 6573K CBX3XXX22 CBTXXX22 D2 NN 1 95K CBX4XXX22 CBTXXX22 D2 NN 1 4104K CBX5XXX22 CBTXXX22 D2 NN 3 5544K CBX6XXX22 CBTXXX22 D2 NN 3 5544K CBX7XXX22 CBTXXX22 D2 NN 2 4310K

28. 28 Reactive : INCIDENTS/PROBLEMS  EXTA : after BIND  Function used in online transaction YYYYY  YYYYY info 19/05 YYYYY info 19/05  DB2 related ?  CPU -> GP ?  ELP -> IO, wait ?

29. 29 Reactive : INCIDENTS/PROBLEMS  APPTUNE or DWH APPTUNE : after BIND  PLAN YYYYY  12743 = open  YYYYY info 19/05 08:00 hour interval YYYYY info 19/05 08:00 hour interval  PLAN ?  PACKAGE ?  STMT ?

30. 30 Reactive : INCIDENTS/PROBLEMS  EXPLAIN  Tune query  Changed Query Changed Query COST*RATE QB PL MIX QTYPE METH ACC MTCH IX TBNAME IXNAME 11.083908 1 1 0 SELECT 0 N 2 N CBTXXX22 CBX7XXX 1.118052 1 2 0 SELECT 1 I 1 N CBTXXX03 CBX1XXX 1.118566 1 3 0 SELECT 1 I 2 N CBTXXX02 CBX3XXX 1.293676 1 4 0 SELECT 3 0 N OK

31. 31 Reactive : INCIDENTS/PROBLEMS  EXTA after fix:  Function used in online transaction YYYYY  YYYYY info 20/05 YYYYY info 20/05

32. 32 Reactive : INCIDENTS/PROBLEMS  APPTUNE or DWH APPTUNE after fix  PLAN YYYYY  12741 = open  YYYYY info 20/05 08:00 hour interval YYYYY info 20/05 08:00 hour interval

33. 33 Reactive : INCIDENTS/PROBLEMS  CONCLUSION  Change Query : FROM CBVXXX02 B, CBVXXX03 C, CBVXXX22 D WHERE C.LOG_BSD_REG_DS = D.LOG_BSD_REG_DS AND B.OLOG_BSD_REG_DS = C.OLOG_BSD_REG_DS AND D.OLOG_BSD_REG_DS BETWEEN :WS-WV.WWV-END-DT-LZN AND :WS-WV. WWV-END-DT-ITV AND D.OLOG_BSD_REG_DS <= :WS-WV.WWV-HPOS-KEY-REG-DS AND D.OVK_NR IN(:WTA-OVK-NR-STR-ZN.WTA-VMT-OVK-NR-1, :  In AND D.OLOG_BSD_REG_DS >= :WS-WV.WWV-END-DT-LZN AND D.OLOG_BSD_REG_DS <=:WS-WV.WWV-END-DT-ITV  I also changed cardinality of PRD_TYPE_KD from 15 to 1005, to force DB2 to use the correct index on CBTXXX02.

34. 34 Reactive : INCIDENTS/PROBLEMS  Next slides hidden :  Describes steps to investigate : -Batch incidents -Online incidents

35. 35 Reactive : INCIDENTS/PROBLEMS  Batch :  Current incident/problem :  Look at job / steps in job -Long running job/BMP ? -CPU versus ELP  Look into Online SQL Performance Monitor -Plan : DB2 dependent ??? -Package : most consuming -STMT : most consuming -Object : most consuming

36. 36 Reactive : INCIDENTS/PROBLEMS  Batch :  Current incident/problem :  Look at access path catalog manager -Explain info -Check access path & Indexes  Try to fix problem -If possible without touching application -> budget !!!

37. 37 Reactive : INCIDENTS/PROBLEMS  Batch :  Yesterday or before :  Look at job / steps in job -Long running job/BMP ? -CPU versus ELP  Look into native ITSV/DB2 derived data -DB2 dependent ? -IO problem ? -CPU problem ? -Locking problem ? -Executing lots of SQL ? -Object problem ? (free space, …)

38. 38 Reactive : INCIDENTS/PROBLEMS  Batch :  Yesterday or before :  Look into Online SQL Performance Monitor or DWH SQL PM (DAY -1) -Plan : DB2 dependent ??? -Package : most consuming -STMT : most consuming  Look at access patch catalog manager -Explain info : LINK -Check access path & Indexes  Try to fix problem -If possible without touching application -> budget !!!

39. 39 Reactive : INCIDENTS/PROBLEMS  Online Transaction :  Current Incident/problem :  Look into Online SQL performance Monitor -Plan : DB2 dependent ? -Package : most consuming -STMT : most consuming IO / CPU / LOCKING / COMPLEX SQL ?  Look at access patch catalog manager -Explain info -Check access path & Indexes  Try to fix problem -If possible without touching application -> budget !!!

40. 40 Reactive : INCIDENTS/PROBLEMS  Online Transaction :  Yesterday or before :  Look into native ITSV/DB2 derived data LINK -DB2 dependent ? -IO problem ? -CPU problem ? -Locking problem ? -Executing lots of SQL ?

41. 41 Reactive : INCIDENTS/PROBLEMS  Online Transaction :  Look into online SQL Performance Monitor or DWH SQL PM -Plan : DB2 dependent ? -Package : most consuming -STMT : most consuming -Look at access patch catalog manager -Explain info -Check access path & Indexes  Try to fix problem -If possible without touching application -> budget !!!

42. 42 Reactive : INCIDENTS/PROBLEMS  Conclusion Access path problems  Much more difficult to force access path in DB2 10 and DB2 11 -We (THINK) know what DB2 has to do -So we try to let DB2 choose that access path -Let DB2 know all you know !!! (extra predicates can help)  Statistics more and more important to DB2 -Cardinality columns in indexes -> very important (1 e column) -Index Levels  Query Rewrite process !!! (DB2 11 study)  Even order of tables in stmt can matter  >= and <= better than BETWEEN  …

43. 43 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

44. 44 Proactive Application Performance Tuning  Sometimes we (applicative DBAs) are really asked to participate in a new projects  Not that often  Reason :  Very important project where performance is an issue by design  Large Workload  Large Throughput needed  Locking can kill system  Large DB system  Very fast respons expected

45. 45 Proactive Application Performance Tuning  We give advise to the DEV team how to design the application, taking into account the previous issues  We design the database model together with the DEV team  Lots of rows (> 1.000.000.000)  High volume insert  Active/Passive data (archive)  Complex overview queries  Index design (larger index pages, include columns, …)  We co-write SQL, or at least screen the access paths  We monitor the application’s behaviour from test over acceptance to production  And are co-responsible for the performance of the application

46. 46 Proactive Application Performance Tuning  We set up performance test together  Follow up these tests  Test  Acceptance  Production

47. 47 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

48. 48 Ad Hoc Initiatives Applicative DBA-group  Performance days :  On regular base lookup bad processes via EVAL Reporting -Analyse -Solution -Fix If possible without touching application -> budget !!! But sometimes redesign is really needed -> negotiate with DEV  DB2 10 optimizations :  Look at index Usage/Design -Lots of random access via indexes with number of levels > 3 Check benefit of larger index pages Fix

49. 49 Ad Hoc Initiatives Applicative DBA-group  Performance days  Look at index Usage/Design -Tables with lots of indexes Indexes used ? Possible to delete indexes ? Use include columns in existing indexes  100 Cases done : -Conclusion Saved a lot of CPU/ELP TIME Difficult to predict what the effective benefit will be Predicted value always > the real value In some cases good estimated values (look per STMT instead of PACKAGE or PLAN)

50. 50 Ad Hoc Initiatives Applicative DBA-group  Performance days  What’s in for ourselves (Applicative DBA’s) ? -We share knowledge -Train each other -Try to get everyone on a higher DB2 knowledge level -…

51. 51 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

52. 52 Application Performance Tuning Organization @ KBC  Only DBA’s (both system & applicative) + system guys 1 1

53. 53 Application Performance Tuning Organization @ KBC  KIT Quality Program  Performance becomes very important  LOOKUP & SOLVE current performance problems  Training DEV-people by applicative DBA’s -Performance Course : Why Performance Working of DB2 Explain/STAGE 1/STAGE 2/Access Paths/Joins Performance Tools 1 2 2

54. 54 Application Performance Tuning Organization @ KBC  KIT Quality Program  Birth of : -EVAL Reporting & Incident Proces -RapitWeb -PFA (Performance Aanspreekpunt) (Performance Contact) Performance specialists @DEV 1 2 2

55. 55 Application Performance Tuning Organization @ KBC  Role PFA :  Has to become a performance specialist @ DEV -Education by applicative DBA’s -1 per department  Knowlegde -> DEV  Proactive Perf role in every new project (obliged !!!)  Follow up EVAL reporting  They do lots of very good work ONION model DBA (S&A) SYST PFA DEV

56. 56 Application Performance Tuning Organization @ KBC  PFA become PFM  Also reporting role about Performance per department  Less problems to solve  No fulltime PFM need anymore  So have to do other DEV tasks  Performance less important 1 3 2 3 anager M

57. 57 Application Performance Tuning Organization @ KBC  Less PFM  New PFM  PFM not a real role anymore 1 3 2 4 4

58. 58 Application Performance Tuning Organization @ KBC  Performance back to DBA  Reactive  Triggered (no continu follow up)  EVAL Reporting (no incidents process)  More performance issues 1 3 2 4 5 5

59. 59 Application Performance Tuning Organization @ KBC  Performance becomes more and more important  Agile  Mobile  More need for PFA 1 3 2 4 6 5 6

60. 60 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

61. 61 New Performance Challenges  AGILE  New way of Development -Not always started with Global Technical Design -Build on build on build on … repetitive Not always first Database Design in mind -If needed first Database Design not corrected -Misuse DB2 objects (columns, number of rows,…) -Less communication with DBA’s Only little things applied at a time No big changes -Performance not always in mind during repetitive DEV process

62. 62 New Performance Challenges  Mobile/Online workload  Much more sessions  Much more executions  Workload  Throughput  Locking

63. 63 New Performance Challenges  More Data  More tables  More rows, lots of rows !!!  Data has to be active much longer  Sizing objects much more difficult  Follow up usage % object  Problems table-PT full -Acute increase in the number of rows -Other hotspots

64. 64 New Performance Challenges  More Data  Problems index-PT full -Usage larger columns in IX (CHAR 50 and > 50) -In combination with no compression -IX larger than TS/PT Only follw up TS/PT space usage !!!

65. 65 New Performance Challenges  BIG DATA  DWH-environment  Link IDAA  Data Analists -Play in DB2 -Own tables -Keep all data Forever ???

66. 66 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

67. 67 Current Optimizations  Redesign EVAL process :  Production problems -Visable in EVAL since acceptance -Not followed up  DEV wants better EVAL process : -INCIDENTs ? NO : Applicatieve DBA group cannot afford 1 person fulltime being busy with that -Better EVAL process, designed together with DEV : OK ;o) -Exception reporting : Problem parked till date xx/xx/xxxx -Only reporting per SUBSYSTEM Not all problems to everyone

68. 68 Current Optimizations  Quality Assurance DB2 objects :  Better follow up process -Not only follow up TS/PT -Also IX Usage of large columns 50 and > 50  Better weekly process : -Runstats Better follow up exceptions -Reorgs -Evaluate DSNACCOX rules versus our own rules based on RTS Lots of Differences !!!

69. 69 Agenda  Why Application Performance Tuning @ KBC  Tools Used @ KBC  Kinds of Application Performance Tuning @ KBC  Reactive  Proactive  Ad Hoc initiative Appl DBA’s  Application Performance Tuning Organization @ KBC  History  New Challenges  Current optimizations  Conclusions

70. 70 Conclusions  We are going back in the right direction.  Performance becomes important again  Is Jan (System Engineer) satisfied with our current Performance Setup ?  Jan ?  Is our current Performance Setup Bulletproof ?  Jan ?

71. 71 Questions ???