1. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 1 Too Deep & Too Wide Using Soft Limits To Avoid Hard Crashes NY/NJ Websphere MQ Family User Group IBM 590 Madison Ave, New York City Tuesday March 25, 2003 T.Robert Wyatt, Vice President Bank Of America Differentiated Service Websphere MQ Support

2. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 2 Objectives Demonstrate why the default queue depth and message length values pose a potential risk Discuss one approach to mitigating that risk

3. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 3 Scenario For Disaster Unrealistic requirements: “Need to be able to queue up an entire day’s worth of large XML messages” –Increase MAXDEPTH –Increase MAXMSGL

4. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 4 Whoops! One day the channel goes down and the system performs exactly as designed: –Queues begin to fill up –Underlying filesystem fills –MQ begins to cut FDC files –MQ and possibly the OS are halted

5. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 5 Recovery Free up disk space any way possible –Move any diagnostic files –Delete non-essential files –Move older MQ logs if linear

6. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 6 Recovery When (or if) QMgr becomes responsive again: Stop all channels RCDMQIMG if linear Shut down the QMgr

7. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 7 Recovery Restarting the QMgr: Delete files or add space –Clean up the file system –Remount the filesystem to larger disk and copy files Results in extended outage, high probability of client impact

8. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 8 What Happened? 5,000 x 4mb messages = 20GB 5k x 4mb persistent msgs > 40GB Def DeadQ 640,000 x 4mb = 2.5TB! When the capacity specified by soft limits exceeds the physical limits, then the soft limits do not exist.

9. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 9 Hard Limits Application constrained only by limits of physical disk space –Failure is sudden –Failure may be catastrophic Affects OS & co-resident apps –Slow manual recovery –High client impact

10. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 10 Soft Limits Application constrained by MQ –System degrades gracefully –Failure isolated to specific apps within MQ –Recovery is automated – normal operation of MQ restores service –Minimal client impact

11. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 11 Tuning Discipline Insure soft limits < physical limits –Reduce MAXDEPTH, MAXMSGL or both –Reduce settings on SYSTEM.DEF* as well as application queues –Reduce DeadQ Settings!

12. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 12 Tuning Discipline Disk Space Available * MAXMSGL MAXDEPTH DSA = MAXMSGL x MAXDEPTH MAXDEPTH = DSA / MAXMSGL MAXMSGL = DSA / MAXDEPTH * Use ½ DSA for persistent msgs

13. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 13 Tuning Discipline Know your messages! –Soft limits must be appropriate to the business to be enforceable –Standardize, preferably automate –Verify on each machine –No exceptions Too shallow? Too narrow? Buy more disk!

14. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 14 Apply System Wide System-wide tuning parameters –Channel MAXMSGL –QMgr MAXMSGL –Max uncommitted messages –Log file size and extents –Use separate, preferably dedicated, filesystems for /var/mqm/qmgrs and /var/mqm/logs

15. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 15 Resources IBM Websphere MQ Manuals http://www-3.ibm.com/software/ts/mqseries/library/manualsa/ –Quick Beginnings Guide –System Administration Manual Support Pacs http://www-3.ibm.com/software/integration/support/supportpacs/ –Capacity planning pacs –Performance reports

16. March 25, 2003NY/NJ MQ Family Users’ GroupToo Deep & Too Wide - 16 Resources MQSeries.net: http://www.mqseries.net http://www.mqseries.net Vienna List Server: http://www-3.ibm.com/software/ts/mqseries/support/usergroups/listserver.html Download presentation: http://www.t-rob.net/Downloads/2Deep2Wide.ppt T.Rob: T.Rob@t-rob.net T.Rob@t-rob.net