IBM Tivoli Support Technical Exchange Event: Troubleshooting Gateways for TNPMW By: Sharina Shahir September 17, 2009

Agenda Debugging Techniques Replicating a gateway installation Case Studies Recommended Practices Q&A

Debugging Techniques Investigate the gateway’s audit and log files Increase the LOG_LEVEL in the properties file Activate DEBUG=debug or

Replicating a gateway installation Having multiple gateway installations for the same vendor data and release. You can achieve this by duplicating a gateway config which will process the data using its own set of properties file You only duplicate within a gateway config You don’t want to duplicate vendor gateways or gateway framework 

Replicating gateway: When you may need it Current processing There are performance data files which are in substantial not only in size but volume as well. (i.e. xml) Due to this nature, configuring a single gateway may be insufficient to process the entire data within the time allocated. Time allotted here means if the collection is 15-min intervals, the gateway runs must complete the stages of parsing within 15 minutes. If it goes beyond that, then it will continue to be in a state of backlog. TransferConfig.pm likely enabled Backlog processing Anytime there is a scenario that prevents the production gateways from processing data, you know there will be backlog data on top of current data that will eventually need to be processed. TransferConfig.pm is usually disabled. Debugging A production gateway is already in placed and you don’t wish to tamper with the settings already provided.

Replicating gateway: How to replicate Notice the copy command used “cp –p <source> <destination>” It copies not only the files but the original timestamps of the files copied Gives a good indication if my EngineConfig.pm and UserConfig.pm are exactly the same in each of the related sub-folder In the duplicate gateway, edit the properties file and configure the appropriate settings.

Putty sessions: Case studies Case Study 1: Missing data Case Study 2: Please, that’s not my name. Case Study 3: I lost my ID so that makes me bad? Case Study 4: PERLIZE me!

Recommended Gateway Practices Gently stop the gateway process. Create an empty file named stop_gateway in the IN_DIR folder. The gateway will stop the engine stage and will not parse the remaining raw files in the IN_DIR folder. The gateway will process the post parsing stage to output the LIF. Executing a kill on the process ID may leave unprocessed PIF that may need to get re-parsed again in the future. Avoid unnecessary tampering of the Perl modules located in the gateway-framework and modules folders. Any customization work may get over-written when performing gateway upgrades or when applying fixpack. For gateway config files, keep track of the changes you make by always making a backup of the original. Recall the “cp –p” command. Having the original also enables you to perform a diff against the newer configuration allowing you to identify the changes made. Practice consistent naming convention Recommended path and naming convention LOG_FILE=/appl/virtuo/logs/gways/<vendor-subsys>_<release>.log AUDIT_FILE=/appl/virtuo/logs/gways/<vendor-subsys>_<release>.audit

Recommended Gateway Practices (continued) Monitor your disk usage. Monitor the spool, logs and archive folders to prevent the 100% disk usage mark! Ensure housekeeping are performed on logs, archived data, pt and bad files. Determine their retention period as you don’t want to keep them indefinitely. Zip archive data where possible – set COMPRESS in properties file May need to investigate when disk usage exceeds a certain threshold. Monitor the audit and log files Frequently check logs to detect any major errors being reported Activate features for debugging only when necessary Permanently increasing the LOG_LEVEL will significantly produce more message and may affect the performance of the gateway Set LOG_LEVEL=1 (or 2 if necessary) and DEBUG=0 when debugging is not required. Have a back-up plan ready when any stages of the parsing process gets affected. Downtime on gateway box will cause backlog processing. Consider duplicating gateways to handle backlog gateway processing. Downtime on related machines, i.e. database server downtime to say, due to maintenance work will prevent the loading of LIF. Consider allocating additional disk space to temporarily store LIF as you should allow gateway processing to continue to run as per normal. This reduces the amount of backlog work where you would only need to handle LIF backlog loading.

Recommended Gateway Practices (continued) TNPMW supports distributed architecture. Consider dedicating a server for your gateways for a more robust solution Gateways are disk I/O intensive as we deal with massive amount of data so splurge on hard disk/file system where possible Spread the gateway processing Avoid the parsing process to run on the same file system as where the $GATEWAY_ROOT and configuration of the gateways reside Avoid lumping all the parsing process onto a single /spool folder system if you have several gateways processing UMTS-type data. Consider the option of having say /spool01 and /spool02 to balance the load. [virtuo]ls -la /spool01 drwxr-xr-x 2 virtuo virtuo 2 Jul 2 17:33 ericsson-utran drwxr-xr-x 2 virtuo virtuo 2 Jul 2 17:33 alcatel-bss drwxr-xr-x 2 virtuo virtuo 2 Jul 2 17:43 huawei-mscs [virtuo]ls -la /spool02 drwxr-xr-x 2 virtuo virtuo 2 Jul 2 17:50 ericsson-mgw drwxr-xr-x 2 virtuo virtuo 2 Jul 2 17:50 ericsson-bss drwxr-xr-x 2 virtuo virtuo 2 Jul 2 17:55 nortel-bss Benefits to this would: cushion the effects when say, /spool01 was damaged allowing /spool02 to continue with the parsing of other gateway processing. Reduce disk I/O and bottleneck issues

Thank you for your attention Questions?