1. IBM Tivoli Support Technical Exchange Event: Troubleshooting Gateways for TNPMW
By: Sharina Shahir
September 17, 2009

2. Agenda Debugging Techniques Replicating a gateway installation
Case Studies
Recommended Practices
Q&A

3. Debugging Techniques Investigate the gateway’s audit and log files
Increase the LOG_LEVEL in the properties file
Activate DEBUG=debug or

4. Replicating a gateway installation
Having multiple gateway installations for the same vendor data and release.
You can achieve this by duplicating a gateway config which will process the data using its own set of properties file
You only duplicate within a gateway config
You don’t want to duplicate vendor gateways or gateway framework 

5. Replicating gateway: When you may need it
Current processing
There are performance data files which are in substantial not only in size but volume as well. (i.e. xml) Due to this nature, configuring a single gateway may be insufficient to process the entire data within the time allocated.
Time allotted here means if the collection is 15-min intervals, the gateway runs must complete the stages of parsing within 15 minutes.
If it goes beyond that, then it will continue to be in a state of backlog.
TransferConfig.pm likely enabled
Backlog processing
Anytime there is a scenario that prevents the production gateways from processing data, you know there will be backlog data on top of current data that will eventually need to be processed.
TransferConfig.pm is usually disabled.
Debugging
A production gateway is already in placed and you don’t wish to tamper with the settings already provided.

6. Replicating gateway: How to replicate
Notice the copy command used “cp –p <source> <destination>”
It copies not only the files but the original timestamps of the files copied
Gives a good indication if my EngineConfig.pm and UserConfig.pm are exactly the same in each of the related sub-folder
In the duplicate gateway, edit the properties file and configure the appropriate settings.

7. Putty sessions: Case studies
Case Study 1: Missing data
Case Study 2: Please, that’s not my name.
Case Study 3: I lost my ID so that makes me bad?
Case Study 4: PERLIZE me!

8. Recommended Gateway Practices
Gently stop the gateway process.
Create an empty file named stop_gateway in the IN_DIR folder.
The gateway will stop the engine stage and will not parse the remaining raw files in the IN_DIR folder.
The gateway will process the post parsing stage to output the LIF.
Executing a kill on the process ID may leave unprocessed PIF that may need to get re-parsed again in the future.
Avoid unnecessary tampering of the Perl modules located in the gateway-framework and modules folders. Any customization work may get over-written when performing gateway upgrades or when applying fixpack.
For gateway config files, keep track of the changes you make by always making a backup of the original. Recall the “cp –p” command. Having the original also enables you to perform a diff against the newer configuration allowing you to identify the changes made.
Practice consistent naming convention
Recommended path and naming convention
LOG_FILE=/appl/virtuo/logs/gways/<vendor-subsys>_<release>.log
AUDIT_FILE=/appl/virtuo/logs/gways/<vendor-subsys>_<release>.audit

9. Recommended Gateway Practices (continued)
Monitor your disk usage.
Monitor the spool, logs and archive folders to prevent the 100% disk usage mark!
Ensure housekeeping are performed on logs, archived data, pt and bad files. Determine their retention period as you don’t want to keep them indefinitely.
Zip archive data where possible – set COMPRESS in properties file
May need to investigate when disk usage exceeds a certain threshold.
Monitor the audit and log files
Frequently check logs to detect any major errors being reported
Activate features for debugging only when necessary
Permanently increasing the LOG_LEVEL will significantly produce more message and may affect the performance of the gateway
Set LOG_LEVEL=1 (or 2 if necessary) and DEBUG=0 when debugging is not required.
Have a back-up plan ready when any stages of the parsing process gets affected.
Downtime on gateway box will cause backlog processing. Consider duplicating gateways to handle backlog gateway processing.
Downtime on related machines, i.e. database server downtime to say, due to maintenance work will prevent the loading of LIF. Consider allocating additional disk space to temporarily store LIF as you should allow gateway processing to continue to run as per normal. This reduces the amount of backlog work where you would only need to handle LIF backlog loading.

10. Recommended Gateway Practices (continued)
TNPMW supports distributed architecture.
Consider dedicating a server for your gateways for a more robust solution
Gateways are disk I/O intensive as we deal with massive amount of data so splurge on hard disk/file system where possible
Spread the gateway processing
Avoid the parsing process to run on the same file system as where the $GATEWAY_ROOT and configuration of the gateways reside
Avoid lumping all the parsing process onto a single /spool folder system if you have several gateways processing UMTS-type data. Consider the option of having say /spool01 and /spool02 to balance the load.
[virtuo]ls -la /spool01
drwxr-xr-x 2 virtuo virtuo Jul 2 17:33 ericsson-utran
drwxr-xr-x 2 virtuo virtuo Jul 2 17:33 alcatel-bss
drwxr-xr-x 2 virtuo virtuo Jul 2 17:43 huawei-mscs
[virtuo]ls -la /spool02
drwxr-xr-x 2 virtuo virtuo Jul 2 17:50 ericsson-mgw
drwxr-xr-x 2 virtuo virtuo Jul 2 17:50 ericsson-bss
drwxr-xr-x 2 virtuo virtuo Jul 2 17:55 nortel-bss
Benefits to this would:
cushion the effects when say, /spool01 was damaged allowing /spool02 to continue with the parsing of other gateway processing.
Reduce disk I/O and bottleneck issues

11. Thank you for your attention Questions?