1. Troubleshooting with cim/trm logs
CIM TGSC
Dec 2014

2. Introduction Audience
This presentation is intended for CS Level 1 and Level 2 support. The information presented within this presentation is based on information from CIM and TRM versions and is presented in efforts to better support CIM/TRM at support levels 1 and 2.

3. Types of CIM/TRM Logs There are three basic types of CIM/TRM logs:
The TRM system log
TRM Error log
Processing Engine (PE) Logs
The TRM system log is the main application log file.
The TRM error log simply contains all of the generated errors.
Rarely used (insufficient context to diagnose / resolve issues).
PE logs contain the log messages from offline (i.e batch) job activity.
TRM System and TRM Error logs are considered ‘online’ logs while PE logs are considered ‘offline’ or ‘batch’ logs

4. Logging Configuration Files / Settings
<DeploymentDir>\trm\WEB-INF\classes\log4j.xml:
Controls logging for the CIM/TRM system and error log contents.
It does not control processing engine logging, which is hard-coded into the application itself in earlier versions but later versions do allow log4j.xml enabled changes for processing engine logging.
See the CIM 7.x Object Processing Guide: PE Logging Settings in log4j.xml section
3 Key Object Types Configured with the log4j.xml:
Appenders – Determine target destination for log messages.
The name of the destination log file
Can the maximum size and number of backups (versions) through the MaxFileSize and MaxBackupIndex parameters
Pre-configured CIM/TRM appenders:
SYSLOG – trm-system.log file
ERRLOG – trm-error.log file
CONSOLE – send to console display (e.g. Windows console (cmd.exe))
Max logging detail level (Threshold parameter) can be specified. This can limit the logging level specified by a logger
Layout – Control general formatting of log messages.
Loggers – Determine what gets written and where:
Name - specifies the source of the logging information (java class(es))
Level - specifies the logging level
Appender-ref - specifies which appender will receive the log messages (i.e. where the log message will be written).
Log setting changes normally occur through adjustments to the loggers. Appenders and Layouts are rarely modified. Changes typically involve logging levels or additional logging.

5. Changing Logging Levels
From most highest (most granular) to lowest (least granular):
TRACE
DEBUG
INFO
WARN
ERROR
FATAL
Default logging level is normally INFO level or lower
Some issues require more detailed logging.
Must modify log4j.xml.
Change logging level by modifying the level value on the appropriate logger.
Most commonly modified loggers:
<logger name="com">
All TRM classes
<logger name="org">
All TRM classes plus 3rd party classes used by TRM.
<logger name="com.teradata.tal.qes.ioc.jdbc.advice.SqlTrace">
Logs SQL’s submitted to the database by TRM prior to 6.6.x versions
<logger name="jdbc.sqltiming">
Logs SQL’s submitted to the database by CIM 6.6.x and 7.x versions when the jdbc.properties has been updated to have dbcp.logSQL = true.
May require modification of Threshold parameter on corresponding Appender.

6. Changing Logging Levels (Cont.)
Loggers are located under the following comment within log4j.xml:
<!--
===== loggers ============================================
-->
Example:
Assume sqltrace logging is set to INFO in TRM 6.5 (no logging of SQL statements in trm-system.log) and we wish to change it to DEBUG level so that SQL statements are put into the trm-system.log:
Change the ‘level’ to DEBUG
<logger name="com">
<level value="INFO" />
<appender-ref ref="SYSLOG" />
<appender-ref ref="ERRLOG" />
</logger> .
<logger name="com.teradata.tal.qes.ioc.jdbc.advice.SqlTrace">
<level value=“DEBUG" />
Notice there is no appender-ref. Loggers have inheritance relationships. Can inherit appenders from other loggers.
For example, "com.x.y" inherits the appenders of the logger "com".
Save log4j.xml changes and restart the application server
Some application servers are configured to automatically refresh but recommend restarting application server to ensure changes are activated.

7. Where to Find Logs
CIM/TRM System and error logs - The logs are named trm-system.log or trm-error.log, respectively (default names). The location of these logs depends on which application server is running CIM/TRM
For Tomcat, the default location is <tomcat root>\dist\logs sub-directory. where <tomcat root> is the installation path of the Tomcat installation directory.
For Websphere, the default location is <WAS install root>\AppServer\profiles\AppSrv01\dist\logs where <WAS install root> is the installation path of the Websphere installation directory.
As discussed on a previous slide, the name and location of the log files can be controlled by the appender object settings in the log4j.xml file.

8. Identifying the Correct Logs
GUI issues/errors will require the trm-system.log.
There may be multiple instances (archives) of the trm-system.log with the additional instances ending in a numeric suffix
ex. Trm-system.log.1
Some basic information is needed to locate the correct trm-system file and section within the log file.
The CIM/TRM User that experienced the issue
The CIM/TRM module the user was in and the specific steps the user performed leading up to the issue. In other words, the detailed steps to reproduce the issue.
The date & time in which the issue occurred.
The exact error message and/or symptoms that the user observed
Correct trm-system log file version will be the one where the date/time of the issue falls between its Modified date/time and the Modified date/time of the next older log file version.

9. Identifying the Correct Logs (Cont.)
Example: Date/Time of issue is December 7, 2010 at 1:30 PM

10. Identifying the Correct Logs (Cont.)
Job (batch) issues/errors will typically require the processing engine logs. Easier to identify.
User receives notification
Failure s include PE logs from job as zipped attachment.
Includes critical information such as
run_id
job_id
job name
CIM/TRM object name (e.g. communication’s name)
Date / time the job executed
will only include the PE logs if the job physically fails.
Other information is still included in the .

11. Identifying the Correct Logs (Cont.)
When we need to gather the PE logs manually:
The PE log files default to different locations depending on the webserver:
Tomcat is <tomcat install root>\webapps\dist\trm\processing-logs
WebSphere is <WAS root>\profiles\<AppSrv>\dist\trm\processing-logs
Under this base path, the logs are separated by different folders and each folder’s name consists of the date in which its logs were generated.
For example, PE logs from April 30, 2007 will be in the log folder named: " "
.. dist\trm\processing-logs\
The PE log files are named with the convention:
PE-run<run id>-job<job id>-<jobtype>-<time>.log
where:
Run id, job id: the run id and job id, respectively
jobtype is a descriptive name for job type, e.g. CommLeadGeneration or SegPlanRealize
time created in HH:mm:ss format (HH = 24 hour clock)

12. Identifying the Correct Logs (Cont.)
Now we can use the information from the user’s notification to locate the correct PE logs.
run_id
job_id
job name
Date / time the job executed
We use jobs execution date to locate the correct date folder:
.. dist\trm\processing-logs\<yyyy.mm.dd>
Next, we use the run id to locate all of the files that are associated with the run:
.. dist\trm\processing-logs\<yyyy.mm.dd>\PE-run<run id>*
For example, information is:
run date=Jan 4, 2011
run id=10006d005rzc
Then we want all of the files matching:
.. dist\trm\processing-logs\ \PE-run10006d005rzc*

13. Logging Configuration Files / Settings Cont.
The number of PE logs saved is controlled by a CIM Setting.
PURGE_WAIT_DAYS_NUM
A value of 0 disables the purging of processing engine logs.
A value >0 specifies the number of days that must pass before logs are eligible for purging.
This setting can be managed via the MAU (Refer to the Settings and Codes section of the CIM Metadata guide)

14. Using Logs to Locate Problems
Normally look for error messages or stack traces
Some are safe to ignore (drop table errors)
Important to know WHEN the issue occurred so the correct error can be identified and correlated to the reported issue. User id is often necessary as well.
Use stack trace / error message to search knowledge base
How to determine search keywords from stack trace
Look for root error message, may be multiple layers down in stack trace.

15. Example of Using Logs
Customer receives an error while attempting to create a Cross Segment analysis profile. They are using TRM version
Which type of TRM log would be most relevant to the investigation of this issue?
What additional information might we need from the customer to locate the correct log file?

16. Example (Cont.)
Which type of TRM log would be most relevant to the investigation of this issue?
The TRM System log
What additional information might we need from the customer to locate the correct log file? We need to know
the user’s TRM id
The approximate date / time the issue occurred
We already know steps to reproduce (see previous slide)
Let’s assume the following values for the required data:
TRM id = jaboyd
Date / time = Jan 13, 2011 / approx 12:22pm
Find section of log using date / time issue occurred and the TRM user id. Log entries typically have timestamp. For example:
:48:02,723 :[pool-2-thread-6] Task are001 boundary end: Remaining system memory after =

17. Example (Cont.) Note the following:
Here we find the appropriate section of the log: .
:22:18,187 INFO jaboyd:http-8080-Processor24 [com.teradata.trm.common.web.TRMRequestLogger] --> 1 /trm/crossSegment.do create [ ]==================method: GETquery string: novalidate=true&dispatch=create&componentType=54parameters: componentType = 54 dispatch = create novalidate = true
:22:18,437 DEBUG jaboyd:http-8080-Processor24 [com.teradata.trm.common.qes.ioc.jdbc.advice.SqlTrace] sql(ms)=141 SQL TRACE: "lock row for access select this_.STANDARD_RENDERER_ID as STANDARD1_100_0_, this_.UPDATE_DTTM as UPDATE2_100_0_, this_.GROUP_ID as GROUP3_100_0_, this_.UPDATE_USER as UPDATE4_100_0_, this_.DEFAULT_VALUE_TXT as DEFAULT5_100_0_, this_.STANDARD_ATTRIBUTE_ID as STANDARD6_100_0_ from CR_STANDARD_DEFAULT_RENDERER this_ where this_.GROUP_ID=?--parameters=[ vcrj3],userId:jaboyd,time: ,dispatch:create,ver: ,url:/trm/crossSegment.do,thread:http-sor24;"
:22:19,093 ERROR jaboyd:http-8080-Processor24 [com.teradata.trm.common.web.command.DefaultCreateCommand] [Assertion failed] - this argument is required; it must not be nulljava.lang.IllegalArgumentException: [Assertion failed] - this argument is required; it must not be null
at org.springframework.util.Assert.notNull(Assert.java:112)
at org.springframework.util.Assert.notNull(Assert.java:123)
at com.teradata.trm.analysis.web.struts.action.AnalysisFormHandler.updateFormWithDateUsageTypes(AnalysisFormHandler.java:160)
Note the following:
The timestamp of the error log message (highlighted in blue) is consistent with the reported date / time of the user activity.
The problem occurred under the reported TRM user (jaboyd – highlighted in green)
We know that it is related to the reported module (highlighted in red)
We now have an error in which to search the knowledge base (highlighted in yellow)

18. Resources for Additional Information
Standard Documentation
TRM v6 Supportability Orange Book
CIM/TRM Metadata Guides
TRM Knowledge Base
The TGSC attempts to write Knowledge Articles for reported issues and supply workarounds where possible. Therefore, the Knowledge Base and JIRA can be searched for known issues.
A few additional Knowledge Articles
KAP3B12AA: Basic Information Required to Submit a TRM Incident
KAP314FB56: Incident Create Checklist – Should be used by customers when creating incidents in
KAP315BC86: Incident Research and Escalation Checklist – Should be used by GTS and CS regional associates before escalating incidents to the GSC
KAP1BAAB2: How Can I Turn on TRM SQL Logging?