1. Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA

2. Copyright 2009 Trend Micro Inc. Classification 9/19/2015 2 Agenda |  Frequent Case  Debug log and information  Troubleshooting  Q&A

3. Copyright 2009 Trend Micro Inc. No report was generated Logs need to collect on Daemon Server –/opt/TrendMicro/tdss/tdes/log/iae_log.txt –/var/log/cron Information –Customer expiration date –Device register to TMSP

4. Copyright 2009 Trend Micro Inc. No report was generated Cont. Normal debug log for log correlation and report generation Crontab task cron_iae.sh will be executed at 2:15 am every day

5. Copyright 2009 Trend Micro Inc. No report was generated Cont. Report will not be generated if Customer service get expired Customer without device registered

6. Copyright 2009 Trend Micro Inc. No incident in report Logs need to collect on Daemon Server –/opt/TrendMicro/tdss/tdes/log/iae_log.txt Information –Check if TDA log has uploaded Latest log time

7. Copyright 2009 Trend Micro Inc. Cannot access Admin console (err 404) Logs need to collect on Daemon Server –/var/log/httpd/access_log –/var/log/httpd/error_log Information –ps –ef | grep httpd –Netstat –anp | grep httpd

8. Copyright 2009 Trend Micro Inc. No Rsync log uploaded Logs need to collect on Access Server –Log receiver /home/tdalog/log /pre-post-exec.log /home/tdalog/log/ db_import_tda.log /home/tdalog/log/ db_import_tdm.log –Authentication (describe in next sides) Information –Ps –ef | grep tmsshd –Netstat –anp | grep tmsshd Listen on port 22

9. Copyright 2009 Trend Micro Inc. No Rsync log uploaded Cont. Normal procedure debug log of TDA log processing –/home/tdalog/log/pre-post-exec.log

10. Copyright 2009 Trend Micro Inc. CAS server case Problems caused by CAS failure : –Device register to TMSP fail –Customer portal login fail –Log uploading fail through RSYNC Logs need to collect on Access Server: –/var/log/messages –/var/log/cas_8000.log –/var/log/cas_8001.log –/var/log/cas_8002.log Information: Ps –ef | grep pound Ps –ef | grep rubcasd Normal log of CAS authentication –/var/log/cas_8000.log

11. Copyright 2009 Trend Micro Inc. CAS server case – Service down TDA register fail Check /var/log/messages Recover –If pound or cas service is down /etc/init.d/pound start /etc/init.d/rubcasd start

12. Copyright 2009 Trend Micro Inc. Data Gateway Case Problems caused by Data Gateway failure : –OCS Heartbeat / OCS log real-time cannot upload –T2 / T3 mitigation request cannot deliver to TMTM –SIC sample cannot upload from TMTM Logs need to collect on Access Server: –/opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/dg/WEB- INF/logs/dg.log –$APACHE_HOME/logs/ssl_request_log_dg –$APACHE_HOME/logs/error_log –/opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out Information: Ps –ef | grep httpd Ps –ef | grep tomcat Netstat –anp | grep 443 Netstat –anp | grep 8009

13. Copyright 2009 Trend Micro Inc. Data Gateway Case – DB disconnect TDA register fail Check apache error log: /usr/apache/logs/error_log

14. Copyright 2009 Trend Micro Inc. Customer portal cannot login Logs need to collect on Access Server: –/opt/TrendMicro/dg/apache-tomcat-6.0.18- 1/webapps/tms2/WEB-INF/logs/tms.log –$APACHE_HOME/logs/ssl_request_log_portal –$APACHE_HOME/logs/error_log –/opt/TrendMicro/dg/apache-tomcat-6.0.18- 1/logs/catalina.out Information Ps –ef | grep httpd Ps –ef | grep tomcat Netstat –anp | grep 443 Netstat –anp | grep 8009

15. Copyright 2009 Trend Micro Inc. Cannot get eMail notification Exclude Mail server problem, collect debug logs: –/root/infomation.log –/var/log/cron No Subscription DB connection fail

16. Copyright 2009 Trend Micro Inc.

17. FAQ Q: How to re-generate report manually? Login TDES machine, change dir to "/opt/TrendMicro/tdss/tdes/"; –1. Daily Report #php gendailydata.php user_id YEAR MONTH DAY example: "php gendailydata.php trend 2009 01 04" generate daily report of 2009.01.04 for customer "trend“ –2. Executive Report (Weekly / Monthly) #php genexecdata.php user_id START_DATE START_DATE yes m/w START_DATE: report start data with format “YYYY-MM-DD” START_DATE: report end data with format “YYYY-MM-DD” yes: if this report will be imported into DB m/w: monthly or weekly example: "php genexecdata.php trend 2009-01-01 2009-01-31 yes m" to generate monthly report for customer "trend" of 2009-01 –3. Upsell Report # php genupselldata.php user_id START_DATE END_DATE no example: " php genupselldata.php trend 2009-01-01 2009-01-31 no" to generate upsell report for customer "trend" from 2009-01-01 to 2009-01-31 Note: Upsell report will not be imported into DB and cannot download from admin console

18. Copyright 2009 Trend Micro Inc. FAQ Q: What the difference between two service type “TDS” and “TLMS”? A: “TDS” service is set to the customer, who has TDA deployed. “TLMS” service is set to the customer, who has both TDA and TMTM deployed. TMSP provide different report for these 2 service type, and for TLMS, TMSP provide additional UI for abnormal endpoints handling process.

19. Copyright 2009 Trend Micro Inc. FAQ Q: Why AE status show N/A on Admin console? A: AE (Abnormal Endpoint) status is designed for the customers, who has TMTM deployed, to show the mitigation failed client on TMSP UI. That means, only the customer with service type “TLMS” will show the AE details in TMSP admin UI as link.

20. Copyright 2009 Trend Micro Inc. FAQ Q: Whether the IAE will be updated ? A: Yes, TMSP IAE rules will be updated timely to detect new threats. Currently, the rule is updated monthly. In TMSP side, we get new threat rule from threat team and deploy to each site after verification.

21. Copyright 2009 Trend Micro Inc. FAQ Q: Can TMSP generate DOC format report? A: No, currently, TMSP can only generate report in PDF format.

22. Copyright 2009 Trend Micro Inc. FAQ Q: How to backup Database? A: Here are two type of DB backup, Full Backup and Lightweight Backup. Full backup : Backup full database of TMSP #mysqldump -uroot -p"$password" tdors_new > tdors_new_backup.sql

23. Copyright 2009 Trend Micro Inc. FAQ Lightweight DB backup: Backup customer info, configuration and some correlation result. # mysqldump -h $host -u$account -p"$password" tdors_new access_code admin_log admins contacts customer customer_expiration_conf customer_expiration_notif notification ocs_ast_event ocs_malware scsd_case scsd_case_followup scsd_case_status scsd_pattern scsd_pattern_status scsd_sample_info scsd_sample_summary t2t3_dce_result tb_devprofile> SJDC2.5.sql After backup entire database, it is recommended to store the backup file into another physical storage.

24. Copyright 2009 Trend Micro Inc. Q&A

25. Copyright 2009 Trend Micro Inc. Classification 9/19/2015 25 THANK YOU!