1. © 2007 IBM Corporation IBM Global Engineering Solutions IBM Blue Gene/P Trouble Shooting

2. IBM Blue Gene/P System Administration What are System Logs for Blue Gene/P  A System Admin needs to refer to these logs for troubleshooting or failure analysis.  Components of BG/P control system will store all output into the BG/P system log directory, /bgsys/logs/BGP.  bgpmaster  mcserver  mmcs_db_server  navigator  mpirund  IO node logs

3. IBM Blue Gene/P System Administration Naming of system logs except for IO node logs  Each system log except for IO node logs will be stored under /bgsys/logs/BGP whose name is based on the timestamp of its start time. A fine difference based on specification is explained below.  naming of past logs for bgpmaster and mcserver – - -yyyy-mmdd-hh:mm:ss.log e.g. ibmsn-mcserver-2007-0406-11:31:52.log  naming convention of past logs for mmcs_db_server, navigator, and mpirund – -bgdb0- -yyyy-mmdd-hh:mm:ss.log e.g. ibmsn-bgdb0-mmcs_db_server-2007-04-09-10:41:21.log,  Each ongoing log file is linked by a file whose name is *-current.log in the same directory. e.g. ibmsn-bgpmaster-current.log, ibmsn-bgdb0-navigator-current.log

4. IBM Blue Gene/P System Administration Naming of IO node logs  Each IO node log will be stored under /bgsys/logs/BGP whose name is based on the location.  naming of logs for IO node –Rxx-My-Nzz-Jnn.log e.g. R00-M0-N00-J00.log The following logs also exist in this directory. diagnostics logs in /bgsys/logs/BGP/diags installation logs ex. VerifyCables-yyyy-mmdd-hh:mm:ss.log InstallServiceAction-yyyy-mmdd-hh:mm:ss.log service action logs ex. ServiceNodeCard-Rnn-Mn-Nnn-yyyy-mmdd-hh:mm:ss.log ServiceLinkCard-Rnn-Mn-L0n-yyyy-mmdd-hh:mm:ss.log ServiceBulkPowerModule-Rnn-B-P0n-yyyy-mmdd-hh:mm:ss.log

5. IBM Blue Gene/P System Administration Access to System Logs via command line  The system logs can be accessed from command line  only to go /bgsys/logs/BGP, and find the log of the time zone of the purpose target system component $ bgpadmin@bgpdd1sys1:~> cd /bgsys/logs/BGP/ $ bgpadmin@bgpdd1sys1:~> ls bgpdd1sys1-mcserver-2007-1010-14:41:46.log bgpdd1sys1-mmcs_db_server-2007-1010-14:11:51.log bgpdd1sys1-mpirund-2007-1010-14:11:51.log R10-M1-N08-J00.log bgpdd1sys1-navigator-2007-1010-13:45:20.log bgpdd1sys1-bgdb0-mmcs_db_server-current.log@ bgpdd1sys1-bgpmaster-2007-1010-12:11:51.log bgpdd1sys1-bgdb0-mpirund-current.log@ bgpdd1sys1-bgdb0-navigator-current.log@ bgpdd1sys1-mcserver-current.log@ bgpdd1sys1-bgpmaster-current.log@ bgpdd1sys1-bgpmaster-2007-1009-15:28:18.log

6. IBM Blue Gene/P System Administration Access to System Logs via BG Navigator access to current system logs  The system logs can be also accessed from the BG Navigator 1.Click “System Logs” on the top page of BG Navigator 2.Click “Current”, or selected by default, on the System Logs home page, if you are looking for a current log You can also chose the other tabs, for “All” or “Old” logs as following page 3.Click the log file name which you are looking for e.g. mmcs_db_server current log 4.Display the log file via Log Viewer on a new pop-up window 1 2 3 4

7. IBM Blue Gene/P System Administration Access to System Logs via BG Navigator access to past system logs 1.Click “System Logs” on the top page of BG Navigator as above 2.Click “All” or “Old” tab, on the System Logs home page, if you are looking for a past log 3.If needed, expand the bullet of the filter option and enter the timestamp 4.Click the log file name which you are looking for 5.Display the log file via Log Viewer on a new pop-up window 1 2 3 4 5

8. IBM Blue Gene/P System Administration Viewing logs via Log Viewer  Log Viewer has these features which assist you in the following areas: 1. Movement: paging backwards and forwards, moving to a specific timestamp 2. Refresh: refreshing your log manually or automatically, in viewing a current log 3. Filtering: selecting only the specific log file lines, can be applied to the entire log file or to the lines that are currently being viewed. 4. Highlighting: –auto highlighting, for example, error message in red character –manual highlighting particular word(s) 5. Hints: Hover your mouse over any of the log file viewer icons, and the viewer provides a hint that describes that icon’s function 1 2 3 4-1 4-25