Getting SP Collects (What are SP Collects?)

SP collects – SP Logs Each SP is a Windows machine underneath They have a perl script installed on all SPs that gather and generate a log file in c:\dumps The time it takes to generate logs depends on how big the array is, how busy it is, and how eventful it’s life may have been. We’re going to step through how to kick off SP collects, monitor them while building and retrieval

Starting SP collects remotely From the Control Station /nas/tools/.get_spcollect /nas/sbin/navicli /nas/tools/collect_support_materials -getsp RemotelyAnywhere From the Windows Console ssh session to the block on VNX Computer Management command prompt Unisphere Unisphere Service Manager RCM_Unisphere naviseccli over ESRS

Please note… It doesn’t make any difference which method you use to start or retrieve the logs – it’s the exact same script (spcollect.pl) running on the SPs For example, you can start SPA SP collect using naviseccli from the desktop, SPB from Unisphere and retrieve SPA’s SP logs using RemotelyAnywhere and SPB’s logs using navicli

Control Station /nas/sbin/navicli Get the IP addresses of the SP: grep SP /etc/hosts Test SP connection by getagent on both SPs /nas/sbin/navicli –h SPA_IP getagent /nas/sbin/navicli –h SPB_IP getagent Start SP collect script on both SPs /nas/sbin/navicli –h SPA_IP spcollect /nas/sbin/navicli –h SPB_IP spcollect

Get the SPs IP addresses Test connectivity and the management server on both SPs using getagent Start the SP collects on both SPs

Control Station /nas/sbin/navicli To monitor: To retrieve /nas/sbin/navicli –h SPA_IP managefiles –list | grep runlog To retrieve /nas/sbin/navicli –h SPA_IP managefiles -retrieve /nas/sbin/navicli –h SPB_IP managefiles -retrieve

SPA SP Logs gathering SPB SP Logs gathering SPA SP Logs gathering SPB SP Logs gathering Both SPs Logs gathered

Control Station /nas/tools/.get_spcollect It operates serially : Start SP collects on SPA Monitor SPA log gathering until logs gathered Transfer SPA log to /nas/var/log Start SP collects on SPB Monitor SPB log gathering until logs gathered Transfer SPB log to /nas/var/log Combine SPA & SPB logs into one zip file /nas/var/log/SPCOLLECT.zip

Starting Monitoring SPA Retrieval Starting Monitoring SPB Retrieval Combine Pros Does the whole thing for you! Cons Slower as done serially. Can time out if it takes too long. Sometimes retrieves an incorrect file instead of the expected one.

Control Station /nas/tools/collect_support_materials -getsp This tool gathers File logs and optionally get SP collects. It start SP collects, then gathers file logs and when done waits for the SP Collects to finish gather before amalgamating them in one zip file in /nas/var/emcsupport

Pros Does everything for you Cons Also gathers file logs which we are less interested in.

RemotelyAnywhere From the Windows console Login to RemotelyAnywhere on each SP (clariion1992/clariion1992) Remote Control Login to the SP (clariion/clariion!) Open a console window (cmd) Run ‘spcollect.pl’

RemotelyAnwhere

RemotelyAnwhere To retrieve the SP collect file once the script has finished running… Logout* of the console (Windows GUI) Select File Manager Use the File Manager to transfer the file back to your desktop from c:\DUMPS on the SP. It’s easiest to select the file by ordering by most recently modified. If there’s a runlog file, then wait and refresh. * It is vitally important not to select Shutdown instead of Logout – you would shutdown the SP

RemotelyAnwhere Transfer to: (Local) C:\DUMPS SPCollect

RemotelyAnwhere ssh session to the block on VNX Login as clariion1992/clariion1992 Run spcollect.pl We need to collect the SP Collects using a different method, such as RemotelyAnywhere File Manager

RemotelyAnwhere Computer Management command prompt Login to Remotely Anywhere, Computer Management, Command Prompt. Run spcollect.pl

Unisphere Client Ensure you are on the correct system using the Dashboard

Unisphere Client System, Diagnostic Files, Generate Diagnostic files for both SPA & SPB separately.

Unisphere Client To monitor, Get Diagnostics Files… Order by Date Transfer to here runlog: still gathering Refresh periodicially until runlog is gone

Unisphere Client To retrieve the files, Select the Log and Transfer Order by Date Select Log Transfer Confirm Monitor Transfer Ignore RP files

Unisphere Service Manager Verify we’re on the correct system (S/N) Diagnostics, Capture Diagnostic Data

Unisphere Service Manager Save the SP Collects once gathered When Logs are gathered click Save As Choose your location to save to Save

RCM_Unisphere Login to the array ftp://ftp.emc.com/pub/rcm/vnx/tools/rcm_unisphere/RCM_Unisphere_1.25.exe Login to the array

RCM_Unisphere Update your default log location setting (Only needs to be done once)

RCM_Unisphere Start SP Collects Health checks Generate Diagnostics for both SPs

RCM_Unisphere To Monitor Get Diagnostic Files (SP A & B) Look for runlog If runlog exists, logs are still gathering Close and reload the page to get an update.

RCM_Unisphere To Retrieve Select the SP Collect Select the type, pre/post. Click transfer

naviseccli Using cmd.exe To start Verify connectivity getagent Start SP collects on both SP spcollect

naviseccli Using cmd.exe To monitor Look for the runlog file The grep utility is very useful – copy it from \\corpeufs9\Techsupp\GTS RCM\_Daniel\tools\grep.exe into C:\Program Files (x86)\EMC\NGTRiiAGE

naviseccli Using cmd.exe To retrieve managefiles -retrieve

Summary There are many ways of starting SP Collects There are many ways of retrieving SP Collects All yield the exact same log files. Running in parallel is faster and more reliable, but takes more real time.

Recommendations Block only Unified Unisphere client /nas/tools/.get_spcollect parallel

Troubleshooting Triiage is blank for an SP File was transferred correctly. Restart the management server on that SP and take fresh logs. File may not be transferred correctly. Check the file size on the SP against the file on your PC and retransfer if different Yes No Zip Valid? PCE: Schedule an SP reboot UE: Proactive SP reboot Customer permission is required. If possible verify host connectivity to the other SP via triiage or Unisphere client beforehand. No OK now? Yes Done

Questions

Restarting Management Agent http://SP_IP/setup Login with Unisphere credentials

Restarting Management Agent Select ‘Restart Management Agent’ Confirm Management agent restart Note: There is no production impact to management agent restart

Restarting Management Agent If on a control station and getagent is working… navicli networkadmin -restartcimom