1. 1EMC CONFIDENTIAL—INTERNAL USE ONLY Unified PUHC Issues Troubleshooting Manfred Zhuang 2014/9/30 Space Issues & Running Tasks

2. 2© Copyright 2014 EMC Corporation. All rights reserved. How to clean up /nbsnas – Go through the KBs – Manually check files and directories How to clean up running tasks – Go through the KBs Q&A Agenda

3. 3© Copyright 2014 EMC Corporation. All rights reserved. Two basic KBs to clean up /nbsnas – Manually clean up Celerra and VNX: How to clean up /nas or /nbsnas for a NAS or File OE upgrade https://support.emc.com/kb/173937 – Automatic clean up script Celerra and VNX: How to clean up the /nbsnas directory before proceeding with a VNX file upgrade. https://support.emc.com/kb/184085 Space Issues How to clean up /nbsnas

4. 4© Copyright 2014 EMC Corporation. All rights reserved. Two useful KBs to space issues – Pre-Upgrade Health Check (PUHC) fails with 'Error HC_CS_14505083116: Command failed. * Command: /nas/sbin/check_inflight_tasks -puhc_format.' https://support.emc.com/kb/84131 Usually can get 1%-2% free space – Celerra and VNX: How to reduce the time that it takes for the hourly nasdb_backup to complete. https://support.emc.com/kb/9731 Got 17% free space once Space Issues How to clean up /nbsnas (Cont’d)

5. 5© Copyright 2014 EMC Corporation. All rights reserved. List all large files in /nbsnas # find /nbsnas -xdev -size +20M -ls 63375 27396 -rwxr-xr-x 1 nasadmin nasadmin 28018326 Apr 9 06:47 /nbsnas/bin/nas_cmd 31688 23360 -rwxr-xr-x 1 root root 23890725 Apr 9 06:46 /nbsnas/dosfs/bin/nas.exe 110940 37880 -rwxr-xr-x 1 nasadmin nasadmin 38743399 Apr 9 06:46 /nbsnas/sbin/statsd 110739 76684 -rwxr-x--x 1 nasadmin nasadmin 78439648 Apr 9 06:47 /nbsnas/sbin/apl_task_mgr 110819 47676 -r-xr-xr-x 1 nasadmin nasadmin 48764460 Apr 9 06:46 /nbsnas/sbin/mgmtd Space Issues How to clean up /nbsnas (Cont’d)

6. 6© Copyright 2014 EMC Corporation. All rights reserved. Space Issues How to clean up /nbsnas (Cont’d) Compare the directory size with lab array (cslabs.emc.com) and check if any files can be deleted/moved – List all directories in /nbsnas and their size: [root@VNX nbsnas]# du -sxm * | sort –n – Pre-check script also checks the difference in size between /nas and /nbsnas. The bigger the difference the more free space needed in /nbsnas.

7. 7© Copyright 2014 EMC Corporation. All rights reserved. Space Issues How to clean up /nbsnas (Cont’d) 6 sys 8 tmp 12 site 12 tasks 13 j_lib 14 bdb 20 etc 25 log 26 server 29 dosfs 39 bin 48 tools 56 dos 86 lib 104 opt 248 http 268 sbin 363 jserver 507 udoctor Example of /nbsnas normal usage: [root@VNX nbsnas]# du -sxm * | sort -n 0 quota 0 rootfs 0 tomcat 1 api 1 cic 1 dev 1 install 1 lock 1 lost+found 1 nasdefault 1 nasdos 1 rdf 1 stat_groups 1 stats 1 symapi 1 symcli 1 var 1 volume 2 man

8. 8© Copyright 2014 EMC Corporation. All rights reserved. Space Issues How to clean up /nbsnas (Cont’d) 0 tmp 0 tomcat 0 var 0 volume 1 api 1 cic 1 lost+found 2 man 13 j_lib 19 etc 25 log 39 bin 86 lib 104 opt 188 tools 199 jserver 233 http 268 sbin 507 udoctor Example of /nas normal usage: [root@VNX nas]# du -sxm * | sort -n 0 bdb 0 dev 0 dos 0 dosfs 0 install 0 lock 0 nasdefault 0 nasdos 0 quota 0 rdf 0 rootfs 0 server 0 site 0 stat_groups 0 stats 0 symapi 0 symcli 0 sys 0 tasks

9. 9© Copyright 2014 EMC Corporation. All rights reserved. Space Issues How to clean up /nbsnas (Cont’d) Reboot the CS

10. 10© Copyright 2014 EMC Corporation. All rights reserved. KB184085 (Script) KB173937 (Step 4-7) KB84131KB9731 Check the files in /nbsnas and /nas Compare the directory size Space Issues How to clean up /nbsnas (Cont’d) Recommended troubleshooting sequence: If none of above works, try rebooting the Control Station

11. 11© Copyright 2014 EMC Corporation. All rights reserved. Space Issues How to clean up / 1 media 1 misc 1 mnt 1 nfs 1 NFS 1 NFS_JD_dir 1 nfsusr2 1 p.gz 1 pre-replication.rpt 1 SCCS 1 screenlog.0 1 selinux 1 srv 1 tmp 2 celerra 2 root 6 boot 7 bin 23 sbin Example of / normal usage: [root@VNX /]# du -sxm * | sort -n 0 file 0 netfile 0 proc 0 sys 0 VNX5700-139.140922_1423 1 amy_nfs3_dir 1 amy_nfs_ckpt_dir 1 amy_nfs_RR 1 cifs 1 ckp 1 ckpt 1 ckpt1 1 clean_nbsnas.tar.gz 1 dev 1 fs 1 ftp_139.cap 1 i386 1 JD_dir 1 lost+found 32 lib 36 etc 47 opt 70 nasmcd 82 home 106 tftpboot 265 var 749 usr 970 nbsnas 974 nas

12. 12© Copyright 2014 EMC Corporation. All rights reserved. Normally, it is caused by the replication task on Data Movers hanging. Very rare situation, it was caused by the scheduled task or hanging tasks on the Control Station. Check the details of running tasks # /nas/sbin/check_inflight_tasks Control Station: Running Tasks: 241847 server_2 : Running Tasks: APM000951023680000_2_272:0 APM000951023680000_2_276:0 APM000951023680000_0_12036:4294979332 Running Tasks How to clean up running tasks

13. 13© Copyright 2014 EMC Corporation. All rights reserved. Apply KB53810 – Pre-Upgrade Health Check fails with 'The following tasks are running in the system: 2250036' message https://support.emc.com/kb/53810 – Check if there are still running tasks before rebooting Control Station and Datamovers If there are many tasks need to be deleted, apply KB79924 – KB79924 can be used to create a script to delete the hung tasks – PUHC and install_mgr fail with error messages https://support.emc.com/kb/79924 – Check if there are still running tasks before rebooting Control Station and Datamovers Running Tasks How to clean up running tasks (cont’d)

14. 14© Copyright 2014 EMC Corporation. All rights reserved. Check if the running tasks are cleaned up # /nas/sbin/check_inflight_tasks Running Tasks How to clean up running tasks (cont’d)

15. 15© Copyright 2014 EMC Corporation. All rights reserved. A special running task type: nasdb_backup – Upgrd-ckv will report error if nasdb_backup is running. – Nasdb_backup runs hourly. It usually starts on the hour. – Check the time on the control station: [root@VNX5700-139 /]# date Tue Sep 23 16:01:49 CST 2014 – If the time is several minutes past the hour, wait for the nasdb_backup to complete. Running Tasks How to clean up running tasks (cont’d)

16. 16© Copyright 2014 EMC Corporation. All rights reserved. Check the details of running tasks check_inflight_tasks KB53810 (KB79924 if many tasks) Check if there are still running tasks before rebooting CS and DMs Reboot CS or DMs Check if the running tasks are cleaned up Running Tasks How to clean up running tasks (cont’d) Recommended troubleshooting sequence:

17. 17© Copyright 2014 EMC Corporation. All rights reserved. Q&A