1. 6-1 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Lesson 2: Troubleshooting Fibre Channel Storage

2. 6-2 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A After this module, you should be able to do the following: Troubleshoot Fibre Channel Storage This lesson was added by instructor. Slides can be downloaded at www.renright.com/work/fc.pptx Learner Objectives

3. 6-3 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Components in a Fibre Channel Storage Area Network SPA SPB ESXi VMKVMK VMKVMK FC HBAs kernel queue FC Switches Storage Processers FE Ports A B C D SPA A B C D 013013 SPB Disk Array Enclosure

4. 6-4 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Storage Processors Storage Processors. SPs consist of three major components – Ports, cache and processors SP are always pair for redundancy. Sometimes they are designated SP A & SP B SPB A B C D 012012

5. 6-5 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Logical front end A B C D A B C D FE Fibre Channel Ports Front End Ports are used for host connectivity

6. 6-6 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Cache Directory Table (TOC) Cache is always configured in pairs. Parity or Mirrored Protected What Should goes through cache? Everything. Reads Writes Commands Cache slots Memory is either configurable or global

7. 6-7 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A SPA SPB 012012 Logical Back End 012012 Back End Ports to connect to physical disks

8. 6-8 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Physical Disks A B C D SPA A B C D SPB Disk Array Enclosure Most arrays use fiber cable through hubs to connect to the disks. Disks are dual ported fibre channel or SSD disks Link Control Cards

9. 6-9 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Cache Function Directory Table (TOC) A B C D SPA A B C D SPB Read Hits, Read Misses, Fast Writes, Delayed Fast Writes Read hit ACK Fast Write 01230123 01230123

10. 6-10 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Module 2: VMware ESX/ESXi Server and Symmetrix VMAX Connectivity - 10 SPC-2 Bit Compliance Devices Required for the proper functioning of: VMware vSphere VMware Site Recovery Manager With SPC-2 enabled, devices are identified by their proper Network Authority Address (NAA) vSphere will identify the devices as naa.DeviceWWN e.g. naa.60000970000194900632533030303444 The NAA identifier is unique for all paths where the ESXi host sees the device

11. 6-11 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Fibre Channel SAN Components VMware vSphere 4.1: Install, Configure, Manage – Revision A FC FE Ports Storage Admin SAN Admin vSphere Admin FC Switch Ports FC HBA Ports

12. 6-12 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A The FC HBA storage array is not supported. Check VMware Compatibility Guide. Verify that the LUN is presented correctly to the ESXi host. The LUN is LUN Masked all the ESXi hosts. The LUN Zoned to the ESXi hosts. The LUN is not set to read-only on the array. HOST ID on the array for the ESXi LUN is between 0 - 255. The storage device is malfunctioning. Use hardware diagnostic tools to identify the faulty component. Mixing multivendor HBAs Never mix Qlogic with Emulex or Brocade Possible Cause: Hardware-Level Issues

13. 6-13 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Located in /proc/scsi/lpfc/# or /proc/scsi/qlaxxxx/# For example: /proc/scsi/qla320/4 & /proc/scsi/qla320/5 The last number is a file without an extension. Reading the file will give you: Vender Model speed WWNN & WWPN FE Ports connected etc Locating Fibre Channel HBA Properties

14. 6-14 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Viewing FC HBA Information

15. 6-15 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Viewing FC Storage

16. 6-16 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A 90% of FC storage problems Check to see that the Limit for maximum LUNs not reduced In advanced options, check to see that disk.MaxLUN is set to 256 Check to see that sparse LUNs not initiated In advance options, check to ensure that disk.SupportSparseLUN is set to 1 Back End Thin Provisioned storage not oversubscribed Ensure that VAAI is utilized Possible Causes - Misconfiguration

17. 6-17 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A NPIV assigns a virtual World Wide Name and virtual N_Port ID to an individual virtual machine. NPIV gives each virtual machine an identity on the SAN. NPIV benefits: Track storage traffic per virtual machine. Zone and mask LUNs per virtual machine. Leverage SAN quality-of-service per virtual machine. Improve I/O performance through per virtual machine array-level caching. Configure NPIV if you have the following requirements: A management requirement to monitor SAN LUN usage at the virtual machine level A security requirement to be able to zone a specific LUN to a specific virtual machine N_Port ID Virtualization WWN ESXi host

18. 6-18 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A NPIV requires the following: Virtual machines use RDMs. Fibre Channel HBAs support NPIV. Fibre Channel switches support NPIV. ESXi hosts have access to all LUNs used by their virtual machines. NPIV cannot be used with virtual machines configured with VMware vSphere® Fault Tolerance (FT). N_Port ID Virtualization Requirements

19. 6-19 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A 19 VMFS Volume Resignaturing – Mechanics (1) AAAA MyDatastore (Signature) naa.123456 With SPC-2, array serial number plus LUN ID are used to create naa ID. naa ID is used to create a disk signature. The disk signature is stored in hidden files, called metadata, at the base of each VMFS datastore. Metadata

20. 6-20 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A VMFS Volume Resignaturing – Mechanics (2) A VMFS datastore name is simply a soft link to a disk signature. Module 5: Data Replication

21. 6-21 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A VMFS Volume Resignaturing – Mechanics (3) While you and I work with the datastore name, the ESXi host works with the disk signature. This allows for the datastore to be renamed without breaking the connection

22. 6-22 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A 22 VMFS Volume Resignaturing – Mechanics (4) Replicating a LUN replicates original disk signature. ESXi will detect it is a replica because the replicas disk signature does not match naa number. BBBB (Computed signature) Computed signature and volume signature not the same. Volume must be a replica. AAAA MyDatastore (Signature) naa.123456 AAAA MyDatastore (Replica) (Signature) naa.654321

23. 6-23 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A 23 ESXi 5.X – VMFS Volume Resignaturing By default, an ESXi host will not display a datastore with a mismatched disk signature. If host can see orginal datastore, it will see two datastores with the same signature. Can then read from or write to either. Can cause inconsistent data. VMware vSphere has the ability to individually resignature and/or mount VMFS volume copies Use vSphere Client Or the CLI utility vicfg-volume for ESXi

24. 6-24 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Module 5: Data Replication - 24 ESX 5.x – VMFS Volume Resignature – vSphere Client

25. 6-25 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A 25 ESXi 5.0 – VMFS Volume Resignature – CLI Cannot be mounted as original is still online – Can be resignatured -m – non persistent mount can only be done via CLI

26. 6-26 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A vCenter Server Storage Filters

27. 6-27 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Fixed (Default for most Active/Active arrays) Works with a preferred path and an active path Most Recently Used (Default for Active/Passive Arrays) Works with an active path on the active storage processor Round Robin (default for some Active/Active Arrays in vSphere 5.1) Assigning Fixed to an Active/Passive array can result in LUN thrashing with prevents access to the LUN. NMP Path Selection Policies

28. 6-28 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A To List SATPs & PSPs esxcli storage nmp satp list esxcli storage nmp psp list Change new paths on A/A arrays to Round Robin: esxcli storage nmp set –s VMW_SATP_DEFAULT_AA –P VMW_PSP_RR By default Round Robin changes paths every 1000 I/Os. To change the I/O or to set it to bytes use esxcli storage nmp psp roundrobin deviceconfig set --type –B XXXX –d naa.xxxxxxxxxxxxxxx Or --type –I 1 –d naa.xxxxxxxxxxxxxxxxxx ESXCLI Commands for changing Path Selection Policies

29. 6-29 © 2013 VMware Inc. All rights reserved VMware vSphere: Troubleshooting Workshop – Revision A Multipathing with Fibre Channel Multipathing allows continued access to SAN LUNs in the event of hardware failure. It also provides load balancing. VMware vSphere: Install, Configure, Manage – Revision A (optional) FC FE Ports SPBSPA