1. Cisco Storage Networking SAN Intelligence
James Long
Storage Networking Systems Engineer

2. Agenda
VSAN
Traffic Management
Diagnostics

3. Virtual SAN (VSAN) VSANs perform two functions:
Traffic isolation
VSAN membership is hardware enforced
No special drivers or configuration required for end nodes (hosts, disks, etc)
Traffic is tagged at Fx_Port ingress and carried across EISL (enhanced ISL) links between switches
Service isolation
Each fabric service (zone server, name server, login server, etc.) operates independently in each VSAN
Each VSAN is configured and managed independently
VSAN = Virtual Switch or Fabric
Fibre Channel Services for Blue VSAN
VSAN header is removed at egress port
Fibre Channel Services for Red VSAN
Trunking E_Port (TE_Port)
Enhanced ISL (EISL) Trunk carries tagged traffic from multiple VSANs
Per-VSAN timers include R_A_TOV, D_S_TOV, E_D_TOV
Up to 256 VSANs per switch (software limit imposed to conserve resources)
Architectural limit is 1024 per switch and 4093 per fabric
Trunking E_Port (TE_Port)
VSAN header is added at ingress port indicating membership
Fibre Channel Services for Blue VSAN
No special support required by end nodes
Fibre Channel Services for Red VSAN

4. VSANs and Zones are Complimentary
Hierarchical relationship
First assign physical ports to VSANs
Then configure zones within each VSAN
VSANs increase scalability
Each VSAN has its own address space
FSFP converges quicker
VSANs increase availability
Service failures are confined per VSAN
Route changes are confined per VSAN
Fabric rebuilds are confined per VSAN
VSANs increase security
Distributed services are confined per VSAN
Traffic is confined per VSAN
VSANs support traffic engineering
Route metrics can be configure per VSAN
ISLs can be restricted per VSAN
Switch 1
VSAN 2
Disk2
Disk3
Zone A
Host1
Disk1
Zone C
Zone B
Disk4
Host2
Zoneset 1
VSAN 3
Zonesets: 1000 per physical fabric
Zones: 2000 per physical fabric
Zone members: per physical fabric
Zone A
Host4
Zone B
Host3
Disk5
Disk6
Zoneset 1

5. Continual VSAN Innovation
VSANs are leveraged from large scale Ethernet expertise (VLANs)
VSANs represent key innovation in SAN design, provisioning and management
Continual enhancements to further drive innovative, cost effective SAN solutions
VSANs enable cost-saving consolidation of SAN Islands
Separate services per VSAN zoneset, FSPF, RSCN, etc
Separate policies per VSAN
VSAN-based SPAN feature
VSAN-based accounting
VSANs are extended to iSCSI and FCIP services
VSAN-assigned iSCSI hosts
VSAN-to-VLAN mapping
VSAN trunking over FCIP
SANOS v1.3
FICON, fabric, and routing enhancements
VSAN-based FICON services
FICON intermix with VSANs
Inter-VSAN routing
VSAN-based fabric timers
VSAN-based QoS services
SANOS v1.2
VSAN-based management enhancements
VSAN-specific roles
SANOS v1.1
SANOS v1.0

6. VSAN Is Now The Standard
Cisco VSAN technology is now standardized by ANSI INCITS T.11
The standard name is Virtual Fabric (VF)
VF is included in
Framing & Signaling specification (FC-FS-2)
Link Services specification (FC-LS)
Switch Fabric specification (FC-SW-4)
Only Cisco MDS 9000 has this capability today
Requires hardware support
Other vendors need new ASICs to support VF (Forklift upgrade)

7. VSANs Enable Traffic Engineering
Virtual SANs enable resource allocation and preference per virtual fabric
Each VSAN runs a separate instance of FSPF routing – independent forwarding decisions per VSAN
EISL trunk links can be tuned for preferential routing per VSAN
Different recovery paths can be configured per VSANs
VSANs can be carried securely across metro and wide area networks via FCIP, SONET, DWDM, or CDWM
Quality of Service (QoS) per VSAN to give preferential treatment at points of congestion in network
VSAN-based fabric Total directors: 4 Total fabric switches: 8 Total client port count:1028 {max 7:1 fan-out} Total ISL port count: 96
Switch B
Domain 100 Domain 200
42 client ports per fabric switch (VSAN-enabled)
16 Gbps EISL Trunk
Metric 50 - Red VSAN
Metric Blue VSAN
8 Gbps EISL Trunk
Metric 50 - Blue VSAN
Metric Red VSAN
168 client ports per director (VSAN-enabled)
Domain 104 Domain 204
Switch A
Preferential routes can be configured per-VSAN to engineer traffic patterns.

8. VSANs Allow Sharing of DR Facilities
Virtual SANs can be carried between data centers over various transport facilities
Allows consolidation of DR facilities while maintaining traffic isolation
Can set preferential usage of DR transport per VSAN – routing metrics
Various wide/metro area facilities can be used securely: FCIP (PoS, ATM, Metro Ethernet), SONET, SDH, DWDM, CDWM
Cisco MDS 9000 family provides traffic statistics per VSAN (chargeback possibility)
Full fabric discovery per-VSAN through Cisco Fabric Manager
Switch A Domain 100 Domain 200
1Gbps EISL Trunk
Metric 50 - Blue VSAN
Metric Red VSAN
8 Gbps EISL Trunk
Metric 50 - Red VSAN
Metric Blue VSAN
DWDM or CWDM
SONET or SDH
IP Routed Network (FCIP)
Domain 104 Domain 204
Switch B

9. VSANs and Non-Cisco Switches
The VSAN feature involves a tagging mechanism which is not understood by 3rd party switches
Cisco MDS 9000 Family supports heterogeneous switch interoperability
Cisco “Interoperability Mode” is configured per VSAN – no loss of functionality in other VSANs
Cisco MDS 9000 switches negotiate an E_Port with non-Cisco switches
The connecting E_Port on the MDS 9000 belongs to a VSAN
The entire 3rd party switch, including all its ports and services, will reside in the VSAN of the connecting E_Port on the MDS 9000
Enhanced ISL (EISL) trunks carrying numerous VSANs
Simple ISL links
Non-Cisco Fabric Switches
E_Ports

10. WWN Based VSANs Port Based VSANs WWN Based VSANs
Move requires
Reconfiguration
on SW2
Move without
reconfiguration
VSAN membership based on pWWN of server/target
Fabric-wide distribution of configuration using CFS
No re-configuration is required when a server/target moves
VSAN membership based on physical port of switch
Reconfiguration is required when a server or target moves

11. VSAN Numbering Rules VSAN 0 Reserved VSAN - not used VSAN 1
Automatically configured by the switch as the default VSAN
All ports are originally in VSAN 1
VSAN
User-configurable VSANs
A maximum of 256 VSANs per switch can be created in this number range
VSAN 4094
Reserved VSAN
Called the “isolated VSAN”
Used to isolate ports whose VSAN has been deleted
Not propagated across switches
VSAN 4095
Configured VSANs
Cisco MDS 9000 with VSANs
VSAN 10
VSAN 20
VSAN 30
Trunking E_Port (TE_Port)
In the figure, VSAN 30 is not propagated across EISL, because it is not configured in the local switch but is configured on the remote switch. Instead of the host device on the local switch being able to connect to the remote switch, it has been placed in the isolated VSAN 4094 because the port’s VSAN (VSAN 30) has been deleted from the local switch configuration.
Trunking E_Port (TE_Port)
Port is in VSAN 4094 (Isolated VSAN)
VSAN 10
VSAN 20
Host is isolated from the fabric
VSAN 30
Configured VSANs

12. Inter-VSAN Routing (IVR)
Industry First!
Inter-VSAN Routing (IVR)
Routes traffic between adjoining VSANs directly
Routes traffic between separated VSANs via transit VSAN
Requires unique domain IDs throughout entire SAN
VSAN 1
IP Network
VSAN 3
VSAN 4
Transit VSAN
VSAN 2

13. Inter-VSAN Routing (IVR) Sharing Resources Across VSANs
Industry First!
Inter-VSAN Routing (IVR) Sharing Resources Across VSANs
Allows sharing of centralized storage resources across VSANs without merging VSANs
Works for all MDS 9000 switches with a software upgrade to SAN-OS 1.3
Preserves the benefits of VSANs yet selectively allows traffic between VSANs
Enables multi-VSAN access for blade servers
Transparent to third-party switches
256 active VSANs per switch
Engineering VSAN 1
Marketing VSAN 2
IVR
Up to 256 VSANs per switch (software limit imposed to conserve resources)
Architectural limit is 1024 per switch and 4093 per fabric
Blade Center VSAN 1
Marketing HR
IVR
IVR
Shared Tape VSAN 3
HR VSAN 3
Marketing VSAN 2

14. Inter-VSAN Routing (IVR) Resilient SAN Extension
Industry First!
Inter-VSAN Routing (IVR) Resilient SAN Extension
MAN/WAN circuit failures do not trigger Principal Switch Selection
Changes to fabric services are not propagated between sites
Restrict fabric control traffic such as SW-RSCNs and Build Fabric/Reconfigure Fabric (BF/RCF) to local VSANs
Works with any transport service (FC, FCIP, FC/SONET, FC/xWDM)
Inter-VSAN Connection with Completely Isolated Fabrics
EISL#1 in Port Channel
Replication VSAN_1
Replication VSAN_4
SONET or Metro DWDM
IVR
Transit VSAN_3
IVR
Local VSAN_2
Local VSAN_5
EISL#2 in Port Channel

15. IVR with FCID Translation
Enable Heterogeneous SAN Consolidation
MDS
VSAN 1
Core Switching/Routing
Superior to Brocade LSANs
VSAN 2
VSAN 3
McData Fabric # 3
Brocade Fabric # 1
Brocade Fabric # 2
No restriction of unique Domain IDs

16. Example: IVR with FCID Translation
10.1.1
20.1.1
VSAN 2
VSAN 1
50.1.1
30.1.1
VSAN = 1
SID =
DID =
VSAN = 2
SID =
DID =
10.1.1
20.1.1

17. Agenda
VSAN
Traffic Management
Diagnostics

18. Head-of-Line (HOL) Blocking Explained
HOL blocking can adversely affect performance across the entire fabric
Host A is issuing write operations to 100MB/sec
Host B is issuing writes operations to 40MB/sec
The tape interface runs out of BBCs, congestion occurs in switch B
Switch B runs out of BBCs, congestion occurs in switch A
Switch A runs out of BBCs, all hosts slow to 15MB/sec
Disk shelf capable of sustaining 200MB/sec FC A
Congestion
Congestion
Congestion
Tape capable of sustaining 15MB/sec B
Switch A
Switch B
Congestion

19. Cisco’s Non-Blocking Architecture
No switch can completely avoid blocking
MDS 9000 has multiple features to alleviate HOL blocking
Virtual Output Queueing (VOQ) for all ports provides buffering for slow destinations
High bandwidth crossbar with no oversubscription plus line rate arbiter; precludes unnecessary queuing within the switch
255 Buffer-to-Buffer Credits (BBCs) per port on 16-port line card defers blocking on ISLs while congested VOQ drains
Fibre Channel Congestion Control (FCC) provides source quench functionality (see next slide)

20. Fibre Channel Congestion Control (FCC)
Host A is issuing write operations to 100MB/sec
Host B is issuing writes operations to 40MB/sec
The tape interface runs out of BBCs, congestion occurs in switch B
When VOQ of tape port crosses threshhold, switch B detects congestion
Switch B signals Switch A to quench the source
Switch A limits incoming traffic via BBC pacing
Disk shelf capable of sustaining 200MB/sec FC A
Limit host B to 15MB/sec
Congestion
Tape capable of sustaining 15MB/sec B
Switch A
Switch B
Congestion
Ingress port limited
VOQ

21. FCC Details Disabled by default Can work in heterogeneous SANs
Signaling frames recognized by FC-2 header values
Consists of 3 functions
Detection (VOQ monitoring)
Signaling (quench frames)
Control (BBC pacing)

22. Oversubscription Explained
OLTP Servers
Disk
When the demand for bandwidth simply exceeds the available bandwidth, you have oversubscription FC
Switch A
Switch B
Congestion
Tape FC
Legacy switches provide limited visibility and no traffic prioritization
Backup Server

23. Cisco Quality of Service (QoS)
Industry First!
Cisco Quality of Service (QoS)
OLTP Servers
Disk
Traffic Classification
Src/Dst PWWN
Src/Dst FCID
Source port FC
Absolute Priority
High Priority
Medium Priority
Low Priority
VOQs
Absolute Priority
High Priority
Medium Priority
Low Priority
VOQs
Tape FC
Queuing and Scheduling
4 VOQs per egress port
Scheduling algorthim is based on Deficit
Weighted Round Robin (DWRR)
Backup Server

24. FCC and QoS Default priority is 4 FCC is QoS aware
Has a configurable priority threshold
All frames classified with priority less than or equal to the FCC threshold can be quenched
Default priority is 4

25. Ingress Rate Limiting
Limits the rate at which a switch accepts traffic from a Server/Target
Rate is configured as a percentage of the FC port speed
Currently available on 9120, 9140, 9216i and MPS-14/2 only

26. Agenda
VSAN
Traffic Management
Diagnostics

27. RTT Measurement for FCIP
Verify connectivity via specified outbound GE port
Accurately measure the Round Trip Time (RTT) to ensure optimal TCP performance
switch# ips measure-rtt interface gigabitethernet 4/1
Round trip time is 420 micro seconds (0.42 milli seconds)

28. Fibre Channel Ping Similar to ping on IP networks
Used to verify end-to-end FC connectivity and latency
Supervisor sends a Fibre Channel frame to the destination and displays the response
switch# fcping pwwn 21:01:00:e0:8b:27:1e:a bytes from 21:01:00:e0:8b:27:01e:a9 ; time = 310 usec 28 bytes from 21:01:00:e0:8b:27:01e:a9 ; time = 343 usec 28 bytes from 21:01:00:e0:8b:27:01e:a9 ; time = 278 usec 28 bytes from 21:01:00:e0:8b:27:01e:a9 ; time = 251 usec 28 bytes from 21:01:00:e0:8b:27:01e:a9 ; time = 259 usec
5 frames sent, 5 frames received, 0 timeouts Round-trip min/avg/max = 251/288/343 usec
FC Ping is the equivalent to its IP counterpart. FC Ping allows a user to 'ping' a Fibre Channel N_Port. By specifying the FC_ID or pWWN, a user can send a series of frames to the target N_Port. Once these frames reach the N_Port, they are looped back to the source and a time-stamp is created.

29. End-to-End Connectivity Analysis
Confined per VSAN
Can verify redundant paths

30. Fibre Channel Traceroute
Verifies FC network connectivity by using Remote Domain Loopback (RDL) and Time-To-Live (TTL) to trace the FC path
Checks both outbound and return paths as they may be different
Used to verify zone permissions within a large fabric
switch# fctrace fcid 0x260000
Route present for : 0x260000
20:00:00:05:30:00:017:5e ; (0xfffcc1) Port No: 48 Latency: 9158 usec
20:00:00:05:30:00:33:de ; (0xfffc83) Port No: 104 Latency: 2153 usec
20:00:00:05:30:00:35:1e ; (0xfffc26) Port No: 107 Latency: 0 usec
20:00:00:05:30:00:35:1e ; (0xfffc26) Port No: 104 Latency: 2153 usec
20:00:00:05:30:00:33:de ; (0xfffc83) Port No: 112
Trace Route (a.k.a. Remote Loopback) sends a series of RDL frames while incrementing the TTL in each frame. When an RDL frame reaches the destination switch, the switch flips the source address with the destination address and routes the frame back to the source switch. Each switch timestamps the frame.
FC Traceroute is slightly different than its IP equivalent in that the outbound and return paths are both recorded. The result of an FC Traceroute command is two path descriptors that identify the path taken on a hop-by-hop basis including a timestamp at each hop in both directions.
RDL is a value add feature which allows a frame to be looped back at the last switch.
RDL works only with MDS9000 trunking enabled ports. RDL frames do not go across 3rd party devices.

31. Discover LUNs Verify proper LUN masking on the storage array
Verify proper array behavior
switch# discover scsi-target local os all lun
discovery started
The Discover command performs the following sequence:
plogi, prli, inquiry, report luns, test unit ready, read capacity, prlo, logo
Uses a SID of FF.FC.xx (xx is the Domain ID of the switch/vsan)

32. Verify LUNs switch# show scsi-target lun
ST318452FC from SEAGATE (Rev 0004)
FCID is 0x6400e1 in VSAN 1, PWWN is 22:00:00:04:cf:92:74:61
OS LUN Capacity Status Serial Number Device-Id
(MB)
WIN 0x Online 3EV0N78D C:1 A:0 T:3 20:00:00:04:cf:92:74:61
AIX 0x Online 3EV0N78D C:1 A:0 T:3 20:00:00:04:cf:92:74:61
SOL 0x Online 3EV0N78D C:1 A:0 T:3 20:00:00:04:cf:92:74:61
LIN 0x Online 3EV0N78D C:1 A:0 T:3 20:00:00:04:cf:92:74:61
HP 0x Online 3EV0N78D C:1 A:0 T:3 20:00:00:04:cf:92:74:61

33. Switch Configuration Analyzer
On demand switch configuration comparison
Compare to policy file or another switch
Define rules for comparison

34. Zone Merge Analyzer
Confined per VSAN
Compare any 2 switches

35. Switch Health Analyzer
On demand health check
Comprehensive checklist

36. Automatic System Health Monitor
Periodically tests
Active Supervisor in-band loopback
Standby Supervisor arbiter
Boot flash on active/standby Supervisors and switching modules
Management port 0
Ethernet Out-of-Band Channel (EOBC) on active/standby Supervisors and switching modules

37. Cisco Fabric Analyzer
Cisco Fabric Analyzer is a switch-embedded control traffic capture/decode feature
Supports FC and iSCSI
Does not decode FC-0 or FC-1
Cabling, Ordered sets
Decode locally or remotely in real time
Save captured data to file and download to host for decoding via Ethereal

38. Switch Port Analyzer (SPAN) and Remote SPAN (RSPAN)
SPAN and RSPAN provide the ability to transparently copy FC frames to SPAN Destination (SD) ports
SPAN and RSPAN work with all standard FC analyzers
Quickly diagnose protocol-level problems remotely, thereby reducing down-time
Supports FC and GE ports

39. SPAN Details
Non-intrusively monitor FC ports on the local switch using an SD port
SPAN traffic is compatible with off-the-shelf FC analyzers, the Cisco Fabric Analyzer and the Cisco Port Adapter Analyzer
You can configure up to 16 SPAN sessions per switch with multiple ingress (rx) sources
You can configure up to 3 SPAN sessions per switch with one egress (tx) port
Multiple sessions can share the same destination ports
Ignores buffer-to-buffer credits
Allows data traffic only in the egress (tx) direction
In a 32-port switching module, all four ports in a port group must belong to the same session, but it is not required to SPAN all four ports
SPAN frames are dropped if the sum of the bandwidth of the sources exceeds the speed of the destination port
Frames dropped by a source port are not spanned
Supports 1 Gbps and 2 Gbps, but must be manually set (no autonegotiation)
If the SD port is shut down, all shared sessions stop generating SPAN traffic
The port mode cannot be changed if it is being used for a SPAN session
The outgoing frames can be encapsulated in enhanced inter-switch link (EISL) format
The SD port does not have a port VSAN
SD ports cannot be configured using the Advanced Services Module (ASM)
A source can be shared by two SPAN sessions: ingress in one session and egress in the other
SPAN
Source Port SD
SPAN
Destination Port
Copy of FC frame
FC Analyzer

40. RSPAN Details
Non-intrusively monitor FC ports at a remotely located switch using a ST port
RSPAN traffic is tunneled through network using FC-tunnels
RSPAN traffic is compatible with off-the-shelf FC analyzers and MDS 9000 PAA
Destination Switch
Any FC port in a switch can be configured as a ST port
ST ports perform the RSPAN encapsulation of the FC frame
ST ports do not use BB_credits
A ST port can only be bound to one FC tunnel
ST ports cannot be used for any other purpose other than to carry RSPAN traffic
ST Ports cannot be configured using ASM modules
RSPAN tunnel Encapsulation
SPAN
Source Port ST
MDS FC Network SD
Source Switch
FC Analyzer

41. Port Analyzer Adapter v2 (PAA-2)
Cisco PAA provides the ability to capture whole or partial FC frames and send to workstation via IP or Ethernet encapsulation
Leverages SPAN and RSPAN
Eliminates needs for costly FC analyzer (does not capture FC-0 or FC-1 errors)
Works with Ethereal

42. Ethereal
1. Captured Frames
2. Detailed Decode
3. Hex Values

43. Other Troubleshooting Tools
Extensive show commands
Extensive debug commands
Syslog locally or to server
SNMP Traps and Informs
RMON Alarms
Call Home
Accounting log
Port Beacon
FM and DM fault visualization