VIO Server architecture design and build Alain Lentini - CoC Unix Lge July 29th 2008 IBM confidential

Some explainations of the presentation VIOs architecture design : Standard LVM mirroring for VIO rootvg (separate SCSI adapters in each VIO) Both VIOs active for virtual SCSI (used for AIX partitions rootvg mirroring) Active/passive model for Ethernet access Mutual takeover (heartbeat) Failure of one VIO transparent to partitions (a few IP packets lost) Allows hot maintenance of VIOs Each VIO on a different LAN switch Multipathing for SAN access inside each VIO (sddpcm) + pathing failover in the AIX partition between the 2 VIOs (Mpio) Various SAN ACCESS ( SDDPCM for Ess DS8000 and SVC – RDAC ) Alain Lentini / CoC Unix Lge

Virtualization of Rootvg Disk access Alain Lentini / CoC Unix Lge

Rootvg disks access configuration VIOS 1 AIX client partition VIOS 2 vtserv1hd0 rootvg vtserv1hd3 LV xxx PgSp LVM partitionning LVM partitionning LVM partitionning + mirroring vtserv1hd2 vtserv1hd1 hdisk8 hdisk9 hdisk8 hdisk9 hdisk0 hdisk2 hdisk1 hdisk3 Vhost7 Slot 34 Vhost6 Slot 32 Vscsi0 Slot 32 Vscsi2 Slot 34 Vscsi1 Slot 33 Vscsi3 Slot 35 Vhost7 Slot 35 Vhost6 Slot 33 SCSI adapter SCSI adapter POWER hypervisor hdisk9 hdisk8 hdisk9 serv1hd0lv serv1hd2lv naming convention - example of commands on server VIO1 Even slots on VIOs 1 and odd slots on VIOs 2 serv1hd1lv serv1hd3lv hdisk8 mklv -lv serv1hd0lv  vioserver1VG 30G hdisk8 mklv -lv serv1hd2lv  vioserver1VG 20G hdisk9 mkvdev -vdev serv1hd0lv -vadapter vhost6 -dev vtserv1hd0 mkvdev -vdev serv1hd2lv -vadapter vhost7 -dev vtserv1hd2 Alain Lentini / CoC Unix Lge

Rootvg disks configuration on vioservers Create the virtual server device in WebSm mkvg -f -vg vioserver1VG hdisk1 hdisk5 extendvg vioserver1VG hdisk2 hdisk6 Example for partition mopesaxsap15p mklv -lv saxsap15phd0LV  vioserver1VG 30G hdisk1 mklv -lv saxsap15phd2LV  vioserver1VG 20G hdisk5 mkvdev -vdev saxsap15phd0LV -vadapter vhost0 -dev vtsaxsap15phd0 mkvdev -vdev saxsap15phd2LV -vadapter vhost1 -dev vtsaxsap15phd2 on vioserver2 mkvg -f -vg vioserver2VG hdisk1 hdisk5 extendvg vioserver2VG hdisk2 hdisk6 mklv -lv saxsap15phd1LV  vioserver2VG 30G hdisk2 mklv -lv saxsap15phd3LV  vioserver2VG 20G hdisk6 mkvdev -vdev saxsap15phd1LV -vadapter vhost0 -dev vtsaxsap15phd1 mkvdev -vdev saxsap15phd3LV -vadapter vhost1 -dev vtsaxsap15phd3 Alain Lentini / CoC Unix Lge

Rootvg disks configuration on client partition Alain Lentini / CoC Unix Lge

result after the creation of 4 virtual scsi client adapter On client partition Lspath Enabled hdisk0 vscsi0 Enabled hdisk1 vscsi1 Enabled hdisk2 vscsi2 Enabled hdisk3 vscsi3 hdisk0 U9117.570.65A1DA0-V5-C36-T1-L810000000000 Virtual SCSI Disk Drive hdisk1 U9117.570.65A1DA0-V5-C37-T1-L810000000000 Virtual SCSI Disk Drive hdisk2 U9117.570.65A1DA0-V5-C38-T1-L810000000000 Virtual SCSI Disk Drive hdisk3 U9117.570.65A1DA0-V5-C39-T1-L810000000000 Virtual SCSI Disk Drive Alain Lentini / CoC Unix Lge

Shared Ethernet Adapter configuration Alain Lentini / CoC Unix Lge

Shared Ethernet Adapter principle 1/2 Note: Note that a numerically lower priority receives a higher overall priority. Alain Lentini / CoC Unix Lge

Create the Virtual LAN adapter on the Hmc In our example, the VIO is secondary server for the vlan 99 : trunk priority = 2 sur vio1 We share the Network workload between the 2 vios : 1 vlan on each VIO managed with the priority. Alain Lentini / CoC Unix Lge

create the virtual adapter for control channel This will generate a device on the VIO server ( cfgdev or cfgmgr ) To find the device, use the slot number in the command lsdev –slots : U9117.570.65A1DA0-V2-C5 Virtual I/O Slot ent8 U9117.570.65A1DA0-V2-C3 Virtual I/O Slot ent9 Alain Lentini / CoC Unix Lge

SEA command creation mkvdev -sea ent0 -vadapter ent10 -default ent10 -defaultid 101 –attr ha_mode=auto ctl_chan=ent13 ( vlan 9 for exemple ) sea = ent14 Check the priority in VIOs errpt errpt : E136EAFA   0519102508 I H ent12          BECOME PRIMARY ent0 = network physical interface ( vlan 101 ) ent10 = virtual adapter generated by hmc menu config ent13 : control, channel lsdev –slots U9117.570.65A1DA0-V2-C2 Virtual I/O Slot ent10 U9117.570.65A1DA0-V2-C5 Virtual I/O Slot ent13 Alain Lentini / CoC Unix Lge

Shared Ethernet Adapter principle 2/2 This diagram is simplified to show only the logical implementation of the solution: 2 client partitions are represented only 1 customer VLAN and the Heartbeat VLAN are represented when the real system contains 4 VLANs. The real config would contain 3 physical port per VIOs: one for each VLAN. The VLAN high-availability is transparent to the client partition. It's provided by the SEA failover feature of the VIOs. AIX Client LPAR 1 AIX Client LPAR 2 en0 (if) en0 (if) ent0 (virt) ent0 (virt) Hypervisor PVID=100 PVID=100 VLAN 1 PVID=100 PVID=10 Control Channel – VLAN 10 PVID=10 PVID=100 VIOS 1 ent1 (virt) ent3 (virt) VIOS 2 ent3 (virt) ent1 (virt) ent2 (SEA) en2 (if) en2 (if) ent2 (SEA) ent0 (phy) Primary Secondary ent0 (phy) Untagged Untagged Ethernet Switch Ethernet Switch Untagged VLAN Alain Lentini / CoC Unix Lge

SEA detailed configuration example Alain Lentini / CoC Unix Lge

San access configuration Alain Lentini / CoC Unix Lge

Data access on SAN through sddpcm on VIOs principles 1/2 Attention: When using multiple Virtual I/O Servers, and exporting the same LUN to the client partitions, only mapping of hdisks is supported (logical volumes are not supported). Attaching IBM TotalStorage Enterprise Storage Servers to the Virtual I/O Servers, only MPIO using the SDDPCM is supported. Alain Lentini / CoC Unix Lge

Data access on SAN through RDAC on VIOs principles 2/2 On both Virtual I/O Servers, you have to set the hdisk attribute reserve_policy to no. This attribute prevents the Virtual I/O Server setting a reservation flag on the disk at the time of mapping. The MPIO component on the client partition will take the responsibility of managing the disk. Alain Lentini / CoC Unix Lge

Data access on SAN through sddpcm (no mirroring) VIOS a AIX client partition VIOS b vtserv1hd9 vtserv1hd9 datavg LV xxx vtserv1hd8 LVM partitionning vtserv1hd8 SDDPCM SDDPCM hdisk8 hdisk9 Default PCM + MPIO Hdisk70 hdisk71 hdisk70 hdisk71 hdisk70 hdisk71 hdisk70 hdisk71 hdisk8 hdisk9 hdisk8 hdisk9 FC adapter port1 port2 FC adapter port2 port1 Vhost24 Slot 44 Vscsi4 Slot 44 Vscsi5 Slot 45 Vhost24 Slot 45 POWER hypervisor Fabric 1 Fabric 2 Bay 1 hdisk70 hdisk71 naming convention - example of commands on server VIO1 mkvdev -vdev hdisk70 -vadapter vhost24 -dev vtserv1hd8 mkvdev -vdev hdisk71 -vadapter vhost24 -dev vtserv1hd9 Alain Lentini / CoC Unix Lge

Data access on SAN through sddpcm (mirroring) VIOS 1 AIX client partition VIOS 2 vtserv1hd9 vtserv1hd9 datatvg LV xxx vtserv1hd8 vtserv1hd8 SDDPCM SDDPCM LVM partitionning + mirroring hdisk70 hdisk71 hdisk70 hdisk71 hdisk8 hdisk9 hdisk8 hdisk9 hdisk70 hdisk71 hdisk70 hdisk71 FC adapter port1 port2 FC adapter port2 port1 Vhost24 Slot 44 Vscsi4 Slot 44 Vscsi5 Slot 45 Vhost24 Slot 45 POWER hypervisor Basic MPIO command example Lspath|grep hdisk8 Enabled hdisk8 vscsi4 Enabled hdisk8 vscsi5 Fabric 1 Fabric 2 Bay 1 hdisk70 Bay 2 hdisk71 Alain Lentini / CoC Unix Lge

Data access on SAN through RDAC (no mirroring) VIOS a AIX client partition VIOS b vtserv1hd9 vtserv1hd9 datavg LV xxx vtserv1hd8 LVM partitionning vtserv1hd8 RDAC RDAC hdisk8 hdisk9 Default PCM + MPIO hdisk70 hdisk71 hdisk70 hdisk71 hdisk8 hdisk9 hdisk8 hdisk9 FC adapter port1 port2 FC adapter port2 port1 Vhost24 Slot 44 Vscsi4 Slot 44 Vscsi5 Slot 45 Vhost24 Slot 45 POWER hypervisor Fabric 1 Fabric 2 FASTT hdisk70 hdisk71 naming convention - example of commands on server VIO1 mkvdev -vdev hdisk70 -vadapter vhost24 -dev vtserv1hd8 mkvdev -vdev hdisk71 -vadapter vhost24 -dev vtserv1hd9 Alain Lentini / CoC Unix Lge

Data access on SAN through RDAC (mirroring) VIOS 1 AIX client partition VIOS 2 vtserv1hd9 vtserv1hd9 datatvg LV xxx vtserv1hd8 vtserv1hd8 RDAC RDAC LVM partitionning + mirroring hdisk70 hdisk71 hdisk8 hdisk9 hdisk8 hdisk9 hdisk70 hdisk71 FC adapter port1 port2 FC adapter port2 port1 Vhost24 Slot 44 Vscsi4 Slot 44 Vscsi5 Slot 45 Vhost24 Slot 45 POWER hypervisor Fabric 2 Fabric 1 FASTT B2 hdisk70 FASTT B4 hdisk71 Basic MPIO command example Lspath|grep hdisk8 Enabled hdisk8 vscsi4 Enabled hdisk8 vscsi5 Alain Lentini / CoC Unix Lge

VIO SAN Connexions example fcs4 fcs3 fcs2 fcs1 fcs0 fcs1 fcs2 fcs0 fcs3 fcs4 3 3 swds4300b2_01 fabric1 swds4300b2_02 fabric2 5 swb2TMMFC-01 fabric1 3 5 swb2TMMFC-02 fabric2 6 SOD fabric1 SOD fabric2 Alain Lentini / CoC Unix Lge

San disks devices creation On vioservers create the vhost servers adapter on the server and on the client partition mkvdev -vdev hdisk70 -vadapter vhost24 -dev vtserv1hd8 mkvdev -vdev hdisk71 -vadapter vhost24 -dev vtserv1hd9 On the client partition Change priority chpath -l hdisk7 -p vscsi4 -a priority=1 chpath -l hdisk7 -p vscsi5 -a priority=2 Priority Check lspath -AHE -l hdisk7 -p vscsi4 priority 1 Priority True lspath -AHE -l hdisk7 -p vscsi5 priority 2 Priority True Devices check lscfg hdisk40 U9117.570.65A1DA0-V5-C40-T1-La50000000000 Virtual SCSI Disk Drive hdisk41 U9117.570.65A1DA0-V5-C40-T1-La60000000000 Virtual SCSI Disk Drive hdisk42 U9117.570.65A1DA0-V5-C40-T1-La70000000000 Virtual SCSI Disk Drive Change the hcheck_interval parameter on each disk device chdev -l hdisk7 -a hcheck_interval=20 –P hcheck_interval CHECK lsattr -El hdisk7 | grep hcheck_interval hcheck_interval 20 Alain Lentini / CoC Unix Lge

Vio configuration check with padmin user Alain Lentini / CoC Unix Lge

Vio configuration check On vio server : lsdev –slots U9117.570.65A1DA0-V2-C24 Virtual I/O Slot vhost2 U9117.570.65A1DA0-V2-C26 Virtual I/O Slot vhost3 U9117.570.65A1DA0-V2-C28 Virtual I/O Slot vhost4 U9117.570.65A1DA0-V2-C30 Virtual I/O Slot vhost5 U9117.570.65A1DA0-V2-C32 Virtual I/O Slot vhost6 U9117.570.65A1DA0-V2-C34 Virtual I/O Slot vhost7 On vio client partition lspath|grep hdisk28 Enabled hdisk28 vscsi4 Enabled hdisk28 vscsi5 lscfg hdisk40 U9117.570.65A1DA0-V5-C40-T1-La50000000000 Virtual SCSI Disk Drive hdisk41 U9117.570.65A1DA0-V5-C40-T1-La60000000000 Virtual SCSI Disk Drive hdisk42 U9117.570.65A1DA0-V5-C40-T1-La70000000000 Virtual SCSI Disk Drive hdisk43 U9117.570.65A1DA0-V5-C40-T1-La80000000000 Virtual SCSI Disk Drive hdisk44 U9117.570.65A1DA0-V5-C40-T1-La90000000000 Virtual SCSI Disk Drive hdisk45 U9117.570.65A1DA0-V5-C40-T1-Laa0000000000 Virtual SCSI Disk Drive Alain Lentini / CoC Unix Lge

Vio configuration check with lsmap Lsmap –vadapter vhost10 | more SVSA Physloc Client Partition ID --------------- -------------------------------------------- ------------------ vhost10 U9117.570.65A1DA0-V2-C40 0x00000005 VTD vtHA11thd32 Status Available LUN 0x9d00000000000000 Backing device hdisk116 Physloc U7879.001.DQDZLWK-P1-C6-T2-W5005076300C4A620-L511C000000000000 VTD vtHA11thd33 LUN 0x9e00000000000000 Backing device hdisk101 Physloc U7879.001.DQDZLWK-P1-C6-T2-W5005076300C4A620-L5043000000000000 VTD vtHA11thd34 LUN 0x9f00000000000000 Backing device hdisk102 Physloc U7879.001.DQDZLWK-P1-C6-T2-W5005076300C4A620-L5044000000000000 Alain Lentini / CoC Unix Lge

LAN FREE SERVER Alain Lentini / CoC Unix Lge

LAN FREE SERVER ARCHITECTURE design Alain Lentini / CoC Unix Lge