1. VIO Server architecture design and build
Alain Lentini - CoC Unix Lge
July 29th 2008
IBM confidential

2. Some explainations of the presentation
VIOs architecture design :
Standard LVM mirroring for VIO rootvg (separate SCSI adapters in each VIO)
Both VIOs active for virtual SCSI (used for AIX partitions rootvg mirroring)
Active/passive model for Ethernet access
Mutual takeover (heartbeat)
Failure of one VIO transparent to partitions (a few IP packets lost)
Allows hot maintenance of VIOs
Each VIO on a different LAN switch
Multipathing for SAN access inside each VIO (sddpcm) + pathing failover in the AIX partition between the 2 VIOs (Mpio)
Various SAN ACCESS ( SDDPCM for Ess DS8000 and SVC – RDAC )
Alain Lentini / CoC Unix Lge

3. Virtualization of Rootvg Disk access
Alain Lentini / CoC Unix Lge

4. Rootvg disks access configuration
VIOS 1
AIX client partition
VIOS 2
vtserv1hd0
rootvg
vtserv1hd3
LV xxx
PgSp
LVM partitionning
LVM partitionning
LVM partitionning + mirroring
vtserv1hd2
vtserv1hd1
hdisk8
hdisk9
hdisk8
hdisk9
hdisk0
hdisk2
hdisk1
hdisk3
Vhost7
Slot 34
Vhost6
Slot 32
Vscsi0
Slot 32
Vscsi2
Slot 34
Vscsi1
Slot 33
Vscsi3
Slot 35
Vhost7
Slot 35
Vhost6
Slot 33
SCSI adapter
SCSI adapter
POWER hypervisor
hdisk9
hdisk8
hdisk9
serv1hd0lv
serv1hd2lv
naming convention - example of commands on server VIO1
Even slots on VIOs 1 and odd slots on VIOs 2
serv1hd1lv
serv1hd3lv
hdisk8
mklv -lv serv1hd0lv  vioserver1VG 30G hdisk8 mklv -lv serv1hd2lv  vioserver1VG 20G hdisk9 mkvdev -vdev serv1hd0lv -vadapter vhost6 -dev vtserv1hd0 mkvdev -vdev serv1hd2lv -vadapter vhost7 -dev vtserv1hd2
Alain Lentini / CoC Unix Lge

5. Rootvg disks configuration on vioservers
Create the virtual server device in WebSm
mkvg -f -vg vioserver1VG hdisk1 hdisk5
extendvg vioserver1VG hdisk2 hdisk6
Example for partition mopesaxsap15p
mklv -lv saxsap15phd0LV  vioserver1VG 30G hdisk1
mklv -lv saxsap15phd2LV  vioserver1VG 20G hdisk5
mkvdev -vdev saxsap15phd0LV -vadapter vhost0 -dev vtsaxsap15phd0
mkvdev -vdev saxsap15phd2LV -vadapter vhost1 -dev vtsaxsap15phd2
on vioserver2
mkvg -f -vg vioserver2VG hdisk1 hdisk5
extendvg vioserver2VG hdisk2 hdisk6
mklv -lv saxsap15phd1LV  vioserver2VG 30G hdisk2
mklv -lv saxsap15phd3LV  vioserver2VG 20G hdisk6
mkvdev -vdev saxsap15phd1LV -vadapter vhost0 -dev vtsaxsap15phd1
mkvdev -vdev saxsap15phd3LV -vadapter vhost1 -dev vtsaxsap15phd3
Alain Lentini / CoC Unix Lge

6. Rootvg disks configuration on client partition
Alain Lentini / CoC Unix Lge

7. result after the creation of 4 virtual scsi client adapter
On client partition
Lspath
Enabled hdisk0 vscsi0
Enabled hdisk1 vscsi1
Enabled hdisk2 vscsi2
Enabled hdisk3 vscsi3
hdisk U A1DA0-V5-C36-T1-L Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C37-T1-L Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C38-T1-L Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C39-T1-L Virtual SCSI Disk Drive
Alain Lentini / CoC Unix Lge

8. Shared Ethernet Adapter configuration
Alain Lentini / CoC Unix Lge

9. Shared Ethernet Adapter principle 1/2
Note: Note that a numerically lower priority receives a higher overall priority.
Alain Lentini / CoC Unix Lge

10. Create the Virtual LAN adapter on the Hmc
In our example, the VIO is secondary server for the vlan 99 : trunk priority = 2 sur vio1
We share the Network workload between the 2 vios : 1 vlan on each VIO managed with the priority.
Alain Lentini / CoC Unix Lge

11. create the virtual adapter for control channel
This will generate a device on the VIO server ( cfgdev or cfgmgr )
To find the device, use the slot number in the command lsdev –slots :
U A1DA0-V2-C5 Virtual I/O Slot ent8
U A1DA0-V2-C3 Virtual I/O Slot ent9
Alain Lentini / CoC Unix Lge

12. SEA command creation
mkvdev -sea ent0 -vadapter ent10 -default ent10 -defaultid 101 –attr ha_mode=auto ctl_chan=ent13 ( vlan 9 for exemple ) sea = ent14
Check the priority in VIOs errpt
errpt : E136EAFA   I H ent12          BECOME PRIMARY
ent0 = network physical interface ( vlan 101 )
ent10 = virtual adapter generated by hmc menu config
ent13 : control, channel
lsdev –slots
U A1DA0-V2-C2 Virtual I/O Slot ent10
U A1DA0-V2-C5 Virtual I/O Slot ent13
Alain Lentini / CoC Unix Lge

13. Shared Ethernet Adapter principle 2/2
This diagram is simplified to show only the logical implementation of the solution:
2 client partitions are represented only
1 customer VLAN and the Heartbeat VLAN are represented when the real system contains 4 VLANs.
The real config would contain 3 physical port per VIOs: one for each VLAN.
The VLAN high-availability is transparent to the client partition. It's provided by the SEA failover feature of the VIOs.
AIX Client LPAR 1
AIX Client LPAR 2
en0
(if)
en0
(if)
ent0
(virt)
ent0
(virt)
Hypervisor
PVID=100
PVID=100
VLAN 1
PVID=100
PVID=10
Control Channel – VLAN 10
PVID=10
PVID=100
VIOS 1
ent1
(virt)
ent3
(virt)
VIOS 2
ent3
(virt)
ent1
(virt)
ent2
(SEA)
en2
(if)
en2
(if)
ent2
(SEA)
ent0
(phy)
Primary
Secondary
ent0
(phy)
Untagged
Untagged
Ethernet Switch
Ethernet Switch
Untagged VLAN
Alain Lentini / CoC Unix Lge

14. SEA detailed configuration example
Alain Lentini / CoC Unix Lge

15. San access configuration
Alain Lentini / CoC Unix Lge

16. Data access on SAN through sddpcm on VIOs principles 1/2
Attention: When using multiple Virtual I/O Servers, and exporting the same LUN to the client partitions, only mapping of hdisks is supported (logical volumes are not supported).
Attaching IBM TotalStorage Enterprise Storage Servers to the Virtual I/O Servers, only MPIO using the SDDPCM is supported.
Alain Lentini / CoC Unix Lge

17. Data access on SAN through RDAC on VIOs principles 2/2
On both Virtual I/O Servers, you have to set the hdisk attribute reserve_policy to no. This attribute prevents the Virtual I/O Server setting a reservation flag on the disk at the time of mapping. The MPIO component on the client partition will take the responsibility of managing the disk.
Alain Lentini / CoC Unix Lge

18. Data access on SAN through sddpcm (no mirroring)
VIOS a
AIX client partition
VIOS b
vtserv1hd9
vtserv1hd9
datavg
LV xxx
vtserv1hd8
LVM partitionning
vtserv1hd8
SDDPCM
SDDPCM
hdisk8
hdisk9
Default PCM + MPIO
Hdisk70
hdisk71
hdisk70
hdisk71
hdisk70
hdisk71
hdisk70
hdisk71
hdisk8
hdisk9
hdisk8
hdisk9
FC adapter
port1
port2
FC adapter
port2
port1
Vhost24
Slot 44
Vscsi4
Slot 44
Vscsi5
Slot 45
Vhost24
Slot 45
POWER hypervisor
Fabric 1
Fabric 2
Bay 1
hdisk70
hdisk71
naming convention - example of commands on server VIO1
mkvdev -vdev hdisk70 -vadapter vhost24 -dev vtserv1hd8
mkvdev -vdev hdisk71 -vadapter vhost24 -dev vtserv1hd9
Alain Lentini / CoC Unix Lge

19. Data access on SAN through sddpcm (mirroring)
VIOS 1
AIX client partition
VIOS 2
vtserv1hd9
vtserv1hd9
datatvg
LV xxx
vtserv1hd8
vtserv1hd8
SDDPCM
SDDPCM
LVM partitionning + mirroring
hdisk70
hdisk71
hdisk70
hdisk71
hdisk8
hdisk9
hdisk8
hdisk9
hdisk70
hdisk71
hdisk70
hdisk71
FC adapter
port1
port2
FC adapter
port2
port1
Vhost24
Slot 44
Vscsi4
Slot 44
Vscsi5
Slot 45
Vhost24
Slot 45
POWER hypervisor
Basic MPIO command example
Lspath|grep hdisk8
Enabled hdisk8 vscsi4
Enabled hdisk8 vscsi5
Fabric 1
Fabric 2
Bay 1
hdisk70
Bay 2
hdisk71
Alain Lentini / CoC Unix Lge

20. Data access on SAN through RDAC (no mirroring)
VIOS a
AIX client partition
VIOS b
vtserv1hd9
vtserv1hd9
datavg
LV xxx
vtserv1hd8
LVM partitionning
vtserv1hd8
RDAC
RDAC
hdisk8
hdisk9
Default PCM + MPIO
hdisk70
hdisk71
hdisk70
hdisk71
hdisk8
hdisk9
hdisk8
hdisk9
FC adapter
port1
port2
FC adapter
port2
port1
Vhost24
Slot 44
Vscsi4
Slot 44
Vscsi5
Slot 45
Vhost24
Slot 45
POWER hypervisor
Fabric 1
Fabric 2
FASTT
hdisk70
hdisk71
naming convention - example of commands on server VIO1
mkvdev -vdev hdisk70 -vadapter vhost24 -dev vtserv1hd8
mkvdev -vdev hdisk71 -vadapter vhost24 -dev vtserv1hd9
Alain Lentini / CoC Unix Lge

21. Data access on SAN through RDAC (mirroring)
VIOS 1
AIX client partition
VIOS 2
vtserv1hd9
vtserv1hd9
datatvg
LV xxx
vtserv1hd8
vtserv1hd8
RDAC
RDAC
LVM partitionning + mirroring
hdisk70
hdisk71
hdisk8
hdisk9
hdisk8
hdisk9
hdisk70
hdisk71
FC adapter
port1
port2
FC adapter
port2
port1
Vhost24
Slot 44
Vscsi4
Slot 44
Vscsi5
Slot 45
Vhost24
Slot 45
POWER hypervisor
Fabric 2
Fabric 1
FASTT B2
hdisk70
FASTT B4
hdisk71
Basic MPIO command example
Lspath|grep hdisk8
Enabled hdisk8 vscsi4
Enabled hdisk8 vscsi5
Alain Lentini / CoC Unix Lge

22. VIO SAN Connexions example
fcs4
fcs3
fcs2
fcs1
fcs0
fcs1
fcs2
fcs0
fcs3
fcs4 3 3
swds4300b2_01 fabric1
swds4300b2_02 fabric2 5
swb2TMMFC-01 fabric1 3 5
swb2TMMFC-02 fabric2 6
SOD fabric1
SOD fabric2
Alain Lentini / CoC Unix Lge

23. San disks devices creation
On vioservers
create the vhost servers adapter on the server and on the client partition
mkvdev -vdev hdisk70 -vadapter vhost24 -dev vtserv1hd8
mkvdev -vdev hdisk71 -vadapter vhost24 -dev vtserv1hd9
On the client partition
Change priority
chpath -l hdisk7 -p vscsi4 -a priority=1
chpath -l hdisk7 -p vscsi5 -a priority=2
Priority Check
lspath -AHE -l hdisk7 -p vscsi4
priority Priority True lspath -AHE -l hdisk7 -p vscsi5
priority Priority True
Devices check
lscfg
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
Change the hcheck_interval parameter on each disk device
chdev -l hdisk7 -a hcheck_interval=20 –P
hcheck_interval CHECK
lsattr -El hdisk7 | grep hcheck_interval
hcheck_interval 20
Alain Lentini / CoC Unix Lge

24. Vio configuration check with padmin user
Alain Lentini / CoC Unix Lge

25. Vio configuration check
On vio server : lsdev –slots
U A1DA0-V2-C24 Virtual I/O Slot vhost2
U A1DA0-V2-C26 Virtual I/O Slot vhost3
U A1DA0-V2-C28 Virtual I/O Slot vhost4
U A1DA0-V2-C30 Virtual I/O Slot vhost5
U A1DA0-V2-C32 Virtual I/O Slot vhost6
U A1DA0-V2-C34 Virtual I/O Slot vhost7
On vio client partition
lspath|grep hdisk28
Enabled hdisk28 vscsi4
Enabled hdisk28 vscsi5
lscfg
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-La Virtual SCSI Disk Drive
hdisk U A1DA0-V5-C40-T1-Laa Virtual SCSI Disk Drive
Alain Lentini / CoC Unix Lge

26. Vio configuration check with lsmap
Lsmap –vadapter vhost10 | more
SVSA Physloc Client Partition ID
vhost U A1DA0-V2-C x
VTD vtHA11thd32
Status Available
LUN x9d
Backing device hdisk116
Physloc U DQDZLWK-P1-C6-T2-W C4A620-L511C
VTD vtHA11thd33
LUN x9e
Backing device hdisk101
Physloc U DQDZLWK-P1-C6-T2-W C4A620-L
VTD vtHA11thd34
LUN x9f
Backing device hdisk102
Physloc U DQDZLWK-P1-C6-T2-W C4A620-L
Alain Lentini / CoC Unix Lge

27. LAN FREE SERVER
Alain Lentini / CoC Unix Lge

28. LAN FREE SERVER ARCHITECTURE design
Alain Lentini / CoC Unix Lge