BRK3310 Azure IaaS Design & Performance Considerations - Best Practices & Learnings from the field Daniel Neumann TSP – Azure Infrastructure at Microsoft @neumanndaniel

Objectives Selecting the best VM size Managed disks – Why you should use them? Networking – Best practices Encryption – Best practices Azure Backup – Best practices

Selecting the best VM size

Azure compute unit is your friend https://docs.microsoft.com/en-us/azure/virtual-machines/windows/acu ACU is standarized on Standard_A1 being 100 ACU represents approximately how much faster other VM sizes are

Azure compute unit is your friend SKU Family ACU/Core CPU A0 50 A1-A4 / A5-A7 / Av2 100 A8-A11 225 2.6 GHz Intel Xeon E5-2670 D / DS 160 Dv2 / DSv2 210-250* 2.4 GHz Intel Xeon E5-2673 v3 (Haswell) F / Fs G / GS 180-240* Intel Xeon processor E5 v3 family H 290-300* 3.2 GHz Intel Xeon E5-2667 v3 (Haswell) Ls ACUs marked with a * use Intel® Turbo technology to increase CPU frequency and provide a performance boost. The amount of the boost can vary based on the VM size, workload, and other workloads running on the same host.

Azure compute unit is your friend SKU Family ACU/Core CPU Ms 160-180* ** 2.5 GHz Intel Xeon E7-8890 v3 (Haswell) Dv3 / DSv3 160-190* ** 2.3 GHz Intel Xeon E5-2673 v4 (Broadwell) Ev3 ESv3 Bs Burstable 2.4 GHz Intel Xeon E5-2673 v3 (Haswell) or better ACUs marked with a * use Intel® Turbo technology to increase CPU frequency and provide a performance boost. The amount of the boost can vary based on the VM size, workload, and other workloads running on the same host. **Hyper-threaded.

Azure compute unit – example VM size vCPU Memory ACU total $ Price 09/14/17 (West Europe) Standard_A2 2 3.5 GB 200 $90.03 Standard_A2_v2 4 GB $96.72 Standard_D1_v2 1 210-250 $100.44 Standard_F1 2 GB $76.64 Standard_B2s Burstable $26.79 Standard_A4 8 14 GB 800 $482.12 Standard_A8_v2 16 GB $427.80 Standard_D3_v2 4 840-1000 $399.53 Standard_D4_v3 640-760 $315.46 Standard_F4 8 GB $305.79 Standard_B4ms $95.98

Azure VM sizes overview Type Sizes Description General purpose Bs, Dsv3, Dv3, DSv2, Dv2, DS, D, Av2, A0-7 Balanced CPU-to-memory ratio. Ideal for testing and development, small to medium databases, and low to medium traffic web servers. Compute optimized Fs, F High CPU-to-memory ratio. Good for medium traffic web servers, network appliances, batch processes, and application servers. Memory optimized Esv3, Ev3, Ms, GS, G, DSv2, DS, Dv2, D High memory-to-core ratio. Great for relational database servers, medium to large caches, and in-memory analytics. Storage optimized Ls High disk throughput and IO. Ideal for Big Data, SQL, and NoSQL databases. GPU NV, NC Specialized virtual machines targeted for heavy graphic rendering and video editing. Available with single or multiple GPUs. High performance compute H, A8-11 Our fastest and most powerful CPU virtual machines with optional high-throughput network interfaces (RDMA). https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes

Azure VM sizes network performance vCPU Memory: GiB Max NICs Expected network performance Standard_F1 1 2 750 Mbps Standard_F2 4 1,5 Gbps Standard_F4 8 3 Gbps Standard_F8 16 6 Gbps Standard_F16 32 6-12 Gbps Standard_A8_v2 2 Gbps Standard_G5 448 20 Gbps Standard_D15_v2 20 140 25 Gbps (Accelerated) Standard_M128s 128 2048 25 Gbps

Optimizing Azure VM network performance Windows: Not always enabled by default Check if Receive Side Scaling (RSS) is enabled: Get-NetAdapterRss Enable RSS: Get-NetAdapter | % {Enable-NetAdapterRss -Name $_.Name} Linux: Enabled by default https://docs.microsoft.com/en-us/azure/virtual-network/virtual-network-optimize-network-bandwidth

Azure VM sizes storage throughput vCPU Memory: GiB Max data disks Max uncached disk throughput: IOPS / MBps Standard_F1s 1 2 3,200 / 48 Standard_F2s 4 6,400 / 96 Standard_F4s 8 12,800 / 192 Standard_F8s 16 25,600 / 384 Standard_F16s 32 51,200 / 768 https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes

Demo Network bandwidth / Storage throughput

Managed disks Why you should use them?

Storage account limitations Standard storage Premium storage Max IOPS per account 20.000 (50.000)* - Max number of disks w/ max IOPS 40 (100)* Total disk capacity per account 35 TB Max number of disks w/ max capacity 35 Total snapshot capacity per account 10 TB Max bandwidth per account 50 Gbps Max number of VMs w/ max bandwidth Depends on New limits marked with a * are in place since 09/20/2017 in Azure. https://azure.microsoft.com/en-us/blog/announcing-larger-higher-scale-storage-accounts/

Storage account limitations – example G5 G5 VM size: 1 OS disk (500 IOPS) 64 data disks (500 IOPS each) Storage accounts needed: 2 storage accounts with 1st storage account: 1 OS disk and 39 data disks 2nd storage account: 25 data disks

Unmanaged vs. managed disks Unmanaged disks Managed disks RBAC Storage account level Disk level Tags Locks Replication LRS, GRS, RA-GRS LRS Encryption ADE, SSE ADE, SSE on by default Pricing Standard per used GB Premium per disk size Per disk size Storage account placement Manual selection Automatic Storage account visibility Visible Not visible Disk accessibility Storage account name and key SAS One time SAS, generated for export

Unmanaged vs. managed disks

Using Storage Spaces in Azure best practices Single virtual disk per pool with the Simple resiliency type. Create virtual disks using New-Volume or New-VirtualDisk cmdlet to create the virtual disk with the same number of columns as physical disks (in this case, .vhd files).

Using Storage Spaces in Azure best practices Use the Format-Volume cmdlet with the -AllocationUnitSize 64KB -UseLargeFRS parameters to format the volume with a 64 KB NTFS allocation unit size and enable large FRS support. Install hotfix KB3063075 for Windows Server 2012 R2 https://support.microsoft.com/en-us/help/3063075/many-id-129-events-are-logged-when-you-run-a-windows-server-2012-r2-virtual-machine-in-windows-azure

Demo Microsoft Azure storage performance health check script https://gallery.technet.microsoft.com/Azure-storage-performance-3e18fe3d

Networking Best practices

User defined route – DMZ / Firewall Name Address prefix Next hop VNET: Subnet local traffic 172.16.1.0/24 Virtual network VNET: Subnet to subnet traffic 172.16.0.0/22 172.16.0.254 (NVA) All other traffic 0.0.0.0/0

User defined route – KMS & Forced Tunneling Name Address prefix Next hop Forced tunneling 0.0.0.0/0 Virtual network gateway KMS 23.102.135.246/32 Internet

Demo Network Watcher and effective route evaluation

Encryption Best practices

Requirements Azure Disk Encryption requires that your Key Vault and VMs reside in the same Azure region and subscription. Use the latest version of Azure PowerShell to configure Azure Disk Encryption

Example setup – ADE with one AAD SP

Example setup – ADE with two AAD SP

Example setup – SSE

Test parameters Tool: PerfInsights https://www.microsoft.com/en-us/download/details.aspx?id=54915 Changed block size for throughput testing from 512KB to 2048KB https://docs.microsoft.com/en-us/azure/storage/storage-premium-storage-performance#nature-of-io-requests

Test parameters Settings: Azure VM size: Standard_D4_v2 / Standard_DS4_v2 OS: Windows Server 2016 100% write test 1GB test file 30 sec warm-up 90 sec test duration 3 runs each for IOPS and throughput testing

Test parameters OS disk Data disk File system NTFS Allocation unit size 4 KB 64 KB Caching Read / write None IOPS testing block size 8 KB Throughput (MBPS) testing block size - 2048 KB

SSE vs. ADE – Standard Managed Disks Standard_D4_v2 – SSE Standard_D4_v2 – ADE OS disk IOPS (500) 494.71 (99%) IOPS 493.81 (99%) IOPS Data disk IOPS (500) Data disk throughput (60MB) 496.04 (99%) IOPS 60.00 (100%) MB/sec 496.13 (99%) IOPS 59,34 (99%) MB/sec CPU average in % when creating 20 GB fixed VHD on data disk 0.215% 3.200 %

SSE vs. ADE – Premium Managed Disks Standard_DS4_v2 – SSE Standard_DS4_v2 – ADE P10 OS disk IOPS (500) 508.14 (102%) IOPS 508.50 (102%) IOPS P4 data disk IOPS (120) P4 data disk throughput (25MB) 121.96 (102%) IOPS 24.32 (97%) MB/sec 121.94 (102%) IOPS 14.29 (57%) MB/sec P6 data disk IOPS (240) P6data disk throughput (50MB) 243.96 (102%) IOPS 48.65 (97%) MB/sec 243.99 (102%) IOPS 27.88 (56%) MB/sec P10 data disk IOPS (500) P10 data disk throughput (100MB) 509,97 (102%) IOPS 97.26 (97%) MB/sec 509.92 (102%) IOPS 57.01 (57%) MB/sec

SSE vs. ADE – Premium Managed Disks Standard_DS4_v2 – SSE Standard_DS4_v2 – ADE P20 data disk IOPS (2300) P20 data disk throughput (150MB) 2345.82 (102%) IOPS 145.91 (97%) MB/sec 2345.70 (102%) IOPS 145.89 (97%) MB/sec P30 data disk IOPS (5000) P30 data disk throughput (200MB) 5099.61 (102%) IOPS 194.56 (97%) MB/sec 5100.61 (102%) IOPS 194.03 (97%) MB/sec P40 data disk IOPS (7500) P40 data disk throughput (250MB) 7650.23 (102%) IOPS 241.02 (96%) MB/sec 7649.42 (102%) IOPS 242.26 (97%) MB/sec P50 data disk IOPS (7500) P50 data disk throughput (250MB) 7650.05 (102%) IOPS 239.20 (96%) MB/sec 7649.868 (102%) IOPS 241.99 (97%) MB/sec CPU average in % when creating 20 GB fixed VHD on data disk 0.221% 2.934 %

ADE – Premium Managed Disks Changing block size for throughput Block size 2048 KB Block size 1024 KB Block size 512 KB (default) P4 data disk throughput (25MB) 14.29 (57%) MB/sec 17,31 (69%) MB/sec 22,17 (89%) MB/sec P6 data disk throughput (50MB) 27.88 (56%) MB/sec 33,93 (68%) MB/sec 42,20 (84%) MB/sec P10 data disk throughput (100MB) 57.01 (57%) MB/sec 71,54 (72%) MB/sec 90,42 (90%) MB/sec P20 data disk throughput (150MB) 145.89 (97%) MB/sec 145,82 (97%) MB/sec 99,32 (66%) MB/sec P30 data disk throughput (200MB) 194.03 (97%) MB/sec 181,19 (91%) MB/sec 98,59 (49%) MB/sec P40 data disk throughput (250MB) 242.26 (97%) MB/sec 179,13 (72%) MB/sec 103,20 (41%) MB/sec P50 data disk throughput (250MB) 241.99 (97%) MB/sec 180,76 (72%) MB/sec 96,56 (39%) MB/sec

Demo ADE performance test setup script https://github.com/neumanndaniel/iaasperftests

Azure Backup Best practices

Requirements Azure Backup requires that your Recovery Services vault and VMs reside in the same Azure region and subscription, when you want to use Azure VM backup. Backup and restore of encrypted VMs is supported only for VMs that are using ADE with a Key Encryption Key (KEK).

Sizing / Performance Use more than one Recovery Services vault for backup Max. 200 Azure IaaS VMs can be registered per vault Max. 25 Recovery Services vaults, per supported region of Azure Backup, per subscription Backup performance / time also depends on VM size Backup operation Best-case throughput Initial backup 160 Mbps (20 MBps) Incremental backup (DR) 640 Mbps (80 MBps)

Best practices Do not schedule more than 40 VMs to back up at the same time. Schedule VM backups during non-peak hours. Make sure that python version on Linux VMs enabled for backup is 2.7 https://docs.microsoft.com/en-us/azure/backup/backup-azure-vms-introduction#best-practices

Example setup

Demo Microsoft Azure Recovery Services vault performance health check script https://gallery.technet.microsoft.com/Azure-Recovery-vault-2df0ee6e

Take aways

Take aways Selecting the best VM size: Managed disks: Networking: Get as much details as you can for the workload that will run in the Azure VM Do not look only at CPU and memory, cover also network and storage Size the VM appropriate to the workload requirements Managed disks: Always use managed disk (or whenever it is possible) Networking: Make sure RSS is enabled on Windows Server VMs Use NSGs and UDRs whenever it is possible to isolate or optimize network traffic

Take aways Encryption: Backup: SSE has no performance impact ADE has some performance impact CPU: Up to 3% more CPU usage IOPS are not impacted Throughput impact depends on the block size the application / workload uses https://docs.microsoft.com/en-us/azure/storage/common/storage-premium-storage-performance#nature-of-io-requests Backup: Keep the requirements and best practices in mind, when dealing with Azure Backup for Azure VMs

Please evaluate this session Tech Ready 15 9/16/2018 Please evaluate this session From your Please expand notes window at bottom of slide and read. Then Delete this text box. PC or tablet: visit MyIgnite https://myignite.microsoft.com/evaluations Phone: download and use the Microsoft Ignite mobile app https://aka.ms/ignite.mobileapp Your input is important! © 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.