1. PVS 7.7 UEFI Boot Implementation and Troubleshooting
May 23rd 2016
PVS 7.7 UEFI Boot Implementation and Troubleshooting
Peter Svoboda
Lead PVS Escalation Engineer
Good morning and welcome to Synergy 2016
My name is Peter Svoboda and I am a Lead Escalation Engineering with the Provisioning Services Escalation Team from our offices in Ft. Lauderdale Florida.
The purpose of this presentation is provide you with information about implementing and troubleshooting UEFI boot with Citrix Provisioning Services.

2. Tweet about this session with hashtags #TE601 and #citrixsynergy
You can tweet about this session using the hashtag #TE601 or #citrixsynergy

3. About the Speaker Joined Citrix in July 2004
Joined as frontline engineer
Joined the PVS Escalation team in February 2011

4. Agenda UEFI Basics Configuring UEFI for PVS Hypervisor Configuration
Troubleshooting UEFI
References
Questions
Agenda
In today’s session will discuss what UEFI is and what it offers. Next we will discuss how to configure a PVS environment to use UEFI. Then we will cover the hypervisor specifics when using UEFI. Then we will cover how to troubleshoot UEFI environments. I will provide you with a list of resources that I used in creating this presentation in the event that you want to do further research. And lastly I will open the floor to any questions you may have.

5. UEFI Basics
Lets go ahead and cover some of the basics about UEFI.

6. What is UEFI
UEFI (Unified Extensible Firmware Interface) is a standard firmware interface for PCs, designed to replace BIOS (basic input/output system)
Benefits:
Support for modern, 64-bit firmware device drivers that the system can use to address more than billion gigabytes (GB) of memory during startup.
Faster startup times and resuming from hibernation.
Support for drives larger than 2.2 terabytes (TB).
Better security by helping to protect the pre-startup—
or pre-boot—process against bootkit attacks.
Capability to use BIOS with UEFI hardware.
UEFI is a firmware interface for PCs. It is designed to replace the existing BIOS firmware interface which has been around since 1975.
UEFI was developed in mid 1990s to address the some BIOS limitations such a 16 bit processor mode and the 1 MB addressable memory space.
Some of the benefits of using UEFI are:
Support for modern, 64-bit firmware device drivers that the system can use to address more than 17.2 billion gigabytes (GB) of memory during startup.
Faster startup times and resuming from hibernation.
Support for drives larger than 2.2 terabytes (TB).
Better security by helping to protect the pre-startup—
or pre-boot—process against bootkit attacks.
Capability to use BIOS with UEFI hardware.

7. Using UEFI
Systems cannot switch from BIOS to UEFI without some additional steps
Either the image has to be rebuilt using UEFI or it has to be converted
UEFI uses the GPT partition type
Conversion can be done by the use of 3rd party programs such as:
GPTGEN
uefi.aspx
Citrix recommends building a new image using UEFI
Here are some things to consider when implementing UEFI in your environment.
Generally speaking system cannot switch the firmware interface from one to the other without additional step
Usually the image has to recreated from scratch or it has to be converted to UEFI
One of the reasons is the partition type, GPT (GUID Partition Table), as when Windows is installed there is a system partition that is created that corresponds to the underlying firmware.
To change from BIOS to UEFI you need to convert the disk partition type from MBR to GPT
While it is possible to convert an image the process is not foolproof and issue may arise
As a best practice Citrix recommends building a new image using UEFI

8. UEFI and PVS
So lets go over how using UEFI as a firmware interface applies to PVS

9. UEFI and PVS Boot Process
BIOS PVS boot process
IP Acquisition
Bootstrap Download
Login
Single Read Mode
MIO Read Mode
Using UEFI with PVS allows for a faster boot
UEFI PVS Boot Process
IP Acquisition
Bootstrap Download
UEFI Login
MIO Read Mode
The primary way that UEFI affects PVS is during the boot process
As a quick refresher the PVS boot process is broken down into five parts:
IP Acquisition, Bootstrap Download, Login, Single Read Mode, MIO Read Mode
Show of hand, how many of you have ever experienced a slow boot scenario?
In those cases most of you have probably seen that the target takes the longest time when you are staring at a blank screen
This is because the most slow downs we see is during Single Read Mode. And during Single read mode we are limited by the MTU size for throughput.
With the use of UEFI boot we take Single read mode and go directly into MIO read mode. So right after login, BNISTACK is loaded and immediately starts transferring data at a 32 MB burst size
Using UEFI with PVS allows for a faster boot

10. PXE (Pre-boot Execution Environment)
DHCP Server 1 2 3
TFTP Server
Now most of what UEFI offers occurs during the first two stages of the PVS boot process. IP acquisition and Bootstrap download. Lets go ahead and cover that boot process again in greater detail
(CLICK) The client initiates the process by broadcasting a DHCPDISCOVER packet containing an extension that identifies the request as coming from a client that implements the PXE protocol.
(CLICK) A DHCP server or a proxy DHCP server implementing this extended protocol sends the client a list of appropriate Boot Servers.
The client also receives the name of a bootstrap file on the chosen TFTP Server.
(CLICK) The client uses TFTP to download the bootstrap from the TFTP server. (CLICK)
(CLICK) Finally, the client loads the bootstrap and then begins the login process to the PVS server which allows it to stream its vdisk. (CLICK) 4 5
PXE Client 6
PVS Server

11. DHCP – Using Scope Options
DHCP Server
PVS Server
DHCPDISCOVER
DHCPOFFER
DHCP ACK
DHCP REQUEST
Specific to Provisioning Services a PVS PXE client will identify itself with DHCP option 60 signifying that it is a PXE client
This is used in conjunction with the PXE service running on the PVS server
The PXE service, which acts like a DHCP relay will respond to DHCP Discover packets that have option 60 in them and provide the client with the location and filename of the network boot program
DHCP Options 66 & 67 can be used to specify the location of the network boot program
The PVS PXE Service acts as a DHCP relay that responds to DHCP Discover packets that have option 60 specified
TFTP Server Name
Bootfile Name
DHCP Client

12. DHCP – with PXE Service DHCP Server PVS Server DHCPDISCOVER DHCPOFFER
DHCP ACK
DHCP REQUEST
PVS PXE SERVICE
Specific to Provisioning Services a PVS PXE client will identify itself with DHCP option 60 signifying that it is a PXE client
This is used in conjunction with the PXE service running on the PVS server
The PXE service, which acts like a DHCP relay will respond to DHCP Discover packets that have option 60 in them and provide the client with the location and filename of the network boot program
DHCP Options 66 & 67 can be used to specify the location of the network boot program
The PVS PXE Service acts as a DHCP relay that responds to DHCP Discover packets that have option 60 specified
TFTP Server Name
Bootfile Name
DHCP Client

13. Bootstrap Download ARDBP32.BIN – UDP Port 69
Citrix PVS TFTP Service
ARDBP32.BIN – UDP Port 69
Once the target device obtains an IP address, it goes through the phase of downloading the bootstrap file.
For BIOS systems this bootstrap file is called ARDBP32.bin.
UEFI is a complete replacement for the BIOS and requires a new bootstrap, PVSNBPX64.EFI.
Try saying 3 times fast after a week in Vegas
PVSNBPX64.EFI – UDP Port 69
Citrix PVS TFTP Service

14. PXE and PVS
Using a PXE server allows for the simplest topology because the PXE protocol works with multiple architectures.
The Provisioning Services PXE Server recognizes the architecture flag embedded in the DHCP, then discovers and returns the appropriate bootstrap filename. Both legacy BIOS computers and UEFI computers may therefore be located on the same network segment.
If DHCP option 66/67 is chosen, there are two topology options:
On a single segment, use DHCP reservations to specify the bootstrap filename (option 67) for each and every target device. This is feasible for smaller environments but quickly scales out of hand for enterprise environments.
Divide the environment into multiple segments, isolating the legacy devices from the UEFI devices. For each segment, configure a DHCP scope with the appropriate option 67 set.
The introduction of another bootstrap complicates network topologies depending upon how the bootstrap is delivered.

15. Configuring bootstraps
The UEFI bootstrap cannot have embedded settings. DHCP options are therefore used to configure the UEFI bootstrap.
There are two new DHCP scope options that need to be configured:
DHCP Option 11
DHCP Option 17
Note: Either the PXE Service or DHCP Scope Options 66/67 are still required.
The UEFI bootstrap cannot have embedded settings. DHCP options are therefore used to configure the UEFI bootstrap.
There are two new DHCP scope options that need to be configured:
DHCP Option 11
DHCP Option 17
Note: Either the PXE Service or DHCP Scope Options 66/67 are still required.

16. Configuring bootstraps
DHCP Option 11 – RLP Server
Option 11 allows you to specify multiple IPv4 addresses.
Use this option to specify the addresses of the streaming NICs on the Provisioning Services server.
You can specify more than four addresses, to a maximum of 32 IP addresses
The UEFI bootstrap reads all addresses then uses round-robin to select one address to connect to.
DHCP Option 11 – RLP Server
Option 11 allows you to specify multiple IPv4 addresses.
Use this option to specify the addresses of the streaming NICs on the Provisioning Services server.
You can specify more than four addresses, to a maximum of 32 IP addresses
As you know the BIOS bootstrap only allowed up to a maximum of 4 addresses
The UEFI bootstrap reads all addresses then uses round-robin to select one address to connect to.

17. Configuring bootstraps (cont)
DHCP Option 17 – Root Path
The Root Path option is typically used with iSCSI to specify the server and virtual disk to start.
Provisioning Services uses the following format to specify the server address:
pvs:<IPv4>[:17:<port>]   
pvs – Required identifier
IPv4 – Address of a streaming NIC on the Provisioning Services server
17 – Protocol identifier for UDP (required if a logon port is specified)
port – Logon port (not required if the default port of 6910 is used)
DHCP Option 17 – Root Path
The Root Path option is typically used with iSCSI to specify the server and virtual disk to start.
Provisioning Services uses the following format to specify the server address:
pvs:<IPv4>[:17:<port>]   
pvs – Required identifier
IPv4 – Address of a streaming NIC on the Provisioning Services server
17 – Protocol identifier for UDP (required if a logon port is specified)
port – Logon port (not required if the default port of 6910 is used)

18. DHCP Scope option 17 examples
IP Address Examples:
pvs:[ ]
pvs:[ ]:17:7800
Hostname Examples:
pvs:[host001.citrix.com]
pvs:[host001.citrix.com]:17:7800
Note: Existing documentation omits the brackets placed around the IP address or hostname. Without the brackets the syntax will be incorrect.

19. Configuring bootstraps (cont)
DHCP Option 15 – DNS Server
This format uses a hostname instead of an IP address.
If you specify DHCP option 15, DNS Server, the UEFI bootstrap performs a DNS lookup to resolve the hostname to an IP address.
You can also use DNS round-robin in combination with records for each Provisioning Services server to distribute logon connections across all Provisioning Services servers in the site.
DHCP Option 15 – DNS Server
This format uses a hostname instead of an IP address.
If you specify DHCP option 15, DNS Server, the UEFI bootstrap performs a DNS lookup to resolve the hostname to an IP address. You will still need DHCP Option 6 configured for your DNS server.
You can also use DNS round-robin in combination with records for each Provisioning Services server to distribute logon connections across all Provisioning Services servers in the site.

20. Configuring Bootstraps
Option 11 allows for more load balancing and redundancy
Option 17 is used when the need to specify a different port is required
Load Balancing available through DNS Round Robin
Option 17 takes precedence over option 11.
Option 11 allows for more load balancing and redundancy
Option 17 is used when the need to specify a different port is required
Load Balancing available through DNS Round Robin
Option 17 takes precedence over option 11.

21. BOOTPTAB
Use the BOOTPTAB file to associate a target device with a specific bootstrap. At Provisioning Services 7.7, the following changes have been made to the format of the BOOTPTAB file to support mixed legacy and UEFI environments:
The ‘ar’ tag specifies the architecture of the target device’s boot environment. You can make multiple entries for the same MAC address but different architectures. This is for use with hardware that supports both legacy BIOS and UEFI booting.
Wildcards are not supported. If an entry for a given MAC address is not found in the BOOTPTAB file, the registry is searched for an appropriate value for the architecture. If neither is found, a default value is used.

22. BOOTPTAB
The full list of architectures is available from the IETF:  
The format of the BOOTPTAB file is:
<hostname>:ha=<mac_address>:ar=<architecture>:bf=<bootstrap_name>
Examples:
host001:ha= :ar=0:bf=ardbp32.bin
host002:ha= :ar=7:bf=pvsnbpx64.efi
If the architecture flag is missing, 0 is the default value.

23. BOOTPTAB
The registry is used to provide the default bootstrap file name for a given architecture.
The name of the value is the architecture number and the value is a string with the name of the bootstrap file.
The following table lists the architectures supported and entries created by the Provisioning Services installer:
HKLM\Software\Citrix\ProvisioningServices\Boot Services\PXE
Value
Architecture
Bootstrap file name
x86  BIOS
ardbp32.bin 6
x86 UEFI
pvsnbpia32.efi 7
x64 UEFI
pvsnbpx64.efi 9
EBC (for VMware ESX)

24. Secure Boot
Secure Boot is a security standard developed by members of the PC industry to help make sure that your PC boots using only software that is trusted by the PC manufacturer.
When the PC starts, the firmware checks the signature of each piece of boot software, including firmware drivers (Option ROMs) and the operating system. If the signatures are good, the PC boots, and the firmware gives control to the operating system.
The following versions of Windows support Secure Boot: Windows 8.1, Windows Server R2, Windows RT 8.1, Windows 8, Windows Server 2012, and Windows RT.
Secure Boot is a UEFI security feature.
PVS, XenApp and XenDesktop do not support Secure Boot.
Hyper-V Gen2 VMs enable Secure Boot by default so it must be disabled for every Gen2 VM.
Physical UEFI computers can have Secure Boot enabled.
Secure Boot is a security standard developed by members of the PC industry to help make sure that your PC boots using only software that is trusted by the PC manufacturer.
When the PC starts, the firmware checks the signature of each piece of boot software, including firmware drivers (Option ROMs) and the operating system. If the signatures are good, the PC boots, and the firmware gives control to the operating system.
The following versions of Windows support Secure Boot: Windows 8.1, Windows Server 2012 R2, Windows RT 8.1, Windows 8, Windows Server 2012, and Windows RT.
Secure Boot is a UEFI security feature.
PVS, XenApp and XenDesktop do not support Secure Boot.
Hyper-V Gen2 VMs enable Secure Boot by default so it must be disabled for every Gen2 VM.
Physical UEFI computers can have Secure Boot enabled.

25. Limitations
Because of the architectural differences between legacy BIOS systems and UEFI systems, some features of Provisioning Services are not available for UEFI systems:
Boot Device Manager (BDM). This tool, which allows you to configure alternative boot methods for Provisioning Services clients using ISO boot or partition offset, is not available for UEFI environments at this release.
The Boot menu, which allows users to choose from a selection of available disks and/or versions, is not available. If more than one disk or version exists, selection operates as follows:
If a maintenance device boots, it takes (in order of preference) the maintenance version, the highest test version, or a production version, depending on what is available. If updates are applied while the maintenance device is streaming a test or production vdisk, the changes are lost on the next reboot.
If multiple vdisks are assigned to a target device, the first vdisk is automatically selected. The user cannot select any other disk.
Because of the architectural differences between legacy BIOS systems and UEFI systems, some features of Provisioning Services are not available for UEFI systems:
Boot Device Manager (BDM). This tool, which allows you to configure alternative boot methods for Provisioning Services clients using ISO boot or partition offset, is not available for UEFI environments at this release.
The Boot menu, which allows users to choose from a selection of available disks and/or versions, is not available. If more than one disk or version exists, selection operates as follows:
If a maintenance device boots, it takes (in order of preference) the maintenance version, the highest test version, or a production version, depending on what is available. If updates are applied while the maintenance device is streaming a test or production vdisk, the changes are lost on the next reboot.
If multiple vdisks are assigned to a target device, the first vdisk is automatically selected. The user cannot select any other disk.
This issue is addressed in PVS 7.8
RECAP
PVS Support UEFI boot with the use of a new bootstrap
Using UEFI boot with PVS makes targets boot faster

26. Hypervisor Configuration
Now lets talk about the specific configurations needed within the hypervisors in order to UEFI

27. Hypervisor Support Hyper V VMWare XenServer (Not supported currently)
XenServer 7.0 will support UEFI boot to DOM 0
Support for UEFI boot to target VMs is further down the road

28. Hyper-V Prerequisite: UEFI boot only supported on Generation 2 VMs.
SCVMM 2012 R2 is used.
UEFI boot only supported on Generation 2 VMs.
Can only be used with 64-bit operating systems, Windows 8 and newer.
Not compatible with secure boot because Hyper-V cannot support multiple certificates.
Only works with Synthetic NIC.
Prerequisite:
SCVMM 2012 R2 is used.
UEFI boot only supported on Generation 2 VMs.
Can only be used with 64-bit operating systems, Windows 8 and newer.
Not compatible with secure boot because Hyper-v cannot support multiple certificates.
Only works with Synthetic NIC.
Generation 2 virtual machines do not have the legacy NIC anymore

29. Hyper-V UEFI configuration
When you create the VM through SCVMM and specify the Virtual Machine Identity you can specify the VM Generation
Gen1 VMs are used for BIOS systems and need to boot from a Legacy NIC and switch over to the Synthetic NIC after BNISTACK loads
Gen2 VM use the Synthetic NIC from boot time
When a new VM is created Secure Boot is enabled by default and needs to be disabled

30. PVS SCVMM Gen 2 Support XenDesktop Wizard:
The template VM must be either:
The master VM
A copy of the master VM (This has to be made by right-clicking on the VM and choose Create > Clone).
This is required because Hyper-V requires that VMs to have the same GUID for the synthetic NIC in order for them to boot the same vDisk.
If a PVSGen2VMClone VM is present, please delete it.
XDSW disables secure boot in the template cache if the chosen template has it enabled.
XDSW deployments with 7.7 and 7.8 disable secure boot in our template cache if the chosen template has it enabled.
This override action is displayed in the summary page.

31. PVS SCVMM Gen 2 Support Known limitations: BDM is not supported.
Secure Boot is not supported.
Only work with 1 NIC in template VM.
Template VM cannot have an iso attached to it.
Gen 2 VMs are not supported by Streamed VM Wizard.
Known limitations:
BDM is not supported.
Secure Boot is not supported.
Only work with 1 NIC in template VM.
Template VM cannot have an iso attached to it.
Gen 2 VMs are not supported by Streamed VM Wizard.

32. VMWare
The following VMware products officially support running virtual machines with virtual EFI firmware:
ESXi 5.0 and newer.
Required virtual hardware:
At least 96 MBytes of RAM.
At least hardware version 7.  Hardware version 8 or newer is preferred due to an enhanced virtual nonvolatile RAM (NVRAM) device which is more robust against guest failures.

33. VMWare UEFI configuration
VMWare allows for the selection of BIOS or UEFI in the device settings under options > Boot Options
While VMWare does allow you to switch the firmware version keep on mind that the VM will not boot unless it had been converted to use UEFI and GPT

34. Troubleshooting UEFI
Lets talk about how we troubleshoot UEFI boot issues

35. Troubleshooting Common Configuration Issues
Network data capture guidelines
Common Configuration Issues
Network data capture guidelines

36. Common Configuration Issues
Misconfigurations
Missing Option 11 or Option 17
Syntax Incorrect for Option 17
Correct Syntax:
pvs:[hostname]:17:<port>
Examples:
pvs:[host001.citrix.com]
pvs:[host001.citrix.com]:17:7800
The most common issue we run into with UEFI are configuration issues. And especially those related to DHCP scope options.
Remember that in order to use UEFI with PVS you need to include either DHCP option 11 or 17 and Option 17 has to have the brackets around the IP or hostname

37. Common Configuration Issues (cont.)
Misconfigurations
Missing DNS Server (Option 6) in DHCP
Missing DNS Domain Name (Option 15) in DHCP
If you do plan on using a FQDN or a Hostname you will need to make sure DHCP option 6 is enabled
For hostname resolution we also need DHCP option 15 which specifies the DNS domain name

38. Common Configuration Issues (cont.)
Misconfigurations
PVS PXE service not running
DHCP Option 66/67 not configured
Option 67 points to the wrong file
ARDBP32.BIN
PVSNBPX32.EFI
PVSNBPX64.EFI
We also require that either the PVS PXE Service or DHCP scope options are configured other we cannot complete the bootstrap download
If you are using DHCP scope options remember to specify the correct bootstrap name

39. Data Capture Guidelines
Whenever possible capture both a server-side and target side trace Try to capture a good vs. bad trace Get all of the IP/MAC information pertaining to a trace

40. Capture Software

41. Target Side capture methods
VMware
Set V-Switch into promiscuous mode 9
Hyper-V
Setup port mirroring
capture-mirrored-traffic-on-a-hyper-v-virtual-machine/

42. Target Side capture methods – cont.
Physical
Port Mirroring
Hubbing out
Using a Tap

43. PVS BIOS vs UEFI Boot

44. PVS BIOS vs. UEFI Video

45. Updates for PVS 7.9
UEFI BDM Support BDM Update Multiple NIC support for Hyper-V Support for multiple virtual optical drives in Hyper-V

46. PVS PXE Checker

47. References http://windows.microsoft.com/en-US/windows-8/what-uefi d=
mirroring-to-capture-mirrored-traffic-on-a-hyper-v-virtual-machine/

48. Questions?

49. Before you leave…
Conference Surveys are available online at starting Thursday, May 26 at 9:00 a.m. PST
Those who provide feedback by Friday, May 27th will receive a $20 Amazon e-gift card
Download presentations starting Monday, May 30th form the My Event Planning Tool