A managed environment is one which is becoming very common day

Troubleshooting an integrated XenDesktop, PVS and XenServer environment
A managed environment is one which is becoming very common day Karen Sciberras, Escalation Engineer & Keith Mclaughlin, Lead Escalation Engineer Tuesday, May 11th 2010

Agenda Architecture Overview XenDesktop Setup Wizard
Pool Management Service General Issues and Troubleshooting Tools

Architecture Overview
Citrix Confidential - Do Not Distribute

XenDesktop Architecture
Users can connect to a XenDesktop Environment via lan or Wan using Citrix Receiver.

Desktop Delivery Components: The DDC brokers connection requests from endpoint devices, assigning a desktop to each user on demand. The delivery controller also manages licensing and the data store. The VDA runs each desktop that will be delivered to users. The Virtual Desktop Agent provides the ICA service that manages communication between a user’s desktop and endpoint device. The Online Plug-in (Desktop Receiver) runs on the endpoint device and displays a user’s desktop.. Secure remote access, should of course be the Citrix Access Gateway… but could be anything the customer uses to let their users come into the secure network. XenDesktop is not really concerned with this as long as the necessary ports are open.

Desktop Virtualization Components: The Virtual machine infrastructure creates the foundation for delivering virtual desktops and offers management features. To prevent vendor lock-in, XenDesktop can also deliver desktops hosted on Hyper-V or ESX. XenServer / VMWare ESX / Hyper-V Provides the “machines” with advanced management features for XenServer resource pools

Desktop Virtualization Components: Provisioning Services cuts storage costs for virtual desktops through OS provisioning. Rather than storing multiple desktop images, PVS streams a single OS image to multiple target devices stored on the VM infrastructure.

Architecture Overview Boot Virtual Desktop Clients
Provisioning Service PXE Boot Desktop Delivery Controller VDA Clients MAPI Lets take a look at what happens behind the scenes when booting a VDA client. [Click] DDC sends a XAPI command to boot the XenDesktop machine. This command sent by the Pool management service running on the Data Collector DDC in the farm. XAPI boots the Virutal machines. The PXE Process provides the bootstrap and connects to the Provisioning Service Server. Provisioning Service Streams the image and the VDI images boot. Boot Virtual Machine Boot Virtual Machines XenServer XAPI Xen Server Citrix Confidential - Do Not Distribute

Provisioning Service Responsibilities
Creates and manages the image Creates and manages Domain Machine Accounts Delivers image to VDA Clients The roles of Provisioning Service in a XenDesktop Environment is to create the image. It is also responsible for managing the image, meaning it handles all updates to the image allowing for changes to be deployed quickly and efficiently. It handles all domain machine accounts for the VDA clients, Karen is going to take a more detailed look in how that is managed is little later. Lastly it streams the image to the VDA client. Citrix Confidential - Do Not Distribute

XenDesktop Responsibilities
Creates Desktop Groups and VDA clients This is easily obtain using the XenDesktop Setup Wizard Manage the Virtual Machines by: Handling the Power Management for 'managed' desktops Maintaining Pools of idle desktops This is handled by Pool Management Service on DDC XenDesktop two main user interfaces is the DDC and the XenDesktop Wizard located on the PVS Server Using the XenDesktop setup wizard a user can setup Desktop Groups, pick a image and the number of VDA clients that is needed. The DDC will create these VDA clients and boot them up. Behind the scenes this is accomplished through the various API’s of the Hypervisor being used and Provisioning Services.

XenDesktop Setup Wizard

Allows an administrator to quickly create a set of virtual desktop It is installed on the Provisioning Services Server and communicates with: XenDesktop DDC (Desktop Delivery Controller) Provisioning Services Virtual Infrasturcture (Citrix XenServer, Microsoft HyperV or VMWare ESX) What is needed: Virtual Machine Template on the hosting infrastructure (XenServer, ESX or HyperV) A base OS provided by Provisioning Services The XD Setup wizard helps an admin save time as it will automatically create many virtual machines in a very small amount of time. It also communicates with Provisioning Services and the Virtual Infrastructure, to allow the creation of the VMs to be created on all three components. The VM template will consist of all the hardware requirements (RAM, CPU etc) which are needed for the set of Virtual Desktops. The XD Setup Wizard will create X number of XD VM with the same virtual hardware configuration. SetupToolApplication.exe using the logged in user needs access to CitrixManagementServer.exe via 2514 svchost.exe using <NT AUTHORITY\NETWORK SERVICE> needs access to svchost.exe via 135 for Remote Procedure Call (RPC) SetupToolApplication.exe using the logged in user needs access to dllhost.exe via <available port> for COM Surrogate SetupToolApplication.exe using the logged in user needs access to CdsImaProxy.exe via 8000 is what we need from the PVS server -> the DDC in relation to the XenDesktop Setup Wizard (your last had the direction backwards – ccn’g Jagruti to see if she has any further input). I would start by focusing on these links. You might also consider CitrixManagementServer.exe, CdsImaProxy.exe, and CdsPoolMgr.exe logging. This can be enabled by simply adding the <add key=”LogFileName” value=” C:\<logfile directory>\<whatever>” /> entry to the respective .config files.

Select base OS Add Target Device in PVS Each Target Device identified by MAC PVS adds Target Device to AD Obtains list of SIDs for VMs Virtual Infrastructure Creates Desktop Group Add Virtual Machines to Group Maps UUID to SID XenDesktop Setup Tool Provisioning Service (PVS) Connect to XenServer Pool Obtain list of Templates Select a Template Create X number of VMs Step 1: Connect to the hosting infrastructure and create new virtual desktops The XD Setup Wizard connects to the XenServer resource pool via the master XenServer. It instructs XenServer to create X number of VMs. E.G we created 100 VMs. A new MAC address is created for each VM that corresponds to the virtual NIC for the VM. The XD Setup Wizard stores this newly created MAC address for each VM along with the host name specified (CXD1, CXD2, ... CXD100). Step 2: Configure virtual desktops in Provisioning Server The XD Setup Wizard adds a target device in Provisioning Server for each of the virtual desktops. The client name for each of the target devices is the host name. When the VM boots it replaces the host name of the base OS image with this client name. Each target device is uniquely identified by the MAC address which is why we stored the MAC address for each VM in the previous step. Each target device is then set to boot from the specified base OS image (CXD_IMAGE). In addition Provisioning Server adds each target device to active directory. You can either let the XD Setup Wizard add computers to the default location or you can specify a custom OU. Step 3: Add virtual desktops to a new Desktop Group in a Desktop Farm The wizard now creates the new desktop group we called "CXD_GROUP". The 100 virtual desktop VMs created above are now added to this desktop group on the Desktop Delivery Controller (aka the Connection Broker or DDC). The DDC identifies each of the VMs by their AD host name, but when the VMs are added the DDC can only see the VM name and UUID (Universal Identifier). The wizard knows the host name for each VM so it informs the DDC of this automatically. Otherwise the administrator would need to manually associate each VM name / UUID with its corresponding AD host name. Desktop Delivery Controller A MAC address created for each VM Corresponds to the Virtual NIC of VM

Creating Machine Accounts in the domain
Provisioning Service Create VDA Target XenDesktop Setup Wizard VDA Clients MAPI Create VDA Target Add VDA Target This slide will show how the stream process adds a target to the domain during the creation of a VDA target. [CLICK] DDC initializes creation of VDA Target, sends command to Provisioning Service MAPI to create target [VDA1]. Provisioning Service SQL client connects to the SQL DB and adds Target VDA1 to the database. It also generates a Machine Account Password key for that device. Provisioning Service contacts Domain Controller and adds the new target to the DDC OU. It supplies the previously generated key. Xen Server SQL DB VDA1 VDA1 SQL Database Domain Controller Citrix Confidential - Do Not Distribute

Pool Management Service

What is Pool Management?
Feature that controls the power states of ‘machines’ Service that contains the logic for the power state transitions We have three plug-ins which all talk to hypervisors: XenServer VMware ESX Microsoft Hyper-V Idle pool Powers on machines in advance so that users don’t have to wait for them Powers down machines when they are not required

Desktop Life cycles Assigned Pooled Off Suspended 7 Minutes Idle pool
Free Taint Action [immediate] Log on Log off A Pooled desktop group consists of machines which are not owned by any user. Thus a user logging into the systems is provided with a machine form the idle pool which they are free to use until log off. At this point, any information which was placed on this machine the user is lost. When the pooled machine is not in use It is marked as free if it is up and running The pool management service may choose to shutdown the machine if the idle pool count is satisfied When a user logs onto a machine, the machine is marked as busy When a user logs off from that machine The machien is marked as “tainted” We use this state because users have elevated rights on a machine so they change registry settings, break the desktop etc. The machien is now in no state for anyone else to use it. Thus we mark this machine as Tainted and we have the option of either shutting down the machine effectively resetting it or if we know the machine is fine (particularly in enviornments which have locked down their desktops) then the machine is placed back in the idle pool. When a user disconnects from a session the machine is marked a Idle. It will remain in this state until the user reconnects to the session or the machine has reached its Idle timeout. IN this case it is placed back in the Idle Pool. In XenDesktop, we have mainly two kinds of desktop Groups:- Assigned and Pooled. An assigned desktop group contains desktops which are specifically allocated to a particular users. When that user is not logged into that machine the Pool management service: mark the machine as “free” if the machine is up and running. fully shutdown or suspended machine after a specified timeout. When a user connects to that machine, the Pool management service will mark this machine as busy When a users disconnects without logging off, the machine is marked as ”idle” the pool management service may suspend the machine after a specified timeout. Also when the user is not actively connected to the machine, it may be suspended Disconnect In Use Disconnected Reconnect Log off Tainted

Some hidden dials that can be tweaked (with care!!)
LogoffActionDelay (7 mins) / DisconnectActionDelay (5 mins) Lets the user change their mind without waiting for a new VM to boot... MUST be > RegistrationTimeout RegistrationTimeout (3 mins) How long a desktop gets to register (before showing error) RegistrationForceShutdownTimeout (3hrs) How long a desktop gets to register (before we force a power-off) ShutdownTimeout (10 mins) How long we give a VM to shutdown gracefully, before pulling the plug This are settings which we don’t really publisize as they should be configured by people who really understand their environment. However, for certain large environnemtns where the default timeouts are causing you more problems than benefits, here are some tweaks to help you improve the overall performance of your enviornment. Logoff Action delay: This setting ensures that when a machine moves from the free state to the Off state, the Pool Management service will wait 7mins by default before shutting down a machine. This is used mainly for two reasons: It allows a user to change their mind and not have to wait for a new VM to boot It is also used a precaution so that machines are not accidentally switched off due to a race condition or something similar. DisconnectActionDelay A disconnect appears as a machine goes from in use to free. When is it returned to the idle pool, it should not be suspended for at least 5 minutes by default. NOTE these timeouts must be greater than the registration timeout. RegistrationTimeout This timeout comes into play when a machine is not able to register with a chosen Desktop Delivery Controller. After the elapsed timeout, the Pool management service should assume that the managed machine has failed. BY default this is 3 minutes. RegistrationForedShutdownTimeout If the machine failed to register within this timeout, the machine is powered off ShutdownTimeout If a machine is unable to shut down by itself, it will be powered off. This prevents a faulty machine from consuming the resourced of a hypervisor. BY default, this is 10 minutes.

General Issues and Troubleshooting

Known Issues Provisioning Services and Antivirus
Configuration for PVS to handle machine passwords Virtual machines are not mapped to AD account names Hypervisor is Overwhelmed

Provisioning Services and Antivirus
Whitelist PVS Filters Do not scan system drive Antivirus updates Low Level Network Driver: Because the PVS Network driver operates at a low level and essentially sniffs and re-directs incoming packets some Antivirus Software will flag the driver as a virus and shut down the driver. To avoid this we recommend adding all PVS Driver files to a non scan list. Do not can system drive When a image is provisioned only the blocks of the disk that are specifically requested are read and passed through the network. When a antivirus software scans the system drive it is forcing the target to read the contents of the whole disk. This will create a large amount of I/O traffic leading to latency issue l Updates: Make all antivirus updates to the image manually. Disable automatic updates k

Provisioning Services and Antivirus best practices
Limit Antivirus updates to the target. Disable scanning of the write cache location especially if caching on server. Do not scan I/O in real time. Exclude scanning low level PVS drivers BNNS.sys, BNNF.sys, BNPort.sys, and bnistack.sys.BNDevice.exe Install Antivirus before Provisioning Services test updates on a staging image. More information can be found: When working in streamed image mode, new virus updates will get downloaded and applied to each target EVERY time the target reboots. These updates could get large overtime and could cause very long boot times. Citrix recommends manually managing the virus updates by disabling auto updates and downloading the updates on a schedule via private image mode. In limited testing it has been observed that most scanners cannot detect a virus within this location because of their inherit design and the methods used to determine a virus. Scanning a remote cache location in this manner will greatly increase the I/O to the cache location and impact the overall performance. Do not scan your Targets I/O stream in real-time. This can cause excessive retries when the Target expects it’s I/0 and that process is delayed by real-time scanning, there is good potential for a second and maybe more requests for the same packet fragment. Avoid scanning the BNDevice.exe process on the Target. There are a few drivers that should be excluded from scanning, as well, in the <systemroot>\windows\system32\drivers directory you can exclude BNNS.sys, BNNF.sys, BNPort.sys, and bnistack.sys. Some Antivirus installs can overwrite the provisioning service target filter which will prevent the target from booting. We also recommend to test all Antivirus updates on a test image before applying it to Production incase the update re-installs the Antivirus filter drivers.

Configuration for PVS to handle machine passwords
Issue: Users were able to log into their machines but now cannot. Administrator is able to log into the vdisk in private mode. Resolution: Points to the AD machine account not being setup properly in AD AD machine account password has expired Group Policy setting: Computer Configuration -> Windows Settings -> Security Settings -> Local Policies -> Security Options Domain member: Disable machine account password changes: Enable Administrator is able to log into the vdisk in private mode.

Identifying Virtual Machines
Hosting Infrastructure Identifiers look like this: Microsoft path/to/VM VMWare Name-UniqueID XenServer GUID XenDesktop uses Active Directory Machine Identity [Machine SID] We store this mapping as VM meta-data Each of the hypervisors that we support, have different methods of identifying the machines they host. The Microsoft Hyper-V uses a path, the Vmware Esx uses a Name ID while the XenServer uses a GUID. On the other hand, in XenDesktop as we are very much integrated with AD, we identify the machines using the Machine SID. In fact, if you had to take an unmanaged machine for example, there is no Virtual Machien Identifier and thus the machine is just identified by the AD SID. However with managed desktops, we have this additional complexity and thus we need to map the AD SID of a particular machine with its corresponding Identifier. This mapping is stored in the VM Metadata.

VM Meta Data CTXGuestOSID CTXGuestMGTInfo This is the Guest OS SID.
Data is written by the DDC Farm Master. Referenced by the DDC to keep the virtual machine and the Guest SID aligned for the PVS service. CTXGuestMGTInfo Keeps the virtual machine and the Pool management service aligned. This data identifies which pool a machine belongs to Allows the DDC to query power state of the virtual machines in that pool. As such we store two kinds of information in the VM Metatdate of a particular machine. GuestOSID. This is an attribute attached to each VM and it hold the Machine Account SID of that particular machine. Thus when this machine needs to start up, we search for the GuesOSID with the same SID as that of the particular machine. The GuestMGT Info is similar to the GuestOSID but instead of storing the SID of the actual machine, it stores the identifier of the Pool that the machine belongs to. Thus it allows the DDC to find the machine and query its current power state

Virtual machines not mapped to AD account names
Desktop in Active Directory VM in XenServer This is the UI which shows this mapping. In this screenshot, we can see that there are some errors between the mappings. This is usually caused if there are DNS issues in the environment or the machine been cloned and thus are being seen with the same SID. In this case, you will need to do the mapping manually yourself. You can do this by clicking on the X and mapping the device. You can also export these mappings and edit the mappings as needed and then imported them back into the system. The metadata, gets lost, copied when it shouldn’t (clones) and appears in admin consoles (VMW) Multiple Virtual Machines identified as the same AD machine “DuplicateGuestOSIdException” Virtual Machine incorrectly identified Account mappings lost “NoSuchGuestOsIdException” Problems [mouse over]

Hypervisor Overwhelmed
By default, the Pool Management Service will start-up 10% of the default pool size. In large environments, this may be more than the hypervisor/ Provisioning server will be able to handle. To prevent this, the pool management can be configured to stagger the start-up of the Virtual Machine. This is configured in the config file of the Pool Management Service

Hypervisor Overwhelmed
Open C:\Program Files\Citrix\VmManagement\CdsPoolMgr.exe.config Add setting, for example: <?xml version="1.0" encoding="utf-8" ?> <configuration> <appSettings> <add key="LogToCdf" value ="1"/> <add key="LogFileName" value ="C:\cdslogs\VMManager.log"/> <add key="LogDebug" value="1"/> <add key="MaximumTransitionRate" value="20"/> </appSettings> </configuration> Restart the Pool Management Service

Tools Used CDF Tracing PVS Soap Logs Network Tracing e.g. Wireshark
Pool management Logs Setup Logs PVS Soap Logs Network Tracing e.g. Wireshark

Enable CDF Tracing on XenDesktop
Only available in XenDesktop CDF trace information can be written to logfiles in plain text Need to edit a text file to enable Can also use CDF Control (CTX111961)

Modules that can log Filename Path Location CdsImaProxy.exe.config
Citrix\Desktop Delivery Controller Desktop Delivery Controller CdsPoolMgr.exe.config Citrix\VmManagement SetupToolApplication.exe.config Citrix\XenDesktop Setup Wizard Provisioning Services

How to Enable Logging Manually create a directory where to store the log Edit the config file with the following values: Configure the value LogToCDF from 0 to 1 <add key=“LogToCDF” value=“1” /> Add the location where log file will be stored: <add key=“LogFileName” value=“<location.log>” Restart the service Article CTX provides further information

XenDesktop Setup Wizard Logs
INF:(9/30/ :31:02 PM):Retrieving poolName for Pool at address INF:(9/30/ :31:02 PM):Pool name for PoolMaster at address is Karen XenEnv INF:(9/30/ :31:02 PM):Cloned Machine XDVDA1 (UUID : d53-c7c1-bcbb-438a761ff565) , MAC : 92:45:30:22:a8:bf INF:(9/30/ :31:02 PM): XenManager.CloneVm method. - Exit INF:(9/30/ :31:02 PM):Vm cloning for desktop XDVDA1 was successful. INF:(9/30/ :31:02 PM):Vm disk provisioning for desktop XDVDA1 beginning. INF:(9/30/ :31:02 PM):Calling ProvisionDisk of diskManager with args: clonedVmName = XDVDA1. INF:(9/30/ :31:02 PM):-> PVSmanager.ProvisionDisk - Entry INF:(9/30/ :31:02 PM):Entering IDiskManager.ProvisionDisk method. INF:(9/30/ :31:02 PM):The VM XDVDA1 will be provisioned using diskTemplate XP machine. INF:(9/30/ :31:02 PM):Adding new machines to provisioning server. INF:(9/30/ :31:02 PM):Adding new machines to provisioning server. INF:(9/30/ :31:02 PM):-> PVSmanager50.GetSharedDiskTemplates - Entry INF:(9/30/ :31:02 PM):-> PVSManager50.GetSites - Entry INF:(9/30/ :31:02 PM):Adding site XD with Id 05d0bb98-2bfa b42-1c3c a INF:(9/30/ :31:02 PM):All 1 sites successfully retrieved.

Provisioning Services: Enable SOAP Logs
The logging properties can be found in the Server Properties. These settings control the logging for the Stream Process, console, soap service, config wizard and MCLI. Logging level value controls how verbose the logging is. Info Error Debug and Trace are the four level Values. Trace is the most verbose. The stream process is the only log that will report trace level events. When setting logging level to trace, the Console soap config wizard and MCLI default to their highest level which is DEBUG. Logs are set to Error by default. Pros and Cons of upping log files.

Provisioning Services: Adding Device to Farm
:49:47,609 [12] DEBUG SoapServer.ServiceMain - ** Start Command 'Add Device' :49:47,625 [12] DEBUG SoapServer.ServiceMain - Username: XDS\Administrator :49:47,625 [12] DEBUG SoapServer.ServiceMain - deviceName=XD3VDA1 :49:47,625 [12] DEBUG SoapServer.ServiceMain - collectionId=f368555d-4d66-487c-ad1c-c1bdb04a9bbe :49:47,625 [12] DEBUG SoapServer.ServiceMain - deviceMac=7e-d e6 :49:47,625 [12] DEBUG SoapServer.ServiceMain - description= :49:47,625 [12] DEBUG Mapi.CommandType - in CommandAddDevice.Execute :49:47,625 [12] DEBUG Mapi.Command - Add to table Device :49:47,625 [12] DEBUG Mapi.Command - sqlStatement = <INSERT INTO [Device] ([deviceId],[deviceName],[collectionId],[deviceMac],[description]) SELECT DISTINCT :49:47,625 [12] DEBUG Mapi.Command - parameter values are = = = = = > :49:47,625 [12] DEBUG SoapServer.ServiceMain - Command 'Add Device' returned: :49:47,625 [12] DEBUG SoapServer.ServiceMain - return code: 0 :49:47,625 [12] DEBUG SoapServer.ServiceMain - ** End Command Verifying the Device was added in the soap logs

Provisioning Services: SOAP Logs - Adding Device to Domain
:49:48,859 [12] DEBUG SoapServer.ServiceMain - ** Start Command 'MacroSet' :49:48,859 [12] DEBUG SoapServer.ServiceMain - Username: XDS\Administrator :49:48,859 [12] DEBUG SoapServer.ServiceMain - MacroSet 'Set Device' :49:48,859 [12] DEBUG SoapServer.ServiceMain - deviceId=235ab677-f89a efe-317c927054c :49:48,859 [12] DEBUG SoapServer.ServiceMain - adTimestamp= :49:48,859 [12] DEBUG SoapServer.ServiceMain - adSignature= :49:48,859 [12] DEBUG Mapi.CommandType - authGroups = <4c5b1faa-0bc7-478a-a45c-50f3e72d4549> :49:48,859 [12] DEBUG Mapi.CommandType - parameters preVal = <deviceId=235ab677-f89a efe-317c927054c5> :49:48,859 [12] DEBUG Mapi.CommandType - record fields preConv = <adTimestamp= , adSignature=8899> :49:48,859 [12] DEBUG Mapi.CommandType - pre-ValidParms = <deviceId=235ab677-f89a efe-317c927054c5> :49:48,859 [12] DEBUG Mapi.CommandType - post-ValidParms = <deviceId=235ab677-f89a efe-317c927054c5> :49:48,859 [12] DEBUG Mapi.CommandType - pre-ConvertFields = <adTimestamp= , adSignature=8899> :49:48,859 [12] DEBUG Mapi.CommandType - post-ConvertFields = <adTimestamp= , adSignature=8899> :49:48,859 [12] DEBUG Mapi.Command - Set in table Device :49:48,859 [12] DEBUG Mapi.Command - sqlStatement = <UPDATE [Device] SET [adTimestamp] FROM [Device] d WHERE d.[deviceId] :49:48,859 [12] DEBUG Mapi.Command - parameter values are = = = 235ab677-f89a efe-317c927054c5> :49:48,859 [12] DEBUG SoapServer.ServiceMain - Command 'MacroSet' returned: :49:48,859 [12] DEBUG SoapServer.ServiceMain - return code: 0

Packet Sniffers Packet traces can be helpful in determining where in the boot cycle a target fails. More information on the use of Wireshark and packet captures can be found in my other session SUM306: Citrix Provisioning Services stream process architecture and advanced troubleshooting Today at 2:30- 3:20.

A managed environment is one which is becoming very common day

Similar presentations

Presentation on theme: "A managed environment is one which is becoming very common day"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

A managed environment is one which is becoming very common day

Similar presentations

Presentation on theme: "A managed environment is one which is becoming very common day"— Presentation transcript:

Similar presentations

About project

Feedback