SUM303: XenDesktop Advanced Troubleshooting Mick Glover Senior Support Readiness Specialist October 16, 2012

Agenda Overview XD5.x Architecture Basics Logging & Tracing Key Support Tools Overview Using PowerShell to monitor your site Troubleshooting Example(s)

Overview

Before we begin.. A Question Q. To date, what is the most common problem logged with technical support worldwide in 2012? A. VDA Connectivity issues (Registration, Connection/Disconnection…) Its probably not much of a surprise to hear that VDA connectivity issues are still the biggest call generator with Citrix Technical Support WW. Keeping this in mind, much of the focus throughout this deck will be on the relationship between the Controllers and VDAs.

The Good News… Its not all about verbose Logging and Tracing… The following goes a long way… Good Understanding of Product Architecture Knowing what to expect Knowing what to trace and when

XD5.x Architecture Basics

XenDesktop 5 Architecture - .Net SOA Desktop Studio Desktop Director PoSH PoSH WCF [80] WCF [80] WinRM 2.0 [5985/5986] Broker Machine Creation Service Host Service AD Identity Service Machine Identity Service Broker Service Configuration Service Virtual Desktop Agent (VDA) SOA: Service-oriented architecture Ref: http://en.wikipedia.org/wiki/Service-oriented_architecture Each service instance reads and writes to the SQL database periodically using connectionless ADO.net. PVS for VMs Agent Group Policy Engine Machine Creation Services Broker Service Infrastructure Services SQL Server SQL Server SQL Server

Services Key Responsibilities Broker Service Configuration Service VDA registration, Resource allocation, Power MGMT & Licensing enforcement Configuration Service Provides Directory Services, MetaData storage & security Host Service Manages Hypervisor connection & resources When troubleshooting issues in a XenDesktop environment it is important to understand the link between the key services and what functionality they provide. Understanding the relationships even at a high level is crucial when enabling logging and collecting CDF traces for specific issues. Broker Service: The broker service controls most of the day-to-day activity of a controller server in a XenDesktop site. It controls and chooses the resources (applications and desktops) to be served to users, and controls the power state of managed machines according to policy rules and user demands. It is also the conduit through which most run-time status of XenDesktop components is exposed to administration consoles, and any run-time state changes enforced. The broker service is also used to enforce licensing. Configuration Service: This configuration service provides the ability to register services that form part of a XenDesktop deployment and can be used by other services as a directory to locate services that they require to use. This service also provides the ability to store metadata for the deployment. Host Service: The Host service provides the ability to create and manage connections to hypervisors of one of the following types; -XenServer (standalone host or a single XenServer Pool) -VMWare ESX (via VMWare vCenter) -Microsoft Hyper-V (via System Center Virtual Machine Manager (SCVMM)) Once created all connections appear in an identical manner regardless of the hypervisor technology that is under pinning the connection. The connections can be used to browse the hypervisor structure to locate VMs, Storage and Network within whatever structure the specific hypervisor provides.

Services Key Responsibilities Machine Creation Service Creates & manages VMs and Master images Machine Identity Service Creates & manages the disks attached to VMs AD Identity Service Creates & manages AD Computer a/c’s When troubleshooting issues in a XenDesktop environment it is important to understand the link between the key services and what functionality they provide. Understanding the relationships even at a high level is crucial when enabling logging and collecting CDF traces for specific issues. Machine Creation Service: The Machine Creation Service provides the functionality to manage Machine images and provisioned Machines. It manages the VMs and Master Images and utilizes the other Machine Creation Services (AD Identity Service and Machine Identity Service) to create Provisioned Machines. Machine Identity Service: The Machine Identity Service is responsible for the maintenance of storage attached to provisioned VMs. This provides the following functions; -Create OS and Identity disks for VMs -Remove OS and Identity disks for VMs -Update of OS disk (reset VM functionality). AD Identity Service: The AD Identity service records Active directory computer accounts that can be used to provide the identity for Provisioned Machines. These AD computer accounts can be created by the AD Identity Service or existing AD computer accounts can be imported. The AD Computer accounts are collected together into Identity Pools, to provide collections of AD computer accounts that can be associated with Provisioning Schemes (defined in the Machine Creation Service).

Registration Basics VDA registration Registration methods Process of locating a controller and establishing ongoing communication WCF web service endpoints on both ends facilitate the registration process Registration methods Registry base registration AD base registration Quick Deployment registration Registration types Soft Registration Hard Registration VDA registration ((a.k.a. Controller discovery/VDA discovery) refers to two things: VDA finds a DDC and 2. the VDA being in communication with DDC. There are two ways for a VDA to register with a DDC. Registry base registration. AD base registration Use the c:\Personality.ini file (use in Quick Deployment)

WCF Registration port Specifics XD5.x: TCP/IP port 80 Controller and VDA Ports can be set differently on controller and VDA Special consideration when changing the WCF port on controllers if using Registry based registration i.e. ControllerRegistrarPort Notes: XD4: TCP/IP port 8080 (Controller and VDA)

Configuring Registry based registration ListOfDDCs The easiest way to configure registry based registration is to use a HDX policy (Controllers settings) configured through AD. You can however provide a list of Controllers to be used by the VDA during VDA installation by entering each controllers FQDN space seperated.

Checking the WCF port on a Controller BrokerService.exe /show Notes: -C:\Program Files\Citrix\Broker\Service\BrokerService.exe -VDA Port: This is the port used for incoming WCF communications from the VDAs -Use vdaport switch to change port e.g. BrokerService.exe –vdaport 89

Changing the WCF port on a Controller BrokerService.exe –vdaport 89 Notes: BrokerService.exe –vdaport 89: -Changes the port number in the broker service configuration file (‘BrokerService.exe.config’) -Change the windows firewall settings to allow comms on the port -Change the Windows HTTP sharing settings (see ‘netsh http add/remove/show urlacl’) to allow listening on the port/URL combinations needed

Changing the WCF ports on the VDA Re-configure the VDA (add/remove programs) Or HDX Computer Policy Notes: Run gpudate /force /target:computer to apply the new HDX policy for testing purposes.

Changing the WCF ports on the VDA Confirming the changes (registry) Notes: WCF_Port registry Value: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\VirtualDesktopAgent\InstallData\"WCF_PORT“ (REG_SZ) WorkstationAgent.exe.config C:\Program Files\Citrix\Virtual Desktop Agent\ When re-confining the VDA, the meta-installer also carries out some other actions similar to the BrokerService.exe –vdaport <port> -Changes the port number in the Desktop service configuration file (‘WorkstationAgent.exe.config’) -Changes the windows firewall settings to allow comms on the port -Changes the Windows HTTP sharing settings (see ‘netsh http add/remove/show urlacl’) to allow listening on the port/URL combinations needed

Changing the WCF ports on the VDA Confirming the changes (.Config file) Notes: WCF_Port registry Value: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\VirtualDesktopAgent\InstallData\"WCF_PORT“ (REG_SZ) WorkstationAgent.exe.config C:\Program Files\Citrix\Virtual Desktop Agent\ When re-confining the VDA, the meta-installer also carries out some other actions similar to the BrokerService.exe –vdaport <port> -Changes the port number in the Desktop service configuration file (‘WorkstationAgent.exe.config’) -Changes the windows firewall settings to allow comms on the port -Changes the Windows HTTP sharing settings (see ‘netsh http add/remove/show urlacl’) to allow listening on the port/URL combinations needed

ControllerRegistrarPort Details Hardcoded on VDA to use port 80 Value must be manually changed if VDA WCF port is changed on Controller When using AD based registration WCF port is included as part of the URL attribute and therefore this manual change is not required as each controller will update its own details in AD

ControllerRegistrarPort Registry value created after re-configuring the VDA

ControllerRegistrarPort - Known Issue Changing Virtual Desktop Agent Port Using Add/Remove Programs Fails Cause: ControllerRegistrarPort is created incorrectly with an ending space Resolution: Manually delete and recreate Registry value Note: CTX131263 also includes updated binary's which can be deployed to resolve the issue without having to manually edit the registry

VDA Heartbeat Details Value: HeartbeatPeriodMs Originally a 1 minute timeout (60000ms) with the VDA sending a ping every 30 seconds (an interval of half the timeout value) Since the release of XD5 SP1, the default timeout/ping interval has changed to 10 & 5 minutes respectively… Notes: -The controller will consider contact to have been lost, and discard the VDA's registration if no ping is received within the full time specified (i.e. 10 minutes) With XD 5.0 SP1 the default heartbeat timeout was increased from 1 minute to 10 minutes, so keep-alive pings occur every 5 minutes rather than every 30 seconds. Registration: Once a registration of a VDA has been validated, the database is presented with the registration data, and a registration record in the database is created. This operation can result in either a ‘hard’ or ‘soft’ registration, depending on whether the VDA machine is previously known to the database or not. Soft registrations are used when the VDA is not known to the site and is useful when a VDA is installed and set up before being added to the site and avoids unnecessary network traffic, event log message and delays at this point. If the registration is a ‘hard’ registration, the registrar synchronously informs other broker sub-components about the newly registered VDA. If a registration is a ‘soft’ registration the notifications do not happen at registration time but later when/if the registration becomes a ‘hard’ registration. The other sub-components notified are the Configuration Notifier and the Monitor sub-components. While soft registered, only calls via the IRegistrar WCF service are accepted by the VDA, such as ‘ping’ (and possibly ‘probe’ but VDAs don’t use this API). Amongst the data provided by the VDA in its registration request is version data relating to the FullArmor rule blob stored on the VDA (if FullArmor policy is using file-based rather than AD-based rule propagation); this version data is used to later decide whether or not to send a copy of the FullArmor rule blob to the VDA When a VDA has registered with a ‘soft’ registration and a subsequent database action means that the VDA machine is now known to the database, the ‘soft’ registration is converted to a ‘hard’ registration. The registrar sub-component is called (e.g. by the database site service used to monitor this) to indicate that this soft to hard registration transition is required and the appropriate actions are performed.This operation is performed by the broker component (i.e. the registrar, configuration notifier and monitor sub-components) of the DDC on which the site service is running, which may not be the DDC with which the VDA originally registered.

Logging & Tracing

.NET <Service>.exe.Config files Available logging mechanisms "LogToCDF" - if present and set to 1, will log to CDF using the supplied CDF trace module "LogFileName" - if present, will log to the specified file name "OverwriteLogFile" - if present, will overwrite existing log to the file if it already exists Note: More than one sink can be specified, in which case logs will be output to all of them… Note: Some enhancements for Rhone to the logging includes adding the facility to set a threshold logging verbosity level for each of the console/debug/file cases, using the new ‘LogToConsoleLevel’, ’LogToDebugLevel’ and ‘LogToFileLevel’ setting (each defaults to ‘5’). <LogLevel>5</LogLevel> - 0,1,5,9 – Need to test and verify. <appSettings> Defines the logging mechanism to be used by the service "LogToCDF" - if present and set to 1, will log to CDF using the supplied CDF trace module "LogFileName" - if present, will log to the specified file name. Ensure that the service has write permission to the location. "OverwriteLogFile" - if present, will overwrite existing log to the file if it already exists More than one sink can be specified, in which case logs will be output to all of them --> <add key="LogToCDF" value="1" /> </appSettings>  

Out of the box… CDF Tracing enabled for all services <Service>.exe.Config <appSettings> <add key="LogToCDF" value="1" /> </appSettings> By default, all WCF web (.NET) services are configured only to log messages to Citrix Diagnostic Facility (CDF). To disable any service from logging messages to CDF, update the value of the LogToCDF parameter to 0, save the .config file, and restart the appropriate service.

Out of the box… CDF Tracing enabled for all services <Service>.exe.Config <appSettings> <add key="LogToCDF" value="1" /> </appSettings> Clear text logging disabled By default, all WCF web (.NET) services are configured only to log messages to Citrix Diagnostic Facility (CDF). To disable any service from logging messages to CDF, update the value of the LogToCDF parameter to 0, save the .config file, and restart the appropriate service.

WCF Service Logging - Example Enabling clear text logging Edit WorkStationAgent.exe.Config Add the lines in Green below to enable and control clear text logging Set appropriate NTFS permissions Restart Citrix Desktop Service (WorkstationAgent.exe) Citrix Desktop Service runs under the NT AUTHORITY\NetworkService a/c. This a/c needs to be given the appropriate NTFS permissions to create a log file under the specified folder.

Service configuration logging reference - XD4 Module Configuration File Location Controller service cdscontroller.exe.config DDC IMA proxy cdsimaproxy.exe.config Pool Management cdspoolmgr.exe.config AD Config Wizard ADSetupGui.exe.config* WorkStation Agent WorkStationAgent.exe.config VDA XD Setup Wizard SetupToolApplication.exe.config PVS *Have to create file as it does not exists by default Provides verbose output of specified service or tool Does not require TMF files for parsing CdsController.exe.config %Program Files%\Citrix\Desktop Delivery Controller WorkstationAgent.exe.config %Program Files%\Citrix\Virtual Desktop Agent CdsPoolMgr.exe.config %Program Files%\Citrix\Vm Management

Service configuration logging reference - XD5 Configuration File Location Component Broker Service BrokerService.exe.Config %ProgramFiles%\Citrix\Broker\Service DDC Machine Creation Service Citrix.MachineCreation.SdkWcfEndpoint.exe.Config %ProgramFiles%\Citrix\MachineCreation\Service ADIdentity Service Citrix.ADIdentity.SdkWcfEndpoint.exe.Config %ProgramFiles%\Citrix\ADIdentity\Service Configuration Service Citrix.Configuration.SdkWcfEndpoint.exe.Config %ProgramFiles%\Citrix\Configuration\Service Machine Identity Service Citrix.MachineIdentity.SdkWcfEndpoint.exe.Config %ProgramFiles%\Citrix\MachineIdentity\Service Citrix Host Service Citrix.Host.SdkWcfEndpoint.exe.Config %ProgramFiles%\Citrix\Host\Service WorkStation Agent WorkStationAgent.exe.config %ProgramFiles%\Citrix\Virtual Desktop Agent VDA http://support.citrix.com/article/CTX127492

Logs are stored in: PortICA logging PortICA can produce both CDF and clear text output for debugging purposes Disabled by default Create PorticaConfig.XML and place under the following directory on the VM C:\Program Files\citrix\ICAService\XML XML directory must be manually created -Log files not generated as PortICA is not constantly active/listening -PortICA is invoked by WorkStationAgent service Logs are stored in: XP C:\Documents and Settings\LocalService\Local Settings\Temp Vista/W7 C:\Windows\ServiceProfiles\LocalService\AppData\Local\Temp Note: Once PorticaConfig.xml has been created and placed in the correct directory, it must be configured

Configuring PorticaConfig.XML <?xml version="1.0" encoding="utf-8"?> <Config xmlns="Portica.xsd">         <Portica>         <LogFile>             <LogLevel>5</LogLevel>         </LogFile>         <CdfTrace>             <LogLevel>5</LogLevel>         </CdfTrace>         <FunctionTrace>             <LogLevel>5</LogLevel>         </FunctionTrace>     </Portica> </Config> LogFile Level 0,1,5,9 supported 0 = disabled CdfTrace Level 1,5,9 supported FunctionTrace Values 1,5,9 supported Master flow control Copy and paste the above XML output into following text into the fileLogFile LogFile Set to 0, 1, 5, 9. Higher values simply enable all lower trace levels, i.e. anything above 9 outputs all available tracing. Note that only the logfile trace level supports a trace level of 0. Zero is the default for log file tracing - this means that no tracing is logged to file. Any value above 0 will cause the log file to be created in the locations discussed previously. CDFTrace Set to 1, 5, 9. Controls the level of tracing output via CDF. FunctionTrace Set to 1, 5, 9. This is the master flow control for all tracing. If you lower the tracing level here it will reduce the level of tracing to both CDF and the log file. Note: Cdf tracing is enabled by default for PortICA. The above values just control the verbosity of the output.

PortICA clear text logs Logs files are automatically saved to the following locations XP C:\Documents and Settings\LocalService\Local Settings\Temp Win7 C:\Windows\ServiceProfiles\LocalService\AppData\Local\Temp Notes: -Hide protected operating system files must be unchecked in windows explorer to view the LocalSettings folder in XP

Making the process easier… Logging Enabler v2.0 Notes: -Tool developer by Sakthi Ganesh (Lead Escalation Engineer) -Includes option to overwrite existing log file i.e. “OverwriteLogFile” -Prompted to restart services -Logs created under C:\XDLogs folder -Can be used to enable ser4vice logging on controllers and VDAs

Logging Enabler v2.0 VDA Logging enabler can also be run on a VDA and used to automatically enable logging for the WorkStationAgent & PortICA service.

Don’t forget Windows Event Viewer Application Log - VDA XenDesktop 5.5 - VDA Event Log Messages http://support.citrix.com/article/CTX133154

Don’t forget Windows Event Viewer Application Log - Controller XenDesktop 4.0 - VDA and DDC Event Log Messages - http://support.citrix.com/article/CTX133152

Finding & Validating Controllers VDA.log (WorkstationAgent.exe)

Confirming Successful Soft Registration Broker.log (BrokerService.exe)

Tips for searching Logs con’t - If OverwriteLogFile option is not set then each time the any of the .net services are re-started a “start of new log” tag will identity this time point in the log. - Search from the bottom up to find the latest relevant point in the log since the last service re-start.

Soft Registration Soft registration VDAs are not yet belong to a desktop group Only partial registration action done No config sent (XML) or status monitoring initiated No session launching or management is permissible VDA appears in Desktop Studio as registered Avoids unwanted registration delays Avoids network traffic and event logs Soft registration was first introduced in the Sonoma release (XD3.0), it is aimed to fix a number of the issues that were to do with delays in registration. One of the prime reasons for the delay was that at the time when the VDA attempts to register with the DDC, the VDA may not yet be a member of a desktop group. These early registration attempts was refused in the early XD version. When VDA was then subsequently added to a desktop group and there was a significant delay before the VDA’s next registration attempt, possibly of up to 10 minutes. To fix this problem, the soft registration mechanism accepts the registration from the VDA even if the VDA is not a member of a desktop group, but does not yet perform all the normal registration actions. Later, when the VDA was added to a desktop group the soft registration is converted to a ‘hard’ registration by performing the registration steps that were missed out earlier.

Viewing Soft Registered Machines XDDBDiag - XenDesktop Site Summary

Hard Registration Promotes soft to hard registration Once VDA is added to a group Configuration data send to site database Status monitoring initiated (ready or power-off etc.) Note: Registration Hardening is managed by the Broker Service RegistrationHardening site service

CDF Tracing overview The acronym goes by several names Citrix Diagnostic Facility Common Diagnostic Facility Common Diagnostic Function CDF is a system to capture diagnostic output from specific modules in Citrix products which can be very helpful when troubleshooting issues The trace messages are coded into the product and displayed with appropriate status and error codes when run

CDF messages in the source code [ OR This is a pseudo-code example to show how CDF trace messages are embedded within the product We have a function that attempts to load a Citrix feature contained within a DLL We begin the function by outputting a CDF message with a call to PrintCDFMsg() stating that this is the beginning Next we try to load the module with a call to LoadLibrary() Now in the conditional “If” statement we check to see if the variable hCitrixDLL is undefined/null. If this is the case, we print a CDF message indicating failure and return false from the function. Else we print a CDF message for success and return true. 43

Trace Message Format Files (TMFs) How it works Raw CDF events are captured as GUIDs from the Citrix modules. To convert these to human-readable strings, the GUIDs need to be matched with their corresponding descriptions. The mapping of GUIDs to descriptions is accomplished by the use of TMF files which are similar to PDB debug files… Notes: This differs from OutputDebugString in which the debug information can be viewed without the need for conversion with tools such as DebugView from Microsoft. http://msdn.microsoft.com/en-us/library/ff553922%28v=VS.85%29.aspx

TMF Files There are two varieties of TMF files: Public and Private Public & Private There are two varieties of TMF files: Public and Private Public TMF files reveal basic state and function information for customers and partners Private TMFs are for internal Citrix use only and are used by escalation engineers and development while debugging. These files contain references to source code. Public TMF server: http://ctxsym.citrix.com/tmfs/xaxd Public TMFs are also available at ftp.citrix.com for externally use

Key Support Tools

Key Support Tools Tracing & Logging Monitoring & Collection CDFControl CDFCheck CDFMarker CDFMonitor LogEnabler Monitoring & Collection Scout XD Site Checker HDXMonitor Database and Services XDPing XDDBDiag

Tracing & Logging

CDFControl (1/2) The primary tool used to control and consume CDF traces All available Citrix modules Additional features include performance data capture tracing on system startup common error code lookups expert control to zero-in on true errors and failures Available at http://support.citrix.com/article/CTX111961 Features: Module Selection Trace Start/Stop CTL creation - CDFControl create CTL file Remote Tracing - CDFControl Remote Tracing Startup Tracing - CDFControl Startup Tracing ETL file parsing - CDFControl ETL File Parsing Real-time Tracing - CDFControl Real-time tracing Expert Control - CDFControl Expert Info Command line arguments - CDFControl command line switches

CDFControl (2/2) Main screen of CDFControl with modules to select Trace has been captured and parsed in real-time The options dialog allows the trace file settings, trace paths, and TMF search paths to be configured among other settings You may want to browse CTX133900. With the release of FP1, there have been additional categories added to the Trace such as those associated with Universal Print Server: UPS, XTE, and policy specific modules all prefixed with MX

CDFControl remote tracing Start CDFControl Select View -> Remote Providers Verify no firewalls are in place Verify remote registry service running Select modules Tools -> Remote Tracing Saved to sub directory of CDFControl 'CDFControl_Remote_Traces\%computername%' Enter in IP or FQDN of machine If this does not work, verify connectivity to remote machine by: Starting regedit on admin machine and connecting to remote machine Click: Enter one or more IP addresses or FQDN Select Add to add the device to Selected Targets list ***must check ‘Collect trace file after trace is stopped’ at bottom of form. Start!

Remote Providers Error Make sure you can connect to registry on remote machine… Ensure Remote Registry service is started on remote system Ensure firewall is not preventing communications Two main reasons for the above error: -The Remote Registry service is not started on the remote system -A firewall is preventing proper communications between the system where CDFControl is running and the remote system Notes: In an internal test situation when using multiple Domains without specific trusts setup, you will have to update the hosts file on the machine running CDFControl with the name and IP of the remote system. Once done, you will be able to connect using

Multi-Domain/Non-Trust Workaround…

CDFControl startup tracing Start CDFControl – select module(s) Tools -> Options (to set trace file path) Tools -> Startup tracing -> Enable ETL file located at Startup trace file path

CDFMarker v1.3 Stand alone executable CDFMarker.exe Injects a customizable CDF trace statement into CDF trace to help locate narrow search CDFMarker outputs its statements to MF_Hook_MMHook and therefore this module must always be selected CDFMarker is a small utility which allows you to send a customizable CDF trace statement into a running CDF trace, on demand, on either XenApp Servers or XenDesktop Virtual Desktop Agents. There are five ways to send the CDF Trace Statement: a hotkey (Ctrl+M) clicking on the System Tray icon clicking on the System Tray icon and selecting Send CDF Trace Message from the popup menu the Command Line, by passing a string to be used in the CDF Trace Statement, such as c:\cdfmarker.exe Hello! the Command Line, by piping the output of one process to CDFMarker as input i.e. at a Command Prompt type: C:\echo "Up the Dubs!" | C:\cdfmarker.exe results in the following CDF Message being posted to the running CDF trace: CDFMarker: "Up the Dubs!" When a CDF Statement has been written by CDFMarker, the System Tray icon briefly displays a yellow ‘x’ to indicate that the statement has been sent. The CDF Trace is gathered in the normal way. You must ensure that you include the CDF module MF_Hook_MMHook as part of the set of CDF modules being traced. This is because CDFMarker outputs its CDF statements to the MF_Hook_MMHook module. The included TMF files must be used when later parsing the captured CDF trace to be able to read the inserted CDFMarker statements. Ref: CDFMarker On Demand - For XenApp and XenDesktop http://support.citrix.com/article/CTX124577 How to Use CDFMarker to Add CDF Tracing to Batch Files and Scripts http://support.citrix.com/article/CTX125486

CDFCheck CDFChkCmd.exe Suspect a specific binary as root cause of issue Notes: -CDFCheck 1.0 – CMD is a stand-alone executable. There is no installation routine, just copy the CDFChkCmd.exe file to the target system

CDFCheck BrokerService.exe CdsController CdsInterService ControllerDal

Trace Module location HKLM\SYSTEM\ControlSet001\Control\Citrix\Tracing\Modules

Monitoring & Collection

XenDesktop Site Checker V1.1 Features Control Logging (Enable/Disable) Restart Services Check Service Instances Views Site Details Node Details Active Site Services (leases) Assignments (Desktop Groups) The XenDesktop site checker uses the XenDesktop Powershell SDK to enumerate XenDesktop 5 service groups, service instances, active site services, and hosting units. It checks whether or not services are running, it provides the ability to enable logging for each service, and provides detailed information about each component that it checks.

Scout V2.4.0.2 Collects data points and CDF traces Data package(s) uploaded to Citrix Technical Support Auto-Update Help -> Check for updates (Note: To check the current installed version of Scout, click Help > About.) Make sure the customer is using the latest version of the tool . Previous versions do not work with 5.6 and later. Basically, as you can see here, the tool collects data points and traces… And proceeds to package the collected data for upload to TS. Details: -Scout captures data and log files -No system modifications -Uninstall – just delete folder -Submit feedback Help -> Feedback

Scout Data points (1/2) Hardware Software Computer Bios Memory Devices Processor PNP device Logical drive Page file usage Network adapter & configuration Video controller Printers IDE controller SCSI controller OS/Service packs PNP driver Driver Processes MS Hotfix Grop Policy Info (RSOP)

Scout Data points (2/2) Farm Hypervisor Site Information Controllers Catalogs Desktop groups Configured VDA Broker administrators Installed DB version XDDBDiag output Broker hypervisor connection Connection leg

Scout Configuration

Scout CDF Tracing

Scout Collecting & Uploading Data Uploading: -8-digit SR# start with ‘6’ -CDF trace files included with upload -Status report sent to Citrix Tech Support -Confirmation email sent to contact email address

Scout Status Messages “File not found or no read permission error” Click OK Status messages encountered during collection process Status Messages Corrective Action Check Network Check network connection or power status of machine Enable Remote Registry Service Enable Remote Registry service Enable WinRM Enable WinRM on the VDA Enable File Enable File and Printer Sharing (Network Adapter Properties) Here you can see status messages that may possibly be returned via Scout. The first, …. Just click OK. It is a benign message. Next…some valid status messages can be seen here in this table… Enable Remote Registry service by: Start > Run: services.msc, and then select and start Remote Registry Enable WinRM on the VDA by:From a command prompt, type: WinRM quickconfig

Scout Check-Update Compares local version against latest version online If the Local and Server versions do not match, click Download… To check for Updates to Scout From the menu bar, select Help > Check for Updates. If the Local and Server versions do not match, click Download. Note: To check the current installed version of Scout, click Help > About.

Database and Services

XDPing Database & Services CLI based tool Verify configuration settings on DDCs and VDAs Can be run on the local console or remotely Xdping xdping /host MyDDC01.citrix.com Notes: -XD Ping is a great resource tool used in troubleshooting VDA Config Issues. -Automates the process of checking for the causes of common configuration issues in a XenDesktop environment. -The Execution of this tool from a network share, including full UNC paths, is not supported. -If XDPing is executed from a network path, this results in the tool failing to execute correctly. Depending on how the tool is run, and from where, the following checks and information can be displayed: -Information and status of Network Interfaces and Network settings. (Console Only) -Performs DNS lookup and reverse lookup on the IP address of the device. -Information on Time synchronization and time check for Kerberos Authentication (Console Only) -User information for login User (Console Only) -Including User details, Authentication type used, Group Membership. -Machine information (Console Only) -Environment information (Computer Name, operating system version, Domain)

73

XDDBDiag Performs consistency check against DB and summarises findings… Outputs the following diagnostic data into comma-separated value (.csv) files This command line support tool performs a consistency data check on the data and connectivity verification in a XenDesktop 5 database. It also outputs the following diagnostic data into comma-separated value (.csv) files located in a compressed file (.zip) named computername_XDDBDiag_Output.zip to the same directory in which the program is located. Site Information Virtual Desktop Agent Information Current Connections / Connection Log Hypervisor Connections Policy Information Desktop Groups Controller Information SQL Infomation Site Information, Virtual Desktop Agent Information, Current Connections / Connection Log, Hypervisor Connections, Policy Information, Desktop Groups, Controller Information & SQL Information

XDDBDiag Details Consistency check on the database Outputs data into .csv file SQL Data Reader Role required Site Information VDA Info Current Connections Controller Info Hypervisor Connections Policy Info Desktop Groups Connection Log Computername_XDDBDiag_Output.zip located on working directory in which tool is located A quick check to ensure all is good with the DB. Very fast way to identify or eliminate DB issues. Note: there is no dependency on the Broker Service because it queries the db directly!!! Advanced Configuration To disable program update notifications, modify the following line in XDDBDiag.exe.Config: <add key="DisableUpdateNotification" value="true"/> To disable exporting of data, modify the following line in XDDBDiag.exe.Config: <add key="DisableDataExport" value="true"/> For a full list of applicable switches reference: CTX128075

XDDBDiag Advanced Configuration XDDBDiag.exe.Config: To disable program update notifications, modify the following line in XDDBDiag.exe.Config: <add key="DisableUpdateNotification" value="true"/> To disable exporting of data, modify the following line in XDDBDiag.exe.Config: <add key="DisableDataExport" value="true"/>

XDDBDiag output

Honourable Mention HDX Monitor v2.0 Validate the operation of HDX on virtual Desktops Key Features Remote Capabilities Reporting feature Configuration suggestions More Information http://hdx.citrix.com/hdx-monitor

Using PowerShell to monitor your site

PowerShell - Reference Load XenDesktop Snap-ins asnp citrix* Return all available XenDesktop cmdlets Get-command –module citrix.* XenDesktop 5.6 SDK Cmdlet Help http://support.citrix.com/article/CTX127254

PowerShell Remoting Built-in remoting functionality -AdminAddress Parameter Specifies the host name or IP address of the controller to which the PowerShell snap-in will connect Default = “LocalHost” Note: Once a value is specified by any command, this value becomes the new default XD5 cmdlets should not be remoted using PowerShell remoting - instead, the -AdminAddress parameter should be used to specify the name of the Controller.

PoSH - Validating Service Status Use Get-BrokerServiceStatus (Broker Service) Use Get-ConfigServiceStatus (Configuration Service) Use Get-HypServiceStatus (Host Service) Use Get-AcctServiceStatus (AD Identity Service) Use Get-ProvServiceStatus (Machine Creation Service) Use Get-PvsvmServiceStatus (Machine Identity Service)

Get-BrokerServiceStatus - Return Values (1 of 2) Meaning OK The broker is connected to a database that is valid, and the service is running. DBUnconfigured The broker does not have a database connection configured DBRejectedConnection The database rejected the logon from the Broker Service. This may be caused by bad credentials, or the database not being installed. InvalidDBConfigured The database schema is missing (possibly just the stored procedures in it). DBNotFound The specified database could not be located with the configured connection string. DBMissingOptionalFeature The broker is connected to a database that is valid, but it does not have the full functionality required for optimal performance. Upgrading the database is advisable. DBMissingMandatoryFeature The broker is connected to a database that is valid, but it does not have the full functionality required so the broker cannot function. Upgrading the database is required. Ref: http://support.citrix.com/static/kc/CTX127254/help/Get-BrokerServiceStatus.html

Get-BrokerServiceStatus - Return Values (2 of 2) Meaning DBNewerVersionThanService The broker is too old to use the database. A newer version is required. DBOlderVersionThanService The database is too old for the Broker Service. Upgrade the database. DBVersionChangeInProgress A database schema upgrade is in progress. OK PendingFailure Connectivity between the Broker Service and the database has been lost. This may be a transitory network error, but may indicate a loss of connectivity that requires administrator intervention. Failed Connectivity between the broker and the database has been lost for an extended period of time, or has failed due to a configuration problem. The broker service cannot operate while its connection to the database is unavailable. Unknown The Service's status cannot be determined Ref: http://support.citrix.com/static/kc/CTX127254/help/Get-BrokerServiceStatus.html

Setting the DB connection string(s) Required for troubleshooting, DB Migration… To view existing DB connection string Use Get-BrokerDBConnection Server Name of server hosting DB Database Name of XD5 database (CitrixXenDesktopDB = Default)

Setting the DB connection string(s) con’t Trusted_connection=true Enables Integrated Windows authentication The following error is returned if above parameter is not specified: Why? Service instances use the local DDC machine a/c to authenticate to the DB SA authentication is not supported…

Setting the DB connection string(s) con’t To Reset Broker Service DB connection string Set-BrokerDBConnection –DBConnection $null The Broker Services will reset and return to an idle state until a valid new database connection string is specified New value = DBUnconfigured

Setting the DB connection string(s) con’t To Set Broker Service DB connection string: Set-BrokerDBConnection –DBConnection "Server=dbserver;Database=XDDB;Trusted_Connection=True“ New value = OK

Setting the DB connection string(s) - Reference One Cmdlet for each Service Set-BrokerDBConnection (Broker Service) Set-ConfigDBConnection (Configuration Service) Set-HypDBConnection (Host Service) Set-AcctDBConnection (AD Identity Service) Set-ProvDBConnection (Machine Creation Service) Set-PvsvmDBConnection (Machine Identity Service) Update PoSH pic with correct one...

Setting the DB connection string(s) The database connection string is updated to the specified value regardless of whether it is valid or not Specifying an invalid connection string will prevent a service from functioning until the error is corrected Once a connection is configured, it is not possible to alter it without first clearing it (set the connection to $null) When connection is set to $null it is reset and service is returned to an idle state Note: Database connection does not have to be configured in order for this command to be used

Viewing Host Details through PoSH Host Service PoSH provider Name: Citrix.Hypervisor Drive: <XDHyp> pseudo-drive Two Directories Connections HostingUnits Each hypervisor, or pool of linked hypervisors, is described and configured through the XdHyp pseudo-drive

PS XDHyp:\connections> dir Displays detailed information about each defined host connection

PS XDHyp:\hostingunits> dir Displays detailed information about each configured host unit The other child-item of the XDHyp: tree is ‘HostingUnits’. Hosting Units define resources configured to be associated with a hypervisor connection, such as storage repositories and Networks. A hypervisor connection can have multiple Hosting Units associated with it to allow a Hypervisor infrastructure to be partitioned in multiple ways. These partitions can then be the target of ‘Machines’ or ‘Catalogs’ when using Machine Creation Services to provision desktops. Hosting Units are ONLY used when deployments utilize the Machine Creation Services to provision catalogs of machines. The following sample output shows a single Hypervisor connection with 2 hosting units defined.

Checking IntelliCache status UselocalStorageCaching set-item -path xdhyp:\hostingunits\res* -UseLocalStorageCaching $true/$false XenServer 5.6 Feature Pack 1 includes new features that optimize the XenServer hypervisor for use in XenDesktop hosted-virtual machine deployments. IntelliCache is one feature of XenServer that caches temporary and non-persistent files for desktop workloads on the local disk of the host server. By caching locally, a portion of runtime reads and writes of the virtual machine may occur in low-cost server attached storage rather than transmitting across more expensive NAS or SAN resources. As a result, IntelliCache may reduce the requirements for central storage by as much as 90%. The combination of IntelliCache and Machine Creation Services allows administrators to deploy and manage all virtual machines as a thin copy of a master VM and build a cache of that image on host servers. Administrators can still perform live migration of dedicated VMs when needed as the local state is only a cache, so a VM can easily fall back to central storage whenever needed. Also, when the storage capacity of the local disk is reached, IntelliCache will fall back to a safe mode and transfer all reads and writes back to the shared storage. Citrix recommends that you use a high performance local storage device to ensure the fastest possible data transfer. To use IntelliCache you must have at least have both XenServer 5.6 Service Pack 2 and XenDesktop 5 Service Pack 1 installed, and enable the functionality in both XenServer and XenDesktop.

Troubleshooting Example

Error Removing Controller from Site Troubleshooting Example 2 Aim: Remove Controller gracefully from Site using Desktop Studio Issue: “No such host is known” error is returned while the controller appears as offline in Desktop Studio

Check Controller status in Desktop STudio

Use PoSH to check Controller state Get-BrokerController

Enable Desktop Studio Logging Edit MmcSnapin.dll.Config under C:\Program Files\Citrix\Desktop Studio Enable Log To File mechanism “LogFileName” Notes Log Enabler does not include na option to enable Desktop Srtudio L

What we see… Inner exception: System.Net.Sockets.SocketException No such host is known at System.Net.Dns.GetAddrInfo(String name) at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) at System.Net.Dns.GetHostEntry(String hostNameOrAddress) at Citrix.Console.Models.ServiceStateModel.<GetAddressesWithExclusions>b__3(ServiceInstanceStatus address) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__31`3.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at Citrix.Console.Models.ServiceStateModel.GetAddressesWithExclusions(IEnumerable`1 addressesToExclude) at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<GetWorkingServiceInstanceSet>d__25.MoveNext() at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<ExecuteNext>d__c.MoveNext() at Citrix.Console.PowerShellInteraction.PowerShellScript`1.<ExecuteNext>d__9.MoveNext() at Citrix.Console.InteractionCore.ScriptBase.<ExecuteImplementation>d__1.MoveNext() at Citrix.Console.InteractionCore.Scheduler.ExecuteNext(ScriptExecution execution)

What we see… Inner exception: System.Net.Sockets.SocketException No such host is known at System.Net.Dns.GetAddrInfo(String name) at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) at System.Net.Dns.GetHostEntry(String hostNameOrAddress) at Citrix.Console.Models.ServiceStateModel.<GetAddressesWithExclusions>b__3(ServiceInstanceStatus address) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__31`3.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at Citrix.Console.Models.ServiceStateModel.GetAddressesWithExclusions(IEnumerable`1 addressesToExclude) at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<GetWorkingServiceInstanceSet>d__25.MoveNext() at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<ExecuteNext>d__c.MoveNext() at Citrix.Console.PowerShellInteraction.PowerShellScript`1.<ExecuteNext>d__9.MoveNext() at Citrix.Console.InteractionCore.ScriptBase.<ExecuteImplementation>d__1.MoveNext() at Citrix.Console.InteractionCore.Scheduler.ExecuteNext(ScriptExecution execution)

What we see… Inner exception: Notes: System.Net.Sockets.SocketException No such host is known at System.Net.Dns.GetAddrInfo(String name) at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6) at System.Net.Dns.GetHostEntry(String hostNameOrAddress) at Citrix.Console.Models.ServiceStateModel.<GetAddressesWithExclusions>b__3(ServiceInstanceStatus address) at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext() at System.Linq.Enumerable.<SelectManyIterator>d__31`3.MoveNext() at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext() at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at Citrix.Console.Models.ServiceStateModel.GetAddressesWithExclusions(IEnumerable`1 addressesToExclude) at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<GetWorkingServiceInstanceSet>d__25.MoveNext() at Citrix.Console.PowerShellSdk.HealthMonitor.Scripts.ReloadServiceInstancesScript.<ExecuteNext>d__c.MoveNext() at Citrix.Console.PowerShellInteraction.PowerShellScript`1.<ExecuteNext>d__9.MoveNext() at Citrix.Console.InteractionCore.ScriptBase.<ExecuteImplementation>d__1.MoveNext() at Citrix.Console.InteractionCore.Scheduler.ExecuteNext(ScriptExecution execution) Notes: -hostNameOrAddress: The host name or IP address to resolve -GetHostEntry: Method queries a DNS server for the IP address that is associated with a host name or IP address The above exception can be returned if the DNS server does not respond. This exception can also be returned if the name is not an official host name or alias, or it cannot be found in the database(s) being queried.

Researching the Exception… www.google.com Exception Type SocketException Condition An error was encountered when resolving the hostNameOrAddress parameter Sources http://msdn.microsoft.com/en-us/library/ms143998%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/system.net.sockets.socketexception%28v=vs.85%29.aspx

Researching the Exception… www.google.com Exception Type SocketException Condition An error was encountered when resolving the hostNameOrAddress parameter Sources http://msdn.microsoft.com/en-us/library/ms143998%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/system.net.sockets.socketexception%28v=vs.85%29.aspx

Check Machine details in AD DDC256 appears correctly in AD

CDF Output

Checking Service States Get-<Service>ServiceStatus E.G Running Get-BrokerServiceStatus against DDC56.glover.com confirms what the output from the CDf trace is telling us i.e. all services on the controller we want to remove from the site have been reset: Note: -AdminAddress parameter is used to specify which controller the cmdlet is run against (built-in PoSH remoting capability…

Validating Service Status Reference Use Get-BrokerServiceStatus (Broker Service) Use Get-ConfigServiceStatus (Configuration Service) Use Get-HypServiceStatus (Host Service) Use Get-AcctServiceStatus (AD Identity Service) Use Get-ProvServiceStatus (Machine Creation Service) Use Get-PvsvmServiceStatus (Machine Identity Service)

Checking Registered Services As each service on DDC256.glover.com has been reset to DBUnconfigured, running Get-ConfigRegisteredServiceInstance against the same controller will return the following expected error:

Checking Service Instances

Checking Registered Services con’t The problem appears to be that the services for ddc256.glover.com are still registered with the site config service and therefore the site database.

Checking Registered Services con’t Get-ConfigRegisteredServiceInstance –ServiceType The problem appears to be that the services for ddc256.glover.com are still registered with the site config service and therefore the site database.

Step 1 Manually cleaning up the database & removing orphan DB entries Take note of ServiceAccountSid used by service… Note: This is the objectSid of the machine the service is running on

ADExplorer ADExplorer can be used to retrieve the ServiceAccountSid assigned to the orphan services objectSid = ServiceAccountSid

Step 2 Manually cleaning up the database & removing orphan DB entries Create DB eviction script for each service to fully remove them from the site… Ref: XenDesktop 5.6 SDK Cmdlet Help http://support.citrix.com/static/kc/CTX127254/help/index.htm

Step 3 Run config service eviction script against site database

Step 4 Run the rest of the service eviction scripts against the Site DB

Step 5 Refresh Desktop Studio Problematic controller no longer appears in Desktop Studio…

Step 6 The clean up… AD based Registration Registry based registration Run Set-ADControllerDiscovery –sync Registry based registration Update ListOfDDCs registry value via AD Group Policy Computer Policy\Controllers Notes: C:\Program Files\Citrix\Broker\Service\Setup Scripts\Set-ADControllerDiscovery.ps1: -Script to create and/or populate a Site Ourganizational Unit in Active Directory and/or configure the XenDesktop controller for VDAs using AD-based controller discovery.

Optional Step Re-add Controller to same Site As a final step, once the problematic controller has been successfully removed from the site, it can be re-added at a later stage and should appear as fully functional/Active.

Resources

Tools Tracing & Logging CDFControl CDFCheck 1.0 - Cmd http://support.citrix.com/article/CTX111961 CDFCheck 1.0 - Cmd http://support.citrix.com/article/CTX127232 CDFMarker On Demand - For XenApp and XenDesktop http://support.citrix.com/article/CTX124577 LogEnabler http://support.citrix.com/article/CTX118837

Tools Monitoring & Collection Citrix Scout XenDesktop Site Checker http://support.citrix.com/article/CTX130147 XenDesktop Site Checker http://support.citrix.com/article/CTX133767 HDX Monitor 2.0 http://support.citrix.com/article/CTX134879

Tools Database and Services XDPing XDDBDiag for XenDesktop http://support.citrix.com/article/CTX123278 XDDBDiag for XenDesktop http://support.citrix.com/article/CTX128075

How To’s How to Use IntelliCache with XenDesktop http://support.citrix.com/article/CTX129052 How to Enable PortICA Logging http://support.citrix.com/article/CTX118837 How to Enable the Controller Service Logging Feature of XenDesktop 5 http://support.citrix.com/article/CTX127492

Troubleshooting Troubleshooting Virtual Desktop Agent Registration with Controllers in XenDesktop http://support.citrix.com/article/CTX117248

KB Articles - CDFMarker CDFMarker may not send a CDF Trace Message from the Command Line http://support.citrix.com/article/CTX125346 How to Use CDFMarker and PerfMon when a Process Intermittently Spikes http://support.citrix.com/article/CTX125347 How to Use CDFMarker to Add CDF Tracing to Batch Files and Scripts http://support.citrix.com/article/CTX125486 How to Use CDFMarker to Mark an Intermittent End User Issue in a CDF Trace http://support.citrix.com/article/CTX125503

KB Articles - CDFMarker con’t How to Use CDFMarker and PerfMon to Identify Large LHC Writes http://support.citrix.com/article/CTX125634 How to Mark High Bandwidth in a CDF Trace using CDFMarker and PerfMon http://support.citrix.com/article/CTX125724 How to Mark a Service Termination in a CDF Trace using CDFMarker http://support.citrix.com/article/CTX125736 How to Mark when a Process Crash Occurs in a CDF Trace using CDFMarker http://support.citrix.com/article/CTX125750

Before you leave… Conference surveys are available online at www.citrixsummit.com starting Thursday, 18 October Provide your feedback and pick up a complimentary gift at the registration desk Download presentations starting Monday, 29 October, from your My Organiser tool located in your My Account