1. The Data Store and SQL Server Deployment Strategies and Troubleshooting
Frank Marzullo
About the Presenter:
Frank Marzullo, has been with Citrix since 2004 and is currently a member of the Critical Situation response team in the XenApp Support Escalation department. He is also recognized as a subject matter expert on the Independent Management Architecture (IMA), the Data Store and Microsoft SQL Server as it pertains to these XenApp components.
Session short description:
We will review a complex issue reported by customers regarding The Independent Management Architecture (IMA), the Data Store and Microsoft SQL Server. We will discuss the steps taken to determine root cause and the tools used to collect and analyze data along with a quick look into the history of IMA’s past, present and future. We’ll also see how these trouble shooting steps can be applied to almost any IMA or Data Store related issue.
Senior Escalation Engineer, Critical Situation response team
May 8, 2012

2. Tweet about this session with hashtag #SUM305 and #CitrixSummit2

3. Agenda XenApp Data Store deployments and challenges
High Performance Vs. High Availability
Tools and Troubleshooting Methodology
SQL Replication and the identity crisis
XenApp 6.5 Improved IMA performance
XenApp Data Store deployments and challenges
We’ll tackle the most frequently asked questions such as: Where do I put my Data Store? And How do I ensure Data Store uptime ?
We’ll review IMA fundamentals like: IMA Communications, IMA change notifications, What is IMA? What is the Data Store?
High Performance Vs. High Availability
We will talk a bit about why Citrix Admins are faced with this choice and the considerations needed before choosing an option.
Tools and Troubleshooting Methodology
Tools and techniques used by escalation engineers to identify and resolve customer issues.
SQL Replication and the identity crisis
A review of a special Case Study on a SQL Replication issue who’s root cause proved very illusive and frustrating for Citrix admins leveraging this option for its performance enhancing capabilities.
XenApp 6.5 Improved IMA performance
We’ll look at how XenApp 6.5 makes some of these challenges and choices easier with improved IMA performance and less IMA network traffic.
#CitrixSummit 3

4. XenApp Data Store Deployments and Challenges

5. Where do I put my Data Store ?
Quick Access
&
Always Available
Where do I put my Data Store ?
How many put there data store where network access speed is fast ?
How many put there data store where its always available ?
How many had to defer to existing company policies or DBA team for Data Base management?
Before we can move forward with any decision we need to understand a little bit about what IMA is going to be doing and how it uses the Data Store.
In the next slides we’re going to review: IMA Communications, an IMA change notification, What is IMA? and What is the Data Store?
#CitrixSummit 5

6. IMA Communication
IMA Ping – 20 byte per minute if no update data has been received in the past 60 seconds.
Read operations– happen via port 2512
Write operations – happen via port 2513
IMA Change notifications – also via port 2512
IMA Ping:
During normal farm operation, each data collector may send a 20 byte ping every minute to all other data collectors and/or every server in its zone but only if the data collector has not received any data (state update info) from the target server within the last 60 seconds.
#CitrixSummit 6

7. IMA Change Notification
2512
Data
Store
2512
Z D C
Zone 1
Members
Zone 1
2513
2512
2512
In this diagram we can see the flow of an IMA change notification and the process that takes place for changes to be propagated to XenApp servers in a Farm.
A Change is made using the DSC/AMC from “any” server and that server writes the change directly to the Data Store via TCP port 2513.
The same server then forwards a “change notification ID” to the ZDC of its zone via TCP port 2512 (Note – this notification does not contain any data)
The Zone Data Collector then forwards the change notification ID via TCP port 2512 to all member servers in its zone and also to Zone Data collectors in other zones.
These ZDCs in turn send the change notification ID to their member servers and each member server will retrieve the actual change data directly from the Data Store on its own.
2512
2512
Member
Any Zone
Z D C
Zone 2
Members
Zone 2
#CitrixSummit 7

8. What is IMA? Data Store #CitrixSummit 8 What is IMA?
SSL SSL SSL
What is IMA?
At the very heart IMA is a core component of the Citrix application delivery system.
IMA is responsible for providing the common functionality to:
Set up virtual resources
Assign those resources to users
Help users find their resources on demand using any device over any connection from anywhere.
and lastly monitor the resource usage.
#CitrixSummit 8

9. The type of monitoring we are discussing today is “Live resource monitoring” vs “historical usage monitoring & reporting” which is now provided by the EdgeSight product. In XenApp 4.5 we had Resource Manager to provide this functionality and it also was driven by an IMA subsystem. By removing this workload from IMA we effectively simplify functionality at a core level.
In the diagram above we can see that by selecting focus to a server, then selecting the processes tab in the display pane we have the ability to view all process currently running and if we zero in on one in particular the categories listed can be added as a column above.
This can be useful for sorting by “highest value”, for example if we want to see which process for which users is consuming the highest CPU time, we can quickly zero in after sorting this column.
#CitrixSummit 9

10. What is the Data Store ? #CitrixSummit 10
The data store is architected to be a database within a database. A type of layering if you will, the design consist of both a physical and logical layer. The physical layer holds reference and search points to the logical layer where farm data is stored.
This layering configuration creates a flexible design, allowing the Data Store to be ported to different database host products, with minimal effort or architecture redesign.
The only limitation to porting the Data Store to a new hosting solution was, the solution must support large binary objects a.k.a. – “blobs”.
This flexibility comes with one caveat; all data must be interpreted by the application accessing the data, in this case the Independent Management Architecture (IMA) service.
#CitrixSummit 10

11. IMA access method Data Store IMA ODBC #CitrixSummit 11
ODBC
SSL SSL SSL
When the IMA service accesses the Data Store, the access is carried out using an open database connection or (OBDC) for short, via connection parameters that are defined in a Data Source Name file or (DSN).
The DSN file is created for you at install but we have the ability do create, modify or customize this file via the ODBC manager built into windows or by manually editing parameters using any text editor. We can later use the Dsmaint config command to tell IMA to use this new or modified file and this will also serve as a test of the new parameters to confirm connectivity to the Data Store.
#CitrixSummit 11

12. Citrix Confidential - Do Not Distribute
Pre – XenApp 6.0
Post– XenApp 6.0
Stored Procedures
Most of IMA’s commands are carried out using predefined SQL stored procedures.
A Stored Procedure  is a group of Transact-SQL statements compiled into a single execution plan.
Here is a quick look at the difference between XenApp 5.0 and earlier Data Stores and XenApp 6.0 and later Data Stores.
Note the number of stored procedures required for IMA to execute operations has decreased by half. Once again simplifying things at a core level.
Citrix Confidential - Do Not Distribute

13. Citrix Confidential - Do Not Distribute
What is a Schema ?
The storage schema of the Data Store and local host cache are the same
So what is a Schema ? - A schema is nothing more than a diagrammatic representation of something; an outline or model.
If image one is your code logic than image two would be your schema.
Citrix Confidential - Do Not Distribute

14. XenApp 5.0 & 4.5 XenApp 6.0 & 6.5 #CitrixSummit 14
As we discussed earlier, two critical decisions were made which had tremendous impacts on the overall structure of XenApp:
Data/Objects are stored as BLOBs (binary large objects) in the form of attribute-value pairs.
Data/Objects are organized into a hierarchical directory structure.
With the BLOB design, we only need one Key-Table for all the objects types we store. The Key-Table keeps the BLOB data and its hierarchy structure among objects in the data store.
With no doubt, the BLOB storage and the flat single table for all objects would have made it very slow to search objects based on their attributes.  Hence, the birth of the Index Table , which was devised to speed up the searching process.
To assist the synchronization between the central database and local host caches, a Delete Tracker table is added.
This is all we needed, three tables (conceptually). In practice, earlier versions of SQL SERVER did not support really large BLOBs, so then we needed to split large BLOBs into multiple parts and move the BLOB data from the Key Table into a Data Table. Another example of this can be seen for Oracle and IBM DB2, where we had another table, SEQTAB, to generate automatically increased node ids.
In this design, there is a huge mismatch between “in-memory objects” with various attributes and the BLOBs in database. So a marshaling and un-marshaling mechanism was implemented by creating an IMA Directory Subsystem. Do not be fooled by the term of ”Directory” in its name as it has nothing to do with directories of the operating system this is internal to the data store.
With newer versions of SQL the support for large BLOBS was introduced and the consolidation of the Data Table back into the Key Table is now possible. Additionally the normalization of certain data structures considerably reduces the number of records needed to represent particular Farm objects.
#CitrixSummit 14

15. Farm Data Sets Persistent Store Dynamic Store
Farm Data Base hosted on SQL or Oracle
Local Host Cache (LHC)
Dynamic Store
Volatile data stored in memory on Zone Data Collectors or Controllers only.
Static data storage is implemented as a two-level hierarchy. IMA saves all static data, such as configuration data, at a central database server.
Each XenApp server has a local copy of the central database known as a ”Local Host Cache”.
When an item is changed at the central database, the change is broadcasted so ZDC’s and member servers can update their local host cache as we reviewed in the “IMA change notification” slide.
Each member server also periodically synchronize its local host cache with the central database to counter, for the case that broadcast update messages may not be received by a member server (for example, if a server is down due to a reboot when a broadcast message is sent out).
Volatile data examples are: Connected & Disconnected sessions, least loaded server, and zone master ranking.
#CitrixSummit 15

16. High Performance Vs. High Availability

17. Which do I choose ? Why can’t I have both ?
Cost ?
Infrastructure Requirements ?
Infrastructure Management ?
Complexity Vs. Simplicity ?
How many admins (by show of hands) would say that:
Cost – would play or did play a role in this choice ?
How about Infrastructure Requirements ? Like number of third party licenses, hardware ect..
Infrastructure management - Who’s going to own these non Citrix infrastructure resources and how will I be interacting with that owner?
How about complexity vs. simplicity of a particular option ?
Well, by looking around we can see that we are not alone in regards to this challenge but the good news is that there is functionality built in to XenApp itself that by design helps address some of our concerns and all we need to do is apply the proper context to the situation to make sure we are taking full advantage of this functionality in our deployments.
Lets take a quick look at the commonly used SQL High Performance and High Availability solutions and review the context in which each applies.
#CitrixSummit 17

18. SQL Options at a glance High Performance over WAN. Transparent to IMA
Replication
Mirroring
High Performance over WAN.
Transparent to IMA
Recommended for - Geographically separated XenApp Farms.
Special considerations for high latency WAN applications.
Transparent to IMA
Recommended for – Disaster Recovery scenarios.
#CitrixSummit 18

19. Resources for SQL Replication & Mirroring
XenApp 6 and 6.5 SQL Replication Important Considerations
SQL Server 2005 Mirroring and Replication Caveats
Data Store Replication with XenApp 5.0 using SQL 2008 with Windows Authentication
Datastore Replication with XenApp 5.0 using SQL 2005 SP2 with Windows Authentication
Database Mirroring Best Practices and Performance Considerations
Using SQL Database Mirroring to Improve Citrix XenApp Server Farm Disaster Recovery Capabilities
Resources
XenApp 6 and 6.5 SQL Replication Important Considerations
SQL Server 2005 Mirroring and Replication Caveats
Data Store Replication with XenApp 5.0 using SQL 2008 with Windows Authentication
Datastore Replication with XenApp 5.0 using SQL 2005 SP2 with Windows Authentication
Database Mirroring Best Practices and Performance Considerations
Using SQL Database Mirroring to Improve Citrix XenApp Server Farm Disaster Recovery Capabilities
Citrix Confidential - Do Not Distribute

20. Options at a glance Very High TCO #1 HA solution for LAN’s
Physical Server Clusters
Virtual Server Instances
Very High TCO
#1 HA solution for LAN’s
Transparent to Application
Lowest TCO with XenServer
Dynamic Server Provisioning
Fast recovery from Snap Shot
#CitrixSummit 20

21. Resources for virtualizing SQL on XenServer
Virtualizing SQL Server 2008 with XenServer - Blog
Virtualizing SQL Server 2008 with XenServer – White Paper
Resources:
Virtualizing SQL Server 2008 with XenServer
White Paper
Citrix Confidential - Do Not Distribute

22. Deprecated features in SQL 2012
Deprecated Features of Transactional Replication - The following transactional replication features were deprecated in SQL Server 2005 or SQL Server 2008.
Updatable subscriptions including immediate updating and queued updating with snapshot and transactional publications ¹
(Microsoft) recommends that you use peer-to-peer transactional replication instead.
If you used data store replication in previous XenApp deployments, note that in XenApp 6.5:
Replication is no longer required because of IMA architectural changes that have significantly improved WAN performance.
Future versions of Microsoft SQL Server no longer include the only replication model that IMA can tolerate (transactional replication with immediate updating subscribers).
Therefore, although you can replicate a XenApp 6.5 data store on SQL Server 2008 R2 and earlier, you do not need to, and you will not be able to with later SQL Server versions.
Microsoft Documentation:   
Deprecated Features of Transactional Replication
The following transactional replication features were deprecated in SQL Server 2005 or SQL Server 2008.
Updatable subscriptions including immediate updating and queued updating with snapshot and transactional publications ¹
We recommend that you use peer-to-peer transactional replication instead.
#CitrixSummit 22

23. #CitrixSummit 23 This link is incorrect:
To enable updating subscriptions
On the Publication Type page of the New Publication Wizard, select Transactional publication with updatable subscriptions.
SCREEN SHOTS FROM BOTH SQL 12 RTM & SQL 2008R2 SERVERS ARE ON NEXT SLIDE FOR COMPARISON
NOTE – the option no longer exists.
#CitrixSummit 23

24. Tools and Troubleshooting Methodology
In this section we will learn about the: 
• Steps necessary to determine root cause when IMA related problems occur. • Tools used by support to collect and analyze data 
Most IMA related issues can be resolved by collecting and analyzing the following data points captured simultaneously during a reproduction of the symptom or problem.

25. Tools and Troubleshooting Methodology
Citrix Diagnostic Facility Trace (CDF)
SQL Server Profiler Trace
CDFControl is an event tracing controller/consumer, geared towards capturing trace messages that are output from the various Citrix tracing providers.
This tracing leverages Windows (ETL) tracing mechanism which is part of the operating system.
SQL Profiler is an activity trace tool built into SQL and it will show things like: what stored procedure, query or operation is being executed in real-time and display any relevant error messages. One of the really neat things about the combination of these two tools is that while CDF will not capture and display SQL profiler messages it can and will display SQL error messages that come from the built in ETL message outputs. The profiler then becomes useful in lining up the operation being executed and the timeline of interest.
#CitrixSummit 25

26. Tools and Troubleshooting Methodology
Msg 548, Level 16, State 2, Line 134 The insert failed. It conflicted with an identity range check constraint in database 'YourDB', replicated table 'dbo.yourTABLE', column 'nodeid'.
Here is a closer look at the message section from the two traces in the previous slide.
#CitrixSummit 26

27. SQL Replication and the “Identity Crisis”

28. Unique Identifier Track Changes Added by SQL #CitrixSummit 28
SQL Server requires that all articles in a publication allowing updatable subscriptions contain a uniqueidentifier column named 'MSrepl_tran_version' used for tracking changes to the replicated data. SQL Server will add such a column to published tables that do not have one. 
#CitrixSummit 28

29. SQL Replication and the identity crisis
What is the “Identity Crisis” ?
Who does it affect ?
How do I deal with it if it happens to me ?
What is the “Identity Crisis” ?
SQL identity ranges are created by default in SQL at the time replication is set up and they are managed automatically by SQL through the log reader agent.   If the log reader agent cannot auto grow  the range fast enough to keep up with the bulk inserts, the constraint is exceeded and you cannot write additional data to the database.  On the Citrix side, the impact could include but not be limited to the following symptoms:
Adding servers to the farm fails
Publishing applications fails
IMA service fails to start
Access Management Console /Delivery Services Console or Presentation Server Console /XenApp Advanced Configuration Console functionality is lost.
Who does it affect ?
In a scenario where a Citrix data store is hosted on a SQL server with Replication enabled as per Citrix documentation, it has been noted in large and older (long running) Citrix farms that if you perform a task that generates a large number of data inserts at one time, (like adding several new servers at once) the SQL Replication “identity range constraint ”might be exceeded.
How do I deal with it if it happens to me ?
provides background on the issue, steps on how to determine if you are having an identity crisis and how to mitigate and prevent future incidents with the expert guidance of a trained DBA or Microsoft SQL Support.
#CitrixSummit 29

30. Citrix Confidential - Do Not Distribute
Resources
CDF Trace - CTX111961
SQL Profiler Templates for IMA issues
SQL 2005 Template
SQL 2008 Template
Data Store Replication checker tools
For XenApp 5.0 and earlier - CTX111656
For XenApp 6 - CTX124815
HOW TO: Troubleshoot Application Performance with SQL Server
SQL Diag & SQL Diag configuration Tool
Citrix Confidential - Do Not Distribute

31. He who fails to plan is planning to fail.
Now that we have a better understanding of IMA and the Data Store and we’ve looked at the tools and techniques used by Citrix Support to diagnose these components, its important not to overlook the value of a good maintenance and backup plan for our Data Stores.
- Winston Churchill (WW II)

32. Best Practice & Learning Resources
Advanced Concepts - XenApp Farm Maintenance
Best practices for creating the Citrix Presentation Server data store on SQL 2005
Best Practices for XenApp Administrators
Advanced Concepts - XenApp Farm Maintenance
Best practices for creating the Citrix Presentation Server data store on SQL 2005
Sizing
Connection failure
Maintenance plans
Best Practices for XenApp Administrators
#CitrixSummit 32

33. Learning Resources SQL Back ups – recovery models
Working with tempdb in SQL Server 2005
SQL Server Index Fragmentation and Its Resolution
SQL Back ups – recovery models
Working with tempdb in SQL Server 2005
SQL Server Index Fragmentation and Its Resolution
#CitrixSummit 33

34. XenApp 6.5 Improved IMA performance

35. XenApp 6.5 Improved IMA performance
XenApp 6.5 introduces the Dynamic Data Center Provisioning
In XenApp 6.5, servers can be configured in Session-host mode
By Default XenApp servers are configured as Controllers
Session-only XenApp servers run in a light-weight mode whose sole function is to host XenApp sessions and nothing more. 
Speeding up Farm Deployments with XenApp 6.5 – Part 1
This feature allows XenApp servers to join a farm in significantly less time with substantial bandwidth savings.
(also known as Session-only mode).
Which are similar to XenApp servers as we know today.
For the full details and great 3 part blog written by Lenny Soletti (Manager Product Development)
Please go to
#CitrixSummit 35

36. XenApp 6.5 Improved IMA performance
The table below shows the functionality of the Controller vs. Session-only roles.
How it works When a XenApp server joins a farm, it performs numerous read and write operations to the IMA data store as well as a download of the farm data to its Local Host Cache (LHC). In previous releases of XenApp, all member servers of the farm were required to download all farm data to their LHC during a join, resulting in an excess amount of data store transactions and bandwidth consumption. In XenApp 6.5, a select few servers are dedicated as XenApp Controllers which are responsible for farm management tasks, while the remaining member servers are Session-only servers whose sole task is to host user sessions. XenApp Controllers need to sync all of the farm data while Session-only servers only need to sync a subset of the information to their LHC. These changes result in fewer database transactions, less bandwidth consumption, and faster IMA startup performance.
#CitrixSummit 36

37. Much of this time saving is due to the reduced number of SQL transactions the IMA service needs to perform on the data store. Compared to previous releases, XenApp 6.5 has made substantial optimizations to IMA to reduce the number of SQL transactions and speed up the IMA start time.
These figure illustrates just how much the number of SQL transactions are reduced in XenApp 6.5, how much faster IMA starts and how much less data is transferred during IMA operations.
#CitrixSummit 37

38. Citrix Confidential - Do Not Distribute
One last resource
XenApp 6.5 Enterprise Scalable XenApp Deployments
This Scalability Document created by our product engineering and test team provides granular details regarding the test scenarios and results shared in previous slide and new improvements seen during RTM testing of XenApp 6.5
Citrix Confidential - Do Not Distribute

39. Key session takeaways:
XenApp Data Store deployments and challenges
Tools and Troubleshooting Methodology
XenApp 6.5 Improved IMA performance
Citrix Confidential - Do Not Distribute

40. 1 2 3 Tools as a Service Data Collection http://Taas.Citrix.com/Beta
TaaS is an online auto-analysis service from Citrix Support that provides troubleshooting, optimization and proactive recommendations tailored to your environment and implementation.
We wanted to make sure information collection was easy, fast and efficient. Push one button or run one command and collect complete environment information. Then simply log into TaaS.citrix.com/beta and upload the information to Citrix
More info for the speaker:
Our ‘Scout’ collection tool can be used to capture XenDesktop information
The ‘Show TechSupport’ Command can be run on a NetScaler to gather information.
The Server Status report from within the XenCenter console can be used to gather XenServer information.
2) The uploaded data will be auto analysed against a list of known Citrix issues and best practices.
3) Finally, we return environment information and recommendations in an easy to consume format; providing clear instructions and next steps… All tailored to your environment.
Quickly collect and upload your data
Auto analysis health check
Recommendations tailored to YOU

41. Find out how to rev up environment maintenance
Tools as a Service
Find out how to rev up environment maintenance
See your Citrix pit crew in the expo hall with the
The TaaS team can be found at one of the Citrix booths in the Expo hall. Please go see them to take a closer look to see how TaaS can help your environment.
checkered racing shoes

42. We value your feedback!
Take a survey of this session now in the mobile app
Click 'Sessions' button
Click on today's tab
Find this session
Click 'Surveys'

43. Before you leave…
Conference surveys are available online at starting Thursday, May 10
Provide your feedback and pick up a complimentary gift at the registration desk
Download presentations starting Monday, May 21, from your My Organizer tool located in your My Account