1. Distributed Replay Testing with your data your way!
Melody Zacharias
SQL Saturday #464, Melbourne
20th February 2016
Testing With Your Data, Your Way!

2. Housekeeping Mobile Phones Evaluations Wifi Details
please set to “stun” during sessions
Evaluations
complete online to be in the draw for fantastic prizes
Wifi Details
SSID: eduroam Login: ext-sqlsat Password: sqlsaturd4y
SESSIONS
EVENT
sessions/sessionevaluation.aspx
eventeval.aspx

3. Connect with the Community
Event staff, volunteers and speakers are here to help and answer questions. Scan the QR code on the speaker badges to connect and network with them.
speaker
@SQLDropbear
I attack SQL challenges by dropping onto them from above.

4. Chapter Leader / Regional Mentor for Canada
Melody Zacharias
ClearSight Solutions
Chapter Leader / Regional Mentor for Canada
@SQLMelody
ca.linkedin.com/in/melodyzacharias

5. distributed replay? is it used for? do we set it up? Agenda Why What
During our session today we will discuss the Why, What and How of Distributed Replay as well as touch on a few Tips at the end just before we get to the questions.
How

6. Why Why Distributed Replay? production like work load
monitor and evaluate potential performance, upgrade or functional issues as well as capacity planning.
Why not a third party tool?
Typically third party tools do not use your data.
approval to run a Microsoft tool in an environment that is already using SQL Server is a much easier sell to management.
As a consultant I recently went to a client to ask to run this in their production environment.
Both Profiler and Distributed Replay can capture trace data. So why not just use profiler?

7. Why
Bottle Neck
bottle neck for high levels of multi threaded processes.
With a single computer replay, OLTP applications on high thru-put, or multiple connection applications can cause a bottle neck.
If the testing you are doing is impacted by the bottleneck such as if you are doing performance testing then it is not an accurate test.
The throttling of the replay on profiler can skew your test results.

8. Why Distributed Replay is truly distributed.
Multiple computers for concurrent processing making it excellent for performance testing and capacity planning in OLTP environments.
Distributed Replay supports up to 16 Physical or virtual computers.
real work flow, makes it fantastic for testing indexes and performance issue whether they are hardware or software related.

9. Why - Limitations Step Run to Cursor Toggle Breakpoint
Of course no product is perfect
These are features of Profiler which are not available in Distributed Replay.
Line by line debugging a favorite for developers is not available.

10. What? What makes up Distributed Replay.
It is a console tool and it requires some setup and configuration.
Lets start by looking at the different parts that make up Distributed Replay.

11. What Administration Tool Replay Controller Replay Client(s)
Target Server
sections of the Distributed Replay tool and talk about how they work individually as well as collectively.

12. What
This will be the first of many sides of diagrams outlining different areas of Distributed Replay. This will give you a visual representation of what parts of the tool are affected with each action of the tool.

13. What – Administration Tool
Lets start with the Administration Tool
Installed as part of the management tools in the feature selection of SQL Server 2012.
a console application that initiates, monitors or cancels operation on the controller.
The Management Tools for SQL Server 2012 can be installed on a SQL Server 2005,2008 or 2008 R2 server, this allows the collection of data from instances as old as 2005, however, all replay must be done on 2012 and greater instances.

14. What - Controller
The Distributed Replay controller Service coordinates the multiple client.
There can only be one controller and one administration tool per installed Distributed Replay environment.
I will elaborate more on that later when we discuss domains.

15. What - Clients
A Distributed Replay Client Service is required on each client server.
either Physical or virtual and will work together to simulate the workload on the target server.
Distributed Replay allows for a maximum of 16 client servers.

16. What – Target
The Target Server must be a SQL Server 2012 or higher instance and it is recommended that it be a test environment.

17. HOW
We have covered what Distributed Replay is and what the different parts are, but how does it all work?

18. How Configuration Collection Replay Results
Lets look at the different pieces of Distributed Replay and see what they do and how they work.
As a console tool you may be concerned that the functionality will be limited, however,
each stage of processing and replay of the tool has its own XML configuration file to enable customization.

19. Configuration installed as part of Management Tools 2012 or higher.
For ease of explanation I have diagramed each of the pieces of the tool on separate servers.
The blue box here outlines the services that can be combined on a single server.
Each service has its own configuration file to allow for customization.

20. C:\Program Files (x86)\Microsoft SQL Server\110\Tools
Configuration
C:\Program Files (x86)\Microsoft SQL Server\110\Tools
Controller configuration file
DReplayController.config
Client configuration file
DReplayClient.config
Preprocess configuration file
Dreplay.exe.preprocess.config
Replay configuration file
Dreplay.exe.replay.config

21. Configuration - Preprocess
level of control each of these configuration files give us.
Within the Preprocess configuration file, we can modify which system session activities are included and
Idle time only

22. Configuration - Controller
level of control each of these configuration files give us.
Logging level with options of Informational, Warning, and Critical

23. Configuration - Controller
Yes I did say they are XML (EXtensible Markup Language) files but don’t worry if you are not familiar with XML, it follows a standard pattern with start and end tag and your data in the middle.
The controller file with where the only thing we can configure is the Logging level with options of
Informational, Warning, and Critical.
This is the easiest place to learn.
You can see here between the greater than and less than symbols where to make the change to the logging level.

24. Configuration - Client
With in the Client configuration we can modify the
Controller file location
Working Directory
Result Directory
Logging Level.
This file will be a slightly more complex however the basic principles still apply.

25. Configuration - Replay
replay configuration file which is located on the same server as the administration tool has many configurable options.
used by the administration tool to control all the actions of the coordinated replay with in all the clients.

26. Configuration Replay Configuration Target instance Sequencing
Stress scale granularity
Connect timescale
Think time scale
Use connection pooling
Health monitor interval
Query time out
Threads per client

27. Configuration
The configuration files are engaged and validated when the related service starts on each server that has the service installed
the Preprocess and Replay configuration files are not loaded but used by the controller at a particular stage in the replay.

28. Configuration Restart the services on all the servers
NET STOP “SQL Server Distribute Replay Controller”
NET START “SQL Server Distribute Replay Controller”
NET STOP “SQL Server Distributed Replay Client”
NET START “SQL Server Distributed Replay Client”
On each server that has the services installed
Here are some examples of a quick way to restart each service.
Just to reiterate, The configuration files are engaged and validated when the related service starts on each server that has the service installed.
So please note that when you make a change to either the client or configuration file you must restart the service on the server that the configuration file is located on.

29. Configuration The Rest of the Story.......
The Rest of the Story was a Monday-through-Friday radio program originally hosted by Paul Harvey.
Little known or forgotten facts
So within msdn or technet, that is essentially the end of the story. What we often forget as DBAs or developers is that there are system and security issues that allow servers to talk to each other. For each of these servers to communicate and allow access we need to do a few more things.

30. Configuration Check for errors
C:\Program Files (x86)\Microsoft SQL Server\110\Tools\DReplayClient\Log
Failed to connect controller with error code 0x
Error code 0x means “Access is denied”
If after the last configuration file is loaded we try to run our replay, we will get his error in the error log.
Access Denied.
Access Denied to what?
Access denied to DCOM - Microsoft Distributed Component Object Model. It is a component distributed across networked computers used for communication of software components, in this case the Distributed Replay.

31. Component Services

32. Configuration Start -> Run and type dcomcnfg
To fix the access error we will need to go to Component Services. If like me you do not use Component Services often it is handy to remember that you can access it by selecting
Start -> Run and type dcomcnfg

33. Configuration
Right click on DReplayController to open the properties, and select Security tab

34. Configuration
Here we can grant our service account the proper access permissions
Edit“Launch and Activation Permissions”
Grant “sqlservice” domain user account
“Local Activation” &
“Remote Activation” permissions.
Note:  launch and activation permissions apply the next time that the application is started on the server.
Also note that this is on the client server. This process will need to be done on each client you are running.

35. Configuration Add the “sqlservice” Domain user account to the
“Distributed COM Users” group.
This is a local user group and will need to be done on each client as well.

36. Configuration Restart the services on all the servers
NET STOP “SQL Server Distribute Replay Controller”
NET START “SQL Server Distribute Replay Controller”
NET STOP “SQL Server Distributed Replay Client”
NET START “SQL Server Distributed Replay Client”
On each server that has the services installed
With any changes to the configuration, restart the services to reload the configuration files.

37. Configuration Check the Distributed Replay Client log file
C:\Program Files (x86)\Microsoft SQL Server\110\Tools\DReplayClient\Log
“Registered with controller <ControllerServerName>”
“Failed to connect controller with error code 0x800706BA
<ClientServername> client may not connect to controller “
Once the configuration files are restarted, you will likely see one of 2 different messages. If you get the first one, you are ready to move on , however if you see the second one then you need open the firewall.

38. Fire Wall

39. Configuration In server manager on our client again we
Open Windows Firewall with Advanced Security
Click Inbound Rules
1. Right click and select New Rule to open the new rule wizard that will walk us thru the process

40. Configuration
The rule we are setting up us a program. You want to let the Distributed Replay entry thru the firewall, to ensure a connection can be made from the Client to the controller

41. Configuration Select “This program path”
Browse to the Dreplayclient.exe
“C:\Program Files(x86)\ Microsoft SQLServer\110\Tools\ DReplayClient\DReplayClient.exe”
and click next.
For security reasons we want to be specific to our distributed Replay Client.

42. Configuration
Select “Allow the connection” and click next

43. Configuration
Select only Domain to limit security issues.

44. Configuration Type a name that is meaningful
“Allow Distributed Replay Client”
and click Finish to finalize the opening in the firewall.

45. Configuration Type a name that is meaningful
“Allow Distributed Replay Client”
and click Finish to finalize the opening in the firewall.

46. Configuration Restart the services on the server
NET STOP “SQL Server Distributed Replay Client”
NET START “SQL Server Distributed Replay Client”
restart the services.
Repeat the process of opening the firewall , adding the service account to the “Distributed COM Users” group.
And setting up the service account with the proper DCOM permissions
for every Client Server
Then again for the Controller Server.
There are many steps to getting Distributed Replay configured, however you will only need to do it once for your environment.

47. Configuration - Verify Log files
<ControllerServerName>
Check the Distributed Replay Client log file and see the message
“Registered with controller <ControllerServerName>”
And When we verify the controller log file we will see this text displayed
<ControllerServerName>

48. Let the games begin

49. Collection
Collection – Is the process of gathering and preparing the data we want to use.
It takes place on within both the client and controller depending on the stage.
This section defines the collection stages
Lets begin with the Trace file,
The process to create a trace file maybe familiar to you if you have used profiler before.

50. Demo

51. Collection
1. Once we have the trace run on our server, it is still not ready for replay.
The next step is the preprocess .
2. This step will produce the remaining files needed to do our replay.
3.The Preprocess is part of the Administration Tool, and prepares the trace file for replay against the target instance.

52. Collection
1. Once we have the trace run on our server, it is still not ready for replay.
The next step is the preprocess .
2. This step will produce the remaining files needed to do our replay.
3.The Preprocess is part of the Administration Tool, and prepares the trace file for replay against the target instance.

53. Collection DReplay preprocess -i input_trace_file
-d controller_working_dir
[-m controller]
[-c config_file]
[-f status_interval]
The command to run the preprocess is launched from the administrator tool and has minimal required parameters, here in blue.
The Items in green are optional.
Parameter:
-I Gives the full path name of the input trace file. Note it can not be a UNC path. So if the admin tool is on a different computer than the controller you will have to move the trace file so it is on the local server.
-d Location of the working directory, must be local and fully qualified. This is where the Dispatch file is stored
-m Servername of the controller, default is local
-c an alternate configuration file if you want to change the filtering or max idle time. The default is the DRReplay.exe.preprocess.config
-f the default interval. The default is to display status messages every 30 seconds.

54. Replay
The last step in Distributed Replay is the replay its self. It looks like a simple step but this is also where all the additional action happens. If you are testing with scripts, jobs, third party monitoring tools or other items, this is where everything comes together to make your testing happen. Notice how the launch and synchronization happen on the controller, that is so that it can synchronize all of the clients. I have just one client shown here but the controller is synchronizing the run of the replay on every client at the same time. 1. this is what makes Distributed replay so powerful.

55. Replay DReplay replay -d controller_working_dir -w clients [-o]
c:\>DReplay replay –d c:\DistRePlay –w client1, client2
DReplay replay
-d controller_working_dir
-w clients [-o]
[-m controller] [-s target_server]
[-c config_file] [-f status_interval]
The command to run replay is launched from the administrator tool and has minimal required parameters, here in blue.
The Items in green are optional.
Parameters
-d Location of the working directory, must be local and fully qualified. Where intermediate files are stored.
-w list of the names of the client computers that should participate in the distributed replay. IP addresses are not allowed. And all clients listed must be already registered with the controller( this happens when the client service starts)
-o – indicates an output file, if not indicated summary info is written to the screen only. Location of output file is in the  <ResultDirectory> element in the client configuration file, DReplayClient.xml.
-m Servername of the controller, default is local
-s Target server specified in the fully qualified format of Server_name\instance_name. Local host is not acceptable.
If used the <server> option in the <Replayoptions > of the Dreplay configuration will not be used. Conversely if not specified the option in the configuration file must be declared.
-c an alternate configuration file if stored in a different location. The default is the DRReplay.exe.preprocess.config
-f the default interval. The default is to display status messages every 30 seconds.

56. Results
If during the replay command you specified an output file the results of the replay will be stored on each client participating in the replay, in the directory specified in the command. If no output file is specified, the results are returned to the console where the command was initiated.

57. Uses
Rarely does everything go right the first time. These tips are the ones I find are most useful when using Distributed Replay.

58. Usage and Presentation
Uses
Results
Application load testing
Physical vs Virtual computing
Metrics for distributed computing
TDE load
System upgrades
Hardware upgrades
Index tuning
Tables
Graphs
Analytics
Metrics

59. Results – Read Latency

60. Tips
Rarely does everything go right the first time. These tips are the ones I find are most useful when using Distributed Replay.

61. PowerShell

62. Change in Domain Change in Domain
production environment and test environment are on different domains.
The steps to running in this scenario are the same , however,
once the trace file is created, it must be moved onto the controller setup in the test domain

63. Change in Domain
Controller configuration file
DReplayController.config
Changes location when you change domains
Client configuration file
DReplayClient.config
Modify location in the configuration
The Client Configuration file specifies the computer name of the controller.
The client will attempt to register with the Distributed Replay environment by contacting the controller. When the service is started.
Remember you can not use UNC paths for this it must be local to the controller. You can not use the same controller on different domains.
For a successful run, ensure that every client configuration file has the test domain location of the controller configuration file and not the one in the production environment.
Note again that for any Controller or Client configuration files that are changed, the services will need to be restarted

64. Common issues
Error DReplay The client ‘ClientServerName’ is not a registered distributed replay client. Make sure that the SQL Server Distributed Replay Client services is running on ‘ClientServerName’ , and that the client is registered with the controller ‘localhost’.
Possible Solutions
This is one of the most common errors encountered, particularly if files are being moved between domains or even test systems.
First, Verify location of Controller file and ensure it is the Location listed in Client configuration file and then
Restart client service, to ensure that any changes that were made are applied.

65. Common issues Empty Result set 3 – Empty results or nothing happens
Possible Solutions
Ensure that the
Login IDs
Users
User Default Databases(even if the name is different)
Database IDs
are the same for the both the target and trace databases.
Replaying events associated with missing or incorrect logins results in replay errors, but the replay operation continues.
If possible database IDs should match on the target, however if this is not possible matching on name can be a suitable substitute.
If you have a restored backup copy of the system you collected your data from you will not likely encounter many of these errors unless there have been significant changes to the environment. **
To set the default database of the login, use the sp_defaultdb system stored procedure.

66. QUESTIONS

67. Resources Distributed Replay: http://bit.ly/1OQUhGA
Requirements including events:
Parameters for Replay:
My Blog:
SQLMelody.blogspot.ca My
This is the end of this session however, additional information on the items mentioned and this topic in general are listed here.

68. Questions?
Please make sure you visit our fantastic sponsors:

69. How did we do?
Please complete an online Evaluation to be included the draw for a fantastic prize! There is a prize for each session timeslot and for the overall event survey – so the more feedback you provide the more chances you have.
Post-Event Survey
eventeval.aspx
Session Surveys
sessions/sessionevaluation.aspx