2. DMI314

3. Determine storage performance limit Validation of storage design Hardware burn-in Build-out tests

4. Use Jetstress during solution design to understand precisely how storage will behave Use Jetstress during build out to check for build issues and hardware defects

5. Jetstress uses ESE.DLL to generate an Exchange workload

6. Test a disk subsystem throughput (Recommended) Easy to configure Database configuration manually set Workload controlled by thread count Test an Exchange mailbox profile Uses Profile for configuration Database configuration manually set Workload controlled by thread count

7. Performance Test “Strict” mode (<= 6 hour test) Average Database Read Latency: 20ms Average Log File Write Latency: 10ms Max Database Read Latency: 100ms (6 x Spikes) Max Log File Write Latency: 100ms (6 x Spikes) Stress Test “Lenient” mode (> 6 hour test) Same Read/Write Latency Max Database Read Latency: 200ms (6 x Spikes) Max Log File Write Latency: 200ms (6 x Spikes)

8. What is it? Proving that the storage platform will perform adequately, even if a common failure scenario is experienced What type of failures? Single spindle failure (Raid) Multi-Path failures Dual controller What should I expect? The test should still pass*

9. Copy ESE Files Install Jetstress Configure XML Create Databases Set Thread count Run 2hr test Run 24hr test Run degraded Evaluate results Remove Jetstress Remove data Reboot

10. Use latest version Copy ESE files (ESE.DLL, ESEPERF.HXX, ESEPERF.XML, ESEPERF.INI) into installation directory. Jetstress treats everything as an Active database The test must account for every Active, Standby or Lagged database Use “Test disk subsystem throughput” Easier to configure

11. Initialisation takes roughly 24hrs per 10TB of Data on SAN* Try to arrange your testing schedule to kick this off over a weekend Jetstress generates one database and then copies the rest in parallel With JBOD The more disk spindles you have, the faster the copy process will be Copy throughput can very high on DAS (950MB/sec; ~70TB in 24Hrs) DATA (TB)1TB2TB5TB10TB50TB100TB TIME (Hours)2.44.812.024.1120.3240.6 TIME (Days)0.10.20.51.05.010.0

12. Sett thread count Start low and work up until the test fails Use short test duration (0.5 = 30 minutes) to set thread count Jetstress generates roughly 30 Random IOPS/thread Perform 2hr test Perform 24hr test Perform degraded mode 2hr test (If appropriate) Raid array rebuilding Degraded IO paths Degraded storage controllers

13. The following data is from a real customer test Thanks Boris Walkthrough of a test report

15. Copy test data somewhere “safe” Uninstall Jetstress Remove test databases There are some scripts hidden in the field guide that can help Create-JetstressDataFolders.ps1 Remove-JetstressDataFolders.ps1 Both require Jetstress.XML file for parsing Remove Jetstress installation folder Reboot

16. The Event log is captured and logged. Errors are logged against the volume A single IO error anywhere will fail the test. Detects -1018, -1019, -1021, -1022, -1119, hung IO, DbtimeTooNew, DbtimeTooOld. Threads, which generate IO, are now controlled at a global level. This means Auto-Tuning should work again* Cannot use Jetstress 2013 with Exchange 2010 Basically things are the same as Jetstress 2010 with some bugs fixed and better error handling.

17. CU1 DLL’s and more than 38 DB’s Server stack trace: at Microsoft.Exchange.Jetstress.Performance.PerfLog.AddCounterWildcard(String wildcardPath) at System.Collections.Generic.List`1.ForEach(Action`1 action) at Microsoft.Exchange.Jetstress.Performance.PerfLog..ctor(String fileName, Boolean binaryLog, Boolean includeJetDatabase, Int32 millisecInterval, TextWriter outWriter, TextWriter errorWriter) at Microsoft.Exchange.Jetstress.Core.StressEngine.ExecuteTest() at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase) at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData) at System.Threading.ThreadStart.EndInvoke(IAsyncResult result) at Microsoft.Exchange.Jetstress.Core.StressEngine.EndExecuteTest() at Microsoft.Exchange.Jetstress.MainConsole.Main(String[] args) Use SP1 ESE.DLL to workaround this issue.

18. Faulty Logical Disk Performance Counters… 22.10.2013 17:18:00 -- Microsoft Exchange Jetstress 2013 Core Engine (version: 15.00.0775.000) detected. 22.10.2013 17:18:00 -- Windows Server 2012 Standard (6.2.9200.0) detected. 22.10.2013 17:18:00 -- Microsoft Exchange Server Database Storage Engine (version: 15.00.0712.008) was detected. 22.10.2013 17:18:00 -- Microsoft Exchange Server Database Storage Engine Performance Library (version: 15.00.0712.008) was detected. 22.10.2013 17:18:58 -- Jetstress testing begins... 22.10.2013 17:18:58 -- Preparing for testing... 22.10.2013 17:18:59 -- Attaching databases... 22.10.2013 17:18:59 -- Preparations for testing are complete. 22.10.2013 17:18:59 -- Jetstress testing failed. Error: Jetstress found the following faulty logical disk performance counters: C:\ExchangeDatabases\DAG01-MDB- 1. Ensure that all logical disk performance counters are working correctly with System Monitor. Error: Instance 'C:\ExchangeDatabases\DAG01-MDB-1' does not exist in the specified Category.

19. How to uncover the secrets of Disk Latency Session: MNG.302 Date: Wednesday Time: 4:45 PM - 6:00 PM Room: MR 19ab Session