1. Microsoft Ignite 2016
11/11/2018 7:03 PM
BRK4021
Investigate tools and techniques for Exchange Performance Troubleshooting
Jeff Mealiffe
Nasir Ali
Bob Samer
Justin Turner
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Session Objectives Takeaways
11/11/2018
Session Objectives
How customers can prevent performance issues?
Share tools and troubleshooting methodologies for performance issues
Demo of tools used by Escalation Services
Takeaways
Increased knowledge of architectural recommendations and best practices for Exchange 2013/2016 deployments
Improved awareness of advanced tools and techniques for troubleshooting Exchange
Showcase some of the most challenging problems that support encounters
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Agenda Scripts, Tools and Data collection
11/11/2018
Agenda
Best Practices for Planning & Sizing
Tools and Techniques for detecting and Analyzing performance
Case Study: Troubleshooting Performance Issue
Exchange Online Latency and Efficiency
Scripts, Tools and Data collection
Health Checker
CPU Sizing Checker
Exchange Log Collection
Log Parser Studio
Windows Performance Recorder/Analyzer
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Preventing Issues: Planning & Sizing
11/11/2018 7:03 PM
Preventing Issues: Planning & Sizing
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. 11/11/2018 7:03 PM
Use the calculator
Role Requirements Calculator turns published sizing guidance into a modeling tool
Try out various failure scenarios
Understand the impact of different hardware & storage choices
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Considerations of scale: up or out?
11/11/2018 7:03 PM
Considerations of scale: up or out?
How big is too big?
Our testing & production deployment focuses on mid-range, 2 socket servers
Design for scale-out, not scale-up
Better alignment with intentions & design points of engineering
Ideally focus on “commodity” 2U servers as a platform to help minimize deployment risk
We don’t push the “top end” today – and don’t want you to either
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Scale guidance updates for Exchange 2016
11/11/2018 7:03 PM
Scale guidance updates for Exchange 2016
Updated memory guidance for Exchange 2016 just released
New max recommended memory = 192GB
No update to max recommended processor cores
Doesn’t impact existing guidance for Exchange 2013
HPE ProLiant Gen9 servers BIOS defaults not optimal for Exchange
NUMA Group Size Optimization = “Clustered” -> Change to “Flat”
More info:
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Virtualization best practices
11/11/2018 7:03 PM
Virtualization best practices
Absolutely do not use memory overcommit
Major cause of perf problems on virtualized Exchange
Hyperthreading is OK, but size based on physical processor cores
Never oversubscribe CPU, CPU constraints cause pain
Delivery throughput reduction = queue growth
Content indexing throughput reduction = increased IOPS
Store ROP processing throughput reduction = RPC latency & end-user pain
Size using guidance for physical, add CPU overhead
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. 11/11/2018 7:03 PM
Balance at CAS layer
Load balancing handles traffic distribution across CAS components
Select a solution that is application aware for high availability
Select a traffic distribution policy carefully
Goal is equal spread of inbound client requests
\Web Service(Default Web Site)\Current Connections
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. 11/11/2018 7:03 PM
Balance within DAG
Always aim for well-balanced equal distribution of active copies
Utilize DB copy ActivationPreference and MaximumPreferredActiveDatabases parameters to ensure *-over maintains balance
Consider regular utilization of RedistributeActiveDatabases.ps1 to maintain balance
During localized high load events, redistribute active copies to migrate load
\MSExchange Active Manager(_total)\Database Mounted
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Balance within databases
11/11/2018 7:03 PM
Balance within databases
Always aim for equal utilization of databases (space & activity)
Spread out heavy and light users across databases
Can be based on heuristics around job role, or statistics like message send/receive rates
Monitor space utilization and rebalance databases regularly via mailbox moves
Don’t forget about whitespace when evaluating available DB space
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Balance within mailboxes
11/11/2018 7:03 PM
Balance within mailboxes
Exchange 2013 improved server-side perf for high item count folders
Legacy clients still have issues with high item counts
Look at Office 365 limits as best practice
More folders with fewer messages
Set-mailbox can enforce mailbox shaping quotas
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Guidance & Best Practices Summary
Microsoft Ignite 2016
11/11/2018 7:03 PM
Guidance & Best Practices Summary
Stay up to date on Exchange & .NET aka.ms/ExchSept2017
CPU consumption increase aka.ms/E2016Sizing
Scalability guidance aka.ms/sizematters
Use the calculator! aka.ms/e2013calc
Perf thresholds published on TechNet aka.ms/excounters
Perf health checker script aka.ms/perfscript
Proper sizing is critical for a healthy deployment
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Detecting Performance Issues on premises
11/11/2018 7:03 PM
Detecting Performance Issues on premises
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Common Causes of Performance Issues
Microsoft Ignite 2016
11/11/2018 7:03 PM
Common Causes of Performance Issues
Cumulative Update Version, Best Practices
.NET Framework Version
Power Management
Sizing/Load Distribution
Scale up (Hardware)
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. 11/11/2018
.NET Framework Version
Performance issues or delays when you connect to Exchange Server 2013 that is running in Windows Server
Fix included in but requires an explicit reg key
Fix implicit in 4.5.2
4.6.2 is the latest supported version
Support for latest .NET Framework
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Power Management Gotchas
11/11/2018
Power Management Gotchas
Misconfigured power management
Applicable to Exchange 2010, 2013 and 2016
Throttle power to the CPU and NIC
CPU frequency can be stepped back in order to save power.
Processor Information(_total)\% of Maximum Frequency
% of Maximum Frequency
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Power Management - Solution
11/11/2018
Power Management - Solution
Physical Hardware:
BIOS/Hardware level – allow OS to manage power
Windows – Set power plan to “High performance”
Virtual Machine:
Check power management settings at the VM host or hardware layer
“High performance” power plan still recommended but not as impactful
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Troubleshooting Sizing
11/11/2018
Troubleshooting Sizing
Most sizing issues we encounter in support are CPU bound
Do the math! What are my available megacycles and how many megacycles do I need?
Variables – Message profile, SPECInt 2006 rating, server role, # of active/passive mailboxes, and multiplier
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. SRV Garbage Collection – Yes or No?
11/11/2018
SRV Garbage Collection – Yes or No?
What is GC? What are SRV and WKS GC?
Should we change it?
.NET CLR Memory\Allocated Bytes/sec > 50MB
.NET CLR Memory\% Time in GC > 10
Why not change it?
Increased # of managed heaps = increased memory footprint
Masks the underlying issue
At best it should be viewed as a troubleshooting step, not a solution
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Health Script Demo Nasir Ali Microsoft Ignite 2016 11/11/2018 7:03 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Case Study: Microsoft Exchange Health Manager Service Crash on Start
TechReady 23
11/11/2018 7:03 PM
Case Study: Microsoft Exchange Health Manager Service Crash on Start
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. TechReady 23
11/11/2018 7:03 PM
Symptoms: MSExchangeHM stuck in the stopping state. Perf counter categories missing from Perfmon. Failed to add perf counters
How we Analyze data and Solve the issue?
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Microsoft Ignite 2016
11/11/2018 7:03 PM
What data is needed?
HealthCheck Script output from the servers exhibiting the problem.
Daily Performance log from the servers exhibiting the problem.
Performance counters XML.
System and Application Event logs from the servers exhibiting the problem.
ProcMon Logs
Watson dump
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Data Analysis Time: 7/26/2017 6:12:41 AM ID: 4999 Level: Error
Microsoft Ignite 2016
11/11/2018 7:03 PM
Data Analysis
Time: 7/26/2017 6:12:41 AM
ID:
Level: Error
Source: MSExchange Common
Message: Watson report about to be sent for process id: , with parameters: E12IIS, c-RTL-AMD64, , MSExchangeHMWorker, M.O.Datacenter.WorkerTaskFrameworkInternalProvider, M.O.D.W.WindowsPerformanceCounterFactory.Create, System.InvalidOperationException, d590,
ErrorReportingEnabled: False
Time: 7/26/2017 6:08:26 AM
ID:
Level: Warning
Source: MSExchange Common
Message: Performance counter updating error. Counter name is Percentage of GLS Shared Cache Expired Entry Hit Last Minute, category name is MSExchange Global Locator Processes. Optional code: 3. Exception: System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
at System.Diagnostics.PerformanceCounter.InitializeImpl()
at System.Diagnostics.PerformanceCounter.get_RawValue()
at Microsoft.Exchange.Diagnostics.ExPerformanceCounter.set_RawValue(Int64 value)
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Checking the Exchange perf counters
11/11/2018 7:03 PM
Checking the Exchange perf counters
0:000> !pe c1c148a840
Exception object: c1c148a840
Exception type: System.InvalidOperationException
Message: Category does not exist.
InnerException: <none>
StackTrace (generated):
SP IP Function
000000C1C013E4D FFD09CE782F System_ni!System.Diagnostics.PerformanceCounterLib.CounterExists(System.String, System.String, System.String)+0x202edf
On a good server from PowerShell
[PS] C:\Program Files\Microsoft\Exchange Server\v15\Scripts>[System.Diagnostics.PerformanceCounterCategory]::CounterExists("Poisoned Workitem Count","MSExchangeWorkerTaskFramework",".")
True
On the bad server
Exception calling "CounterExists" with "3" argument(s): "Category does not exist."
At line:1 char:1
+ [System.Diagnostics.PerformanceCounterCategory]::CounterExists("Perce ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : InvalidOperationException
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. 11/11/2018 7:03 PM
So what’s next?
We can’t access exchange counters, although the counters are on the server and are not disabled
Process monitor shows the below getting loaded in the perfmon during initialization
Microsoft Monitoring Agent
InterceptCounters.dll
Check for APM (Application Performance Monitoring)
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Solution! Reinstall the SCOM agent with APM disabled
© Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Case Study: Exchange Perf and AD
TechReady 23
11/11/2018 7:03 PM
Case Study: Exchange Perf and AD
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Symptoms DC side symptoms Client side symptoms
Primarily: High Local Security Authority Subsystem Service (LSASS) CPU Utilization (could be high memory)
Client side symptoms
Timeouts, application failures due to slow/no DC response
Slow Lightweight Directory Access Protocol (LDAP) bind
Repeated prompts for credentials

31. Cause Excessive or inefficient workloads Other bottleneck conditions
LDAP; Security Accounts Manager (SAM); Local Security Authority (LSA); Change notification (LDAP); scripts that leverage expensive WMI queries
Other bottleneck conditions
MaxConcurrentAPI (MCA); Null domain Auth; Null domain lookups

32. Client DC Client DC Resolution Reduce Optimize Redistribute
Modify app/script
Implement a cache
Apply updates
and rollups
Optimize
Index
Modify configuration
Client DC
Client DC
Redistribute
AD Site configuration
Increase Capacity
More DCs, CPUs etc.

33. Troubleshooting Workflow
“Peeling the Onion”
DC-Side Data Collection
Active Directory Diagnostics Data Collector Set (With DC workload review)
1644 Event Logging
Client-Side Data Collection

34. Troubleshooting Process
Reduce, Optimize, Distribute, Increase Capacity
Client and DC-side methods
If it hurts, stop doing it.
Client-Side Data Collection
Identify application, process, or script
Tasklist, netstat, network trace, process monitor, etc.
DC-Side Data Collection
Map workload to client-side caller
Active Directory Diagnostics, 1644 Events, network trace, etc.

35. DC-Side Data Collection
Active Directory Diagnostic Data Collector Set (SPA)
1644 Event Logging – Tracking Inefficient / Expensive Queries
Network trace, netstat –anob, tasklist /svc, Netlogon.log, LSP log, event logs (DS SDP if possible)

36. DC Workload Directory operations Active Directory Replication
Search and Bind
Kerberos ticket operations
WMI queries, DSCrackNames, Account operations
Sid2Name and Name2Sid
NTLM operations

37. 1644 Event Logging
Enable logging of expensive and inefficient searches in Event ID 1644
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\15 Field Engineering
Set to a value of 0x5 to log one event per LDAP search that exceeds the threshold (Increase the size of the Directory Service Event Log)
Thresholds
If no threshold value is specified (registry value not set) then the following values are applied:
Data analysis is difficult when looking at individual events
Event 1644 script available from TechNet Script Gallery
Extracts 1644 events into Excel with pivot tables to make analysis easier
Registry Path
Data Type
Default value OS
Comment
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Expensive Search Results Threshold
DWORD
10,000
ALL
Using the default values, a search is considered expensive if it visits more than 10,000 entries
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Inefficient Search Results Threshold
1,000
A search is considered inefficient if the search visits more than 1,000 entries and the returned entries are less than 10 percent of the entries that it visited.
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Search Time Threshold (msecs)
30,000
Windows Server 2012 R2 or later or MSKB is installed (Windows Server 2008, Windows Server 2008 R2, Windows Server 2012)
Event is logged if search exceeds 30,000 milliseconds (30 seconds) – probably a bit too long for a threshold

38. Common Issues
CPU Load
LDAP Search vs ATQ Threads
SAM calls (SAMenum*)
Slow / Hang (slow LDAP, Kerb auth, ATQ Thread exhaustion)
Null Domain Lookups (LSP log), Null Domain Auth (Netlogon.log)
How long queued requests take to be serviced
Requests are being queued
Max ATQ Threads is equal to ATQ Threads Total

39. Exchange Online Latency and Efficiency
Microsoft Ignite 2016
11/11/2018 7:03 PM
Exchange Online Latency and Efficiency
Bob Samer
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

40. Exchange Online Cloud (EXO)
Largest deployment of Exchange on the planet
Hundreds of thousands of servers hosted in hundreds of DCs across the globe
Millions of users in 100s of countries generating billions of requests each day
Growth is off the charts, we are growing across every dimension (users, requests, servers, processes, # of builds released, etc.)
Our mission is to continuously decrease costs and reduce latency without slowing down!

41. Testing IS Production No labs or test passes involved!
Leverage our massive scale to test scalability and latency
Multiple builds a day rolled out to production with intelligent ramp that halts and alerts on issues.
Real user traffic in each ring used to find latency problems, crash-fests, CPU issues, memory leaks, IO abuse, etc.
RING 1
RING 2
RING 3
RING 0
WORLDWIDE
FEATURE TEAMS
OFFICE 365 TEAM
MICROSOFT
SLICE OF PROD

42. How do we manage cost/latency increases?
Integrated system of fabric and applications that combine to:
Prevent most increases from reaching production
Contain remaining increases from spreading in production
Constantly analyze and optimize
RING 1
RING 2
RING 3
RING 0
WORLDWIDE
FEATURE TEAMS
OFFICE 365 TEAM
MICROSOFT
SLICE OF PROD
These systems process literally PBs of data per day to triangulate issues as well as enable ongoing optimizations

43. Perf System Fabric EDS CPR Latency Cubes
Configuration driven service with extensible analyzer model running on every production machine
Processes information from logs/perf counters and executes triggers
Pushes aggregate data to SQL Store (Azure) for further analysis
EDS
Low overhead, continuous profiling running on every production machine
Daily aggregates of CPU, Memory Allocation, Exception sample counts and stacks aligned on key pivots (process, build, machine)
Data used for trending, call stack aggregation and drill down to pinpoint specific code causing production resource usage
CPR
Processing platform extracting end full latency information for all O365 protocols.
Allows for near real time alerting for latency regression and cost increases in addition to long term trending over time/location
Data also used to create “checkpoints” for protocols to quickly pinpoint what layer of the stack is introducing an increase.
Latency Cubes

44. Perf Applications (Stability)
Stop runaway costs from destroying individual servers
Smart alerting when components are nearing limits on an individual machine
Debugging views that gives cost breakdowns across units of capacity

45. Perf Applications (Efficiency)
Find costly call stacks across the entire fleet
Provide engineers Real-time Feedback in VS IDE

46. Perf Applications (Latency)
Find slowest call stacks across the entire population of user requests
Layer cakes showing individual component’s contribution to overall cost

47. So what…how does this help on-prem?
Cloud and On-Prem share builds and platforms*
Cloud optimize cycle benefits On Prem via:
Exchange/O365 Software fixes
Pushing fixes into our dependencies
Performance Feature work
Advice on recommended HW and SW configs

48. Q&A Microsoft Ignite 2016 11/11/2018 7:03 PM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

49. Resources Use the latest TechNet documentation
11/11/2018
Resources
Use the latest TechNet documentation
Exchange Server 2013 Performance Recommandations
Exchange 2013 Sizing and Configuration Recommendations
Exchange 2013 Performance Counters for troubleshooting
Performance Troubleshooting tools
ExMon
IIS Logs and Log Parser Studio Reports
Exchange Performance Data Collection tool
Exchange 2013 Performance Health Checker Script
Windows Performance ToolKit (WPT)
Performance Analysis of Logs (PAL) Tool
Windows SysInternals
Generating user message profiles for use with the Exchange Calculators
Change Computer Power Plan/Profile Script
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

50. AD Performance: Additional Resources
Creating More Efficient Microsoft Active Directory-Enabled Applications
LDAP Query Optimizer changes
1644 Event Improvements
1644 Reader PowerShell script
AD Data Collector Sets
ATQ Performance Counters
How to Find Expensive Inefficient Queries using 1644 and script
LDAP

51. Please evaluate this session
Tech Ready 15
11/11/2018
Please evaluate this session
From your
Please expand notes window at bottom of slide and read. Then Delete this text box.
PC or tablet: visit MyIgnite
Phone: download and use the Microsoft Ignite mobile app
Your input is important!
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.