1. Help! AX IS SLOW!
Grant Wilson
@NAVUG

2. Agenda Introductions Expectations AX Performance Introduction
Configuration
Virtualization
Finding Bottlenecks
Prevention
Discussion

3. Introductions Who am I? ERP Solutions Architect
More than 10 years experience in AX.
Both functional and technical experience.
End user and consultant
AXUG Board Chairman, Programming Committee, Speaker, All-Star…
Still learning…

4. Expectations I hope you walk out feeling ... Interactions
Don’t be afraid to ask a question
Don’t be afraid to answer a question
Slides will be available
I hope you walk out feeling ...
Like they learned something
Like this time was worthwhile
Housekeeping
Cell phone use
If you have to leave …

5. AX Performance Introduction
What does slow mean?
Everyone is locked out, a group can’t function, a user’s screen is frozen, a report takes too long?
Who determines what is slow?
Can the issue be replicated in a test environment?
Should be “easy” to fix
Sometimes nothing is “wrong” with AX
Is someone watching a HD movie on Netflix across the network?
Did someone print a 1000 page report?
Are SQL, AOS, Terminal, WAN, Desktop all just a little loaded?

6. Configuration
Make sure your AX environment is properly configured for a production environment. This means SQL, SSRS, SharePoint, AOS, and Terminal Servers are properly setup. Some specific notes:
Verify your SQL Configuration -
For your AOS, make sure debugging is not enabled, hot swapping is not enabled, X++ and CIL compile without error.
Latest available patches are installed
If you are not yet live, and have some Visual Studio experience on staff, I highly recommend using the Benchmark Toolkit - us/download/details.aspx?id=39082 to simulate a full load on your system. Additionally, make sure to test your Terminal/Citrix servers at full load.

7. Configuration Recommended SQL Trace Flags
Everyone running AX should be using this. It changes the behavior of statistics auto- update. The link to the managing SQL indexes talks about statistics being more important than indexes. Details on this flag from Microsoft
Use this if you run Master Planning, especially while other processes are running. It changes the lock escalation in SQL and reduces the number of deadlocks.
This will ensure the size of multiple tempdb files will stay the same. As per the recommendation from Microsoft on SQL configuration, you should have one tempdb file for each CPU on the SQL server, it is important that they grow the same size.
Changes the behavior of SQL commands with parameters (which is how AX runs). Microsoft has a good description here of how this helps.
Use to enable SQL hotfixes.

8. Virtualization ~20% performance hit for Virtualization
When running VMware
Make sure the SQL disks are "Thick Provisioned - eager zeroed"
Using VMotion will temporarily cut the network connection, if you do this on an AOS or SQL server, AX will go down. VMotion is one of the main benefits of virtualization, but should only be used in emergency as it will stop your environment.
Watch the CPU load on each machine. Not the load reported load in task manager, which is wrong on virtual machines, but the actual load. VMware has performance counters for this, look to see actual speed and CPU stolen.
Configuring Storage on a Virtual SQL server is different than a Physical one. If all the storage is running on the same SAN back end, you don't need the same drive separation. Talk with your storage vendor.

9. Virtualization Watch the Network load on the host servers.
Try and keep the SQL and AOS servers close together, on the same physical host is the best.
Make sure you do your tests while the host servers are under normal load. If they are being used for anything else, this will impact your performance.
Performance Monitoring on Virtual Servers requires different statistics as the standard Windows counters are not aware of Host load.
Look at VMWare specific counters for CPU and compare to regular CPU
%CPU Stolen is important counter

10. Finding Bottlenecks
The main idea here is that you are in production and are seeing problems and are looking for the current bottleneck. Here is the path I use to to find the issue:
Is the issue being seen by one user or multiple users:
Multiple users:
1. Check SQL for Blocking/Locking, RAM, CPU, Disk Space and Disk utilization. There are many good packages to monitor SQL and your DBA should be monitoring this, however many companies don't have dedicated resources for this. I always check here first. For RAM and CPU issues, those should be easy to address. Running out of disk space happens, make sure to check it. For disk utilization - that one is complex depending on your setup. If you are seeing locking/blocking regularly I recommend two things - first, make sure that nothing is running from a SQL standpoint (Backup, Maintenance, etc). Second, use the Dynamics Performance analyzer to determine the source - Lastly, check to make sure that any users have access to the SQL server are not running expensive queries. As a general rule, no one should have ODBC access to your production database.
Sample Pal Report
2. Check the AOS server(s). Look at RAM, CPU and Disk Space. You should regularly be monitoring the RAM usage of the AOS service to determine what is normal for your environment. If overloaded, an AOS restart will fix the problem but root cause analysis will be required. If you are regularly seeing Memory issues, check your buffer settings are appropriate, check to make sure that a table caching is appropriate for the number of records in the tables. It is possible the settings from go-live are no longer appropriate based on record growth.
3. Terminal/Citrix - Look for CPU/RAM/Hard Disk issues. Check the number of users or maybe other applications on the servers are eating up the system resources.
4. SSRS - Check CPU/RAM/Hard Disk. The service restarts regularly and can slow report processing down. Here is a tool to make sure they keep running fast - those-ssrs-servers.aspx
5. Check batch processing in AX - are there new batch processes which are using a lot of resources?
6. Check infrastructure - Network, Switches, etc
7. Client issues - Are patches being applied or recently applied? Is a weekly virus scan running?
Printing – remote vs mapped
8. Is it month end or year end? The month end process involves a lot of intensive reports and processes. Can you move any of these to low activity times via batch processing?

11. Finding Bottlenecks
Here is the path I use to start and find the issue:
Is the issue being seen by one user or multiple users:
Multiple users:
Check SQL for Blocking/Locking, RAM, CPU, Disk Space and Disk utilization. Running out of disk space happens, make sure to check it. If you are seeing locking/blocking regularly I recommend two things - first, make sure that nothing is running from a SQL standpoint (Backup, Maintenance, etc). Second, use the Dynamics Performance analyzer to determine the source - Lastly, check to make sure that any users have access to the SQL server are not running expensive queries. As a general rule, no one should have ODBC access to your production database.

12. Sample report from Dynperf

13. Finding Bottlenecks
Check the AOS server(s). Look at RAM, CPU and Disk Space. You should regularly be monitoring the RAM usage of the AOS service to determine what is normal for your environment. If overloaded, an AOS restart will fix the problem but root cause analysis will be required. If you are regularly seeing Memory issues, check your buffer settings are appropriate, check to make sure that a table caching is appropriate for the number of records in the tables. It is possible the settings from go-live are no longer appropriate based on record growth.
Terminal/Citrix - Look for CPU/RAM/Hard Disk issues. Check the number of users (30) or maybe other applications on the servers are eating up the system resources.

14. Finding Bottlenecks
SSRS - Check CPU/RAM/Hard Disk. The service restarts regularly and can slow report processing down. Here is a tool to make sure they keep running fast - those-ssrs-servers.aspx
Check batch processing in AX - are there new batch processes which are using a lot of resources or any currently running?
Check infrastructure - Network, Switches, etc
Client issues - Are patches being applied or recently applied? Is a weekly virus scan running?
Is it month end or year end? The month end process involves a lot of intensive reports and processes.

15. Finding Bottlenecks Single User:
What was the user doing in AX? Did they trigger a long running process? Have they recently closed AX via CTRL-ALT-DEL and are now trying the same process they were doing before? This can cause SQL issues. If not, see if you can replicate the issue in a test environment and determine via profiling and trace parser what the issue is.
Check printer settings
Like above, check the Terminal Server, Infrastructure, Patching and AV scans.
Note: If your system AOS/SQL recently restarted, there can be a significant warm- up time to rebuild the cache. 

16. Prevention
Code can cause many issues from too many database calls, poor use of indexes, over complexity... Code needs to be performance tested. The AX trace parser and tracing cockpit are very effective at finding the issue.
microsoft-dynamics-ax-2012.aspx
SQL:
Check the server and logs everyday
Updates the statistics daily
Make sure your indexes are appropriate. Both adding missing indexes and disabling indexes you do not use.
Check max degree of parallelism is set to 1
Check growth rate of data file and logs - should be a set size, usually between 200 and 500 MB

17. Prevention Client: AOS:
Client performance settings - client-performance-options.aspx
Database logging - this causes significant overhead and every setting should be scrutinized that the loss in performance is made up with the data.
AOS:
Make sure that the AOS is periodically restarted
Make sure that the clients have the same kernel version as the AOS
Check batch processing has enough power
Check Number Sequence allocations - sequence-consumption-rates.aspx

18. Prevention Daily checks
AX Notifications/Alerts - go through all your alerts
AX Batch Processing - make sure everything that is supposed to run did and that there are no batches that should not be there.
Event logs on AOS, SQL Server and Terminal Servers - standard windows event logs will provide a good view into potential issues.
SQL Backup and Maintenance plans (more info here -
Server RAM and HD space - do you have enough.
SQL Log - check for errors or unusual entries.

19. Prevention Weekly Checks Monthly Checks
AX Number Sequences - sequences.html - if you run out of numbers, it can stop your system.
Windows/SQL patching status - are your servers up to date? What is your patching strategy not only to keep up to date, but also to make sure that no updates cripple a production system.
Monthly Checks
AX Table size - look for abnormal growth of tables and row count in tables.
SQL Database size of the AX and TempDB databases and logs. This also assumes your SQL Database is properly configured.
Clear out old alerts in AX - do your users really need an unread alert that is 6 months old? This should also be done with a review of what alerts are being generated.

20. Prevention Monthly Checks Other items
Clear out old AX Database Log entries - consider what strategy is required for keeping log entries.
Clear out old batch processing entries - do you really need to know that a batch process ended successfully 3 months ago?
Other items
Make sure you are also monitoring your WAN/LAN connections.
 SSRS and SharePoint will flush their data cache every day. This means the first user of the day will require these to reload. Use the links below to learn how to Warm these up everyday.
Warm up SSRS
Warm up SP

21. Questions – grant.wilson@microsemi.com
Discussion
Questions –