1. SQL Server on Linux Troubleshooting tips and tricks
Suresh Kandoth
SQL Server on Linux Troubleshooting tips and tricks

2. What exactly are you troubleshooting?

3. diagnostic info & basic logs
Identify installed software
SQL Server packages -> yum info mssql-server / dpkg -l mssql-server
select
systemctl status mssql-server
SQL Server logs
default location -> /var/opt/mssql/log
Database engine -> errorlog
Database engine ->
default trace [log*.trc] & default system session [system_health*.xel]
SQL Agent -> sqlagent.out
Operating System logs
centralized system logging -> journalctl
kernel messages -> dmesg
application or daemon logs -> syslog / messages
Other key logs-> /var/logs

4. permissions & accounts
binary
/opt/mssql/
ownership = root
fixed path
data
/var/opt/mssql/
ownership = mssql
customizable
config
service daemon
process startup context
mssql config file

5. troubleshooting file activity
client application, sql server error log, journalctl
error messages
specific process, specific path, typical syscalls
tracing errors - strace
writes & reads
normal activity - strace

6. troubleshooting memory consumption
free / sar
swap file
OOM Killer settings
operating system
pidstat
mssql-conf [memory.memorylimitmb]
multiple components inside single process
Process
DMV [dm_os_process_memory, dm_os_memory_nodes, dm_os_memory_clerks]
DBCC MEMORYSTATUS
SSMS Memory consumption report
Extended Event [page_allocated and page_freed]
sp_configure ‘max server memory’,
memory grants [RG/hints]
database engine

7. Performance monitoring
OS diagnostics
overview of system (top / vmstat / uptime / pmstat); process view (pidstat)
cpu (mpstat); memory (free / sar –r); disk (iostat / iotop); network (sar –n)
long term collection: sar or pcp
collectd/InfluxDB/Grafana [mssql-monitoring]
telegraph [cross platform monitoring agents for SQL Server]
SQL diagnostics
Activity monitor [SSMS]
Performance dashboard [SSMS]
Dashboard insights [sqlops]
Extended Events

8. db engine performance troubleshooting
Query Store
runtime statistics
plans used at different times
wait categories
DMV’s
queries [sys.dm_exec_requests & sys.dm_exec_query_stats]
waits [sys.dm_os_waiting_tasks & sys.dm_os_wait_stats]
PSSDIAG and SQL Nexus utility
unified data collection
Linux collector on GitHub
import data into database for analysis
sys.dm_os_performance_counters can be used to query performance counters of db engine

9. deeper investigations
sqldumper.exe
exceptions in db engine
dbcc stackdump
dumptrigger / -y / create_dump event
Memory dumps
gdb or paldumper
exceptions in sql pal
generate-sql-dump.sh <pid> <target_dir>
Core dumps
Track activity of SQLPAL components
/var/opt/mssql/logger.ini
Trace written to log file
PAL logger

10. Availability Groups pcs cluster status pcs cluster start / stop
cluster management
pcs resource show --full
pcs status / crm_mon
pcs resource update
resource management
pcs resource move
pcs resource clear
pcs resource unmanage
failover resources

11. high availability logs
/var/log/cluster/ or /var/log/corosync/
corosync.log
/var/opt/mssql/log/
errorlog
var/opt/mssql/log/
AlwaysOn_health*.xel

12. /var/log/sssd KRB5_TRACE
AD authentication
SPN configuration
(setspn)
keytab file
(ktutil)
kerberos
(id, kinit, klist)
networking issues
SSSD+NSS
/var/log/sssd KRB5_TRACE

13. Thank you!

14. PSSDIAG for SQL Server on Linux
shell script based
unified data collection
SQL Server instances
host instance
container instances
OS distributions
RHEL, Ubuntu & SLES
customize & extend
analyze data using same tools [SQL Nexus]
framework
performance
static logs
memory dumps
scenarios
OS data points
performance data using sysstat
configuration information
common OS logs
SQL data points
perfstats scripts
DMV’s & catalog views
SQL performance counters
extended events
SQL log files
data collection
start_collector.sh
stop_collector.sh
scenario config [*.scn files]
control scripts

15. Live Monitoring : mssql-monitoring

16. PAL logging
Content of /var/opt/mssql/logger.ini to trace libos activity
Content of /var/opt/mssql/logger.ini to trace kerberos activity
[Output.FileOutput] type=File filename=/var/opt/mssql/log/palstart_$(_pid).log [Logger] level=silent [Logger.libos.trace] level=debug outputs=FileOutput
[Output.FileOutput]
type=File
filename=/var/opt/mssql/log/palkerb_$(_pid).log
[Logger]
level=silent
[Logger.security.kerberos]
level=debug
outputs=FileOutput
trace output is captured in /var/opt/mssql/log/pal*.log; don’t leave the trace running, can get very big