1. SQL Saturday San Diego #802
September 22nd 2018
Many of us became a DBA by accident.
Since safety is no accident, let’s ensure
that your systems are running smoothly,
and send you some alerts when things
look as if they’re about to become one.

2. First, a thank you to our sponsors
We wouldn’t be able to build any of this without your support

3. What just happened? It’s not a matter of if, it’s a matter of when
Nothing is foolproof
Plan ahead
By a show of hands, who here monitors? (It better be the whole f’n room) Who here does not? … Why not?
Who here doesn’t care? Who here doesn’t raise hands?
Source:

4. Your foreman PATRICK PARTIN @GINGERDBA
20 years doing everything, no, really :
- Web Developer, , Network, Linux, Storage, Windows Admin, and believe it or not, some hybrid DBA/Engineer/Architect.
I have both graduated, and dropped out from the University of Rhode Island
- Currently the Manager of Data and Systems Infrastructure at SmartDrive, a large Transportation Analytics company.
This guy is totally awesome, so, be like him and monitor your stuff.
He went to public schools and uses simple words, so, if he can do it, so can you.
TWITTER
SQL SLACK
COMMUNITY
@GINGERDBA
@PADDYRICK

5. What I will cover In Windows, quick install of:
Telegraf – Data Collection Agent
Influx – Time-Series Database
Grafana – Alerting/Visualization
NSSM – Enables running as a Windows Service
Creating some quick Dashboards and Charts with Grafana
Creating your own Slack, and hooking it up to Grafana

6. Monitor what? Ev. Ery. Thing.
Seriously though, monitor everything you can get your hands on. Everything.
You never know when you’re going to be stuck on something stupidly difficult, having these charts and graphs will help you discover or eliminate what is going on.
Not just production either. Wouldn’t you love for your developers to know what you know in production, and be able to see problems earlier on? What’s saying you can’t pull from their servers and just copy the setup so they have their own instance, and show them how to make their own graphs? The fun doesn’t stop there, because of how Influx works, they can make some simple API calls, and with instrumentation into their programs themselves, the possibilities are endless.

7. How can you monitor? Plenty of programs and tools available
Apex SQL Monitor
Idera SQL Diagnostic Monitor
Red Gate SQL Monitor
SentryOne SQL Monitor
Solarwinds SQL Monitor
Plenty of services available
Too many to count, and it seems they’re multiplying
Most ready to go, right out of the box
Most are competitively priced
If you’re not doing this, even in the slightest, give these guys a shout, they have trials, it’s simple, and they’ve got it set.

8. But, what if … What if you’re in a mixed environment, or polyglot?
What if you’re looking at more than just databases?
What if you need Custom Metrics? Business KPI?
By a show of hands, who here has a normal server? environment?
Yeah, didn’t think so.

9. What about alerting? Don’t be the Maytag Man
When in doubt, send ‘em out!
, IM/SMS/Text, Disk Space, Slack
These things are cheap
These things are fast
Know what isn’t cheap and fast?
Updating your resume
Looking for a new job
If a stacked bar chart trends down in the middle of a forest, and nobody is around to see it… do you get paged?

10. Building your own
For one or many reasons, you may decide to build your own
Plenty of open-source choices.
Which is right for you, depends on your use case
There are hundreds of open-source programs, some geared towards Networking, some Webhosting, some Databases, and so on.
PS: Most, if not all of these are totally free.
Push/Pullers – CollectD, Logstash, FluentD, Telegraf, StatsD
Databases – Elasticsearch, Influx, Prometheus
Visualization – Chronograf, Grafana, Graphite, Kibana
(mostly) All-in-ones – Cacti, Nagios, Zabbix

11. Get and Install Influx
Influx is made by Influx Data, and written in Go
_windows_amd64.zip
Unzip to a folder of your choosing
Check your Windows Firewall settings
Influx defaults to Port 8086
Execute InfluxD.exe, and voila! (Just as long as you never close it)
I had originally created this presentation for the San Diego SQL Server User Group in June, and Influx was at 1.5.3,
As of last night, Sept 21st, they’re on 1.6.3, adding new features what seems to be every day.

12. Run it as a service? Assuming you’re going to use this more than once…
Download/Install NSSM (Non-Sucking Service Manager)
Copy nssm.exe from either win## to your Influx folder
Create nssmInstall.bat with only the following: nssm install InfluxDB
Save and run the Executable

13. NSSM Install (Part 2) Fill out the Application Tab
Fill out the Details Tab
Click Install Service

14. NSSM Install Create nssmStart.bat nssm start InfluxDB
Create nssmStop.bat nssm stop InfluxDB
Create nssmStatus.bat nssm status InfluxDB pause
Create nssmEdit.bat nssm edit InfluxDB
Finally, run the nssmStart.bat
After all those, test and verify:
Run Influx.exe from a cmd prompt/shell
Check to see if you’ve got a firewall blocking 8086, or if
using Windows 10 Creators Update, get the updated NSSM ver

15. Get and Install Telegraf
Telegraf also by Influx, acts as our data collection agent
_windows_amd64.zip
Unzip to a folder of your choosing
Let’s use c:\Program Files\Telegraf\ (they prefer it there by default)
Make sure you can connect to your SQL Server(s) by TCP
SQL 2017 Developer has TCP/IP defaulted to off
1.7.0 was back in June, current version is as of Sept 22nd
They’re adding new features, inputs and outputs all the time, go git you some.
Git it?
LOL

16. Config the config Open up telegraf.conf in Notepad++, Wordpad
Just not Notepad, which has issues with the line endings
Interval = “10s”
This is applied to everything defined within this configuration file.
If changed, update flush_interval to match
Logfile = “/Program Files/Telegraf/telegraf.log”
Need to make this match, wherever you installed it to.

17. Adding SQL Server in Telegraf
In the Telegraf.conf file, let’s copy/paste (but change IP, etc)
# Read metrics from Microsoft SQL Server [[inputs.sqlserver]] ## Specify instances to monitor with a list of connection strings. ## By default, the host is localhost, listening on default port, TCP ## for Windows, the user is the currently running AD user (SSO). ## See for detailed connection ## parameters. servers = [ "Server=localhost;Port=1433;User Id=telegraf;Password=realtoughpw-sqlsat802;app name=telegraf;log=1;", ] ## Optional parameter, setting this to 2 will use a new version ## of the collection queries that break compatibility with the original ## dashboards. query_version = 2

18. Run it as a service
From an Admin Command Prompt, run the following: telegraf.exe --service install
Test it out by running the following: telegraf.exe --config -test
Did you see a wall of text fly past?
If so, congrats, those counters and values would have been inserted

19. Adding Telegraf in to your SQL Server
Now that Telegraf is configured to call our database, let’s configure our database to accept those calls.
Edit/Change/Run the following: USE master; GO CREATE LOGIN [telegraf] WITH PASSWORD = N’realtoughpw-sqlsat802’; GO GRANT VIEW SERVER STATE TO [telegraf]; GO GRANT VIEW ANY DEFINITION TO [telegraf]; GO
Still have that command prompt open?
net start telegraf

20. Checking your work
Go to your Influx folder, and run Influx.exe, their CLI show databases use telegraf (assuming that’s what we called it) show series (shows all the time series we’ve got) ctrl + D (quit)
Checkpoint – So far, we have:
Influx running as a service, taking our data
Telegraf running as a service, fetching our data
Next up, Grafana, our visualization engine
Take a look at the show series output, and you can see exactly what we’re getting:
sys.dm_db_resource_stats - sys.dm_io_virtual_file_stats - sys.dm_io_memory_clerks - sys.dm_os_performance_counters - server properties, and wait stats

21. Installing Grafana
Download Grafana (5.2.4)
Extract to a directory of your choosing (c:\Grafana for me)
Go to Conf folder, and copy sample.ini to custom.ini
In custom.ini:
Remove the ; before http_port = 3000
Might want to change to another port, like 8080
Reminder: We want to work on the custom.ini, never the default or sample files. Depending on the version, Windows gets pretty fussy about using port numbers and ranges is pretty well accepted, but certainly test it out in your environment, your mileage may vary.

22. Starting up Remember NSSM?
Pretty much the same thing all over again
for more info
Let’s run the Grafana-server.exe from the bin folder
User: admin
Pass: admin
Yeah, Admin/Admin. Reminder – You need to change this.

23. Add a Datasource Click the Add Data source InfluxDB Details
Name your data source something meaningful
Change the type to InfluxDB
Use the proper URL (port 8086 is the default)
InfluxDB Details
Enter our database name, we used telegraf
Save & Test

24. Create some charts Demos on
Graphs
Single Stats
Debug/troubleshoot a graph with Query Inspector
Check the underlying data
Definitely lying. I’ve been using this stuff and I’m still going to find a way to do it wrong. Practice hasn’t made me perfect, but perhaps it will for you. If showing off variables, show them this one: show tag values on "telegraf-sql-prod" from "sqlserver_server_properties" with key = "sql_instance"

25. Helpful hints
Graphs
- $tag_instance in alias field, group by tag(instance)
Debug/troubleshoot a graph with Query Inspector
Check the underlying data
Definitely lying. I’ve been using this stuff and I’m still going to find a way to do it wrong. Practice hasn’t made me perfect, but perhaps it will for you. If showing off variables, show them this one: show tag values on "telegraf-sql-prod" from "sqlserver_server_properties" with key = "sql_instance"

26. By you, for you, filled by you
By now, hopefully you’ve got a few graphs under your belt, try to write some powershell to extract information from your SQL Server, and push it into your own measurement.
If scripting isn’t your thing, there are a few community scripts:
Write-data-3b1bf982
I’ve been using this stuff and I’m still going to find a way to do it wrong. Practice hasn’t made me perfect, but perhaps it will for you. If showing off variables, show them this one: show tag values on “your-influxdb" from "sqlserver_server_properties" with key = "sql_instance"

27. Adding Slack http://docs.grafana.org/alerting/notifications/#slack
Go to give an and Get Started
Check your , enter in the verification code
Administration > Manage Apps
Build an App
Incoming Webhooks
Copy that Token

28. Alerting / Notifications
Click the Alerting Bell
Configure your channel

29. Attaching it to Graphs Edit your graph, go the Alert tab
Click Notifications tab

30. Recap We installed Influx to store our data, and NSSM to keep it up
We installed Telegraf to pull and push Influx our data
We installed Grafana and created some charts
We configured Grafana to send us alerts via Slack

31. Free Resources / Mentioned
InfluxData – Latest InfluxDB and Telegraf
Grafana -
NSSM – Non-Sucking Service Manager
SQL Server 2017 Developer Edition (totally free as of 2017!)
!TCP/IP is disabled by default in SQL 2017 Developer!
SQL Configuration Manager -> Network Config -> TCP/IP, Choose to Enable

32. Further Reading (offering, not pressuring)
Telegraf does more than just SQL Server, LOTS more
changelog/
nters - Lots of counters for AD, DNS, IIS, .NET/www
sp_WhoIsActive (11.30)
DBATools.io (0.9.something)
CentOS 7 (totally free!)

33. Bonus / Further Reading
Chronograf + Kapacitor - portal.influxdata.com/downloads
Chronograf also doubles as Admin GUI, and quick dashboarding
Kapacitor does the alerting, and without needing the graphs
Nerd cred, I was at the very first #InfluxDays
Next up: November 2018, San Francisco

34. Questions?
If you can’t think of any, or something comes to you later, hit me up.

35. Thank you!
Thank you all for your time this evening. I hope you enjoy making your first dashboards, and if you make something super sweet, you’ve got to let me know all about it.