SQL Saturday San Diego #802 September 22nd 2018 Many of us became a DBA by accident. Since safety is no accident, let’s ensure that your systems are running smoothly, and send you some alerts when things look as if they’re about to become one.

First, a thank you to our sponsors We wouldn’t be able to build any of this without your support

What just happened? It’s not a matter of if, it’s a matter of when Nothing is foolproof Plan ahead By a show of hands, who here monitors? (It better be the whole f’n room) Who here does not? … Why not? Who here doesn’t care? Who here doesn’t raise hands? Source: https://www.colsafety.com/

Your foreman PATRICK PARTIN @GINGERDBA 20 years doing everything, no, really : - Web Developer, E-mail, Network, Linux, Storage, Windows Admin, and believe it or not, some hybrid DBA/Engineer/Architect. I have both graduated, and dropped out from the University of Rhode Island - Currently the Manager of Data and Systems Infrastructure at SmartDrive, a large Transportation Analytics company. This guy is totally awesome, so, be like him and monitor your stuff. He went to public schools and uses simple words, so, if he can do it, so can you. TWITTER SQL SLACK COMMUNITY @GINGERDBA @PADDYRICK

What I will cover In Windows, quick install of: Telegraf – Data Collection Agent Influx – Time-Series Database Grafana – Alerting/Visualization NSSM – Enables running as a Windows Service Creating some quick Dashboards and Charts with Grafana Creating your own Slack, and hooking it up to Grafana

Monitor what? Ev. Ery. Thing. Seriously though, monitor everything you can get your hands on. Everything. You never know when you’re going to be stuck on something stupidly difficult, having these charts and graphs will help you discover or eliminate what is going on. Not just production either. Wouldn’t you love for your developers to know what you know in production, and be able to see problems earlier on? What’s saying you can’t pull from their servers and just copy the setup so they have their own instance, and show them how to make their own graphs? The fun doesn’t stop there, because of how Influx works, they can make some simple API calls, and with instrumentation into their programs themselves, the possibilities are endless.

How can you monitor? Plenty of programs and tools available Apex SQL Monitor Idera SQL Diagnostic Monitor Red Gate SQL Monitor SentryOne SQL Monitor Solarwinds SQL Monitor Plenty of services available Too many to count, and it seems they’re multiplying Most ready to go, right out of the box Most are competitively priced If you’re not doing this, even in the slightest, give these guys a shout, they have trials, it’s simple, and they’ve got it set.

But, what if … What if you’re in a mixed environment, or polyglot? What if you’re looking at more than just databases? What if you need Custom Metrics? Business KPI? By a show of hands, who here has a normal server? environment? Yeah, didn’t think so.

What about alerting? Don’t be the Maytag Man When in doubt, send ‘em out! Email, IM/SMS/Text, Disk Space, Slack These things are cheap These things are fast Know what isn’t cheap and fast? Updating your resume Looking for a new job If a stacked bar chart trends down in the middle of a forest, and nobody is around to see it… do you get paged?

Building your own For one or many reasons, you may decide to build your own Plenty of open-source choices. Which is right for you, depends on your use case There are hundreds of open-source programs, some geared towards Networking, some Webhosting, some Databases, and so on. PS: Most, if not all of these are totally free. Push/Pullers – CollectD, Logstash, FluentD, Telegraf, StatsD Databases – Elasticsearch, Influx, Prometheus Visualization – Chronograf, Grafana, Graphite, Kibana (mostly) All-in-ones – Cacti, Nagios, Zabbix

Get and Install Influx Influx is made by Influx Data, and written in Go https://dl.influxdata.com/influxdb/releases/influxdb- 1.6.3_windows_amd64.zip Unzip to a folder of your choosing Check your Windows Firewall settings Influx defaults to Port 8086 Execute InfluxD.exe, and voila! (Just as long as you never close it) I had originally created this presentation for the San Diego SQL Server User Group in June, and Influx was at 1.5.3, As of last night, Sept 21st, they’re on 1.6.3, adding new features what seems to be every day.

Run it as a service? Assuming you’re going to use this more than once… Download/Install NSSM (Non-Sucking Service Manager) https://nssm.cc/download Copy nssm.exe from either win## to your Influx folder Create nssmInstall.bat with only the following: nssm install InfluxDB Save and run the Executable

NSSM Install (Part 2) Fill out the Application Tab Fill out the Details Tab Click Install Service

NSSM Install Create nssmStart.bat nssm start InfluxDB Create nssmStop.bat nssm stop InfluxDB Create nssmStatus.bat nssm status InfluxDB pause Create nssmEdit.bat nssm edit InfluxDB Finally, run the nssmStart.bat After all those, test and verify: Run Influx.exe from a cmd prompt/shell Check to see if you’ve got a firewall blocking 8086, or if using Windows 10 Creators Update, get the updated NSSM ver

Get and Install Telegraf Telegraf also by Influx, acts as our data collection agent https://dl.influxdata.com/telegraf/releases/telegraf- 1.8.0_windows_amd64.zip Unzip to a folder of your choosing Let’s use c:\Program Files\Telegraf\ (they prefer it there by default) Make sure you can connect to your SQL Server(s) by TCP SQL 2017 Developer has TCP/IP defaulted to off 1.7.0 was back in June, current version is 1.8.0 as of Sept 22nd They’re adding new features, inputs and outputs all the time, go git you some. Git it? LOL

Config the config Open up telegraf.conf in Notepad++, Wordpad Just not Notepad, which has issues with the line endings Interval = “10s” This is applied to everything defined within this configuration file. If changed, update flush_interval to match Logfile = “/Program Files/Telegraf/telegraf.log” Need to make this match, wherever you installed it to.

Adding SQL Server in Telegraf In the Telegraf.conf file, let’s copy/paste (but change IP, etc) # Read metrics from Microsoft SQL Server [[inputs.sqlserver]] ## Specify instances to monitor with a list of connection strings. ## By default, the host is localhost, listening on default port, TCP 1433. ## for Windows, the user is the currently running AD user (SSO). ## See https://github.com/denisenkom/go-mssqldb for detailed connection ## parameters. servers = [ "Server=localhost;Port=1433;User Id=telegraf;Password=realtoughpw-sqlsat802;app name=telegraf;log=1;", ] ## Optional parameter, setting this to 2 will use a new version ## of the collection queries that break compatibility with the original ## dashboards. query_version = 2

Run it as a service From an Admin Command Prompt, run the following: telegraf.exe --service install Test it out by running the following: telegraf.exe --config -test Did you see a wall of text fly past? If so, congrats, those counters and values would have been inserted

Adding Telegraf in to your SQL Server Now that Telegraf is configured to call our database, let’s configure our database to accept those calls. Edit/Change/Run the following: USE master; GO CREATE LOGIN [telegraf] WITH PASSWORD = N’realtoughpw-sqlsat802’; GO GRANT VIEW SERVER STATE TO [telegraf]; GO GRANT VIEW ANY DEFINITION TO [telegraf]; GO Still have that command prompt open? net start telegraf

Checking your work Go to your Influx folder, and run Influx.exe, their CLI show databases use telegraf (assuming that’s what we called it) show series (shows all the time series we’ve got) ctrl + D (quit) Checkpoint – So far, we have: Influx running as a service, taking our data Telegraf running as a service, fetching our data Next up, Grafana, our visualization engine Take a look at the show series output, and you can see exactly what we’re getting: sys.dm_db_resource_stats - sys.dm_io_virtual_file_stats - sys.dm_io_memory_clerks - sys.dm_os_performance_counters - server properties, and wait stats

Installing Grafana Download Grafana (5.2.4) http://docs.grafana.org/installation/windows/ Extract to a directory of your choosing (c:\Grafana for me) Go to Conf folder, and copy sample.ini to custom.ini In custom.ini: Remove the ; before http_port = 3000 Might want to change to another port, like 8080 Reminder: We want to work on the custom.ini, never the default or sample files. Depending on the version, Windows gets pretty fussy about using port numbers and ranges. 8080 is pretty well accepted, but certainly test it out in your environment, your mileage may vary.

Starting up Remember NSSM? Pretty much the same thing all over again http://docs.grafana.org/installation/configuration/ for more info Let’s run the Grafana-server.exe from the bin folder http://localhost:8080 User: admin Pass: admin Yeah, Admin/Admin. Reminder – You need to change this.

Add a Datasource Click the Add Data source InfluxDB Details Name your data source something meaningful Change the type to InfluxDB Use the proper URL (port 8086 is the default) InfluxDB Details Enter our database name, we used telegraf Save & Test

Create some charts Demos on Graphs Single Stats Debug/troubleshoot a graph with Query Inspector Check the underlying data Definitely lying. I’ve been using this stuff and I’m still going to find a way to do it wrong. Practice hasn’t made me perfect, but perhaps it will for you. If showing off variables, show them this one: show tag values on "telegraf-sql-prod" from "sqlserver_server_properties" with key = "sql_instance"

Helpful hints Graphs - $tag_instance in alias field, group by tag(instance) Debug/troubleshoot a graph with Query Inspector Check the underlying data Definitely lying. I’ve been using this stuff and I’m still going to find a way to do it wrong. Practice hasn’t made me perfect, but perhaps it will for you. If showing off variables, show them this one: show tag values on "telegraf-sql-prod" from "sqlserver_server_properties" with key = "sql_instance"

By you, for you, filled by you By now, hopefully you’ve got a few graphs under your belt, try to write some powershell to extract information from your SQL Server, and push it into your own measurement. If scripting isn’t your thing, there are a few community scripts: https://github.com/markwragg/PowerShell-Influx https://gallery.technet.microsoft.com/scriptcenter/InfluxDB- Write-data-3b1bf982 I’ve been using this stuff and I’m still going to find a way to do it wrong. Practice hasn’t made me perfect, but perhaps it will for you. If showing off variables, show them this one: show tag values on “your-influxdb" from "sqlserver_server_properties" with key = "sql_instance"

Adding Slack http://docs.grafana.org/alerting/notifications/#slack Go to http://www.Slack.com give an e-mail and Get Started Check your e-mail, enter in the verification code Administration > Manage Apps Build an App Incoming Webhooks Copy that Token https://api.slack.com/bot-users

Alerting / Notifications Click the Alerting Bell Configure your channel

Attaching it to Graphs Edit your graph, go the Alert tab Click Notifications tab

Recap We installed Influx to store our data, and NSSM to keep it up We installed Telegraf to pull and push Influx our data We installed Grafana and created some charts We configured Grafana to send us alerts via Slack

Free Resources / Mentioned InfluxData – Latest InfluxDB and Telegraf https://portal.influxdata.com/downloads Grafana - https://grafana.com/grafana/download?platform=windows NSSM – Non-Sucking Service Manager https://nssm.cc/download SQL Server 2017 Developer Edition (totally free as of 2017!) https://www.microsoft.com/en-ie/sql-server/sql-server-downloads !TCP/IP is disabled by default in SQL 2017 Developer! SQL Configuration Manager -> Network Config -> TCP/IP, Choose to Enable

Further Reading (offering, not pressuring) Telegraf does more than just SQL Server, LOTS more https://github.com/influxdata/telegraf https://docs.influxdata.com/telegraf/v1.8/about_the_project/release-notes- changelog/ https://github.com/influxdata/telegraf/tree/master/plugins/inputs/win_perf_cou nters - Lots of counters for AD, DNS, IIS, .NET/www sp_WhoIsActive (11.30) http://whoisactive.com/downloads/who_is_active_v11_30.zip DBATools.io (0.9.something) https://dbatools.io/download/ CentOS 7 (totally free!) https://www.centos.org/download/

Bonus / Further Reading Chronograf + Kapacitor - portal.influxdata.com/downloads Chronograf also doubles as Admin GUI, and quick dashboarding Kapacitor does the alerting, and without needing the graphs Nerd cred, I was at the very first #InfluxDays Next up: November 2018, San Francisco

Questions? If you can’t think of any, or something comes to you later, hit me up.

Thank you! Thank you all for your time this evening. I hope you enjoy making your first dashboards, and if you make something super sweet, you’ve got to let me know all about it.