1. SQL Server on Linux:The How, the What and the Why?
Patrick Flynn

2. Please Support our Sponsors
SQL Saturday is made possible with the generous support of these sponsors. You can support them by opting-in and visiting them in the sponsor area.

3. Don’t forget to silence your phone

4. Who am I Patrick Flynn MCM – SQL Server 2008 MCSM – Data Platform
MCM – SQL Server 2008
MCSM – Data Platform
Production DBA for 10+ years.

5. Agenda HOW? WHAT? WHY? Version 1.0 of SQL Server Released 1989 on OS/2
Version 4.21a in 1993 for Windows NT
Current Version v14

6. Disclosure: This Matchbox can contain Everything I know about Linux

7. Output of Query run against various Linux and Windows Instances
2 Virtual Linux Boxes
2 Azure Linux Instances
2 Docker Images
1 Windows Instance on Windows 2016

8. SQL Server on Linux: How?
Requirements
Same high quality code & security standards
Same value in terms of functionality and scale
Application compatibility between SQL Server on Windows and Linux
Features and fixes need to appear immediately across platforms
Lay a foundation for other SQL Server services

9. SQL Server on Linux: How?
(a) Pure Port to Linux
Rewrite 70+ Million Lines C++ Code
Chasing a constantly changing codebase
Multi Year Tedious Plumbing task.
No Guarantee of Success or Performance
SQL Server RDBMS and other services that ship with it in the SQL Server product suite account for more than 40 million lines of C++ code.
If you include all of products (SSIS, SSRS etc) then have mort than 70 million lines of C++ code.
Pure port could take years just to get compiling and booting let alone figuring out things like performance and feature parity with SQL Server on Windows
SQL Server innovation is happening in the same codebase and would have been a daunting task and keep the team in a close to endless catch-up game.
Getting to run only fraction of work need to deliver Product Also need to account for Lab Automation, Testing, Packaging, Servicing, Security, High Availability, Support Tools & Third Party Tools etc

10. SQL Server on Linux: How?
(b) Find a way to reuse majority of code
SQLOS
Drawbridge
Slava Oks + +
SQLOS – Created For SQL Server 2005
- Purpose to Abstract the operating system from the Rest of SQL Server Engine
- Provides user-level operating system services to rest of Server
- Built to Take advantage of Win32 Subsystem. Substituted some services but Win32 calls blead through entire engine
If just used SQLOS need to either bring Win32 awareness deep inside or re-platform all existing Win32 calls across full code base
Drawbridge – Microsoft Research project from 2011
- Initially designed as a Container Technology
- Enables ntoskrnl to run in User Mode – Called LibOS (Library OS)
- Provides the Win32 Subsystem right inside your Process
Slave Oks - Responsible for Initial Design of SQLOS
- Spent 2017 to 2014 working on MIDORI operating system
- While developing Midori utilised Drawbridge to provide Windows Programs inside Midori
- Developed Concept of combining SQLOS with Drawbridge to create SQLOSV2 or SQLPAL
- Joined Project Helsinki in early 2005
- Produced a “running version” of SQL Server on Linux on 28 February 2005

11. SQLPAL Platform Abstraction Layer
LibOS
As mentioned earlier, Drawbridge contains a complete user space implementation of the Windows system calls and the APIs needed in Windows. So Drawbridge implemented a type of user space version of the NT kernel (known as NTUM - "user mode NT"). Plus, LibOS has a library which represents the Windows APIs in the user space. Compared to the actual implementations in a normal Windows environment, both elements are kept much leaner, but they result in a run-time environment which does not behave any differently than a conventional Windows one.
SQL PAL
In combination, Drawbridge and SQLOS allow for MSSQL servers to be run in Linux. But in their implementation, they go way too far in some respects and partly solve the same problems. For this reason, the team at Microsoft decided to extract the relevant parts and use Drawbridge, LibOS and the SQLOS abstraction layer to construct a new platform abstraction layer (PAL). This one only contains the ABI and APIs relevant for SQL servers and features everything that's required in order to provide a run-time environment for MSSQL. The bottom layer consists of a Linux binary (in the user space) which initiates the rest of the environment.
To-do list for SOS and LibOS
The current status is that MSSQL is already running smoothly in Linux, but the Microsoft team is not yet finished consolidating SOS and LibOS. So for the time being, there are two versions of SOS / LibOS within the SQL server for Linux. The "upper" version forwards calls to SQL PAL, which then uses the newer version of SOS to send queries to the Linux kernel via a host bridge. Microsoft is currently working intensively on eliminating the need for these two versions and harmonising these two instances so that, in the end, the entire abstraction can be performed in SQL PAL.
The final architecture at a glance
As already mentioned, Drawbridge and SQLOS have a few overlaps which Microsoft is still working on eliminating. Overall, SQL PAL comprises the run-time environment for MSSQL and the RPM/DEB packet brings all executables, as well the necessary additional Windows libraries along with it in order to be able to run MSSQL, based on SQL PAL. According to Microsoft, around 81MB uncompressed Windows libraries will be included in the supply (that's just about 1% of a full Windows installation). The SQL PAL binary is about 8MB.
The software is installed in /opt and is managed with systemd. Microsoft supplies everything ready-to-use and with man pages included.
Generally, the process for starting MSSQL on Linux can be described as follows: the host component (standard Linux binary) first starts SQL PAL, which then initiates the MSSQL server in the "emulated" Windows environment.
The final architecture looks about like this:

12. SQLPAL Platform Abstraction Layer
SOSv2
SOS and LibOS merged
81 MB of uncompressed Windows libraries, SQL PAL itself ~8MB
The sqlserver.exe, sqlmin.dll, … are the same, exact, PE binaries we ship for Windows (our traditional box product), the first question is always:  “How can the Linux kernel understand a PE image?”  
ELF image format is the image format know to the Linux kernel.
PE image format is the image format known the Windows kernel.

13. Process View
API View
Microsoft SQL Server runs on Linux through a containerised approach called Drawbridge
Drawbridge implements a Linux loader and a minimal set of ABI calls to allow an in-process NT user mode kernel to run
SQL Server runs on top of a SQL platform layer (called SQL OS) that could be ported to run on Drawbridge
SQL Server had supportability commands added to allow the state of the system to be measured with SQL calls
A number of efficiency gains were applied to both the Drawbridge components and the SQL Server code to bring performance to within 20% of the equivalent process running on Windows

14. Overhead for transition between PE and ELF < 100 instructions
This is just Engineering needed to get running on SQL Server
All of supporting tools, documentation needs to be updated
600,000 + Unit Tests needed to be modified
Support Engineers need training
Debuging Tools needed to be upgraded – DBGBridge (1 hour presentation and Blog Posts from Bob Dorr)

15. What is SQL Server on Linux?
It is just SQL Server. Everything just works
Initially aimed at Database Engine
Currently around 95% features fully supported.
March 7th 2016 announced plans to bring SQL Server to Linux.
Private Preview with > 20,000
We are bringing the core relational database capabilities to preview today, and are targeting availability in mid-2017.
And despite the fact that it was only announced in spring 2016, you shouldn’t think of SQL Server on Linux as being a subset of the full Windows offering, says Rohan Kumar, general manager of the Microsoft database systems group.
“It’s the same SQL Server product,” he tells CIO.com. “We have one SQL Server product, and Windows and Linux are your deployment choices.”
Some features fall into Category of Partially Supported
SQL Agent – Partial Support (CMD shell support)
Some features planned but not yet available
FullText Search
Some features will never be supported
FileTable – depends on Windows

16. Supported Platforms & Features
Ubuntu 16.10, Ubuntu LTS
Red Hat Enterprise Linux 7.3 Workstation, Server, and Desktop and earlier
SUSE Enterprise Linux Server v12 SP2
Docker Engine 1.8 and later
Features
Core Database Engine
Including CLR, In-Memory OLTP, Columnstore Index, Query Store, DMV, Security and Encryption features
Tooling
Administration: SQLCMD, SSMS, mssql-conf, Powershell
Development: SSMS, Visual Studio Code, SSDT, Connectivity libraries (C#, Java, PHP, Node.js, Python, Ruby, C++)
It is just SQL Server
Includes 2016 SP1 Improvements
TRIM
CONCAT_WS
STRING_AGG
TRANSLATE
sys.dm_exec_query_statistics_xml (with TF 7412)

17. Unsupported Features (CTP 2.0)
Database Engine
High Availability
Security
Services
Replication
Database Mirroring
Active Directory Authentication
SQL Server Browser
FileTable
Windows Authentication
SQL Server R Services
Stretch DB
Extensible Key Management
SQL Server Python Services
Polybase
Use of user-provided certificate for SSL or TLS
StreamInsight
Distributed Query
Analysis Services
CLR assemblies with the EXTERNAL_ACCESS or UNSAFE permission set
Reporting Services
System extended stored procedures (XP_CMDSHELL, etc.)
Integration Services
Data Quality Services
Master Data Services
Dedicated Administrator connection is not enabled
Cannot start in Single-User mode (Can’t restore
SQL Server Database Engine (CTP 1.3)
Indirect checkpoint performance improvements.
Cluster-less Availability Groups support added.
Minimum Replica Commit Availability Groups setting added.
Availability Groups can now work across Windows-Linux to enable cross-OS migrations and testing.
Temporal Tables Retention Policy support added,
New DMV SYS.DM_DB_STATS_HISTOGRAM
Online non-clustered columnstore index buill and rebuild support added
5 new dynamic management views to return information about Linux process. For more information, see Linux Process Dynamic

18. Installing SQL Server on Linux
No Separate Installation Media or Program
Software Install uses the Linux Package Management System
In Linux distributions, a package refers to a compressed file archive containing all of the files that come with a particular application. The files are usually stored in the package according to their relative installation paths on your system. Most packages also contain installation instructions for the OS, as well as a list of any other packages that are dependencies, prerequisites required for installation.
Common types of Linux packages include .deb, .rpm, and .tgz. Since Linux packages do not usually contain the dependencies necessary to install them, many Linux distributions use package managers that automatically read dependencies files and download the packages needed before proceeding with the installation. Some examples of package managers are APT, YUM, and zipper
Packages are downloaded into the local repository and then installed using the Package Manager.
Full instructions at :
No Internet then manually download from :
Packages for Ubuntu, RHEL and SLES

19. Can also install Linux version of:
SQLCMD
BCP
Similar process to installing Server

20. Post Install Configuration and Mangement
sudo /opt/mssql/bin/sqlservr-setup
Run Configuration Package
#> sudo /opt/mssql/bin/mssql-conf setup
Modify Configuration
mssql-conf (Native Command Line Tool)
SSMS
Powershell
You can use this utility to set the following parameters:+
TCP port: Change the port where SQL Server will listen for connections.
Default data directory: Change the directory where the new SQL Server database data files (.mdf).
Default log directory: Changes the directory where the new SQL Server database log (.ldf) files are created.
Default dump directory: Change the directory where SQL Server will deposit the memory dumps and other troubleshooting files by default.
Default backup directory: Change the directory where SQL Server will send the backup files by default.
Set traceflags: Set the traceflags that the service is going to use.
Set collation: Set a new collation for SQL Server on Linux. +

21. Upgrade from CTP 1.2 Azure Red Hat 1.2 Azure Red Hat 2.0
sudo yum update mssql-server
sudo yum install mssql-server-agent
sudo systemctl restart mssql-server
Completed in < 5 minutes

22. SQL Server on Linux HA Options:
Clustering
Linux to Linux
Availability Groups
Can be Linux to Linux
Can be Windows to Linux
Backup and Restore
Databases can be backed up on Windows and restored on Linux
Databases can be backed up on Linux and restored on Windows
Log Shipping
Released with CTP 1.3

23. SQL Server on Linux HA Failover Cluster Instance on RHEL 7.3
based on Red Hat Enterprise Linux (RHEL) HA add-on
Uses Corosync and Pacemaker
Log shipping can be set-up using CronTab and CronJobs

24. SQL Server on Linux HA
Mission-critical HADR with Always On Availability Groups
Run mission-critical application using SQL Server running on Linux
Log shipping can be set-up using CronTab and CronJobs

25. SQL Server on Linux HA
Mission-critical HADR with Always On Availability Groups
Workload load balancing for increased scale and performance
Log shipping can be set-up using CronTab and CronJobs

26. SQL Server on Linux HA
Mission-critical HADR with Always On Availability Groups
Seamless cross-platform migration
Log shipping can be set-up using CronTab and CronJobs

27. Performance: TPC-H

29. Bringing SQL Server to Linux is one of the bigger steps in Microsoft’s plan to help developers build any app for (and from) any platform.
SQL Server on Linux will provide customers with even more flexibility in their data solution. One with mission-critical performance, industry-leading TCO, best-in-class security, and hybrid cloud innovations – like Stretch Database which lets customers access their data on-premises and in the cloud whenever they want at low cost – all built in.
Offer its well-known and trusted database to an expanded set of customers.
By taking this key product to Linux Microsoft is proving its commitment to being a cross platform solution provider. This gives customers choice and reduces the concerns for lock-in.
“SQL Server’s proven enterprise experience and capabilities offer a valuable asset to enterprise Linux customers around the world,” said Paul Cormier, President, Products and Technologies, Red Hat. “We believe our customers will welcome this news and are happy to see Microsoft further increasing its investment in Linux. As we build upon our deep hybrid cloud partnership, spanning not only Linux, but also middleware, and PaaS, we’re excited to now extend that collaboration to SQL Server on Red Hat Enterprise Linux, bringing enterprise customers increased database choice.”

30. Why? Meet Customer Requests and Provide Flexibility
Expand the Potential Customer Base
Deliver a consistent data platform across Windows Server and Linux, as well as on-premises and cloud.
Docker
25% of VM’s in Azure
Most new development in Linux
Project NAMI – Wordpress running on SQL Server
Conversion from Oracle on same platform

32. Thank You! Questions? Linux Images Azure – RedHat 7.2 - Suse 12 SP2
OSBoxes.org

33. Links and Other Resources
Microsoft Documentation:
cing-sql-server-on-linux/
server-getting-started-with-ms-sql-server-on-ubuntu.aspx
server-on-linux-how-introduction/
aka-project-helsinki-story-behind-the-idea/
and-pe-images-just-work/

34. Links and Other Resources
Linux OS Links (Linux OS Fundamentals for the SQL Admin)
be&utm_content=buffer76a68&utm_medium=social&utm_source=t witter.com&utm_campaign=buffer
Installation Videos
(Ubuntu)
(Docker)
Drawbridge
old-drawbridge-helped-microsoft-bring-sql-server-to-linux/