Always Encrypted in the Real World Daniel de Sousa Always Encrypted in the Real World

About Me Daniel de Sousa Business Intelligence Specialist @ large pizza chain Currently working on data compliance project between multi countries and continents Been working with data since 1994(dBase III Plus, all SQL servers) Currently using SQL 2016 SP1  https://twitter.com/dcasousa Currently living in Brisbane, Australia https://www.linkedin.com/in/daniel-de-sousa-0b419828/

Agenda Scenario Challenges Solution Technology used Limitations Demo/Feature use

Scenario

Scenario Large Global Pizza Chain Headquarters: Franchises: Australia Belgium Germany France Netherlands Japan New Zealand

Scenario Process: Stores provide daily data to HQ Data is received, grouped and processed (in AU) Data is loaded into datawarehouse (in AU) Franchisees can access data(Cube/Report/Dashboard) Global Reports are generated

Challenges

Challenges Although they have same system, it is used differently between countries Data coding(varchar x nvarchar) Reports built based on AU data Any delay/failure in AU process, delays more data access anywhere Europen data(pii) cannot leave EU without encryption

Solution

Solution Datawarehouse stays in AU Azure VM on regions SQL 2016 SP1 Azure functions Azure KeyVault Always Encrypted Encryption

Technology used

Technology used Azure VM on regions 4x VMs in Western Europe SQL 2016 SP1 SSIS 2016 SSRS 2016 SSAS 2016 3x VMs in Asia

Technology used Azure Functions Functions were created to trigger the data from the store at the end of the day as well as to consume the data received Staged the received data(more for control)

Technology used Azure KeyVault The security guys create a secret for each “type” of application Avoid inter-dependence and total failure in case of small failures Easier to rotate in case a secret is exposed Easier to rotate on a time basis

Technology used Always Encrypted SQL Server 2016 onwards All editions(from SP1) Client Side Encryption – ADO .Net Requires .Net 4.6 driver

Technology used Always Encrypted The Certificate protects the key and is needed for encryption/decryption The Certificate is needed on the client, not the server Certificates can be created by SQL Server Makecert New-SelfSignedCertificate Certificate Store – Current User Certificate Store – Local computer Azure Key Vault Key Store Provider (CNG) 1) indicates the Current User certificate store location in the Windows Certificate Store, which is your personal store. 2) indicates the Local computer certificate store location in the Windows Certificate Store. 3) you will need to sign in to Azure (click Sign in). Once you sign in, you will be able to pick one of your Azure subscriptions and a key vault. 4) indicates a key store that is accessible via a key store provider (KSP) that implements the Cryptography Next Generation (CNG) API. Typically, this type of a store is a hardware security module (HSM). After you select this option, you will need to pick a KSP. Microsoft Software Key Store Provider is selected by default. If you want to use a column master key stored in an HSM, select a KSP for your device (it must be installed and configured on the computer before you open the dialog). 5) a key store that is accessible via a cryptographic service provider (CSP) that implements the Cryptography API (CAPI). Typically, such a store is a hardware security module (HSM). After you select this option, you will need to pick a CSP. If you want to use a column master key stored in an HSM, select a CSP for your device (it must be installed and configured on the computer before you open the dialog).

Technology used Always Encrypted Column Master Key(CMK) Column Encryption Key(CEK) Encrypt Column (AEAD_AES_256_CBC_HMAC_SHA_256) Drivers(.NET or ODBC 13.1) Connection String CMK Protects column encryption keys.Must be stored in a trusted key store.Stored in the database in system catalog views. CEK Encrypt sensitive data stored in database columns.Column can be encrypted using a single column encryption key.Encrypted values of column encryption keys are stored in the database in system catalog views.Backup column encryption keys in a secure/trusted location CMK protects the CEK and it’s stored in trusted key stores such as Windows Certificate Store(WCS), Azure keyVault, Hardware Security Modules or even custom key store if necessary. IMPORTANT: In a production environment, the certificate management must be done with separate, secured user accounts and, preferably, on separate servers. CEK protects the encrypted columns and it’s stored encrypted in the database. Data in the db is always stored encrypted using AEAD_AES_256_CBC_HMAC_SHA_256. All decryption is done by client application which requires AE-enabled client driver. Deterministic: same encrypted value for any unencrypted value which allows creation of indexes, point-lookup searches, equality joins and grouping. Only one that can be indexed. Randomized: it generates a new value for each encryption CONNECTION STRING: Driver={ODBC Driver 13 for SQL Server};server=localhost;trusted_connection=Yes;database=DIME;columnencryption=Enabled;keystoreauthentication=KeyVaultClientSecret;keystoreprincipalid=XXXxxxXXXxxxXXXxxx;keystoresecret=fxfxfxfffkhskdjfskdfgskdfhskdjvsdf;sodfhskudgfksdj

Technology used Always Encrypted SSMS(Advanced SSMS SSMS 17 (config/parameterization) Column Encryption Setting = Enabled SSMS 16.3 To enable/disable Parameterization for Always Encrypted for the current Query Editor window: + Select Query from the main menu. Select Query Options…. Navigate to Execution > Advanced. Select or unselect Enable Parameterization for Always Encrypted. Click OK. + To enable/disable Parameterization for Always Encrypted for future Query Editor windows: + Select Tools from the main menu. Select Options…. Navigate to Query Execution > SQL Server > Advanced. SSMS 17 This allows the underlying .NET Framework Data Provider for SQL Server to detect data targeting encrypted columns, and to encrypt such data before sending it to the database Without parameterization, the .NET Framework Data Provider passes each statement, you author in the Query Editor, as a non-parameterized query. If the query contains literals or Transact-SQL variables that target encrypted columns, the .NET Framework Data Provider for SQL Server will not be able to detect and encrypt them, before sending the query to the

Technology used

Technology used

Technology Used Encryption Text Data is varbinary Create a number to represent(Encrypt) Processed in Australia Flow back to Region Map number back(Decrypt)

Limitations

Limitations Deterministic Encryption requires _BIN2 collation. A CEK can have two encrypted values (for key rotation) Queries can only perform operations on deterministic encryption Only the equals (=) operation is allowed in queries. (No >, <, <>, like, etc.) Queries must pass values as parameters, not literals. Limited data types Key columns in indexes only allow deterministic encryption NO CDC NO FOR XML or FOR JSON PATH During Encryption process, there can be no inserts or updates as it may result in data corruption And More can be checked at Microsoft Docs(reference)

Demo A look at the feature Create CMK Create CEK Create table Encrypt column Create table with Encrypted column Insert data Select data without setting Select data with setting Select data with CLAUSE

References Always Encrypted (Database Engine) https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-database-engine Always Encrypted https://msdn.microsoft.com/en-us/library/mt163865.aspx Column Encryption Metadata https://blogs.msdn.microsoft.com/sqlsecurity/2015/07/06/always-encrypted-key-metadata/

Questions?

THANK YOU