With Temporal Tables and More Tracking Data Changes With Temporal Tables and More

What is this presentation about? Introduction What is this presentation about? The how's and whys of tracking changes to the data in your database using SQL Server change tracking technologies available in SQL 2016 Emphasis on temporal tables and as time permits Change Data Capture Change Tracking Triggers

Tracking data changes Overview Possible Whys Audit trail - who changed what when Analysis - maintaining a data warehouse Time travel - reconstructing state of the data at a point in time in the past Data sync – one and two way sync’ing Repairing row-level data - recovering from accidental data changes and application errors

Tracking data changes overview Things to consider when preparing to track changes Identifying what to track and what not to track Costs of tracking / not tracking Legal requirements New vs legacy systems What will your hardware / software support

Tracking data changes overview Dealing with date and time Time zones and UTC Format of Assertion(History) vs version (Validity) vs bi-temporal Does the date/time represent when the change occurred or when the information in the row was valid. Who or what is the date/time relative to?

Temporal Tables Part of the ANSI SQL 2011 standard. First introduced with SQL Server 2016 Designed to keep a full history of data changes at the row level using a ‘current’ table and a history table System-versioned because the start and end dates for each row is managed by the system Assertion / historical in nature meaning that the lifetime of a is record based on the physical dates the record was changed

Temporal Tables Good for Not so good for Audit trail Point in time analysis Maintaining a data warehouse Repairing row-level data Not so good for Data synchronization Short term data history requirements Column level change tracking Tables that have frequent schema changes

Requirements for the base table Temporal tables - demo Requirements for the base table Must have two explicitly defined datetime2 non-nullable columns (period columns) Used exclusively by the system to record ‘life-span’ of a row Names of columns can be customized  Date/time stored as coordinated universal time (UTC) Can be hidden Must have a PERIOD FOR SYSTEM_TIME statement.  Identifies the period columns Must have a primary key INSTEAD OF triggers are not allowed on the base table. AFTER triggers allowed. Foreign keys with cascade update or delete are not allowed Cannot have any computed columns based on a user defined function

Requirements for the history table Temporal tables - demo Requirements for the history table Must have the same schema as base table No primary key, foreign keys, unique indexes, table constraints or triggers. Cannot be configured for change data capture, change tracking, transactional or merge replication By default, a clustered index created using period columns Can be in a different schema than the base table but the same database Data cannot be modified.  This includes INSERT, UPDATE, DELETE and TRUNCATE

Change data capture First introduced in SQL 2008, Change Data Capture (CDC) captures DML changes on a tracked table. Enterprise only unless you have SQL 2016 SP1 Uses SQL Server Agent jobs to asynchronously read the transaction log to track and record DML transactions When enabling CDC on a table, an additional table is created containing the same columns as tracked table along with metadata needed to understand the changes Table-valued functions are also created to allow access the change history CDC was written by the SQL Server Replication team so it works with transaction replication

Not so good for Good for Change data capture Maintaining a data warehouse Short term audit trail Data synchronization (using net changes) Not so good for Long term storage of change history Point in time analysis Solutions that require real time data capture

How it works Change data capture SQL Server Agent jobs One used to populate the change tables One responsible for change table cleanup   SQL Server Agent must be running for data capture and clean up to work  Transaction logs SQL Server Agent jobs need to be running for the transaction log to get truncated Change data is not available until after change is committed to source table and capture job has processed the related log entries

Change Data Capture - demo Validity interval Changes captured by CDC has a finite lifespan called the validity interval. Important because the extraction interval for a request must be fully covered by this time period. Default is 3 days measured in mins Metadata __$start_lsn identifies the commit log sequence number (LSN) that was assigned to the change. __$end_lsn not supported. Future compatibility not guaranteed. Column always NULL (SQL 2012) __$seqval used to order changes that occur in the same transaction __$operation records the operation associated with the change: 1 = delete, 2 = insert, 3 = update (before), and 4 = update (after) __$update_mask variable bit mask with one defined bit for each captured column. Insert and delete - all bits set. Update - bits set correspond to changed columns

Change Data Capture - demo Requirements Does not play nice with memory-optimized tables SQL Server Agent must be running Net changes: source table has a primary key or an unique index referenced by index_name when setting up CDC on the table.

Change tracking A lightweight solution first introduced in SQL 2008 that provides an efficient mechanism to identify records in a user table that have changed on account of DML transactions Tracks the fact that a row has changed but not how many times the row has changed or the values of any intermediate changes. Changes are tracked as they occur Clean up of internal tables used to keep track of what records have changed are managed by the system Built-in functions are available to easily query to data changes

Change tracking How it works Once enabled, any DML statement will cause change tracking information for each modified row to be recorded. Each table enabled for Change Tracking has an internal on-disk table created on the same filegroup.  This table is used to record table change versions and the rows that have changed since a particular version. When a change occurs the row’s primary key, list of impacted columns (optional) and DML command are recorded in the internal tracking table. An auto cleanup thread purges expired content of the internal tracking tables based on the retention period setting of the database.

Change tracking Good for Not so good for Data synchronization Any solution that requires that each data change is tracked and recorded.

Change tracking - demo Requirements The database compatibility level must be set to 90 (SQL 2005) or greater. Snapshot isolation is strongly recommended to help ensure that all change tracking information is consistent. Tables being tracking must have a primary key.   DDL operations are not tracked Deletes caused by a TRUNCATE statement are not tracked. However, this will cause the minimum valid version to be updated. Client applications would then need re-initialize their data.

Triggers A special kind of stored procedure attached to an event.  When the event happens, SQL Server executes the trigger’s code 4 types: DML, DDL, CLR, Logon INSTEAD OF and AFTER DML triggers in SQL Server work at the statement-level. This means that one trigger instance fires per statement regardless of how many rows were affected, even zero Triggers extend the life of a transaction. While a trigger is at work, the associated transaction will not commit until the trigger completes Due to their event driven nature, issues caused by triggers can be difficult to troubleshoot

Triggers Good for Not so good for Auditing One audit table per user table or one audit table for all user tables Anything if you want to put the work into building the solution Any environment that is SQL Server 2005 or earlier Not so good for Scenarios where simplicity is required or other change tracking technologies would do the job just as well or better

Triggers - demo

Conclusion The tools available natively in Microsoft SQL Server to track data changes have gotten better and better over time. With SQL 2016 SP1 most if not all these tools are available to everyone with a SQL standard license or better. There is really no reason why you should be suffering with a system that does a poor job of tracking changes in your DB. In addition to the tools that are offered natively in SQL Server there are also 3rd party venders that provide off the shelf solutions for tracking data changes. For example, one of the sponsors of today’s event ApexSQL has a product called ApexSQL Audit that tracks and reports events on SQL Server by auditing access and changes to the SQL Server instance and its objects.