1. With Temporal Tables and More
Tracking Data Changes
With Temporal Tables and More

2. What is this presentation about?
Introduction
What is this presentation about?
The how's and whys of tracking changes to the data in your database using SQL Server change tracking technologies available in SQL 2016
Emphasis on temporal tables and as time permits
Change Data Capture
Change Tracking
Triggers

3. Tracking data changes Overview
Possible Whys
Audit trail - who changed what when
Analysis - maintaining a data warehouse
Time travel - reconstructing state of the data at a point in time in the past
Data sync – one and two way sync’ing
Repairing row-level data - recovering from accidental data changes and application errors

4. Tracking data changes overview
Things to consider when preparing to track changes
Identifying what to track and what not to track
Costs of tracking / not tracking
Legal requirements
New vs legacy systems
What will your hardware / software support

5. Tracking data changes overview
Dealing with date and time
Time zones and UTC
Format of
Assertion(History) vs version (Validity) vs bi-temporal
Does the date/time represent when the change occurred or when the information in the row was valid.
Who or what is the date/time relative to?

6. Temporal Tables
Part of the ANSI SQL 2011 standard. First introduced with SQL Server 2016
Designed to keep a full history of data changes at the row level using a ‘current’ table and a history table
System-versioned because the start and end dates for each row is managed by the system
Assertion / historical in nature meaning that the lifetime of a is record based on the physical dates the record was changed

7. Temporal Tables Good for Not so good for Audit trail
Point in time analysis
Maintaining a data warehouse
Repairing row-level data
Not so good for
Data synchronization
Short term data history requirements
Column level change tracking
Tables that have frequent schema changes

8. Requirements for the base table
Temporal tables - demo
Requirements for the base table
Must have two explicitly defined datetime2 non-nullable columns (period columns)
Used exclusively by the system to record ‘life-span’ of a row
Names of columns can be customized 
Date/time stored as coordinated universal time (UTC)
Can be hidden
Must have a PERIOD FOR SYSTEM_TIME statement.  Identifies the period columns
Must have a primary key
INSTEAD OF triggers are not allowed on the base table. AFTER triggers allowed.
Foreign keys with cascade update or delete are not allowed
Cannot have any computed columns based on a user defined function

9. Requirements for the history table
Temporal tables - demo
Requirements for the history table
Must have the same schema as base table
No primary key, foreign keys, unique indexes, table constraints or triggers.
Cannot be configured for change data capture, change tracking, transactional or merge replication
By default, a clustered index created using period columns
Can be in a different schema than the base table but the same database
Data cannot be modified.  This includes INSERT, UPDATE, DELETE and TRUNCATE

10. Change data capture
First introduced in SQL 2008, Change Data Capture (CDC) captures DML changes on a tracked table. Enterprise only unless you have SQL 2016 SP1
Uses SQL Server Agent jobs to asynchronously read the transaction log to track and record DML transactions
When enabling CDC on a table, an additional table is created containing the same columns as tracked table along with metadata needed to understand the changes
Table-valued functions are also created to allow access the change history
CDC was written by the SQL Server Replication team so it works with transaction replication

11. Not so good for Good for Change data capture
Maintaining a data warehouse
Short term audit trail
Data synchronization (using net changes)
Not so good for
Long term storage of change history
Point in time analysis
Solutions that require real time data capture

12. How it works Change data capture SQL Server Agent jobs
One used to populate the change tables
One responsible for change table cleanup  
SQL Server Agent must be running for data capture and clean up to work 
Transaction logs
SQL Server Agent jobs need to be running for the transaction log to get truncated
Change data is not available until after change is committed to source table and capture job has processed the related log entries

13. Change Data Capture - demo
Validity interval
Changes captured by CDC has a finite lifespan called the validity interval.
Important because the extraction interval for a request must be fully covered by this time period.
Default is 3 days measured in mins
Metadata
__$start_lsn identifies the commit log sequence number (LSN) that was assigned to the change.
__$end_lsn not supported. Future compatibility not guaranteed. Column always NULL (SQL 2012)
__$seqval used to order changes that occur in the same transaction
__$operation records the operation associated with the change: 1 = delete, 2 = insert, 3 = update (before), and 4 = update (after)
__$update_mask variable bit mask with one defined bit for each captured column.
Insert and delete - all bits set. Update - bits set correspond to changed columns

14. Change Data Capture - demo
Requirements
Does not play nice with memory-optimized tables
SQL Server Agent must be running
Net changes: source table has a primary key or an unique index referenced by index_name when setting up CDC on the table.

15. Change tracking
A lightweight solution first introduced in SQL 2008 that provides an efficient mechanism to identify records in a user table that have changed on account of DML transactions
Tracks the fact that a row has changed but not how many times the row has changed or the values of any intermediate changes.
Changes are tracked as they occur
Clean up of internal tables used to keep track of what records have changed are managed by the system
Built-in functions are available to easily query to data changes

16. Change tracking How it works
Once enabled, any DML statement will cause change tracking information for each modified row to be recorded.
Each table enabled for Change Tracking has an internal on-disk table created on the same filegroup.  This table is used to record table change versions and the rows that have changed since a particular version.
When a change occurs the row’s primary key, list of impacted columns (optional) and DML command are recorded in the internal tracking table.
An auto cleanup thread purges expired content of the internal tracking tables based on the retention period setting of the database.

17. Change tracking Good for Not so good for Data synchronization
Any solution that requires that each data change is tracked and recorded.

18. Change tracking - demo Requirements
The database compatibility level must be set to 90 (SQL 2005) or greater.
Snapshot isolation is strongly recommended to help ensure that all change tracking information is consistent.
Tables being tracking must have a primary key.  
DDL operations are not tracked
Deletes caused by a TRUNCATE statement are not tracked. However, this will cause the minimum valid version to be updated. Client applications would then need re-initialize their data.

19. Triggers
A special kind of stored procedure attached to an event.  When the event happens, SQL Server executes the trigger’s code
4 types: DML, DDL, CLR, Logon
INSTEAD OF and AFTER
DML triggers in SQL Server work at the statement-level. This means that one trigger instance fires per statement regardless of how many rows were affected, even zero
Triggers extend the life of a transaction. While a trigger is at work, the associated transaction will not commit until the trigger completes
Due to their event driven nature, issues caused by triggers can be difficult to troubleshoot

20. Triggers Good for Not so good for Auditing
One audit table per user table or one audit table for all user tables
Anything if you want to put the work into building the solution
Any environment that is SQL Server 2005 or earlier
Not so good for
Scenarios where simplicity is required or other change tracking technologies would do the job just as well or better

21. Triggers - demo

22. Conclusion
The tools available natively in Microsoft SQL Server to track data changes have gotten better and better over time. With SQL 2016 SP1 most if not all these tools are available to everyone with a SQL standard license or better. There is really no reason why you should be suffering with a system that does a poor job of tracking changes in your DB. In addition to the tools that are offered natively in SQL Server there are also 3rd party venders that provide off the shelf solutions for tracking data changes. For example, one of the sponsors of today’s event ApexSQL has a product called ApexSQL Audit that tracks and reports events on SQL Server by auditing access and changes to the SQL Server instance and its objects.