1. Distributed Availability Groups
Jennifer Brocato
Distributed Availability Groups

2. About Me
Lead DBA for the brown truck company 25+ years experience with Enterprise Development Worked with SQL Server since 2000 My article on SQLServerCentral.com on SQLDAG

3. What we will cover Availability group refresher The HA/DR solution
Distributed availability groups – what is it
Configuring a SQLDAG solution
Failover a SQLDAG
Monitoring and health

4. Availability Groups Refresher

5. Availability Group Requirements
Requires a Windows Server Failover Clustering (WSFC) cluster.
An instance of SQL Server must reside on a WSFC node
Each availability replica of a given availability group must reside on a different node of the same WSFC cluster.
Relies on WSFC to monitor and manage the current roles of the availability replicas
Overall health is based on all nodes in the cluster
Quorum configuration – node and disk majority
sys.dm_hadr_cluster
sys.dm_hadr_cluster_members

6. Example Configuration (GEO Cluster)

7. Configuring Quorum

8. Availability Group Basics
Contains one or more databases
Database must be in Full Recovery Model
One Unit of Failover
SQL Server Agent Jobs, Logins, Linked Servers do not fail over
Create Availability Groups then add database

9. Creating the HA/DR Solution
Make sure Failover Clustering is installed on nodes by going to Administrator Tools, Failover Cluster Manager
Perform a Cluster Validation
Configure Quorum Node Votes (powershell script)
Primary Data Center Nodes have vote set to 1
Secondary Data Center Nodes have vote set to 0
Install stand-alone or failover cluster instance
Enable AlwaysOn Availability Group for each SQL Server Service

10. Creating the HA/DR Solution
Add login and grant connect to SQL
Create endpoint
Create database or restore to primary instance
Create full/log backup
Create availability group
Restore full/log backup to each secondary
Join availability group
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]

11. Primary Replica with 4 Secondaries

12. Distributed Availability Group (SQLDAG)
One of the new features in SQL2016 is the ability to distribute availability groups across clusters. This solution makes high availability and disaster recovery geographically dispersed.
Distributed Availability groups allows you to associate availability groups on two different Windows Server Failover Clusters.

13. Distributed Availability Group
Two or more clusters
Mix of standalone and FCIs
Secondary cluster only knows that it is a secondary and does not know which is primary (DMV is coming for visibility with this)
No GUI
No Alerts
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]

14. Creating the HA/DR Solution using SQLDAG (…continued)
Add login and grant connect to SQL
Create endpoint
Create database or restore to primary instance
Create full/log backup
Restore full/log backup to each secondary including secondary cluster(s)
Create availability group
Join availability group on primary cluster (may contain just one replica)
Create availability group on secondary (primary instance) cluster
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]
ALTER AVAILABILITY GROUP AG_1 GRANT CREATE ANY DATABASE (for automatic seeding)

15. Now for the FUN PART! WITH (DISTRIBUTED)
Go to the Primary Cluster Primary Instance and create SQLDAG
CREATE AVAILABILITY GROUP [distributedag]
WITH (DISTRIBUTED)
AVAILABILITY GROUP ON
'AG_1' WITH (
LISTENER_URL = 'tcp://<virtualname>:5022', --Use listener name when there is a standalone
AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
FAILOVER_MODE = MANUAL ),
'AG_2' WITH
LISTENER_URL = 'tcp://<virtualname>:5022', --Use SQLVNN not listener name when there is an FCI --check port to make sure it is 5022/5023/5024 etc ); GO
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]

16. Now for the FUN PART…again!
Go to the Secondary Cluster Primary Instance and join SQLDAG
CREATE AVAILABILITY GROUP [distributedag]
JOIN
AVAILABILITY GROUP ON
'AG_1' WITH (
LISTENER_URL = 'tcp://<virtualname>:5022', --Use listener name when there is a standalone
AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT,
FAILOVER_MODE = MANUAL ),
'AG_2' WITH
LISTENER_URL = 'tcp://<virtualname>:5022', --Use SQLVNN not listener name when there is an FCI --check port to make sure it is 5022/5023/5024 etc ); GO
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]

17. Now Let’s Get Synchronized
ALTER DATABASE D000DB1 SET HADR AVAILABILITY GROUP = xxxx
Then WAIT….
Check the health of the distributed availability group to see if all replicas are synchronizing
select r.replica_server_name, r.endpoint_url,
rs.connected_state_desc, rs.last_connect_error_description,
rs.last_connect_error_number, rs.last_connect_error_timestamp
from sys.dm_hadr_availability_replica_states rs join sys.availability_replicas r
on rs.replica_id=r.replica_id
where rs.is_local=1
There is no GUI for SQLDAG so you will need to rely on DMVs
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]
ALTER AVAILABILITY GROUP AG_1 GRANT CREATE ANY DATABASE (for automatic seeding)

18. Gathering Information About Availablity Groups
dm_hadr_automatic_seeding
dm_hadr_physical_seeding_stats
dm_hadr_availability_replica_cluster_nodes
dm_hadr_availability_group_states
dm_hadr_availability_replica_states
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]
ALTER AVAILABILITY GROUP AG_1 GRANT CREATE ANY DATABASE (for automatic seeding)

19. Manual Failover of SQLDAG
ALTER AVAILABILITY GROUP of SQLDAG (secondary cluster) to SYNCHRONOUS_COMMIT
Verify from dm_hadr_database_replica_states that status is SYNCHRONIZED and end_of_log_lsn matches from primarys in both clusters
On instance that hosts primary SQLDAG, set SQLDAG role to Secondary
ALTER AVAILABILITY GROUP xxx SET (ROLE = SECONDARY)
At this point the SQLDAG is unavailable
Verify again readiness to failover
Issue failover – ALTER AVAILABILITY GROUP xxx FORCE_FAILOVER_ALLOW_DATA_LOSS
After this step the SQLDAG is available again
Set SQLDAG back to ASYNCHRONOUS_COMMIT
USE [master] GO
CREATE LOGIN [US\DataCtrSQLSvc] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
use [master]
GRANT CONNECT SQL TO [US\DataCtrSQLSvc]
--Use AES encryption, RC4 is depricated
CREATE ENDPOINT [hadr_endpoint]
STATE=STARTED
AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
FOR DATA_MIRRORING (ROLE = ALL, AUTHENTICATION = WINDOWS NEGOTIATE
, ENCRYPTION = REQUIRED ALGORITHM AES)
GRANT CONNECT ON ENDPOINT::[hadr_endpoint] TO [US\DataCtrSQLSvc]
ALTER AVAILABILITY GROUP AG_1 GRANT CREATE ANY DATABASE (for automatic seeding)

20. Configuration Example
2 sets of clustered VMs
Availability group in each VM cluster
Automatic Failover
Synchronous
Distributed availability group between each cluster set
ESX host cluster A in data center 1
ESX host cluster B in data center 1
ESX host cluster C in data center 2
ESX host cluster D in data center 2
VM1 on ESX host cluster A
VM2 on ESX host cluster B
VM3 on ESX host cluster C
VM4 on ESX host cluster D
Cluster VM1 and VM2 – Cluster X
Cluster VM3 and VM4 – Cluster Y
VM1 and VM2 set up with AG with automatic failover and synchronous
VM3 and VM4 set up with AG with automatic failover and synchronous
SQLDAG between cluster X and cluster Y

21. Monitor SQLDAG Replication Health State
SELECT * FROM (SELECT… FROM sys.availability_replicas r LEFT JOINsys.availability_groups ds ON r.group_id = ds.group_id WHERE ds.name = '<AGName>' ) a inner join ( SELECT … FROM sys.availability_groups ag JOIN sys.dm_hadr_availability_group_states as ags ON ag.group_id=ags.group_id LEFT JOIN sys.dm_hadr_database_replica_states ds ON ds.group_id = ag.group_id ) b ON b.availability_group = a.replica_server_name
select CAST(r.replica_id as VARCHAR(36)) as replica_id, CAST(r.group_id as VARCHAR(36)) as group_id,
replica_server_name, endpoint_url, ISNULL(ds.is_distributed,0) as is_distributed, ds.name
from sys.availability_replicas r left join sys.availability_groups ds on r.group_id = ds.group_id
where ds.name = '<AGName>'
) a
inner join (
select availability_group=cast(ag.name as varchar(30)),
primary_replica=cast(ags.primary_replica as varchar(30)),
primary_recovery_health_desc=cast(ags.primary_recovery_health_desc as varchar(30)),
synchronization_health_desc=cast(ags.synchronization_health_desc as varchar(30)),
CAST(ag.group_id as VARCHAR(36)) as group_id ,
CAST(ag.resource_id as VARCHAR(36)) as resource_id, CAST(ag.resource_group_id as VARCHAR(36)) as resource_group_id, ag.failure_condition_level,
ag.health_check_timeout, ag.is_distributed,
ds.recovery_lsn, ds.truncation_lsn, CAST(ISNULL(ds.last_sent_lsn,0) AS VARCHAR) last_sent_lsn,
ISNULL(ds.last_sent_time,0) as last_sent_time, CAST(ISNULL(ds.last_received_lsn,0) AS VARCHAR) as last_received_lsn,
ISNULL(ds.last_received_time,0) as last_received_time,
CAST(ISNULL(ds.last_hardened_lsn,0) AS VARCHAR) as last_hardened_lsn,
ISNULL(ds.last_hardened_time,0) as last_hardened_time,
ds.last_redone_lsn, ds.last_redone_time, ds.log_send_queue_size,
CAST(ISNULL(ds.log_send_rate,0) AS VARCHAR) as log_send_rate, ds.redo_queue_size,
CAST(ISNULL(ds.redo_rate,0) AS VARCHAR) as redo_rate,
CAST(ds.end_of_log_lsn AS VARCHAR) as end_of_log_lsn, CAST(ds.last_commit_lsn AS VARCHAR) as last_commit_lsn, ds.last_commit_time,
automated_backup_preference_desc=cast(ag.automated_backup_preference_desc as varchar(10))
from sys.availability_groups ag join sys.dm_hadr_availability_group_states ags on ag.group_id=ags.group_id
left join sys.dm_hadr_database_replica_states ds on ds.group_id = ag.group_id
) b on b.availability_group = a.replica_server_name

22. Health State
Metric Collection timestamp, LSN information and Health State

23. Health State
LSN Hardened, RedoRate, LSN Commit.

24. Alert on Unhealth SQLDAG State
Record health state with query in a set interval using a job (5 minutes is a suggestion).
Use a job to check latest health state in comparison to prior health state
Repeated unhealthy SQLDAG replication to trigger alerting mechanism (such as ).

25. Worth Mentioning SQL2017 Read-scale availability groups
Not for HA or DR, just for synchronization, no WSFC
The official abbreviation for distributed availability groups is not DAG. DAG is used for Exchange Database Availability Groups.

26. Questions

27. Thank You