1. Ondřej Ševeček | GOPAS a.s. | MCM: Directory Services | MVP: Enterprise Security | ondrej@sevecek.com | www.sevecek.com |

2. Active Directory Replication

3. Central Database  LDAP – Lightweight Directory Access Protocol  database query language  similar to SQL  TCP 389, SSL TCP 636, GC TCP 3268, GC SSL TCP 3269  Windows NT 4.0 SAM  SMB/CIFS TCP 445 (or NetBIOS)  password resets, SAM queries  Kerberos  UDP/TCP 88

4. Design Considerations  Distributed system  DCs disconnected for very long times  several months  Multimaster replication  with some FSMO roles

5. Design Considerations  Example: Caribean cruises, DC/IS/Exchange on board with tens of workstations and users, some staff hired during journey. No or bad satelite connectivity only. DCs synced after ship is berthed at main office.  Challenge: Must work independently for long time periods. Different independent cruise- liners/DCs can accomodate changes to user accounts, email addresses, Exchange settings. Cannot afford lost of any one.

6. Database  Microsoft JET engine  JET Blue  common with Microsoft Exchange  used by DHCP, WINS, COM+, WMI, CA, CS, RDS Broker  %WINDIR%\NTDS\NTDS.DIT  ESENTUTL  Opened by LSASS.EXE

7. Installed services LSASS Security Accounts Manager TCP 445 SMB + NamedPipes Kerberos Key Distribution Center UDP, TCP 88 Kerberos Active Directory Domain Services UDP, TCP 389 LDAP NTDS.DIT

8. Installed services LSASS SAM TCP 445 SMB + NamedPipes KDC UDP, TCP 88 Kerberos NTDS UDP, TCP 389 LDAP NT4.0 NTLM Pass-through Windows 2000+ LDAP/ADSI Client Connect to Domain

9. Restartable AD DS  Windows Server 2008  Active Directory Domain Services service  LSASS.EXE  Can log on DS Restore Mode Admin  HKLM\System\CurrentControlSet\Control\LSA  DsrmAdminLogonBehavior = 1

10. DNS Best Practice DC1 DNS DC2 DNS AD

11. Active Directory Replication

12. Logical Structure  Partitions  separate “subdatabases”  replication domains  RootDSE (Root Directory Services Enterprise)  Schema  Configuration  Domain  can contain user accounts  Application  can contain user accounts  Global Catalogue

13. Replication domains Domain A Config Schema RootDSE1 Domain B Config Schema RootDSE5 RootDSE6 App1 Domain B Config Schema RootDSE4 App1 RootDSE3 Domain A Config Schema RootDSE2 App1 App2

14. Dublin Global Catalogue London Paris Prague kamil@idtt.comMBX1 judith@idtt.comMBX2 helen@idtt.comMBX1 ian@idtt.comMBX3 Exchange HUB SMTP

15. Dublin Global Catalogue London Paris Prague kamil@idtt.com judith@idtt.com helen@idtt.com ian@idtt.com Exchange HUB SMTP GC kamil@idtt.comMBX1 judith@idtt.comMBX2 ian@idtt.comMBX3 helen@idtt.comMBX1

16. Global Catalogue (DC data) DC object GUIDDNDisplay name Tel.OfficeMemberCustom data user#1CN=Kamil...Kamil Sevecek 555-666C 915CN=Sales... CN=People... user#2CN=Judith...Judith Hava 777-888D 308CN=Sales... CN=People... global group #3CN=Sales...SalesCN=Kamil... CN=Judith... universal group #4CN=People...PeopleCN=Kamil... CN=Judith... ou#5OU=London...GPO share#6CN=share...\\srv8\doc dns record #7CN=pc31...10.10.0.71

17. Global Catalogue (GC data) DC object GUIDDNDisplay name Tel.OfficeMemberCustom data user#1CN=Kamil...Kamil Sevecek 555-666 user#2CN=Judith...Judith Hava 777-888 global group #3CN=Sales...Sales universal group #4CN=People...PeopleCN=Kamil... CN=Judith... ou#5OU=London... share#6CN=share... dns record #7CN=pc31...

18. London DC1 Paris GC and Logon Paris Prague SRV Kamil DL G U GC A B C UEKamil UD UC UEJudith SID #1 SID #2 SID #3 DC2 DLE DC3 UD DC4 UE

19. London DC1 Paris GC and Logon Paris Prague SRV Kamil DL G U GC A B C UEKamil UD UC UEJudith SID #1 SID #2 SID #3 DC2 DLE DC3 UD DC4 UE

20. Ticket London DC1 Paris GC and Logon Paris Prague SRV Kamil DL G U A B C U Kamil UC SID #1 SID #2 SID #3 DC2 DLE DC3 UD DC4 UE GBKamilSID #4 EKamil UD DLEKamilSID #5

21. Active Directory Replication

22. Attribute Types  string, integer, datetime, boolean, binary  DN reference  multivalue  up to 5000 items  linked multivalue  unlimited, requires 2003 Forest Level  backlink  memberOf  computed  primaryGroupToken, tokenGroups, lastLogonTimestamp  write/only attributes  unicodePwd

23. Group membership Sales CN=Kamil,OU=London,DC=... CN=Judith,OU=Paris,DC=... CN=Victor,OU=London,DC=... CN=Stan,OU=London,DC=... member Judith CN=Sales,OU=Groups,DC=... CN=IS Access,OU=Groups,DC=... memberOf Link Backlink

24. (Not)replicated attributes  Not replicated  logonCount  badPasswordCount  badPasswordTime  lastLogon  lastLogoff  Replicated  pwdLastSet  lockoutTime  lastLogonTimestamp (since 2003)

25. Logon timestamps (2003 DFL) Client DC lastLogon11:38 lastLogon9:00 lastLogon- lastLogonTimestamp11:00 lastLogonTimestamp11:00 lastLogonTimestamp11:00

26. lastLogonTimestamp  Requires 2003 domain level  Updated only once per 14-random(5) days  DC=idtt,DC=local  msDS-LogonTimeSyncInterval  1+ – minimum without randomization  5+ – randomization starts  14 – the default ...

27. Password changes Password Change Immediate Replication password hash Normal replication DC PDC Client

28. Password changes Client DC PDC pwdLastSet

29. Authentication failures DC PDC pwd1 DC pwd1 Client

30. Authentication failures DC PDC pwd1 DC pwd2 Client pwd2

31. Authentication failures Client PDC pwd2 DC pwd2 DC pwd1

32. Authentication failures ClientDC badPasswordCount3 2 PDC badPasswordCount7 lockoutTime DC badPasswordCount2

33. Security Principals  Users  login, password, SID + SID history  Computers  user + computer attributes  Service Accounts  computer + specific attributes  Groups  login, SID + SID history

34. Computer Password Age

35. Active Directory Replication

36. Intrasite Replication Topology DC1 DC2 DC4 DC3

37. Originating Updates and Notifications DC1 DC2 DC4 DC3 15 sec 3 sec

38. Notification and Replication DC1DC2 I have got some changes Kerberos Authenticated DCOMTCPRandom Give me your replica Kerberos Authenticated DCOMTCPRandom

39. Intrasite Replication – 3 Hops max. DC1 DC4 DC3 DC5 DC6 DC7 DC2

40. Intersite Replication (no Bridgeheads) DC1 DC2 DC3 DC5 DC6 DC7 DC4

41. Intersite Replication (no Bridgeheads) DC1 DC2 DC3 DC5 DC6 DC7 DC4 15 sec 3 sec schedule

42. Intersite Replication with a Bridgehead DC1 DC2 DC3 DC5 DC6 DC7 DC4 15 sec 3 sec schedule

43. Intrasite Replication  Uses notifications by default (originating/received)  300/30 sec on Windows 2000  15/3 sec on Windows 2003  Occurs every hour as scheduled  nTDSSiteSettings  At this frequency KCC detects unavailable partners  HKLM\System\CCS\Services\NTDS\Parameters  Replicator notify pause after modify (secs)  Replicator notify pause between DSAs (secs)

44. Intrasite Replication DC1 DC2 notification random TCP download changes random TCP 15 sec download changes random TCP schedule

45. Intersite Replication DC1 DC2 download changes random TCP schedule

46. Intersite Replication  Does not use notifications by default  siteLink: options = USE_NOTIFY (1)  Compression used  siteLink: options = DISABLE_COMPRESSION (4)  Bridge all site links

47. Static TCP for Replication  HKLM\System\CurrentControlSet\Services  NTDS\Parameters  TCP/IP Port = DWORD  Replication  Netlogon\Parameters  DCTcpipPort = DWORD  LSASS (Pass-through)  NTFRS\Parameters  RPC TCP/IP Port Assignment = DWORD  DFSRDIAG StaticRPC /port:xxx /Member:dc1

48. Urgent Replication (Notification)  Intrasite only  intersite also if notification enabled  Do not wait for delay (15/3 sec)  In the case of  account lockout  password and lockout policy  RID FSMO owner change  DC password or trust account password change

49. Immediate Replication (Notification)  Password changes  from DCs to PDC  Regardless of site boundaries  PDC downloads only the single user object  all changed attributes but only single object  From DC/PDC further with normal replication

50. Example Replication Traffic  Atomic replication of a single object with a one byte attribute change  Notification + replication  intersite compressed  Overall 7536 B  30 packets ~10 round trips  50 ms round trip means 500 ms transfer time  consumption at 120 kbps  Useful data ~80 B

51. Bridge All Site Links On Olomouc London Prague Paris Roma Cyprus B BA  site links are transitive  can be disabled on IP transport A A A A

52. Bridge All Site Links Off Olomouc London Prague Paris Roma Cyprus A A  site links are not transitive  Cyprus partition is cut off A A A B B

53. GC Replication Olomouc London Prague Paris Roma Cyprus A A A A A  one-way: from the source NC into the nearest GC  two-way: GCs between themselves B GC

54. Roma London GC Replication Olomouc Prague Paris Cyprus A A A A B A B  one-way: from the source NC into the nearest GC  two-way: GCs between themselves GC

55. Subnetting in AD (Apps) 10.10.x.x / 16 10.10.0.248 / 29 DC1 DC2 DC3DC4 DC5 Exchange

56. Subnetting in AD (Recovery) 10.10.x.x / 16 Recovery Site 10.10.0.7 / 32 DC1 DC2 DC3DC4 DC5

57. Active Directory Replication

58. Modification operations  Create new object  Modify attributes  change/delete value  change distinguishedName = rename  Rename container  all subobjects renamed as well

59. Replication Metadata  REPADMIN /ShowObjMeta  all attributes  when  originating DC

60. Replication conflicts  The later action wins  if no one is later then random (USN)  Attribute modified on two DCs “simultaneously”  only one change wins  Linked multivalue attribute modified  merged (on 2003+ forest level)  Object/container deleted and object modified  deleted  Object moved into a deleted container  CN=lost and found  Two objects with the same sAMAccountName, cn or userPrincipalName created  object renamed, logins duplicit

61. Linked Multi-values

62. DC1 Replication Kamil10:00 Helen11:00 DC2 DC19:00 11:05

63. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC111:30 Kamil10:00 Helen11:00 11:30

64. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC111:30 Kamil10:00 Helen11:00 Judith12:00 12:05

65. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC112:30 Kamil10:00 Helen11:00 Judith12:00 Judith12:00 12:30

66. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC112:30 Kamil10:00 Helen11:00 Judith12:00 Judith12:00 DC1 DC3 Marie11:00 Me 12:30

67. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC112:30 Kamil10:00 Helen11:00 Judith12:00 Judith12:00 DC1 DC3 DC110:30 DC27:00 Kamil10:00 DC1 Marie11:00 Me 12:30

68. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC112:30 Kamil10:00 Helen11:00 Judith12:00 Judith12:00 DC1 DC3 DC110:30 DC27:00 Kamil10:00 DC1 Marie11:00 Me 13:30

69. DC1 Replication Basics Kamil10:00 Helen11:00 DC2 DC112:30 Kamil10:00 Helen11:00 Judith12:00 Judith12:00 DC1 DC3 DC112:30 DC213:30 Kamil10:00 DC1 Marie11:00 Me 13:30

70. DC1 Replication Basics Kamil10:00 Helen11:00 Kamil10:00 Helen11:00 Judith12:00 Judith12:00 DC1 DC3 DC112:30 DC213:30 Marie11:00 DC2 14:15

71. USN  Each object modification increments USN for that object and for the whole DC  Each DC remembers USNs of its replication partners  repadmin /showutdvec

72. USN 2 USN 5001 3 USN 3001 1 USN 1001 25001 33001 11001 33001 11001 25001

73. USN 2 USN 5001 3 USN 3001 1 USN 1003 25001 33001 1 3 11001 25001 Kamil1002 John1003 1001

74. USN 2 USN 5001 3 USN 3001 1 USN 1003 25001 33001 1 3 11001 25001 Kamil1002 John1003 Notify Give me 1002, 3 1001

75. USN 2 USN 5003 3 USN 3001 1 USN 1003 25001 33001 11003 33001 11001 25001 Kamil5002 John5003 Kamil1002 John1003

76. USN 2 USN 5004 3 USN 3001 1 USN 1003 25001 33001 11003 33001 11001 25001 Kamil5002 John5003 Maria5004 Kamil1002 John1003

77. USN 2 USN 5004 3 USN 3004 1 USN 1003 25001 33001 11003 33001 11003 25004 Kamil3002 John3003 Kamil5002 John5003 Maria5004 Maria3004 Kamil1002 John1003

78. 2 1 1 1 1 USN 2 USN 5004 3 USN 3004 1 USN 1003 25001 33001 11003 33001 11003 25004 Kamil John Kamil 1002 John1003 Kamil John Maria Kamil John 5002 5003 5004 2 1 1 Kamil John Kamil John Maria 3002 3003 3004

79. 2 1 1 1 1 USN 2 USN 5004 3 USN 3004 1 USN 1003 25001 33004 11003 33001 11003 25004 Kamil John Kamil 1002 John1003 Kamil John Maria Kamil John 5002 5003 5004 2 1 1 Kamil John Kamil John Maria 3002 3003 3004 Maria2

80. Active Directory Troubleshooting

81. Delete operations  Delete only removes most attributes from an object  tombstone  Replicates as normal object change/move  Deleted by individual DCs after tombstoneLifetime  CN=Directory Services,CN=Services,CN=Configuration,...

82. Reanimating objects  LDP  Options – Search  Extended  Return deleted objects  View – Tree  CN=Deleted Objects

83. Tombstone lifetime  Windows 2000  60 days  Windows 2003 SP1+  180 days  upgrade keeps the 60 days from previous version

84. Tombstone lifetime  CN=Directory Services,CN=Windows NT,CN=Services,CN=Configuration,DC=idtt,DC= local  tombstoneLifetime  garbageCollPeriod (12 hours by default)  Garbage collection does not delete white space from the database  only offline defragmentation  the amount can be logged by setting HKLM\System\CCS\Services\NTDS\Diagnostics 6 Garbage Collection = 1

85. AD Recycle bin  Optional feature with Windows 2008 R2 forest level  Preserves all attributes on deleted objects for the tombstone lifetime  after that, the object becomes normal tombstone for another lifetime  Does not preserve attribute changes  recovery site still useful  Keeps deactivated links (group membership)

86. Active Directory Replication

87. The Three Problems  Single DC offline for a long time  not so long as tombstone!  authentication problem  Tombstone lifetime  two separate DC zones  not a “business” consistency problem  USN rollback  restore from snapshot, image, manual backup  total inconsistency!

88. DC Offline for Long Time DC1 DC2 DC3 DC2PWD21 DC3PWD31 PWD21 Month 0 OLD PWD- PWD31 OLD PWD-

89. DC Offline for Long Time DC1 DC2 DC3 DC2PWD21 DC3PWD31 PWD22 Month 1 OLD PWD21 PWD32 OLD PWD31

90. DC Offline for Long Time DC1 DC2 DC3 DC2PWD21 DC3PWD31 PWD23 Month 2 OLD PWD22 PWD33 OLD PWD32

91. PWD 21 DC Offline for Long Time DC1 DC2 DC3 DC2PWD21 DC3PWD31 PWD23 Month 3 OLD PWD22 PWD33 OLD PWD32 Kerberos KDC Ticket

92. PWD 23 DC Offline for Long Time DC1 DC2 DC3 DC2PWD21 DC3PWD31 PWD23 Month 3 OLD PWD22 PWD33 OLD PWD32 KDC Disabled Ticket Kerberos KDC

93. Lingering Objects  When DC didn’t replicate during the tombstoneLifetime, it halts replication  Can be restored by Allow Replication with Divergent and Corrupt Partner  HKLM\System\CCS\Services\NTDS\Parameters  turn on, replicate, turn off

94. DC4 DC3 DC2 DC1 Objects and Tombstones Frank Stan Tania Frank Stan Tania Frank Stan Tania Frank Stan Tania

95. DC4 DC3 DC2 DC1 Objects and Tombstones Frank Stan Tania Frank Stan Tania Frank Stan Tania Frank Stan Tania

96. DC4 DC3 DC2 DC1 Objects and Tombstones Frank Stan Tania Frank Stan Tania Frank Stan Tania Frank Stan Tania

97. DC4 DC3 DC2 DC1 Objects and Tombstones Frank Stan Tania Frank Stan Tania Frank Stan Tania Frank Stan Tania

98. DC4 DC3 DC2 DC1 Objects and Tombstones Frank Tania Frank Tania Frank Tania Frank Tania

99. DC4 DC3 DC2 DC1 Lingering Objects Frank Stan Tania Frank Stan Tania Frank Stan Tania Frank Stan Tania

100. DC4 DC3 DC2 DC1 Lingering Objects Frank Stan Tania Frank Stan Tania Frank Stan Tania Frank Stan Tania

101. DC4 DC3 DC2 DC1 Lingering Objects Frank Tania Frank Stan Frank Tania Frank Stan Tania

102. DC4 DC3 DC2 DC1 Lingering Objects Frank Tania Frank Stan Frank Tania Frank Stan Tania

103. Lingering Objects

104.  Strict Replication Consistency  HKLM\System\CCS\Services\NTDS\Parameters  1 – do not replicate  0 – request full copy from source  By default only on new Windows 2003+ installations

105. Lingering Object found/deleted

106. Correct Registry Settings  Long term normal operation  Strict consistency = 1  Allow divergent partner = 0  Temporary repair operation  Strict consistency = 1  Allow divergent partner = 1

107. USN Rollback  May or may not be detected  Cannot be repaired  not always lingering objects!  DC must be denoted/repromoted  unplug network  DCPROMO /forceremoval  NTDSUTIL Roles  NTDSUTIL Metadata Cleanup

108. USN Rollback 1001 DC1 2 USN 5001 1 33001 Snapshot 1001

109. USN Rollback Kamil1002 John1003 Judith1004 Helen1005 1001 DC1 Eva1006 2 USN 5001 1 33001 Snapshot 1001

110. USN Rollback Kamil1002 John1003 Judith1004 Helen1005 1001 DC1 Eva1006 2 USN 5001 11006 33001 Snapshot Kamil1002 John1003 Judith1004 Helen1005 Eva1006

111. USN Rollback (Detectable) 1001 DC1 2 USN 5001 11006 33001 Restore Kamil1002 John1003 Judith1004 Helen1005 Eva1006 Frank1002 Stan1003

112. USN Rollback (Non-detect.) Frank1002 Stan1003 1001 DC1 2 USN 5001 11006 33001 Tania1004 Mark1005 Martin1006 Victor1007 Leo1008 Restore Kamil1002 John1003 Judith1004 Helen1005 Eva1006

113. USN Rollback (Non-detect.) Frank1002 Stan1003 1001 DC1 2 USN 5001 11008 33001 Tania1004 Mark1005 Martin1006 Victor1007 Leo1008 Restore Victor1007 Leo1008 Kamil1002 John1003 Judith1004 Helen1005 Eva1006

114. Restoring VM Snapshots  Restore offline  HKLM\System\CurrentControlSet\Services\NTDS  Database Restored from Backup = DWORD = 1  Restart NTDS service  changes InvocationID of the database instance

115. Active Directory Troubleshooting

116. Stealth  Microsoft’s internal tool  test only, not supported  Must run from writable location  C:\ etc.  creates temporary DB

117. Database attributes ATTxyyyattributeATTxyyyattribute m3cnk589827objectSID m4snk589826objectGUID m11ouk589825name m13descriptionm590045sAMAccountName m42givenNamem590480userPrincipalName m8statem131218company m6countrym131203country m19physicalDeliveryOfficeNamem131328address m20telephoneNumberm12jobTitle m131213departmentm7city m131682employeeNumber m131085displayName m1376259email

118. Database structure  DNT – Distinguished Name Tag  unique identification inside the table  NCDNT – Name Context DNT  reference to which partition the object belongs  RDNT – Parent DNT  reference to parent OU/CN object  objects do not store their whole DN, the hierarchy is built by the references

119. link_table  Contains linked multivalue references  LINK_DNT  DNT of the referencing object (group)  BACKLINK_DNT  DNT of the referenced object (user, phantom)  Phantom  GUID, NC reference, name

120. Active Directory Troubleshooting

121. Backup  Windows 2003-  streaming backup  must have read the data from the database using API and built a new database into the backup media  Windows 2008+  Volume Shadow Copy  AD Writer available  makes the database into clean shutdown

122. Restore  Not supported  to restore manually only the database  to restore from disk image  to restore from virtual machine snapshot  may be partially recovered by the Database Restored from Backup registry  Supported  restore whatever you want first  then restore System State

123. Recovery Site  You can separate one or more DCs into a slowly replicating site to avoid the need for offline restore  Authoritative Restore can be done even on non-restored DCs  the database must still be brought offline

124. Authoritative Restore  Marks some objects as authoritative  Replicates all their attributes over any other objects found on other DCs  When restoring whole OUs, consider their relationship to GPOs linked

125. Authoritative Restore Problems  old computer, user, service and trust passwords restored  membership in remote Domain Local and Universal groups lost  exports.LDF file containing the group memberships  connections to GPO lost at OU level  orphaned GPOs

126. Ondřej Ševeček | GOPAS a.s. | MCM: Directory Services | MVP: Enterprise Security | ondrej@sevecek.com | www.sevecek.com |