1. Ing. Ondřej Ševeček | GOPAS a.s. | MCM: Directory Services | MVP: Enterprise Security | ondrej@sevecek.com | www.sevecek.com |

2. Active Directory Troubleshooting

3. Client Applications  Kerberos and NTLM authentication  Secure Channel  password changes, NTLM pass-through, Kerberos PAC validation  Group Policy client  DFS client  Certificate Autoenrollment client

4. Client Applications  NPS (IAS), RRAS, TMG (ISA), RD Gateway (TS Gateway)  group membership, Dial-In tab  RD Host (Terminal Server)  Remote Control tab etc., Licensing servers  DHCP Server  authorization  IIS  account and group membership for SSL certificate authentication  WDS  computer MAC addresses or GUIDs

5. Site Design Scenarios Central Branche

6. Site Design Scenarios Office

7. Site Design Scenarios Central Branche

8. Network Interactions Racap (DC Location) Any DC 2000+ Client 2000+ LDAP UDP SRV: Any DC List Get My Site DNS SRV: My Side DC My Site DC 2000+

9. Network Interactions Recap (2008/Vista+ DC Location) Any DC 2008+ Client Vista+ LDAP UDP SRV: Any DC List Get My Site DNS SRV: My Site DC Next Closest Site Close Site DC 2000+ My Site DC 2000+ SRV: Close Site

10. Network Interactions (Network Logon) DC 2000+ Client 2000+ Kerberos Server 2000+ App Traffic DC 2000+ SMB D/COM TGT: User In-band TGS: Server NTLM Occasional PAC Validation TGS: Server D/COM Dynamic TCP NTLM Pass-through

11. Connection Properties  Bandwidth (Mbps)  forget about this  Latency (ms)  round-trip-time (RTT)  SMB, D/COM, SQL  Packet Loss (per sec., per Mb)  packet loss rate (PLR)  VPN such as PPTP, SSTP, IP-HTTPS

12. Timeouts  DNS  primary DNS = 1 sec.  secondary DNSs = 2 sec. ... 1 2 2 4 8...  ARP ... 600 ms 1000 ms  LDAP UDP Site Location  600 ms  TCP  SYN = 21 sec. (3x retransmission)  PSH/ACK = 93 sec. (5x retransmission) ... 3 6 12 24 48...  Kerberos (TCP, 3 attempts, KdcSendRetries)  63 sec.

13. Basic DC location  Know the DNS name of the domain  Query general DNS DC SRV records  _ldap._tcp.dc._msdcs.idtt.local  Ping DC  Windows 2003-  LDAP UDP (ping) DC  to get the client’s site/close site

14. Site DC Location  Site unaware lookup  NSLOOKUP  SET Q=SRV  _ldap._tcp.dc._msdcs.idtt.local  Site specific lookup  NSLOOKUP  SET Q=SRV  _ldap._tcp.Paris._sites.dc._msdcs.idtt.local

15. Lab: Finding DCs Manually  Use NSLOOKUP to query for the generic DC list  NSLOOKUP  SET q=SRV  _ldap._tcp.dc._msdcs.idtt.local

16. London 10.10.x.x Site Example – Single Site DC1DC2DC3 Client DC4 DC5

17. Paris 10.20.x.x London 10.10.x.x Site Example – Multihomed DC (DNS Bitmask Ordering OK) DC1DC2DC3 Client DC4 DC5

18. Roma 10.30.x.x Paris 10.20.x.x London 10.10.x.x Site Example – Multihomed DC (DNS Bitmask Ordering Error) DC1DC2DC3 Client DC4 DC5

19. DNS Record Priority and Weight

20. Berlin 10.50.x.x Paris 10.20.x.x Roma 10.30.x.x London 10.10.x.x Site Awareness DC1DC2DC3 DC4 DC5 DC6 Client where I am?AnonymousLDAPUDP

21. General Operation  Use DNS to find generic DC list  Ping selected DC  Windows 2003-  Anonymous LDAP (UDP) to determine site  DC defines site from the request source IP address (NAT?)  Use DNS to find close DC in site  Ping or LDAP UDP to determine availability

22. DC Locator  NetLogon Service  nltest /sc_query:idtt  no network access  nltest /sc_verify:idtt  tries to authenticate with the DC  nltest /sc_reset:idtt  always performs new DNS lookup  nltest /dsgetsite  anonymous query against selected DC

23. Lab: Check NLTEST Usage  Try the NLTEST to query, verify and reset secure channel from Seven2 to its London DCs

24. Berlin 10.50.x.x Paris 10.20.x.x Roma 10.30.x.x London 10.10.x.x Limit UDP Site Location to a Central Site? DC1DC2DC3 DC4 DC5 DC6 Client where I am?AnonymousLDAPUDP

25. Limiting Generic DC List  Limit creation of generic DC DNS records  GPO: Computer Configuration – Administrative Templates – System – Netlogon – DC Locator DNS Records  DC Locator DNS Records not Registered  Dc Kdc

26. Limiting Generic DC List (Wise?) Central Branche

27. Limiting Generic DC List (Wise?) Office

28. DFS Client (MUP)  Multiple UNC provider (MUP) driver  Determines its own DFS server referrals  obtains the list of DFS root servers from AD using the default DC from Netlogon  SYSVOL may be accessed from a different DC  DFSUTIL /PKTINFO  Windows Server 2003/Windows XP  DFSUTIL CACHE REFERRAL  Windows Server 2008/Windows Vista

29. DFS Context Menu

30. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Empty Site DC1DC2DC3 DC4DC5 DC6 DC7 Berlin 10.50.x.x Client DC4DC5

31. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Empty Site DC1DC2DC3 DC4 DC5 DC6 DC7 Berlin 10.50.x.x Client DC4DC5 DC1 DC2 DC3

32. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Empty Site DC1DC2DC3 DC6 DC7 Berlin 10.50.x.x Client DC4DC5 DC1 DC2 DC3 cost 50 cost 100

33. Automatic Site Coverage  Each DC registers itself for its neighboring empty sites  HKLM\System\CurrentControlSet\Services\N etlogon  AutoSiteCoverage = DWORD = 1/0  GPO: Sites Covered by the DC Locator DNS SRV Records

34. Active Directory Troubleshooting

35. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Out of Site DC1DC2DC3 DC4DC5 DC6 DC7 Berlin 10.50.x.x Client 10.100.0.7

36. Super-netting or Sub-netting

37. Out-of-site Clients

39. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Limiting Generic DC List DC1DC2DC3 Berlin 10.50.x.x Client 10.100.0.7

40. DC Stickiness  When one close selected, client sticks to it  even when moved into a different site  must reset secure channel  Force rediscovery interval GPO  Vista+  hotfix for Windows XP  also registry value ForceRediscoveryInterval

41. London 10.10.x.x Site Example – Until Restart/24 hours DC3 DC1DC2 Client

42. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Moving Client DC1DC2DC3 DC4DC5 DC6 DC7 Berlin 10.50.x.x Client DC4DC5 previously in Paris

43. Lab: Moving the Client  On Seven2 verify the current DC in use  NLTEST /sc_query:idtt  Move the client into Paris and update group policy  GPUPDATE  Verify the current DC in use again  the client should use the same DC still although in remote site (stick)  Reset the secure channel several times and determine the result  NLTEST /sc_reset:idtt

44. Active Directory Troubleshooting

45. Berlin 10.50.x.x Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Failed DC DC1DC2DC3 DC4 DC5 DC6 DC7 Client

46. Lab: Client Failover  Move the client into Cyprus  Reset the secure channel and verify it has been connected to DC5  Unplug DC5 from network  Update group policy  GPUPDATE  Verify the resulting DC in use  NLTEST /sc_query:idtt

47. Non-close Site DC  Close site  client’s site  next closest site if enabled  If there is not DC available in the close site, rediscovery every 15 minutes  HKLM\System\CurrentControlSet\Services\Netlog on\Parameters  CloseSiteTimeout = REG_DWORD = x seconds

48. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Next Close Site DC1DC2DC3 DC4DC5 DC6 DC7 Berlin 10.50.x.x Client

49. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Close Site DC1DC2DC3 DC4DC5 DC6 DC7 Berlin 10.50.x.x Client cost 50 cost 100

50. Paris 10.20.x.x Cyprus 10.40.x.x Roma 10.30.x.x London 10.10.x.x Site Example – Close Site DC1DC2DC3 DC4DC5 DC6 DC7 Berlin 10.50.x.x Client cost 100 cost 50

51. Try Next Closest Site  First get any DC name from DNS  Second query the DC for clients site name  returns the clients site  plus the closest site (determined by the DC)  Then query DNS for DCs in its current site and then tries to use the DCs  If none responds, the client queries DNS for its next closest site and tries to use the found DCs

52. Try Next Closest Site  Does not consider RODC sites by default  Can be change in registry  NextClosestSiteFilter  Windows 2003- cannot return the next closest site information  problem if the hit “any DC” is Windows 2003-  it is then going to be used regardless of its site

53. Lab: Next Closest Site  Enable Try next closest site in a GPO  Have DC5 unplugged from network  Update group policy  Check the resulting DC in use  NLTEST /sc_query:idtt

54. Client Rules Recap  Windows 2003-  In current site  In any site  Windows Vista+ with Next closest site  In current site  In the closest site  In any site  If the client is out of any site, find any dc  consider creating subnets for VPNs etc.

55. General Best Practice  Use only AD DNS servers on clients  Do not use multi-homed DCs  Define all IP ranges in AD  may use super-netting if necessary  Limit the generic DC list  site UDP location, out-of-site clients, DC failure  may use static GPO Site assignment  Force rediscovery  Try next closest site

56. Active Directory Troubleshooting

57. Read/only DC  Physically insecure locations  Only specified password hashes  Read/only database  other DCs are not willing to replicate back from the RODC  Local Administrator  Managed By tab in the DC properties

58. RODC scenario Cyprus 10.40.x.x London 10.10.x.x DC1DC2DC3 DC5 SRV CL1 2003 2008 GC 2008

59. Requirements  Forest functional level 2003  Domain functional level 2003  Global catalogue 2003+  understands confidential attributes  At least one writable 2008+ DC

60. RODC and Windows 2003  Windows 2003 does not consider RODC  Do not construct replication connections

61. RODC and Windows 2003  Disable Auto Site Coverage  HKLM\SYSTEM\CurrentControlSet\Services\Netlo gon\Parameters  AutoSiteCoverage = REG_DWORD = 0  or install RODC compatibility pack  Windows 2003, XP (11 issues)  KB 944043  Windows 2003, XP

62. DNS locator records

63. Password caching  Passwords are only cached  once the user logs on using writable DC first time  can be prepopulated  If the login fails on RODC, the request is forwarded to another writable DC  if offline, password expiration is ignored

64. Password caching/forwarding Cyprus 10.40.x.x London 10.10.x.x DC1DC2DC3 DC5 SRV CL1 2003 2008 GC 2008  not cached yet  not cached yet after recent password change  wrong password  expired password  account locked

65. Write referrals Cyprus 10.40.x.x London 10.10.x.x DC1DC2DC3 DC5 SRV CL1 2003 2008 GC 2008  try update on RODC  referral returned  try update on the referred writable DC directly

66. Write Referral Problems  BitLocker  SP1 for Windows 2008/Vista  Managed Service Accounts  SP1 for Windows 2008 R2/Windows 7

67. Account lockout  Accounts locked locally  not replicated  But the failure attempt is also reattempted on a writable DC  so this then replicates

68. Expired passwords  pwdLastSet older than allowed by policy  Logon attempt fails completely  Password must be changed out-of-band and logon then attempted again

69. Expired password DC CL1 logon error: expired logon ok password change pwdLastSet before 3 months pwdLastSet actual

70. Discarding RODC

71. RODC DMZ Scenario  Only RODC has internal domain access  Cannot join domain normally  use a join script (+ RODC compatibility pack)  Cannot change machine passwords  Cannot determine their site from the "any DC list"  HKLM\SYSTEM\CCS\Services\Netlogon\Parameters SiteName = REG_SZ  Cannot update AD account  operating system  service principal names

72. Active Directory Troubleshooting

73. DNS Integration  Clients find DCs by domain/site name  DCs find replication partners according to their GUID  Netlogon de/registers locator records  DNS stores its data in  domain partition  DomainDnsZones application partition  ForestDnsZones application partition

74. Netlogon de/registration  Netlogon de/registers its own records at startup and deregisters them at shutdown  requires DNS registration enabled on at least one network adapter  does not require DNS/DHCP Client service  %windir%\System32\Config\netlogon.dns  It does not touch others’ records  Autosite coverage  turned on by default

75. Netlogon de/registration  Restarting Netlogon  NLTEST /DSREGDNS  force reregistration  NLTEST /DSQUERYDNS  query last status  does not require DNS/DHCP Client service and does not react on /REGISTERDNS

76. AD Integrated Zones  Offer Secure Dynamic Update  Timestamping  trimmed to whole hour  Aging and scavenging  records deleted by default between 14-21 days of their age

77. DNS Application Partitions  Domain partition  CN=MicrosoftDNS,CN=System,DC=...  DomainDnsZones  replicated to all DNS Server which are also DCs for the domain  ForestDnsZones  replicated to all DNS Server which are also DCs for the forest

78. Secure Dynamic Update  Client side feature  DHCP Client on Windows 2003-  DNS Client on Windows Vista+  IPCONFIG /REGISTERDNS  DNS Server must be on DC to authenticate clients with Kerberos  All Authenticated Users  can create new records  When a record is created, only the creator/owner can modify/update it

79. Secure Dynamic Update  Updates done regularly by clients  once a day by default by DNS/DHCP Client  once a day by Netlogon  once a day by Cluster Service  Default TTL is 20 minutes  Disable DHCP dynamic updates  insecure!

80. Dynamic Update Primary DNS Secondary DNS Client DNS 1 3 2 SOA Update

81. Adjust A/PTR Record TTL

82. Dynamic Update and Replication DNS AD DNS 0 sec. 15-21 sec. 0-3 min. schedule

83. Dynamic Update and Replication

84. Speed up the refresh

85. DHCP and dynamic update  DHCP acts only on behalf of its clients  client must provide its name (anonymously)  Domain member computers since Windows 2000 do register themselves  DHCP registers only  workgroup computers, mobile phones  printers, scanners, network devices, crap…  Insecure, chaotic, unnecessary, corrupting

86. Disabling DHCP dynamic update

87. Dynamic DNS Update on RODC  Each writable DC returns itself as a primary DNS  RODC returns either (random) writable DC as the primary DNS

88. Dynamic DNS Update on RODC DNS ADRODC R/O DNS 0 sec. Client SOA Upd 1 2

89. Dynamic DNS Update on RODC DNS ADRODC R/O DNS 0 sec. 0-3 min. Client replicateSingleObject 0 sec. DsRemoteReplicationDelay default 30 sec.

90. DsRemoteReplicationDelay  Determines how long RODC's DNS server waits until it requests replication of the single object  Default = 30 sec.  Minimum = 5 sec.  Do not forget the DsPollingInterval

91. Time stamping/Aging  Record Created  timestamp trimmed to whole hour  No-refresh period starts  by default 7 days  timestamp does not change if the record does not change  Refresh period follows  by default next 7 days  timestamp gets updated at the first update

92. Scavenging  Server wide configuration  Should be done by only one DNS Server as best practice  By default ocurres only once per 7 days

93. DNS Aging and Scavenging  per-zone setting  implemented by all DNS servers  timestamp updates only during the refresh interval  limits replication traffic

94. DNS Aging and Scavenging  per-server setting  should be done only by one of the DNS servers

95. DNS Aging and Scavenging

96. DnsTombstoned = TRUE  Scavenged records remain in AD yet for another time DsTombStoneInterval before they are deleted from AD  default 7 days  checked and potentially deleted everyday at 2:00  Aimed to decrease replication traffic and limit DNT/USN exhaustion

97. DNS Best Practice DC1 DNS DC2 DNS AD

98. DNS Waiting for AD

99. DNS Best-Practice Reasons  Faster boot time without errors and timeouts  Deregistration at shutdown is recorded in live DNS Server  would have problems replicate if sent into shutting-down DC

100. Client DNS balancing  Clients do not balance DNS servers  queries/updates  use the first one always if possible  DHCP server does not use round robin  Configuration must be done “manually”  manual on servers  more DHCP scopes for clients

101. Client DNS non-balancing  Always alternate DNS server IP addresses

102. Client DNS non-balancing DNS1 DNS2 Client1 DNS1 DNS2 Client2 DNS1 DNS2 Client3 DNS1 DNS2

103. DNS Client Settings  HKLM\System\CurrentControlSet\Services\Tc pip\Parameters  Timetouts  DNSQueryTimeouts  Disjoint namespace on multihomed machines  DisjointNameSpace  PrioritizeRecordData  GPO – DNS Suffix appending on Vista+

104. DNS Server UDP Pool  After applying KB 953230, DNS Server reserves 2500 UDP ports  HKLM\System\CurrentControlSet\Services\D NS\Parameters  SocketPoolSize = DWORD = 2500  DNSCMD /Config /SocketPoolSize 2500

105. DNS Cache Pollution  rogue attacker's DNS server: idtt.com, 1.2.3.4  server: idtt.com authoritative DNS server  question: www.idtt.com, type A  answer: no records  authority answer:  idtt.com SOA  idtt.com NS a.gtld-servers.net  a.gtld-servers.net A 1.2.3.4  server: idtt.com authoritative DNS server  question: www.idtt.com, type A  answer: no records  authority answer:  microsoft.com NS ns.idtt.com  ns.idtt.com A 1.2.3.4  Enabled by default since 2000 SP3  SecureResponses

106. DNS Cache Locking  Further limits cache poisoning as already improved by the UDP pool  Records present in the cache cannot be updated before their TTL expires  prevents cache poisoning in some scenarios  frequently visited sites are already in the cache  Windows 2008 R2  enabled by default - 100%  CacheLockingPercent = DWORD = 0-100

107. Performance Considerations  MaxCacheTtl  maximum Ttl limit on cached RRs  by default 1 day maximum  MaxNegativeCacheTtl  by default 15 minutes

108. General Best Practice  More than 2 DNS servers are usually unnecessary for a site  Enable DNS Aging and Scavenging  may decrease DsPollingInterval  may shorten the client update refresh interval  Alter clients’ DNS settings to rotate the DNS server addresses  Disable DHCP dynamic update