ExpressRoute for Office 365 Training Troubleshooting Asymmetric Routes and Connectivity – Day 2 Session 7

Basic requirements Circuit is up Ensure that your prefixes are validated Route traffic to the nearest circuit Hot-potato routing Don’t try to guess SaaS endpoint location Longest prefix match (LPM) does work

Basic requirements (cont’d) For outbound (on-premises->Cloud) connectivity Use different (NAT) prefixes for internet and ExpressRoute If using NAT for ExpressRoute, use different NAT pools per circuit For inbound (Cloud->on-premises) connectivity Ensure that on-premises endpoints targeted by the cloud are available through more than one ExpressRoute circuit without causing path asymmetry

9/8/2018 7:56 AM Baseline Regardless of route, all connectivity to Office 365 services should work All services support being accessed across ExpressRoute if the traffic goes across ExpressRoute Overdeliver routes (Good?) vs. Underdeliver routes (Bad!!) http://aka.ms/o365endpoints How to read the “ExpressRoute for Office 365” column NO = not designed to go across ExpressRoute YES = designed to go across ExpressRoute © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Endpoints Article

Four common failure areas Circuit onboarding (Private/Public/Microsoft) Asymmetric routing (Microsoft) Most difficult to diagnose Prefix validation (Microsoft) General connectivity failures

Circuit onboarding failures Provisioning state CTag – both sides must match STag (VLAN) – both sides must match IP addresses MD5 hash AS number SKU – must be Premium for Microsoft peering

Service Provider State Azure ExpressRoute Status Provisioning state Service Provider State Azure ExpressRoute Status Functional? NotProvisioned Enabled NO Provisioning Provisioned YES Disabling

Missing ARPs One of the following is wrong: CTag/STag Customer is using the 2nd IP of the peer subnets Azure always takes the 2nd IP!

BGP session isn’t Active Typically Idle status, but could be cycling between states Inconsistent use of MD5 between Azure and on-premises Inconsistent ASN between Azure and on-premises Either network cannot handle the prefix count

How do I know it is working? Active BGP session ~600+ prefixes from Microsoft Psping to Office 365 resources

What is asymmetric routing? Asymmetric routing is when the traffic from network A enters network B at one point and exits network B going back to network A through a different waypoint This applies even to internal networks (DMZ vs. core, etc.) Although not technically incorrect (packets can get from A->B), most environments are configured such that asymmetrically routed packets end up getting dropped by firewalls, etc.

Example 1: Cloud to on-premises over the internet 9/8/2018 7:56 AM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution 1: Source NAT 9/8/2018 7:56 AM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Solution 2: Route Scoping

Example 2: Cloud to on-premises over ExpressRoute (two circuits) © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Inbound traffic at risk for asymmetric routing 9/8/2018 Inbound traffic at risk for asymmetric routing ADFS during password validation for sign-in Exchange Server Hybrid deployments Exchange Online mail to an on-premises host SharePoint Online Mail to an on-premises host SharePoint federated hybrid search SharePoint hybrid BCS. Skype for Business hybrid and/or Skype for Business federation Skype for Business Cloud Connector © 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Ways to identify asymmetric routing Pre-work Paper verification Real-time (not preferred, but sometimes necessary) Tracert, psping show ip route x.x.x.x in your core Should show same path out as you expect in

Paper verification Can you prove to yourself from a network diagram that you don’t have asymmetric routing? Can you prove it to someone else? [Ignore Blue, Red, and Green paths for this illustration]

Traceroute Should show you traversing the ExpressRoute circuit 172.22.8.18 is one side of the circuit

Router data ExpressRoute path Internet path 9/8/2018 7:56 AM © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Prefix validation issues Default logic does automatic approval Matches ASN ownership against IP address ownership Will fail if ASN owner != IP owner as per appropriate RIRs/IRRs

Sample manual RIR/IRR check 1.       Go to the WHOIS for the American Registry for Internet Numbers (ARIN) - https://whois.arin.net/ 2.       Example deep link for this client IP - https://whois.arin.net/rest/net/NET-191-0-0-0-0/pft?s=191.236.191.245 But because Net Type = Allocated To LACNIC and Organization = Latin American and Caribbean IP address Regional Registry (LACNIC), that means ARIN is not authoritative for this and you need to check LACNIC. (if you're curious, this is an example where ARIN is authoritative - Net Type = Direct Assignment, and Organization is the company name - https://whois.arin.net/rest/net/NET-40-74-0-0-1/pft?s=40.117.44.231) © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Sample manual RIR/IRR check (cont’d) 3.       So then you go to LACNIC's page - http://www.lacnic.net/ 4.       Example deep link to the WHOIS from LACNIC - https://rdap.lacnic.net/rdap-web/ip?key=191.236.191.245& Under REGISTRANT that Name is Microsoft Informatica Ltda

Validation Needed Engage Azure Support to get the prefixes manually verified Be prepared to provide proof of ownership chain (email thread works) Can (and should) be done ahead of time Azure Support can pre-validate the prefixes BGP peering creation will succeed at that point

Post-deployment issues 9/8/2018 7:56 AM Post-deployment issues Expected traffic isn’t going across ExpressRoute Proxy configuration © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Possible proxy server configuration

Post-deployment issues 9/8/2018 7:56 AM Post-deployment issues Expected traffic isn’t going across ExpressRoute Proxy configuration Unexpected traffic going across ExpressRoute (Overdeliver) This is OK! Just because your circuit is up doesn’t mean you are advertising the routes into your network For Private peering, telnet to 3389/22 of a gallery image Azure VM Public endpoints don’t work Advertising a default route will break public endpoints © 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Summary Basic Requirements Circuit Onboarding Failures ARIN Lookup Testing the Routes are Propagating Asymmetric Routing Prefix validation General Connectivity Failures

© 2016 Microsoft Corporation. All rights reserved © 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.