1. ExpressRoute for Office 365 Training
Troubleshooting Asymmetric Routes and Connectivity – Day 2 Session 7

2. Basic requirements Circuit is up
Ensure that your prefixes are validated
Route traffic to the nearest circuit
Hot-potato routing
Don’t try to guess SaaS endpoint location
Longest prefix match (LPM) does work

3. Basic requirements (cont’d)
For outbound (on-premises->Cloud) connectivity
Use different (NAT) prefixes for internet and ExpressRoute
If using NAT for ExpressRoute, use different NAT pools per circuit
For inbound (Cloud->on-premises) connectivity
Ensure that on-premises endpoints targeted by the cloud are available through more than one ExpressRoute circuit without causing path asymmetry

4. 9/8/2018 7:56 AM
Baseline
Regardless of route, all connectivity to Office 365 services should work
All services support being accessed across ExpressRoute if the traffic goes across ExpressRoute
Overdeliver routes (Good?) vs. Underdeliver routes (Bad!!)
How to read the “ExpressRoute for Office 365” column
NO = not designed to go across ExpressRoute
YES = designed to go across ExpressRoute
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Endpoints Article

6. Four common failure areas
Circuit onboarding (Private/Public/Microsoft)
Asymmetric routing (Microsoft)
Most difficult to diagnose
Prefix validation (Microsoft)
General connectivity failures

7. Circuit onboarding failures
Provisioning state
CTag – both sides must match
STag (VLAN) – both sides must match
IP addresses
MD5 hash
AS number
SKU – must be Premium for Microsoft peering

8. Service Provider State Azure ExpressRoute Status
Provisioning state
Service Provider State
Azure ExpressRoute Status
Functional?
NotProvisioned
Enabled NO
Provisioning
Provisioned
YES
Disabling

9. Missing ARPs One of the following is wrong: CTag/STag
Customer is using the 2nd IP of the peer subnets
Azure always takes the 2nd IP!

10. BGP session isn’t Active
Typically Idle status, but could be cycling between states
Inconsistent use of MD5 between Azure and on-premises
Inconsistent ASN between Azure and on-premises
Either network cannot handle the prefix count

11. How do I know it is working?
Active BGP session
~600+ prefixes from Microsoft
Psping to Office 365 resources

12. What is asymmetric routing?
Asymmetric routing is when the traffic from network A enters network B at one point and exits network B going back to network A through a different waypoint
This applies even to internal networks (DMZ vs. core, etc.)
Although not technically incorrect (packets can get from A->B), most environments are configured such that asymmetrically routed packets end up getting dropped by firewalls, etc.

13. Example 1: Cloud to on-premises over the internet 9/8/2018 7:56 AM
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Solution 1: Source NAT 9/8/2018 7:56 AM
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Solution 2: Route Scoping

16. Example 2: Cloud to on-premises over ExpressRoute (two circuits)
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. Inbound traffic at risk for asymmetric routing
9/8/2018
Inbound traffic at risk for asymmetric routing
ADFS during password validation for sign-in
Exchange Server Hybrid deployments
Exchange Online mail to an on-premises host
SharePoint Online Mail to an on-premises host
SharePoint federated hybrid search
SharePoint hybrid BCS.
Skype for Business hybrid and/or Skype for Business federation
Skype for Business Cloud Connector
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Ways to identify asymmetric routing
Pre-work
Paper verification
Real-time (not preferred, but sometimes necessary)
Tracert, psping
show ip route x.x.x.x in your core
Should show same path out as you expect in

19. Paper verification
Can you prove to yourself from a network diagram that you don’t have asymmetric routing?
Can you prove it to someone else?
[Ignore Blue, Red, and Green paths for this illustration]

20. Traceroute Should show you traversing the ExpressRoute circuit
is one side of the circuit

21. Router data ExpressRoute path Internet path 9/8/2018 7:56 AM
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Prefix validation issues
Default logic does automatic approval
Matches ASN ownership against IP address ownership
Will fail if ASN owner != IP owner as per appropriate RIRs/IRRs

23. Sample manual RIR/IRR check
1.       Go to the WHOIS for the American Registry for Internet Numbers (ARIN) -
2.       Example deep link for this client IP -
But because Net Type = Allocated To LACNIC and Organization = Latin American and Caribbean IP address Regional Registry (LACNIC), that means ARIN is not authoritative for this and you need to check LACNIC.
(if you're curious, this is an example where ARIN is authoritative - Net Type = Direct Assignment, and Organization is the company name -
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Sample manual RIR/IRR check (cont’d)
3.       So then you go to LACNIC's page -
4.       Example deep link to the WHOIS from LACNIC -
Under REGISTRANT that Name is Microsoft Informatica Ltda

25. Validation Needed
Engage Azure Support to get the prefixes manually verified
Be prepared to provide proof of ownership chain ( thread works)
Can (and should) be done ahead of time
Azure Support can pre-validate the prefixes
BGP peering creation will succeed at that point

26. Post-deployment issues
9/8/2018 7:56 AM
Post-deployment issues
Expected traffic isn’t going across ExpressRoute
Proxy configuration
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Possible proxy server configuration

28. Post-deployment issues
9/8/2018 7:56 AM
Post-deployment issues
Expected traffic isn’t going across ExpressRoute
Proxy configuration
Unexpected traffic going across ExpressRoute (Overdeliver)
This is OK!
Just because your circuit is up doesn’t mean you are advertising the routes into your network
For Private peering, telnet to 3389/22 of a gallery image Azure VM
Public endpoints don’t work
Advertising a default route will break public endpoints
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Summary Basic Requirements Circuit Onboarding Failures ARIN Lookup
Testing the Routes are Propagating
Asymmetric Routing
Prefix validation
General Connectivity Failures

30. © 2016 Microsoft Corporation. All rights reserved
© 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.