1. Office 365 ExpressRoute Planning Integration with LANs
Day 2 Session 6

2. Office 365 Connectivity Options for Enterprise Customers
Choosing the right connectivity option is key to obtaining optimal performance levels to Office 365 services
Two main options:
Proxied (Either on Premises or Cloud based)
Direct Routing (ExpressRoute fits this method)
If ExpressRoute is chosen, an internet route is still required

3. Customer Connection Options
12/7/2017
Customer Connection Options
Please make diagram conform to TR Brand and palette, make lines wider etc
Customer LAN/WAN
Public Internet
Proxy
Customer LAN/WAN
Direct (NAT/PAT)
Customer LAN/WAN
ExpressRoute
Microsoft Global Network
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. ExpressRoute Locations
Design decisions around ExpressRoute Circuit/s
Location - Latency is key, is a single circuit going to cause higher latency than local egress? Also consider where your chosen provider has ExpressRoute locations
Multiple Circuits? Is this required? Where is most suitable to connect these?
Sizing – How much Bandwidth is required?
All locations to use ExpressRoute? Or some sites to use Internet routing? Do all services need to go via ExpressRoute?

5. TechReady 23 12/7/2017 7:17 AM SEA SEA1 YQB YMQ YYZ ORD BOS ASE DSM
JFK
DEN
BWI
COS
SFO
LAX1
CLT
OKC
LAX
ATL
PHX
DFW
SAT
HOU
MIA
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. TechReady 23 12/7/2017 7:17 AM SEA SEA1 YQB YMQ YYZ ORD BOS ASE DSM
JFK
DEN
BWI
COS
SFO
LAX1
CLT
OKC
LAX
ATL
PHX
DFW
SAT
HOU
MIA
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Risks
Latency from global locations may be higher than if a local egress was used
Bandwidth - Does the LAN/WAN have the capacity to handle all the Office 365 services?
Cost – Does the customer pay for the amount of data transferred?
Geo DNS – If DNS is resolved locally in each region this will cause performance issues

8. Possible solutions This is where a network assessment pays dividends
Is ExpressRoute the right solution here?
Would using ExpressRoute in multiple locations be a better design?
Would a localised egress in each location provide better performance?
If a single ExpressRoute circuit is the only option, using local egresses in Non-NA sites would likely give the best overall performance
Sending Skype over the ExpressRoute and other services via local egresses may be an possible solution

9. Handling Internet Routed Traffic

10. Internet Egress for non ExpressRoute traffic
What still needs the internet?
DNS Name Resolution
CRL Checks
CDN data
Yammer
Office Video
Office 365 ProPlus client downloads
On-Prem Identity Provider Sign-In eDiscovery
Design normally based on existing internet egress method
Important this route is not missed in planning for scale, latency and configuration

11. Proxied Connection Normally the easiest method to implement
All non Expressroute traffic sent to Proxy IP address
Key to remember, non ExpressRoutable elements can still cause performance problems if not optimized
Localised Proxies are advised where possible
Cloud Based Proxies may be an option where local on-premises proxies aren’t possible

12. Example PAC File
This example sends ER traffic direct and non ER routable traffic to a proxy
Downloadable from “Managing Office 365 Endpoints” page
//EXPRESS ROUTE DIRECT
if (
dnsDomainIs(host, "lync.com")||
dnsDomainIs(host, "microsoftonline.com")||
dnsDomainIs(host, "officeapps.live.com")||
dnsDomainIs(host, "outlook.office.com")||
dnsDomainIs(host, "protection.outlook.com")||
dnsDomainIs(host, "sharepoint.com")||
dnsDomainIs(host, "adminwebservice.microsoftonline.com")||
dnsDomainIs(host, "agent.office.net")||
dnsDomainIs(host, "clientconfig.microsoftonline-p.net")||
dnsDomainIs(host, "domains.live.com")||
dnsDomainIs(host, "hip.microsoftonline-p.net")||
dnsDomainIs(host, "home.office.com")||
dnsDomainIs(host, "login.microsoftonline.com")||
dnsDomainIs(host, "login.windows.net")||
dnsDomainIs(host, "outlook.office365.com")||
dnsDomainIs(host, "portal.office.com")||
dnsDomainIs(host, "provisioningapi.microsoftonline.com")||
dnsDomainIs(host, "smtp.office365.com")||
dnsDomainIs(host, "
{return "DIRECT";}
//All other traffic to proxy
else {return "PROXY :8080";}

13. Direct Routing of Internet Traffic
If this method is used, then a routing decision needs to be taken to separate it from the ExpressRoute traffic (in most cases)
Example scenarios coming up

14. ExpressRoute Design Options

15. ExpressRoute Options Three Main options Routing Override at the Edge
Proxy Level Override
BGP Propagation Design

16. Non BGP Propagation Design
If a customer wishes to use ExpressRoute without managing BGP throughout the managed network, there are a number of solutions
All internet traffic is sent to a network device which makes the routing override
Normally these methods require the lowest amount of internal network configuration to implement ExpressRoute

17. Routing Override at edge
12/7/2017
Routing Override at edge
Public Internet
Please make diagram conform to TR Brand and palette, make lines wider etc
Border Internet Router/NAT- PAT
Customer Site A
Edge Router
Microsoft Global Network
NAT/PAT
ExpressRoute
= BGP Route Propagation
= Internal Corporate Network
Customer Site B
= Internet Traffic
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. Routing Override at edge – Multiple Circuits
12/7/2017
Routing Override at edge – Multiple Circuits
Please make diagram conform to TR Brand and palette, make lines wider etc
Internal Router
NA Edge Router
Chicago ExpressRoute
Microsoft Global Network –Chicago IX
NAT/PAT
Pool 1
Customer North America Site
Border Internet Router/N AT-PAT
Public Internet
Internal
Router
Border Internet Router/NAT-PAT
Microsoft Global Network –London IX
EMEA Edge Router
NAT/PAT Pool 2
London ExpressRoute
= North America Corporate Network
Customer EMEA site
= BGP Route Propagation
= EMEA Corporate Network
= Internet Traffic
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Routing Override at edge
TechReady 23
12/7/2017 7:17 AM
Routing Override at edge
Client sends connection to actual Office 365 IP/Port
No need to handle Microsoft BGP routes internally
Internal routing tables stay small
Routing override is handled on edge router via it’s knowledge of the ExpressRoute BGP information
Internet traffic is sent via a separate internet egress
Lower level of internal implementation requirements
Have to be able to route public IPs to the edge router
All traffic can be sent direct, no need for client proxy configuration for Office 365 traffic
Use BGP routes and not URL & IP page for route management
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. Proxy level override 12/7/2017 Public Internet
Please make diagram conform to TR Brand and palette, make lines wider etc
Internet Proxy
Customer Site A
Microsoft Global Network
Office 365 ExpressRoute Proxy
Internal Router
ExpressRoute
Edge Router
= BGP Route Propagation
= Internal Corporate Network
Customer Site B
= Internet Traffic
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. Proxy level override 12/7/2017 Public Internet Public Internet
NAM Internet Proxy
Public Internet
Please make diagram conform to TR Brand and palette, make lines wider etc
Microsoft Global Network
NAM Office 365 ExpressRoute Proxy
ExpressRoute Chicago
Internal Router
Customer USA Site
Edge Router
= BGP Route Propagation
Public Internet
= Internal Corporate Network
EMEA Internet Proxy
= Internet Traffic
Microsoft Global Network
EMEA Office 365 ExpressRoute Proxy
ExpressRoute London
Internal Router
Customer EMEA site
Edge Router
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Proxy Level Routing Override
No need to handle external IPs on internal routing tables
Internal routing tables stay small
Meets some customers requirements for proxying all exiting traffic or where unable/unwilling to carry public IPs on the internal network
Easy management of multiple ER locations by using different proxy IP per ExpressRoute circuit
Proxies can be a bottleneck and not recommended for Skype for Business
PAC file needs regular maintenance

23. Proxy level override– Client Configuration
PAC file needs to be configured to split ExpressRoute traffic from Internet
URL & IP List should be used to create and maintain this file
Lack of maintenance may mean some traffic flows via the internet egress and not ExpressRoute
If Proxy allows only explicit access, then connectivity may fail if the PAC is not maintained
Most configurations only need the ExpressRoute traffic explicitly defining in the PAC file
Office 365 traffic will pick up the pac file and use it
//EXPRESS ROUTE PROXY TRAFFIC
if (
dnsDomainIs(host, "lync.com")||
dnsDomainIs(host, "microsoftonline.com")||
dnsDomainIs(host, "officeapps.live.com")||
dnsDomainIs(host, "outlook.office.com")||
dnsDomainIs(host, "protection.outlook.com")||
dnsDomainIs(host, "sharepoint.com")||
dnsDomainIs(host, "adminwebservice.microsoftonline.com")||
dnsDomainIs(host, "agent.office.net")||
dnsDomainIs(host, "clientconfig.microsoftonline-p.net")||
dnsDomainIs(host, "domains.live.com")||
dnsDomainIs(host, "hip.microsoftonline-p.net")||
dnsDomainIs(host, "home.office.com")||
dnsDomainIs(host, "login.microsoftonline.com")||
dnsDomainIs(host, "login.windows.net")||
dnsDomainIs(host, "outlook.office365.com")||
dnsDomainIs(host, "portal.office.com")||
dnsDomainIs(host, "provisioningapi.microsoftonline.com")||
dnsDomainIs(host, "smtp.office365.com")||
dnsDomainIs(host, "
{return “PROXY :80";}
//All other traffic to internet proxy
else {return "PROXY :8080";}

24. Internal BGP Route Propagation
12/7/2017
Internal BGP Route Propagation
Public Internet
Please make diagram conform to TR Brand and palette, make lines wider etc
Border Internet Router/NAT- PAT
Customer Site A
Microsoft Global Network
NAT/PAT
Edge Router
ExpressRoute
= eBGP Route Propagation
= IBGP Route Propagation
Customer Site B
= Internet Traffic
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Multi Site - Internal BGP Route Propagation
12/7/2017
Multi Site - Internal BGP Route Propagation
Chicago ExpressRoute
Please make diagram conform to TR Brand and palette, make lines wider etc
Microsoft Global Network
NAT/PAT Pool 4
Customer NAM Site
Edge Router
Border Internet Router/NAT- PAT Pool 1
= eBGP Route Propagation
= Internet Traffic
Public Internet
Border Internet Router/NAT- PAT pool 2
= NAM iBGP Route Propagation
= EMEA iBGP Route Propagation
NAT/PAT Pool 3
Edge Router
London ExpressRoute
Microsoft Global Network
Customer EMEA Site
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. Internal Routing model
Allowing eBGP (External BGP) routes to be learnt internally (iBGP)
How far do you let the routing propagate?
All the way in? Use BGP Communities to manage
Some management of route costing is required on more complex networks or where multiple ExpressRoute Circuits are available
Location based BGP community information is one method.
Manage at the edge, use edge router/Default Gateway to send traffic to ExpressRoute or internet egress
Asymmetric Routing – Ensure one route has priority especially for inbound services

27. Internal Route Propagation Design
Route table sizes. ExpressRoute may add 600+
If routers cannot handle this increase (or the customer does not want to have all routes)
Upgrade Routers
Use BGP Communities to limit routes to used services
Use one of the other methods outlined above instead

28. Notes on Inbound traffic
Mapping of inbound Services (Covered in next sessions)
ADFS during password validation for sign-in
Exchange Server Hybrid deployments
Mail from an Exchange Online tenant to an on-premises host
SharePoint Online Mail send from SharePoint Online to an on-premises host
SharePoint federated hybrid search
SharePoint hybrid BCS
Skype for Business hybrid and/or Skype for Business federation
Skype for Business Cloud Connector

29. Advice for all connection methods
SNAT recommended for incoming connections – helps with symmetric routing by ensuring response goes back to the SNAT device
Outbound – Separate NAT pool each ExpressRoute circuit and Internet egress to reduce risk of Asymmetric routes
NAT pool advertised to Microsoft must not be advertised to the internet
SMTP Mail inbound has to be over the internet
Ensure all egresses are well scaled to cope with peak load
Any devices/methods using the URL&IP list should be kept fully updated
Customers may want to restrict the traffic coming in from ExpressRoute
Don’t use the URL&IP list to restrict traffic
Use BGP Communities to restrict traffic the Office 365 communities

30. Scenario
Contoso wish to use BGP Filters to control inbound traffic and use the URL & IP list to configure this and drop any traffic which is not matched. Contoso do this to ensure all routes originating from Microsoft actually belong to Microsoft. Also wish to ensure a possible configuration issue on the Microsoft side with an excessive number of routes.
Problem:
This will cause outages as on occasion this page is not updated in the day notice period. Also ranges which are non customer impacting (such as new /32 within existing /24) or the addition of a subdomain may take time to be included. Customers should not create access policies for ExpressRoute using this method.

31. Solution – Two Parts Firewall Security Router Protection
To proactively permit traffic to Office 365 via a firewall, use DNS based filtering. DNS records update much less frequently than Ips. Another option is to restrict on Microsoft’s entire IPV4 range
Router Protection
To protect against flooding use the max-prefix property on routers
Restrict on BGP communities to suit services being used

32. GEO DNS & ExpressRoute
Care should be taken to ensure if DNS is resolved locally, a local egress is used for Exchange traffic
Alternatively, resolve DNS where the egress is to ensure a local datacentre to the egress is reached
Scenario – Contoso has a single ExpressRoute circuit in North America where their Tenant is located. European sites use this but also have local DNS resolution.
Outcome – High latency due to EMEA CAS nodes being specified, traffic flows from EMEA > ExpressRoute egress in NA > EMEA CAS node > Mailbox in NA

33. Geo DNS for Exchange
Microsoft DNS
If Outlook client is in the same region as the Tenant, then we connect direct to it 3
Microsoft’s DNS servers return the IP addresses of the regional datacenter
North America Datacenters
Client’s DNS asks the Microsoft DNS Server 2
Portal
EXO
MBX
Customer tenant in US
EXO
CAS
Client’s DNS 5
Exchange Online accesses the datacenter where the tenant resides and proxies the requests
The client asks the local DNS Servers 1
The user accesses the regional datacenter 4
© 2014 Microsoft Corporation

34. Geo DNS for Exchange
DNS Call returns an IP address of a Datacenter local to the user’s location
Outlook connects to that and the data is backhauled over the Fibre network between the tenant location and the local Datacenter
Result is a much fast connection for the client and data stays in tenant location.
Microsoft DNS 3
Microsoft’s DNS servers return the IP addresses of the regional datacenter
North America Datacenters
Client’s DNS asks the Microsoft DNS Server 2
Exchange Online accesses the datacenter where the tenant resides and proxies the requests
Customer tenant in US
EXO
MBX
Client’s DNS 5
The client asks the local DNS Servers 1
EU Datacenters
Portal
The user accesses the regional datacenter 4
EXO
CAS
© 2014 Microsoft Corporation

35. Egress Point Location issues
12/7/2017
Egress Point Location issues
Exchange Online uses GEO DNS
You get a different IP Address from the DNS depending on where in the world you request it
Impacts a multi-country corporate network with multiple Internet connection points
Commonly DNS is only requested at one point and cached
You can get DNS from another part of the globe to where you have Internet connectivity
Internet
egress point
Microsoft datacenter
DNS call
Customer network
Data transfer
MICROSOFT CONFIDENTIAL—NDA ONLY
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

36. Summary Connectivity options Connection locations
Options for Handling Internet bound traffic
ExpressRoute internal Routing options
Routing Override at the edge
Proxy Level Override
BGP Route Propagation
General Advice for all connection options
Geo DNS

37. © 2016 Microsoft Corporation. All rights reserved
© 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.