How the Microsoft Information Technology organization designed the corporate Exchange Server 2007 environment Published: November 2007 Exchange Server 2007 Design and Architecture at Microsoft

Agenda ● Solution overview ● Reasons for Microsoft IT to use Exchange Server 2007 ● Environment prior to Exchange Server 2007 ● Planning and design process ● Architecture and design decisions ● Deployment planning ● Best practices

The costs and general limitations associated with the platforms and technologies used in the Exchange Server 2003 environment prevented Microsoft IT from efficiently meeting emerging messaging and business needs. The costs and general limitations associated with the platforms and technologies used in the Exchange Server 2003 environment prevented Microsoft IT from efficiently meeting emerging messaging and business needs. Solution Overview Business Challenge Solution Results/Benefits With Exchange server 2007 Microsoft IT created new opportunities to drive down costs and system complexities, increase security, and deploy new features not available in previous versions of Exchange Server. With Exchange server 2007 Microsoft IT created new opportunities to drive down costs and system complexities, increase security, and deploy new features not available in previous versions of Exchange Server. Increased reliability. Increased reliability. Larger mailbox sizes. Larger mailbox sizes. Reduced total cost of ownership (TCO). Reduced total cost of ownership (TCO). Increased protection against spam. Increased protection against spam. Reduced topology complexities. Reduced topology complexities. improved regulatory compliance improved regulatory compliance Enhanced remote access and mobility options. Enhanced remote access and mobility options.

Reasons for Microsoft IT to use Exchange Server 2007 ● Increase employee productivity ● Increase operational efficiency ● Decrease security risks ● Decrease costs “Our mission is to deliver value by enabling people with innovative and reliable information technology solutions that seamlessly integrate with, and improve, how people work..” Jim DuBois General Manager, MSIT Microsoft Corporation

Environment Prior to Exchange Server 2007

Environment Prior to Exchange Server 2007 (Directory Infrastructure) ● Multiple forests for various legal and business requirements ● 70% of resources in Corporate forest – over 1 million objects ● 9 domains in corporate forest based on geography ● 202 sites in hub and spoke topology ● Dedicated Exchange site in Redmond

Environment Prior to Exchange Server 2007 (Directory Topology)

Environment Prior to Exchange Server 2007 (Messaging Topology) ● Centralized administration from Redmond ● Four administrative groups (North America, Dublin, Singapore, and Sao Paulo) ● Routing topology correspond to WAN links ● Routing group connectors between routing groups with default option ● Four central bridgehead servers in North America as remote bridgehead servers in the RGC configuration ● Inbound Internet mail messages through two redundant locations

Environment Prior to Exchange Server 2007 (Messaging Topology) Routing group Mailbox servers (clustered) Public-folder servers Bridge-head servers Front-end servers Gateway servers Special purpos e RG_REDMOND- EXCHANGE RG_DUBLIN62220 RG_SINGAPORE52220 RG_SAO PAULO 1200 RG_REDMOND PERIMETER RG_SILICON VALLEY PERIMETER

Planning and Design Process

Architecture and Design Decisions ● Administration and permissions model ● Message routing topology ● Server architectures and designs ● Mailbox storage design ● Backup and recovery ● Client access server topology ● Unified messaging ● Internet mail connectivity “Microsoft IT is our first and best customer. Almost two years prior to RTM, Microsoft IT began with pre-release production deployments to help us build an excellent product. The close relationship with Microsoft IT is so vital to our culture of quality and customer satisfaction that we do not ship products or service packs until Microsoft IT signs off on the enterprise readiness. We shipped Exchange Server 2007 on December 7, 2006, with the confidence and proof in hand that the product delivers on its potential to help customers build reliable enterprise-class messaging environments while reducing total cost of ownership.” Terry Myerson General Manager Exchange Server Product Group Microsoft Corporation

Administration and Permissions Model ● ● Security Principles and Guidelines ● Exclusive Microsoft IT Management ● Centralized System Administration ● Default Permissions Mode ● Formal Approval Process ● Permissions Review

Administration and Permissions Model (Approval Processes)

Message Routing Topology ● Network Infrastructure and Site Consolidation ● Dedicated Exchange Sites in the Active Directory Topology ● Optimized Message Transfer Between Hub Transport servers ● Connectivity to Remote SMTP domains ● Increased Message Routing security ● Coexistence with Exchange Server 2003

Message Routing Topology (Network Infrastructure and Site Consolidation) ● Physical network -> IP routing topology -> Active Directory site topology ● Previous consolidation With Exchange 2003 made planning easier ● Many benefits of consolidated datacenters ● ● Uncomplicated messaging topology ● ● Best possible Hub Transport server utilization ● ● Reduced chance of server communication issues

Message Routing Topology (Dedicated Exchange Sites in the Active Directory Topology)

Message Routing Topology (Optimized Message Transfer Between Hub Transport servers)

Message Routing Topology (Increased Message Routing Security) ● Messaging traffic encryption and lab environment exception ● Technologies used ● IPSec ● Transport layer (TLS) ● Restricted access to SMTP submission points ● Forefront Security on Hub Transport and Edge Transport

Message Routing Topology (Coexistence with Exchange Server 2003) ● Special routing group where Active Directory site topology defines the message routing topology

Message Routing Topology (Coexistence with Exchange Server 2003) Routing group connector Local bridgeheads Remote bridgeheads From RG_REDMOND to EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) Any local server can send mail over this connector. This enables all Exchange 2003 servers to transfer messages directly to the Hub Transport servers without involving Exchange 2003 bridgeheads. All Hub Transport servers located in ADSITE_REDMOND-EXCHANGE From EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) to RG_REDMOND All Hub Transport servers located in ADSITE_REDMOND-EXCHANGE. All Hub Transport servers located in RG_REDMOND From RG_DUBLIN to EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) Any local server can send mail over this connector. All Hub Transport servers located in ADSITE_DUBLIN From EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) to RG_DUBLIN All Hub Transport servers located in ADSITE_DUBLIN. The public-folder servers in RG_DUBLIN, which also function as bridgehead servers From RG_SINGAPORE to EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) Any local server can send mail over this connector. All Hub Transport servers located in ADSITE_SINGAPORE From EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) to the Singapore routing group All Hub Transport servers located in ADSITE_SINGAPORE. The public-folder servers in RG_SINGAPORE, which also function as bridgehead servers

Server Architectures and Designs ● Flexible and Scalable Messaging Infrastructure ● Multiple-Role and Single-Role Server Designs ● Scaling Up Server Designs

Server Architectures and Designs (Flexible and Scalable Messaging Infrastructure)

Server Architectures and Designs (Multiple-Role and Single-Role Server Designs) Server role Red- mond Silicon Valley Dublin Singa- pore Sao Paulo Technology Mailbox Microsoft Windows Clustering and CCR. Network interface card (NIC) teaming by using NICs connected to different switches Edge Transport Domain Name System (DNS) round robin and Mail Exchanger (MX) records with same cost values. Multiple Hub Transport servers as bridgeheads in Send Connector configuration Hub Transport Automatic load balancing through Mail Submission Service. Edge Subscriptions for Hub/Edge connectivity. Client Access Web Publishing Load Balancing (WPLB) on Microsoft Internet Security and Acceleration (ISA) Server Microsoft Network Load Balancing (NLB) internally. Unified Messaging 7022 Automatic round robin load balancing between Unified Messaging servers. Multiple voice over IP (VoIP) gateways per dial plan.

Server Architectures and Designs (Scaling Up Server Designs) ● New scaled-up Mailbox designs after initial rollout ● Up to 6000 users with 500 MB mailboxes ● Quad-core Intel Xeon with 16 GB RAM to eliminate bottleneck

Mailbox Storage Design ● Eliminating Storage as the Single Point of failure ● Reducing Storage Costs and Configuration Complexities ● Optimizing the Storage Design for Reliability and Recoverability ● Standardizing the Storage Design

Mailbox Storage Design (Eliminating Storage as the Single Point of Failure) ● CCR configuration with cluster nodes and the file- share witness in the same Active Directory site

Mailbox Storage Design (Optimizing the Storage Design for Reliability and Recoverability) ● CCR still requires reliability and recoverability provisions at storage and server levels ● Microsoft IT uses these strategies ● ● RAID ● ● Separate transaction logs from database files ● ● No circular logging on Mailbox servers ● ● Configure multiple storage groups per Mailbox server

Mailbox Storage Design ● Standardizing the Storage Design

Mailbox Storage Design ● 6000-user mailbox server with two USBBs per cluster node

Backup and Recovery ● Performing VSS-Based Backups on Passive Node ● Eliminating Backups to Tape ● Optimizing Backup Cycles According to SLAs

Backup and Recovery (Performing VSS- Based Backups on Passive Node) ● Software VSS backups on passive node with DPM

Backup and Recovery (Eliminating Backups to Tape) ● 14 days of online database backups

Backup and Recovery (Optimizing Backup Cycles According to SLAs) ● New 500 MB and 2 GB quotas would overtax existing backup processes ● Weekly full, daily incremental ● Seven storage groups on each LUN Storage group MonTueWedThuFriSatSun SG 1 FullIncIncIncIncIncInc SG 2 IncFullIncIncIncIncInc SG 3 IncIncFullIncIncIncInc SG 4 IncIncIncFullIncIncInc SG 5 IncIncIncIncFullIncInc SG 6 IncIncIncIncIncFullInc SG 7 IncIncIncIncIncIncFull

Client Access Server Topology ● Preserving Existing Namespaces for Mobile Access to Messaging Data ● Increasing Security Based on ISA Server 2006 ● Providing Load Balancing and Fault Tolerance for External Client Connections ● Providing Load Balancing and Fault Tolerance for Internal Client Connections ● Optimizing Offline Address Book Distribution ● Enabling Cross-Forest Availability Lookups

Client Access Server Topology (Preserving Existing Namespaces for Mobile Access to Messaging Data) ● 60,000 Outlook Web Access unique users per month and 30,000 ActiveSync sessions ● Existing Multiple URL namespaces to distribute load that need to be preserved with Exchange 2007 ● Deploy Client Access servers, verify, then migrate users ● Each Active Directory site with Mailbox servers must also include Client Access servers ● Redirect Office Outlook Web Access users to Client Access servers that are local to the user’s Mailbox server via ExternalURL property ● Client Access servers act as proxy servers for local Client Access servers (Exchange ActiveSync, Exchange Web Services)

Client Access Server Topology (Increasing Security Based on ISA Server 2006) ● Stateful inspection and application-layer filtering ● Blocks any traffic that appears out of context, such as requests to initiate a connection on an established session ● SSL bridging process enables ISA Server 2006 to filter invalid data packets before the traffic reaches the Client Access servers ● Externally trusted SSL certificates for both external and internal traffic

Client Access Server Topology (Providing Load Balancing and Fault Tolerance for External Client Connections)

Client Access Server Topology (Providing Load Balancing and Fault Tolerance for Internal Client Connections)

Client Access Server Topology (Optimizing Offline Address Book Distribution)

Client Access Server Topology (Enabling Cross-Forest Availability Lookups)

Unified Messaging (Topology)

Unified Messaging (Redundancy and Load Balancing)

Unified Messaging (Security) ● Many possible security issues: SIP Proxy impersonation, session hijacking, sniffing, etc ● Secure protocols such as MTLS can mitigate risk ● Trusted LANs, VLANs, and other methods of segmentation ● IPSec ● General practices such as strong password

Unified Messaging (Feature and User Considerations) ● Some settings and features with default values, some customized ● Need to customize dial plans, VoIP gateway partners, hunt groups, mailbox policies, etc. ● Need to inform users of changes and provide documentation for self-service ● Microsoft created custom templates ● Custom intranet site with documentation for usage and user self-service

Internet Mail Connectivity ● Inbound and Outbound Message Transfer ● Redundancy and Load Balancing ● Increasing Perimeter Network Security ● Server Hardening ● Optimizing Spam and Virus Scanning ● Optimizing Outbound Message Transfer

Internet Mail Connectivity (Inbound and Outbound Message Transfer)

Internet Mail Connectivity (Redundancy and Load Balancing) ● Multiple Hub Transport servers with Edge Transport servers ● All Hub Transport servers transfer outbound messages to local Edge Transport servers ● Edge Transport servers can transfer inbound messages to Hub Transport servers ● For inbound messages, DNS round-robin and MX records with preference value of 10 ● Edge Transport servers in Europe and North America

Internet Mail Connectivity (Increasing Perimeter Network Security)

Internet Mail Connectivity (Server Hardening) ● Ports ● Services ● File Shares ● Accounts ● Security updates

Internet Mail Connectivity (Optimizing Spam and Virus Scanning) ● ● Connection-filtering configuration ● IP block-list, IP allow-list providers, and Sender Reputation Level ● ● Recipient-filtering configuration ● ● Content-filtering configuration ● ● Store SCL: 5 ● ● Reject SCL:7 ● ● No delete or quarantine SCL ● ● Attachment-filtering configuration with Forefront Security

Internet Mail Connectivity (Optimizing Outbound Message Transfer) ● Built-in protection on SMTP connectors, including header firewall, tarpitting, backpressure, etc ● One receive connector that faces the Internet and one send connector for transferring incoming e- mail to Hub Transport servers ● One receive connector faces Hub Transport servers for outbound messages ● Three send connectors for relaying outbound messages to Internet hosts

Deployment Planning ● Introducing Exchange Server 2007 into the Corporate Production Environment ● Verifying the Successful Integration of Exchange Server 2007 ● Fully Deploying Client Access Servers in North America ● Fully Deploying Hub Transport Servers in North America ● Deploying Mailbox Servers in North America ● Introducing Edge Transport Servers in North America ● Deploying Forefront Security for Exchange Server 2007 ● Deploying Exchange Server 2007 in Regional Data Centers ● Switching the Messaging Backbone to Exchange Server 2007 ● Completing the Transition to Exchange Server 2007

Deployment Planning (Fully Deploying Client Access Servers in North America)

Deployment Planning (Fully Deploying Servers in North America) ● Hub Transport role including SMTP connectors ● Mailbox role and user migration – at least 16,000 mailboxes before other deployment tasks ● Edge Transport coexistence and replacement ● Forefront Security

Planning and Design Best Practices ● ● Clearly define goals ● ● Design for production in mind ● ● Design for peak load days ● ● Test in lab environment ● ● Identify key risks ● ● Develop rollback and mitigation procedures

Server Design Best Practices ● ● Use multiple-core processors and design storage based on both capacity and I/O performance ● ● Use VSS-based backup ● ● Eliminate single points of failure

Deployment Best Practices ● ● Establish flexible and scalable messaging infrastructure ● ● Carefully plan URL namespaces ● ● Manage permissions through security groups ● ● Use fewest permissions necessary ● ● Use Forefront and multiple layers of protection ● ● Place Edge Transport servers in a perimeter network ● ● Use ISA Server 2006 to publish Client Access servers

Summary ● Messaging environment hosts 130,000-plus mailboxes with 500 MB and 2 GB quotas in 4 datacenters on 62 Mailbox servers ● 25 Client Access, 15 Hub Transport, 10 Edge, and 11 Unified Messaging servers ● Many cost and reductions with transition to Exchange Server 2007 ● Migration from SAN to DAS storage with Exchange Server 2007 ● USBBs enable scaling up Mailbox servers ● Eliminated single points of failure ● Increased security and better filtering

For More Information ● Additional content on Microsoft IT deployments and best practices can be found on ● Microsoft IT Showcase Webcasts ebcasts ebcasts ebcasts ● Microsoft TechNet

This document is provided for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. © 2006 Microsoft Corporation. All rights reserved. This technical white paper presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Microsoft Press, Active Directory, ActiveSync, Forefront, Outlook, Windows, and Windows Server are either registered are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.