MCSE Guide to Microsoft Exchange Server 2003 Administration Chapter 13 Monitoring and Troubleshooting the Server

MCSE Guide to Microsoft Exchange Server 2003 Administration 2 Objectives Understand the tools available for monitoring the health of the Exchange Server 2003 environment Understand the different steps used by nightly online maintenance to maintain server Information Stores Describe the maintenance tools used for maintaining, troubleshooting, and fixing an Exchange 2003 server Understand the process involved in troubleshooting server issues

MCSE Guide to Microsoft Exchange Server 2003 Administration 3 Monitoring and Troubleshooting the Server Monitor infrastructure for successful operation –Spot potential problems before they occur –Respond quickly to problems that do occur –Identify trends in usage patterns What to monitor: connections, services, servers, resource usage Use Exchange Server 2003 tools to aid monitoring Skills for troubleshooting Exchange refined by experience

MCSE Guide to Microsoft Exchange Server 2003 Administration 4 Monitoring Monitoring tools: server and link monitors Server monitors: –Check the status of designated services –Check the usage of particular resources Link (connection status) monitors: –Check status of connection between two servers –Status for link monitor is either available or unavailable –Special configuration is unneeded (unlike server monitors) Configure monitors and view status with Exchange System Manager

MCSE Guide to Microsoft Exchange Server 2003 Administration 5 Activity 13-1: Viewing Monitors Time Required: 10 to 15 minutes Objective: View the status of existing monitors Description: View the list of servers within the organization and determine their current status to ensure that they are operating. In addition, you also verify that the connectors you have established between servers are available to transmit messages.

MCSE Guide to Microsoft Exchange Server 2003 Administration 6

7 Monitoring (continued) Server monitors provide status of six resources: –Windows 2000 service –SMTP queue growth –X.400 queue growth –CPU utilization –Free disk space –Available virtual memory Warning or critical event generated if monitored service stops

MCSE Guide to Microsoft Exchange Server 2003 Administration 8 Monitoring (continued) Preconfigured services installed for default monitoring: –Microsoft Exchange Information Store –Microsoft Exchange MTA Stacks –Microsoft Exchange Routing Engine –Microsoft Exchange System Attendant –Simple Mail Transport Protocol (SMTP) –World Wide Web Publishing Service Additional services help monitor overall functionality –Examples: antivirus service or backup agent

MCSE Guide to Microsoft Exchange Server 2003 Administration 9 Activity 13-2: Configuring Service Monitoring Time Required: 15 to 20 minutes Objective: Configure service monitoring on your server Description: Add and configure services to be monitored on your back-end server. You also configure notifications that allow you to be notified when problems occur on any server in your organization. You can configure two types of notifications: an notification or a script notification.

MCSE Guide to Microsoft Exchange Server 2003 Administration 10

MCSE Guide to Microsoft Exchange Server 2003 Administration 11 Monitoring (continued) Continual queue growth indicates a potential problem in system –Message inflow is greater than message outflow –Network may have systemic issues Two server monitors assist with monitoring queues: –X.400 queue monitor –SMTP queue monitor Configuring queue monitoring: –Specify period of acceptable continual queue growth –Trigger warning or critical state if period is exceeded

MCSE Guide to Microsoft Exchange Server 2003 Administration 12 Activity 13-3: Configuring SMTP Queue Monitoring Time Required: 10 to 20 minutes Objective: Configure queue monitoring to monitor for sustained queue growth Description: Configure message queue monitoring on the server to watch for sustained queue growth for extended periods of time. This can be an indicator of network or system problems that might need attention.

MCSE Guide to Microsoft Exchange Server 2003 Administration 13

MCSE Guide to Microsoft Exchange Server 2003 Administration 14 Monitoring (continued) High CPU utilization may indicate several issues –Insufficient processing power –Too many services running on machine Remedying problems is related to CPU utilization –If CPU utilization runs at 100% for an extended period, reboot server –Offload services to another machine Configuring CPU monitoring: –Specify period CPU may run above a certain level –Trigger warning or critical state if period is exceeded

MCSE Guide to Microsoft Exchange Server 2003 Administration 15 Activity 13-4: Configuring CPU Utilization Monitoring Time Required: 10 to 20 minutes Objective: Configure CPU utilization monitoring to monitor for sustained high CPU usage on a server Description: Configure CPU utilization monitoring on the server to monitor the CPU usage on the back-end server. When CPU utilization is too high, the server is unable to process messages effectively or manage other critical functions.

MCSE Guide to Microsoft Exchange Server 2003 Administration 16

MCSE Guide to Microsoft Exchange Server 2003 Administration 17 Monitoring (continued) Disk space uses: –Data storage –Logging –Tracking –Virtual memory Exchange shuts down services if disk space is lacking Closely monitor free disk space to avoid shutdown Responding to notification of diminishing space: –Replace undersized disk drives –Free up space on existing drives –Investigate log usage

MCSE Guide to Microsoft Exchange Server 2003 Administration 18 Activity 13-5: Configuring Free Disk Space Monitoring Time Required: 10 to 20 minutes Objective: Configure disk space monitoring to monitor the free space on your server Description: Configure disk space monitoring on the back-end server to monitor the disk space usage on the server. When disk space runs out, the server can begin to malfunction because it has no place to store the information.

MCSE Guide to Microsoft Exchange Server 2003 Administration 19

MCSE Guide to Microsoft Exchange Server 2003 Administration 20 Monitoring (continued) Brief notes on virtual memory used by Windows –OS perceives memory capacity above physical amount –Virtual memory takes the form of page (swap) file Diminishing disk space indicates memory issues –Determine if server dips below certain usage level –Determine how long server remains below threshold Configuring virtual memory monitoring: –Specify period available virtual memory can remain below threshold –Trigger warning or critical state if period is exceeded

MCSE Guide to Microsoft Exchange Server 2003 Administration 21 Activity 13-6: Configuring Virtual Memory Monitoring Time Required: 10 to 20 minutes Objective: Configure virtual memory monitoring to monitor the available virtual memory on your server Description: Configure virtual memory monitoring on the back-end server to monitor the virtual memory usage on the server. Because Exchange is extremely memory intensive, it is a good idea to keep an eye on available virtual memory. This helps you see if the server is using memory efficiently.

MCSE Guide to Microsoft Exchange Server 2003 Administration 22

MCSE Guide to Microsoft Exchange Server 2003 Administration 23 Preventive Maintenance Run maintenance tasks to reduce chance of problems Example: database consistency and cleanup –Maintenance tasks run by server on a nightly basis Validates integrity of Information Store Ensures unneeded items are removed from database

MCSE Guide to Microsoft Exchange Server 2003 Administration 24 Activity 13-7: Configuring the Maintenance Interval for Exchange Databases Time Required: 5 to 10 minutes Objective: Configure the maintenance interval for the Exchange databases on your server Description: Configure the time at which the maintenance routines run against the databases on your back-end server. By default, Exchange Server 2003 runs maintenance tasks daily from 1:00 a.m. until 5:00 a.m. This activity changes the interval to 11:00 p.m. until 3:00 a.m.

MCSE Guide to Microsoft Exchange Server 2003 Administration 25 Information Store Maintenance Information Store maintenance –Can be configured to run at specific times –Tasks not completed are carried over to the next run

MCSE Guide to Microsoft Exchange Server 2003 Administration 26 Purge Indexes Indexes are used to expedite a search within the store Indexes are created dynamically as the Information Store is used –Ex 1: when information is resorted –Ex 2: when information is requested in a different sort order Excess index growth is managed with expiry time –Information store assigns expiry time to index –Index must be used within expiry time –If expiry time exceeds current time, index is deleted Exchange removes indexes by default after 40 days

MCSE Guide to Microsoft Exchange Server 2003 Administration 27 Tombstone Cleanup Mailbox folder structure organizes and stores mail items Actions following deletion of message from folder: –Message deactivated and marked for deletion in store –Tombstone (marker) is created for the message –Tombstone is propagated to other stores Default setting for tombstone is 60 days –Insures deletions for objects replicate across all stores –Tombstone checked for expiration during maintenance –Objects with expired tombstones are deleted from store

MCSE Guide to Microsoft Exchange Server 2003 Administration 28 Deleted Item Retention Cleanup Deleted messages placed in deleted item retention Examining deleted item retention during nightly maintenance: –Check for messages exceeding deleted item retention period –Permanently delete expired messages –Clean out expired messages from dumpster

MCSE Guide to Microsoft Exchange Server 2003 Administration 29 Public Folder Expiry Exchange checks public folder messages during nightly maintenance –Determines expiration according to Limits tab value –Messages exceeding Limits tab value are removed from public folder store

MCSE Guide to Microsoft Exchange Server 2003 Administration 30 Activity 13-8: Setting the Age Limit on Public Information Stores Time Required: 5 to 10 minutes Objective: Configure the age limit for public folders on the public folder store Description: Age limits are used to control the amount of information that users can post to public folders. Users who exceed the designated limits receive warning messages and are subject to restrictions, such as the inability to post messages.

MCSE Guide to Microsoft Exchange Server 2003 Administration 31

MCSE Guide to Microsoft Exchange Server 2003 Administration 32 Folder Tombstoning Actions following deletion of public folder: –Tombstone created for public folder –Default tombstone setting is 180 days –Tombstone replicated throughout organization –Tombstones are examined during Information Store maintenance cycle –Public folders with expired tombstones are permanently deleted from store

MCSE Guide to Microsoft Exchange Server 2003 Administration 33 Conflict Aging Conflicts may occur when a public folder is modified –Ex: one user overwrites the change made by another Exchange checks for conflict in maintenance cycle –Determines if conflict is resolved by participants Users have opportunity during conflict aging –If not, checks if conflict age limit is exceeded –Exchange takes action on unresolved conflicts

MCSE Guide to Microsoft Exchange Server 2003 Administration 34 Site Folder Check and Version Information Update Public folder database may have replicas of: –System configuration folders –System folder for administrative group Check public folders with system folder replicas –Ensure that duplicate folders do not exist Remove any duplicates that exist –Ensure that version information is up to date Update version information, if necessary

MCSE Guide to Microsoft Exchange Server 2003 Administration 35 Mailbox Cleanup Account deletion does not entail immediate mailbox deletion –Mailbox is in Information Store for another 30 days Purpose for retaining mailboxes after account deletion: –Gives administrators time to re-attach mailbox to re- created account –Employee may return to company within 30 days Mailboxes in Store after 30 day period are deleted

MCSE Guide to Microsoft Exchange Server 2003 Administration 36 Message Table Check Reference count data is maintained for messages Check reference count to ensure database is consistent with message information Reference count of zero: –Indicates message is not currently displayed by any folder –Messages in category are deleted during nightly maintenance

MCSE Guide to Microsoft Exchange Server 2003 Administration 37 Online Defragmentation Online defragmentation is a critical maintenance task Operation of online defragmentation: –Perform full pass over database –Free up pages by compacting records –Database objects no longer used are deleted –Mailbox and public folder store are reorganized Reduces amount of I/O time Suspend incomplete online defragmentation process –I/O intensive operation should not affect production

MCSE Guide to Microsoft Exchange Server 2003 Administration 38 Maintenance Utilities Additional maintenance utilities provided by Microsoft Four important additional utilities: –ESEUTIL –ISINTEG –RPC Ping –MTA Check

MCSE Guide to Microsoft Exchange Server 2003 Administration 39 ESEUTIL ESEUTIL: command line utility –Performs maintenance and repair on Information Stores Important ESEUTIL options: –Integrity check –Offline defragmentation –Repairing a database Use ESEUTIL only if normal backup is ineffective Copy database before applying ESEUTIL options Integrity check: first option to exercise with ESEUTIL –Checks database signatures and signatures on each page –Ensures that.edb file has.stm file association

MCSE Guide to Microsoft Exchange Server 2003 Administration 40 Activity 13-9: Using ESEUTIL to Check the Integrity of Your Database Time Required: 10 to 20 minutes Objective: Run the ESEUTIL utility to check the integrity of mailbox store on your server Description: Check the integrity of the Mailbox Store (SERVERXX), where XX is the number assigned to the back-end server, by running the ESEUTIL utility. When checking the integrity of the database, ESEUTIL checks the database tables, rebuilds all indexes on a temporary database (TEMPTINTEG.edb), and compares the indexes to the original.

MCSE Guide to Microsoft Exchange Server 2003 Administration 41

MCSE Guide to Microsoft Exchange Server 2003 Administration 42 ESEUTIL (continued) Public folder store and mailbox store databases grow as information is added Databases do not shrink when information is removed –Space is markedavailable during nightly garbage collection –New information is written to any available free space –Available space is used before database is expanded May require single files to be broken up for storage Fragmentation: reduced size of blocks of free space Exchange Server 2003 and fragmentation –Online defragmentation manages fragmentation issues –Online defragmentation does not compact database

MCSE Guide to Microsoft Exchange Server 2003 Administration 43 ESEUTIL (continued) ESEUTIL and fragmentation: –Defragments and compacts database when Store is offline When to use ESEUTIL for offline defragmentation: –Moving mailboxes off Information Store –Archiving large number of attachments to Hierarchical Storage Management Solution When to be wary of using offline defragmentation: –Reclaiming free disk space created on a daily basis Database may quickly grow back to original size in daily environment

MCSE Guide to Microsoft Exchange Server 2003 Administration 44 Activity 13-10: Using ESEUTIL to Perform Offline Defragmentation Time Required: 10 to 20 minutes Objective: Perform an offline defragmentation of the mailbox store on your server Description: Perform an offline defragmentation of the mailbox database on your Exchange 2003 server. ESEUTIL defragments the database by moving the used pages in the database into contiguous blocks in a new database. Unused pages are discarded, which is the only way to recover empty space inside the database for other uses.

MCSE Guide to Microsoft Exchange Server 2003 Administration 45

MCSE Guide to Microsoft Exchange Server 2003 Administration 46 ESEUTIL (continued) The role of ESEUTIL in recovering disk space –Use when database is corrupt and backup is not available Recovery mode of ESEUTIL (run first): –Applies records in transaction stores against database –Ensures transactions committed –Ensures ESEUTIL is working with all data Repair mode used to resurrect database: –Checks and repairs critical structures –Does not act to maintain relationships between tables and pages within tables

MCSE Guide to Microsoft Exchange Server 2003 Administration 47 Activity 13-11: Using ESEUTIL to Recover and Repair a Database Time Required: 10 to 20 minutes Objective: Perform a recovery and repair of a mailbox store on your server using ESEUTIL Description: Perform a repair of the Mailbox Store (SERVERXX), where XX is the number assigned to the back-end server, on the back-end server. Prior to repairing a database, it is highly recommended that you perform a recovery operation before attempting any repairs. This is to ensure that all transactions have been committed to the database prior to running the repair.

MCSE Guide to Microsoft Exchange Server 2003 Administration 48

MCSE Guide to Microsoft Exchange Server 2003 Administration 49 ISINTEG Information Store Integrity Check (ISINTEG) –Eliminates errors in Information Store databases –Resolves issues in relationships between tables –Example using ISINTEG with ESEUTIL: ESEUTIL discards unreadable messages in Msgs table Folder in mailbox may have references to messages Run ISINTEG to resolve issue ISINTEG removes references to deleted messages ISINTEG runs in two modes: –Check mode: reports on errors encountered –Check and fix mode: reports and fixes errors

MCSE Guide to Microsoft Exchange Server 2003 Administration 50

MCSE Guide to Microsoft Exchange Server 2003 Administration 51

MCSE Guide to Microsoft Exchange Server 2003 Administration 52 Activity 13-12: Using ISINTEG to Check on the Integrity of a Database Time Required: 10 to 20 minutes Objective: Check the integrity of a mailbox store on the server using ISINTEG Description: Perform an integrity check of the Mailbox Store (SERVERXX), where XX is the number assigned to the back-end server. You also run ISINTEG after running the ESEUTIL utility for database recovery and repair. This is because the ESEUTIL utility only checks the physical well-being of the database.

MCSE Guide to Microsoft Exchange Server 2003 Administration 53 RPC Ping Remote Procedure Call (RPC) –Allows program on one computer to execute a program on another Exchange servers in routing group rely on RPCs Example: MTA opening RPC with remote MTA –Establish a TCP connection with the remote server –Establish RPC connection with remote server’s endpoint mapper –Request port number remote server’s MTA listens to –Perform two-way handshake with the remote server

MCSE Guide to Microsoft Exchange Server 2003 Administration 54 RPC Ping (continued) Exchange clients connect to Exchange servers using RPCs RPC Ping utility has two functions: –Confirms RPC connectivity between two systems –Ensures Exchange servers are responding to requests RPC Ping has two components: –Server component (file named rpings.exe) May use a number of underlying transport protocols –Client component (installed on another computer) Used to test RPC connectivity

MCSE Guide to Microsoft Exchange Server 2003 Administration 55

MCSE Guide to Microsoft Exchange Server 2003 Administration 56 Activity 13-13: Using RPC Ping to Test Connectivity Time Required: 10 to 20 minutes Objective: Test RPC connectivity with the Exchange 2003 server using RPC Ping Description: Make use of the RPC Ping utility to test RPC connectivity between a client workstation and an Exchange 2003 server. The RPC Ping utility is part of the Windows 2003 Resource Kit Tools. As such, you first need to download and install this kit on both the client and server. For the purpose of this activity, the front-end server acts as the client machine.

MCSE Guide to Microsoft Exchange Server 2003 Administration 57

MCSE Guide to Microsoft Exchange Server 2003 Administration 58 MTA Check Message Transfer Agent (MTA) –Handles message transfer for X.400 queue connector –Maintains separate message queue for each connector MTA’s message queues are stored in files with.dat extension –Location: \Exchsrvr\Mtadata directory –Files can become corrupted during improper MTA shutdown MTA Check command line utility (mtacheck.exe): –Tries to fix MTA message queues and messages within –Automatically run when MTA service starts

MCSE Guide to Microsoft Exchange Server 2003 Administration 59 Activity 13-14: Using MTA Check to Verify MTA Message Queues Time Required: 10 to 20 minutes Objective: Use the MTA Check utility to verify the MTA message queues Description: Make use of the MTA Check utility to verify the MTA database, clean up messages, and repair the MTA database, if necessary. To use MTA Check, you first download it from the Microsoft Web site.

MCSE Guide to Microsoft Exchange Server 2003 Administration 60

MCSE Guide to Microsoft Exchange Server 2003 Administration 61 Resolving Server Issues General strategy for troubleshooting problems: –Take top-down approach to investigating problem –Use available tools to assist in diagnosis –Research the problem –Integrate findings with general understanding Eleven guidelines assisting investigation –1. Is issue client or server related? –2. Are clients responding poorly or not at all? –3. What is the frequency of the problem? –4. Which clients are experiencing the problem? –5. Review the event logs.

MCSE Guide to Microsoft Exchange Server 2003 Administration 62 Resolving Server Issues (continued) Eleven guidelines assisting investigation (continued) –6. Investigate the queues –7. Monitor performance counters –8. Check that DNS is properly configured –9. Verify configuration of Active Directory –10. Investigate permissions issues –11. Ensure firewall is configured properly

MCSE Guide to Microsoft Exchange Server 2003 Administration 63 Summary Server monitors check server status/resource usage Link monitors check connection between servers Queue monitoring checks queue growth over time CPU monitoring checks CPU utilization Disk monitoring checks disk space utilization

MCSE Guide to Microsoft Exchange Server 2003 Administration 64 Summary (continued) Virtual memory monitoring checks memory Monitor settings trigger either warning or critical state Nightly Information Store maintenance ensures consistency and performs cleanup ESEUTIL is used to check for syntactic correctness of database ESEUTIL does not verify if database is contextually correct

MCSE Guide to Microsoft Exchange Server 2003 Administration 65 Summary (continued) ISINTEG is used to check if database is contextually correct ISINTEG checks relationships between database tables RPC Ping confirms RPC connectivity between two systems MTA Check utility attempts to fix MTA message queues and messages within Take top-down approach to troubleshooting problems