1. Troubleshooting Exchange Transport Service Miha Pihler MVP – Enterprise Security Microsoft Certified Master | Exchange 2010

2. About Speaker / Trainer Author

3. Agenda Understand how Exchange Transport Service Works – One of the most important services – Without it there is no e-mail at all (Also no spam) ;-) Troubleshoot common Exchange Transport Service

4. Microsoft Exchange Transport Service

5. Exchange Transport Service (cont.) MSExchangeTransport.exe – Exchange 2010 and Exchange 2013 – It is parent service and it spawns a child service EdgeTransport.exe – Child service is actually listening in on TCP port 25 If child service fails parent notices and respawns service

6. Exchange Transport Service (cont.) If child service fails multiple times (twice) parent will check messages in queue and it will move problematic message(s) to special queue Queue is called Poison Queue and you can see it with e.g. get-queue Message will stay in poison queue until it expires or until Administrator performs some action on it – Removes the messages – Re-submits them

7. Exchange Transport Service (cont.) EdgeTransport.exe.config.xml – C:\Program Files\Microsoft\Exchange Server\V14\Bin – Settings from the file are applied at start or re- start of the service

8. Exchange queue DB Queue DB is ESE database – Same rules apply as for any other ESE DB Exchange queue – C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue – C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue You can export messages from queue – export-message – Export-Message srv-exch1\366055\230652 | AssembleMessage -Path "c:\temp\email.eml„ Place the.eml file into Pickup folder – C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Pickup – http://technet.microsoft.com/en-us/library/aa997214.aspx http://technet.microsoft.com/en-us/library/aa997214.aspx

9. Exchange queue DB (cont.) Delete queue DB and generate new one – Stop Exchange Transport Service – Rename Queue folder – Start Exchange Transport Service This will create new clean Queue DB

10. Important log files If you enabled filtering agents on your Exchange than first log to check should be C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\AgentLog What to expect from Agent log – Recipient does not exist If enabled Can be enabled/disabled per Accepted Domain – RBL (Realtime Block List) events (if RBL enabled) – SenderID events E.g. domain does not exist

11. Examples of Agent log events You have to check these logs on all servers!

12. Important log files (cont.) FSEAgentLog – You only need to check this logs if you are using Forefront Protection for Exchange – Similar to AgentLog C:\Program Files (x86)\Microsoft Forefront Protection for Exchange Server\Data\FSEAgentLog

13. Important log files (cont.) Protocol logs – SmtpReceive – SmtpSend – Must be enabled on connector (disabled by default) C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLog C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\FrontEnd\ProtocolLog

14. Examples of Protocol logs

15. What else to look for in protocol logs Authentication errors – This will happen on internal connectors mostly – It will tell you that your receive AND/OR send connectors are not configured correctly – If you have multiple HUB servers they will always require an authentication before they exchange messages – Make sure that send and receive connectors are configured to send and accept authentication

16. What else to look for in protocol logs Authentication errors – If these connectors are not configured correctly there will be no e-mail going in or out and it will be queued at the last / first server – I see this very often when administrators create new connectors – It is also confusing when selecting Internet or Internal Connector from the wizard

17. Pipeline [PS] C:\Windows\system32>Get-TransportPipeline Event TransportAgents ----- --------------- OnConnectEvent {Connection Filtering Agent, Protocol Analysis... OnHeloCommand {} OnEhloCommand {} OnAuthCommand {} OnEndOfAuthentication {} OnMailCommand {Connection Filtering Agent, Sender Filter Agent} OnRcptCommand {Connection Filtering Agent, Address Rewriting... OnDataCommand {} OnEndOfHeaders {Connection Filtering Agent, Address Rewriting... OnEndOfData {Edge Rule Agent, Protocol Analysis Agent, Atta... OnHelpCommand {} OnNoopCommand {} OnReject {Protocol Analysis Agent} OnRsetCommand {Protocol Analysis Agent} OnDisconnectEvent {Protocol Analysis Agent} OnSubmittedMessage {Address Rewriting Outbound Agent, FSE Routing... OnResolvedMessage {} OnRoutedMessage {Address Rewriting Outbound Agent} OnCategorizedMessage {}

18. Pipeline with 3rd party add-on [PS] C:\Windows\system32>Get-TransportPipeline Event TransportAgents ----- --------------- OnConnectEvent {} OnHeloCommand {} OnEhloCommand {} OnAuthCommand {} OnEndOfAuthentication {} OnMailCommand {} OnRcptCommand {} OnDataCommand {} OnEndOfHeaders {} OnEndOfData {} OnHelpCommand {} OnNoopCommand {} OnReject {} OnRsetCommand {} OnDisconnectEvent {} OnSubmittedMessage {Exclaimer Mail Disclaimers Routing Agent, Text... OnResolvedMessage {} OnRoutedMessage {Exclaimer Mail Disclaimers Routing Agent, Tran... OnCategorizedMessage {Exclaimer Mail Disclaimers Routing Agent}

19. Agents and 3rd party agents You can disable 3rd party agents You can‘t disable some built in agents [PS] C:\Windows\system32>Get-TransportAgent Identity Enabled Priority -------- ------- -------- Exclaimer Mail Disclaimers Routing Agent True 1 Transport Rule Agent True 2 Text Messaging Routing Agent True 3 Text Messaging Delivery Agent True 4

20. Problems on the sending side.. What if the problem is on the sending side… – E.g. problem with script/program sending the e- mail One way is to use Wire Shark – But what if we use SMTP with SSL (remember slides before)? Wellfirst we have to disable SSL

21. Pipeline Tracing Helps solve some of the hardest problems (without using Wire Shark and other changes) Can help you view the message in original state as it enters your organization Must be enabled and set for specific sender

22. Pipeline Tracing Set-TransportServer – PipelineTracingSenderAddress miha.pihler@telnet.si – PipeLineTracingEnabled $True miha.pihler@telnet.si C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\PipelineTra cing\MessageSnapshots\ – Will show up once you get a „hit“ on set sender

23. Pipeline Tracing Log Example

24. Pipeline Tracing Can be used for other troubleshooting purposes – For example we want to find specific header information

25. Performance counters A number of usefull performance counters – For different Connectors (send / receive) – For different agents – …

26. Other usefull things to look at Get-ReceiveConnector – Tarpit – Protocol Errors settings Get-TransportServer – Configuration or specific transport server Get-TransportAgent and Get-TransportPipeline – See what agents are installed and where they are used Get-TransportConfig – Mostly configuration for internal mail flow

27. Summary When troubleshooting mail delivery: Know what logs to look at – Agent Log should be first if you are using different filters It will list ALL e-mail that touched your servers – FSE (Forefront) logs should be next – Don‘t forget to check these logs on ALL your servers If needed enable and use Pipeline tracing – This is useful if you are troubleshooting delivery into your organization – See how e-mail looks as it arrives to your systems

28. Q&A Tomorrow morning I have another session Understanding and troubleshooting Kerberos miha.pihler@telnet.si