Field Report: Edge Fails with “454 4.7.0 Temporary authentication failure”

Issue


There was a scenario where it was noticed that outbound mails were not getting delivered. The environment had 2 Edge Servers together with 2 CAS/Mailbox servers, both being Exchange 2013. Upon checking the mailbox queue on the internal servers, it was noticed that the mails were stuck in the queue on the Send connector that was responsible for outbound mail delivery with Edge Server.

The Error reported the following.

451 4.4.0 Primary target IP address responded with: “454 4.7.0 Temporary authentication failure.”  Attempted failover to alternate host, but that did not succeed.  Either there are no alternate hosts, or delivery failed to all alternate hosts”

Additionally, the following tests were done.

  • System Clock was checked between all Edge, Mailbox/CAS and domain controllers. They were in the same time.
  • Checked for any replication issues between the domain controllers. No issues were found.
  • Checked for communication related issues between the Edge Server and the Internal Exchange Servers for the below ports. All required ports were open.
Network interface Open port Protocol Note
Inbound from and outbound to the internal network 25/TCP SMTP This port is required for mail flow to and from the Exchange organization.
Local only 50389/TCP LDAP This port is used to make a local connection to AD LDS.
Inbound from the internal network 50636/TCP Secure LDAP This port is required for EdgeSync synchronization.
Inbound from the internal network 3389/TCP RDP Opening this port is optional. It provides more flexibility in managing the Edge Transport servers from inside the internal network by letting you use a remote desktop connection to manage the Edge Transport server.
  • Checked the EdgeSync for Edge Server 01: The following results were noted.

    EdgeSync service cannot connect to this subscription because of error “No EdgeSync credentials were found for Edge transport server ED01.contoso.com on the local Hub Transport server. Remove the Edge subscription and re-subscribe the Edge Transport server.”

  • Upon checking the Event viewer, the following errors were thrown;
    • Event ID 1032

      Microsoft Exchange EdgeSync can’t find the replication credential on %1 to synchronize with Edge server %2. This may happen if %1 joined the current Active Directory site after subscription for %2 was established. To have this Hub Transport server participate in EdgeSync, re-subscribe %2 to the current Active Directory site.

Resolution


So basically, we see that the Edge Sync is not working as it should be. At the same time, we see that there’s a certificate issue as well. Upon checking the certificate, we identified that it’s from a local CA. So the next steps we would do is to;

  1. Re-assign Exchange services to the existing certificate
  2. Re run Edge Synchronization
  3. Verify

However, upon doing the service re-assigning to the existing services, the following error was thrown.

The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

Though it said the problem has been fixed, we were still unable to get Edge Sync working. So the final thought was to;

  1. Generate new Exchange Server certificate
  2. Assign services to new certificate
  3. Re-run Edge-Sync

Viola!!! No more errors!!

Advertisement

Ignite 2016 Session Viewer for O365/Exchange – Technet Gallery

43

This is a small macro sheet I made to be used as a one-stop real time viewer for Office 365 and Exchange Server related sessions from the Microsoft Ignite 2016. Currently there are more than 170 Office 365 and 50 Exchange related sessions listed on the Microsoft Ignite site. This viewer is not fully updated, however the session list is getting updated and will post changelogs in whats news.

Download

Following sessions are available for viewing;

  • BRK1001 : Maximize your Office 365 administration: tips and tricks
  • BRK1003 : Explore accessibility in Office 365: plans and progress
  • BRK1016 : Address your CXO’s top five cloud security concerns
  • BRK1021 : Unplug with the Microsoft Outlook experts
  • BRK1033 : Build your intranet with Microsoft Office 365
  • BRK1044 : Dive deeper into what’s new and what’s coming in Outlook on the web
  • BRK2008 : Understand your users: what’s new in Office 365 Usage Reporting
  • BRK2009 : Manage Office 365 more effectively: what’s new in Office 365 administration
  • BRK2010 : Implement ExpressRoute for Microsoft Office 365 (step by step)
  • BRK2013 : Keep calm and automate: How we secure the Office 365 service
  • BRK2032 : Identify and illustrate insights with new Microsoft Excel Charts
  • BRK2033 : Discover Office 365 Groups – overview, what’s new and roadmap
  • BRK2035 : Learn about advancements in Office 365 Advanced Threat Protection
  • BRK2035 : Learn about advancements in Office 365 Advanced Threat Protection
  • BRK2044 : Discover what’s new and what’s coming for Office Delve
  • BRK2046 : Learn what to use when: Office 365 Groups, SharePoint Team Sites, Yammer, and OneDrive for Business
  • BRK2050 : Dive into Microsoft Office 365 and SharePoint Hybrid Scenarios
  • BRK2053 : Connect your business critical applications to Outlook and Groups
  • BRK2093 : Design your Exchange infrastructure right (or consider moving to Office 365)
  • BRK2097 : Drive Office 365 adoption: methodology, best practices, and resources from Microsoft
  • BRK2100 : Move to Office 365 and drive adoption – lessons learned from the Carlsberg Group
  • BRK2139 : Protect your business and empower your users with cloud Identity and Access Management
  • BRK2160 : Build business applications with Power Apps, Microsoft Flow, and Office 365
  • BRK2166 : Learn about Office 365 Secure Score: actionable security analytics
  • BRK2170 : Discover what’s new with Microsoft Exchange Public Folders
  • BRK2215 : Debate the top 10 reasons not to move your Exchange on-premises mailboxes to Exchange Online
  • BRK2216 : Unplug with the experts on Exchange Server and Exchange Online
  • BRK2216 : Unplug with the experts on Exchange Server and Exchange Online
  • BRK2217 : Discover modern support in Outlook for Exchange Online
  • BRK2218 : Move from Exchange 2007 to Modern Exchange
  • BRK2219 : Meet twin sons of different mothers – Exchange Engineers and Exchange MVPs
  • BRK2220 : Peer behind the curtain – how Microsoft runs Exchange Online
  • BRK2245 : Transform the way you manage Skype for Business
  • BRK2252 : Understand Microsoft’s Office 365 datacenter strategy and approach
  • BRK2275 : Improve Office 365 adoption: top 10 ways
  • BRK2298 : Plan to drive value and user adoption in Microsoft Office 365
  • BRK3000 : Unplug with the experts on Microsoft Exchange Top Issues
  • BRK3001 : Explore the ultimate field guide to Microsoft Office 365 Groups
  • BRK3003 : Collaborate outside the firewall with Microsoft Office 365
  • BRK3007 : Investigate tools and techniques for Exchange Performance Troubleshooting
  • BRK3015 : Reduce costs and challenges with Office 365 eDiscovery and Analytics
  • BRK3016 : Take control of your data with intelligent data governance in Office 365
  • BRK3017 : Own your data and service – monitor and investigate with Office 365 Auditing, Insights and alerts
  • BRK3018 : Take control of your security and compliance with Office 365
  • BRK3019 : Manage Microsoft Office 365 Groups
  • BRK3022 : Challenge cloud encryption myths and learn about Office 365 BYOK plans
  • BRK3023 : Understand how Microsoft protects you against Spoof, Phish, Malware, and Spam emails
  • BRK3024 : Building security and compliance solutions with the O365 Activity API – a Microsoft IT case study
  • BRK3040 : Own your data with next generation access control technology in Office 365
  • BRK3045 : Use Microsoft Graph to reach users on hybrid Exchange 2016
  • BRK3046 : Build intelligent line-of-business applications leveraging the Outlook REST APIs
  • BRK3074 : Discover what’s new in Active Directory Federation and domain services in Windows Server 2016
  • BRK3083 : Secure Office 365 like a cybersecurity pro—assessing risk and implementing controls
  • BRK3102 : Conduct a successful pilot deployment of Microsoft Intune
  • BRK3109 : Deliver management and security at scale to Office 365 with Azure Active Directory
  • BRK3215 : Dive into Modern Authentication – how it works and what to do when it doesn’t
  • BRK3216 : Plan performance and bandwidth for Microsoft Office 365
  • BRK3217 : Run Microsoft Exchange Hybrid for the long haul
  • BRK3219 : Migrate to Exchange Online via Exchange Hybrid
  • BRK3220 : Deploy Microsoft Exchange Server 2016
  • BRK3221 : Understand the Microsoft Exchange Server 2016 Architecture
  • BRK3222 : Implement Microsoft Exchange Online Protection
  • BRK3227 : Ask us anything about Microsoft Office 365 Groups
  • BRK3242 : Discover a new level of Service Health insights for Office 365
  • BRK3253 : Experience Scott Schnoll’s Exchange tips and tricks
  • BRK3254 : Cert Exam Prep: Exam 70-345: Designing and Deploying Microsoft Exchange Server 2016
  • BRK3281 : Deliver a BYOD program that employees and security teams will love with Microsoft Intune
  • BRK3298 : Secure your Active Directory to mitigate risk in the cloud
  • BRK4000 : Review ExpressRoute for Office 365 configuration (routing, proxy and network security)
  • BRK4015 : Build client-side web parts for Microsoft SharePoint
  • BRK4031 : Overcome network performance blockers for Office 365 Deployments
  • BRK4032 : Dive deep into Microsoft Exchange Server High Availability
  • THR1003R : Take control of your security and compliance with Office 365
  • THR1004R : Empower employees with Microsoft Delve Analytics
  • THR1005R : Dive deeper into what’s new and what’s coming in Microsoft Outlook 2016 for Windows
  • THR1011R : Dive deeper into what’s new and what’s coming in Outlook mobile
  • THR2004R : Manage Microsoft Office 365 from anywhere
  • THR2006R : Get an edge over attackers – what you need to know about email threats
  • THR2007R : Fight back with advancements in Office 365 Advanced Threat Protection
  • THR2009R2 : Roll out Microsoft Office in one of the most demanding environments
  • THR2020R : Deploy successfully : top 10 Office 365 ProPlus installation/activation tips
  • THR2022 : Migrate your data to Microsoft Office 365 – why?
  • THR2190R : Secure your sensitive email with Office 365 message encryption
  • THR2207 : Modernize your clients with Office 365, Windows 10 and Enterprise mobility – the admin experience
  • THR3001R : Migrate DL to Microsoft Office 365 Groups
  • THR3007 : Protect your sensitive information with Office 365 Data Loss Prevention
  • THR3008R : Gain visibility and control with Office 365 Advanced Security Management
  • THR3010 : Help your users collaborate better with Office 365 Groups