The environment was running Exchange Server 2016 (n-1)CU with Office 365 in a hybrid mode. The issue came up when users on-prem noticed that they are unable to receive emails from their own O365 cloud users. Looking more into this, figured out the following user mail flow scenarios;
- On-prem users can send/receive mails within on-prem and to/from internet users
- O365 cloud users can send/receive mails within O365 and to/from internet users
- On-prem users cannot send/receive emails to/from O365 users
So obviously, this had to do something with the hybrid connector since that’s the tunnel that bridges the two environments.
So back to the basics, lets trace down the error messages and see if we can get a clue. In order to see where our mails are stuck, you need to first check the delivery reports. Log in to the O365 Exchange ECP as administrator and navigate to Message Tracing. On the message tracing log, you will see something like below;
We’re almost there now. The above indicates that why O365 is not delivering the mails to on-prem environment is because as on the first item, we have configured to use TLS between the hybrid environments. Thus due to whatever reason, the certificate presented from the on-prem public IP does not match the certificate that is been binded in the Hybrid Configuration. Now in order to validate this, we need to look at two points;
- Ensure that the certificate is not expired
- Check Hybrid Configuration certificate settings: To validate that you have the correct certificate open up the Exchange PowerShell module on your on-prem and run the below command;
Now as you can see, the Hybrid configuration is configured with the correct certificate. So nothing to worry on that.
- Exchange Connector TLS settings: The next connection point would be our internal Exchange Server receive connector. Why this connector is important is that when O365 is trying to connect with the exchange server for mail delivery, it will try out TLS for authentication. This is because we have TLS enabled; meaning we should have the correct certificate binded to the corresponding Receive Connector. In many cases this would be the Default FrontEnd connector. Run the below command and ensure that the property tlscertificatename is set correctly and is the same as the above certificate.
Get-ReceiveConnector -Identity “<ExchangeServerName>\Default Frontend <ExchangeServerName>” | fl
In my case, the certificate was all fine and is set for the one as expected.
So, the certificate itself, exchange setup and office 365 hybrid setup is all configured as expected. Yet somehow, Office 365 is telling me that the certificate is not correct.
This is where we would need to properly test TLS in a more informative manner. We will now see how we can use a method where we can see the TLS communication that is made to our on-premises.
Enter CheckTLS web testing provider. The provider has many tools to check TLS mechanisms but in our case, we are looking at the receiving part. Thus we will head over here.
What it does?
TestReceiver performs all the steps that Internet email systems go through to send email. It records every command and byte of data it sends and every answer and byte of data that the other email system sends. TestReceiver never actually sends an email, it just gets as close as possible, learning as much about the remote system as it can.
Because CheckTLS focuses on security, TestReceiver tries to establish a secure (TLS) connection with the recipient’s system. Along with recording everything, it looks at the security of the recipient’s system for things like: certificate contents and signers, encryption algorithms, key lengths, hostname mis-matches, incorrect wild-card usage, weak cyphers, etc.
So is this safe? absolutely. Since it only checks whatever is already published by your organization. It doesn’t grab your email ID, username or passwords.
- So head over to the site and under the Input fields, type your domain name in the eMail Target
- Ensure that under the Output Format you have selected Detail (this will provide you a verbose log of the connection status, more meaningful)
- Click Run Test.
- Give it some time to run the test and notice the logging which happen in the background. This is the session initiation that happens in real time.
- Once the test is completed, look in the detailed log.
And we have found the culprit. Just have a closer look at the above image. You can see that the TLS authentication does start however the certificate validation fails. The reason is that the issuing certificate is not our Exchange Certificate, but a self signed certificate by the barracuda appliance which is front-ending to external connections.
When Office 365 tries to initiate a TLS session, it is getting this self signed certificate thus the required URLs are not found (mail.domain.com) hence it drops the connection and throws out;
450 4.7.320 Certificate validation failed.
If this is your case, bump up your network team and ask them to bind the correct certificate from the appliance. Once its configured properly, run the below tests again to validate that everything is all good.
- CheckTLS Receive validation – Ensure correct certificate is thrown out
- Office 365 Connector Validation
If everything is successful, send out couple of emails and you will see that mailflow is working as expected. For the mails that were queued, give some time and it’ll start flowing.
Got any inputs? Please feel free to let me know your ideas.