Jude's Blog

Archive for the ‘Microsoft’ Category

450 4.7.320 Certificate validation failed

leave a comment »

Issue


The environment was running Exchange Server 2016 (n-1)CU with Office 365 in a hybrid mode. The issue came up when users on-prem noticed that they are unable to receive emails from their own O365 cloud users. Looking more into this, figured out the following user mail flow scenarios;

  • On-prem users can send/receive mails within on-prem and to/from internet users
  • O365 cloud users can send/receive mails within O365 and to/from internet users
  • On-prem users cannot send/receive emails to/from O365 users

So obviously, this had to do something with the hybrid connector since that’s the tunnel that bridges the two environments.

So back to the basics, lets trace down the error messages and see if we can get a clue. In order to see where our mails are stuck, you need to first check the delivery reports. Log in to the O365 Exchange ECP as administrator and navigate to Message Tracing. On the message tracing log, you will see something like below;

11

We’re almost there now. The above indicates that why O365 is not delivering the mails to on-prem environment is because as on the first item, we have configured to use TLS between the hybrid environments. Thus due to whatever reason, the certificate presented from the on-prem public IP does not match the certificate that is been binded in the Hybrid Configuration. Now in order to validate this, we need to look at two points;

  • Ensure that the certificate is not expired
  • Check Hybrid Configuration certificate settings: To validate that you have the correct certificate open up the Exchange PowerShell module on your on-prem and run the below command;

Get-HybridConfiguration

12

Now as you can see, the Hybrid configuration is configured with the correct certificate. So nothing to worry on that.

  • Exchange Connector TLS settings: The next connection point would be our internal Exchange Server receive connector. Why this connector is important is that when O365 is trying to connect with the exchange server for mail delivery, it will try out TLS for authentication. This is because we have TLS enabled; meaning we should have the correct certificate binded to the corresponding Receive Connector. In many cases this would be the Default FrontEnd connector. Run the below command and ensure that the property tlscertificatename is set correctly and is the same as the above certificate.

Get-ReceiveConnector -Identity “<ExchangeServerName>\Default Frontend <ExchangeServerName>” | fl

In my case, the certificate was all fine and is set for the one as expected.

So, the certificate itself, exchange setup and office 365 hybrid setup is all configured as expected. Yet somehow, Office 365 is telling me that the certificate is not correct.

This is where we would need to properly test TLS in a more informative manner. We will now see how we can use a method where we can see the TLS communication that is made to our on-premises.

Enter CheckTLS web testing provider. The provider has many tools to check TLS mechanisms but in our case, we are looking at the receiving part. Thus we will head over here.

https://www.checktls.com/TestReceiver

What it does?

TestReceiver performs all the steps that Internet email systems go through to send email. It records every command and byte of data it sends and every answer and byte of data that the other email system sends. TestReceiver never actually sends an email, it just gets as close as possible, learning as much about the remote system as it can.

Because CheckTLS focuses on security, TestReceiver tries to establish a secure (TLS) connection with the recipient’s system. Along with recording everything, it looks at the security of the recipient’s system for things like: certificate contents and signers, encryption algorithms, key lengths, hostname mis-matches, incorrect wild-card usage, weak cyphers, etc.

So is this safe? absolutely. Since it only checks whatever is already published by your organization. It doesn’t grab your email ID, username or passwords.

  1. So head over to the site and under the Input fields, type your domain name in the eMail Target
  2. Ensure that under the Output Format you have selected Detail (this will provide you a verbose log of the connection status, more meaningful)
  3. Click Run Test.
  4. Give it some time to run the test and notice the logging which happen in the background. This is the session initiation that happens in real time.
  5. Once the test is completed, look in the detailed log.13

 

And we have found the culprit. Just have a closer look at the above image. You can see that the TLS authentication does start however the certificate validation fails. The reason is that the issuing certificate is not our Exchange Certificate, but a self signed certificate by the barracuda appliance which is front-ending to external connections.

When Office 365 tries to initiate a TLS session, it is getting this self signed certificate thus the required URLs are not found (mail.domain.com) hence it drops the connection and throws out;

450 4.7.320 Certificate validation failed.

If this is your case, bump up your network team and ask them to bind the correct certificate from the appliance. Once its configured properly, run the below tests again to validate that everything is all good.

  • CheckTLS Receive validation – Ensure correct certificate is thrown out
  • Office 365 Connector Validation

If everything is successful, send out couple of emails and you will see that mailflow is working as expected. For the mails that were queued, give some time and it’ll start flowing.

Got any inputs? Please feel free to let me know your ideas.

Cheers..

Advertisements

Windows Server Restarting often with BSOD

leave a comment »

Issue


So here’s the case, during the last two weeks we’ve identified quite a few scenarios where the Domain Controllers kept on restarting out of nowhere. Here are the symptoms;

  • BlueScreen (BSOD)
  • Restarting every 5-10 mins or quite often
  • Windows Server 2012 or Windows Server 2012 R2

Upon going through the dump analysis of those cases and digging in more, the root cause was related to a SRV.SYS windows driver. Apparently this is a driver that handles SMBv1 connections. The restarting is due to a memory overflow in the system.

 

Resolve


 

Check your Updates!!!!! Well, in all our cases the client servers were not patched. The BSOD was due to the SMBv1 memory overflow.

In order to fix the issue, head over to the Microsoft Security Bulletin MS17-010 – Critical website, look for your Operating System, patch it ASAP.

Pretty strange why it was all good until now. Anyhow, whatever said and done another good reminder for everyone who’s lazy on updating their systems.

WannaCrypt attack; Keeping protected.

leave a comment »

Issue


I’m pretty sure you know it already. There’s a serious Ransom-ware spreading across the globe and this one is again I repeat. Pretty darn serious!

Behold! “WannaCrypt” is here. This is so severe, that it has been reported that more than 95k PC’s are already infected.

This slideshow requires JavaScript.

 

The problem is that the ransom-ware if hit, will lock (encrypt) your files. So that you have no way of opening it without unlocking (decrypt). To unlock, the makes of the malware is demanding a ransom. And we never know if they will give you the key as well.

Apart from that this can spread across to other computers. Which makes things worse. The spread is using something called a Remote code execution that existed in Microsoft Server Message Block 1.0 (SMBv1) component.

Resolve


Individual user

  1. Update your Antivirus application
  2. Update your Windows Operating System
    Running Windows XP? There’s good news for you. For older systems like Windows XP,  Windows Server 2003 etc Microsoft no longer provides updates as its support lifecycle is ended. However, Microsoft has been generous to publicly make available the patches for the below operating systems due to the severity of the case. Click on the below to view and download the updates needed.

    Windows XP SP2 x64
    Windows XP SP3 x86 
    Windows XP Embedded SP3 x86
    Windows 8 x86
    Windows 8 x64

  3. Run a full Anti-virus scan of your computer.
  4. Pray!

Corporate/Enterprise/ IT Admins

  1. If you have an enterprise Antivirus protection suite, make sure it’s engine is up to date.
  2. Push the latest AV updates to your client PCs.
  3. Update all your Windows Server Operating systems that includes both client and Servers.
    Thanks to Microsoft they’ve got you covered if you still have Windows Server 2003 Operating Systems as well.
    If you have WSUS or SCCM, get the updates and push it. If not, use the below ‘MS17-010’ link below to get the correct update for your system.

Windows Server 2003 SP2 x64
Windows Server 2003 SP2 x86

MS17-010 – Microsoft Security Bulletin Update

Oh and Windows 10 users, you’re all good. ^_^

Written by judeperera

May 13, 2017 at 9:56 am

Field Report: Edge Fails with “454 4.7.0 Temporary authentication failure”

leave a comment »

Issue


There was a scenario where it was noticed that outbound mails were not getting delivered. The environment had 2 Edge Servers together with 2 CAS/Mailbox servers, both being Exchange 2013. Upon checking the mailbox queue on the internal servers, it was noticed that the mails were stuck in the queue on the Send connector that was responsible for outbound mail delivery with Edge Server.

The Error reported the following.

451 4.4.0 Primary target IP address responded with: “454 4.7.0 Temporary authentication failure.”  Attempted failover to alternate host, but that did not succeed.  Either there are no alternate hosts, or delivery failed to all alternate hosts”

Additionally, the following tests were done.

  • System Clock was checked between all Edge, Mailbox/CAS and domain controllers. They were in the same time.
  • Checked for any replication issues between the domain controllers. No issues were found.
  • Checked for communication related issues between the Edge Server and the Internal Exchange Servers for the below ports. All required ports were open.
Network interface Open port Protocol Note
Inbound from and outbound to the internal network 25/TCP SMTP This port is required for mail flow to and from the Exchange organization.
Local only 50389/TCP LDAP This port is used to make a local connection to AD LDS.
Inbound from the internal network 50636/TCP Secure LDAP This port is required for EdgeSync synchronization.
Inbound from the internal network 3389/TCP RDP Opening this port is optional. It provides more flexibility in managing the Edge Transport servers from inside the internal network by letting you use a remote desktop connection to manage the Edge Transport server.
  • Checked the EdgeSync for Edge Server 01: The following results were noted.

    EdgeSync service cannot connect to this subscription because of error “No EdgeSync credentials were found for Edge transport server ED01.contoso.com on the local Hub Transport server. Remove the Edge subscription and re-subscribe the Edge Transport server.”

  • Upon checking the Event viewer, the following errors were thrown;
    • Event ID 1032

      Microsoft Exchange EdgeSync can’t find the replication credential on %1 to synchronize with Edge server %2. This may happen if %1 joined the current Active Directory site after subscription for %2 was established. To have this Hub Transport server participate in EdgeSync, re-subscribe %2 to the current Active Directory site.

Resolution


So basically, we see that the Edge Sync is not working as it should be. At the same time, we see that there’s a certificate issue as well. Upon checking the certificate, we identified that it’s from a local CA. So the next steps we would do is to;

  1. Re-assign Exchange services to the existing certificate
  2. Re run Edge Synchronization
  3. Verify

However, upon doing the service re-assigning to the existing services, the following error was thrown.

The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

Though it said the problem has been fixed, we were still unable to get Edge Sync working. So the final thought was to;

  1. Generate new Exchange Server certificate
  2. Assign services to new certificate
  3. Re-run Edge-Sync

Viola!!! No more errors!!

Written by judeperera

January 10, 2017 at 11:46 am

Exchange Server Upgrade | Step-by-Step

with one comment

The guide will cover upgrading an Exchange Server 2013 CU8 into Exchange 2013 CU15.

In a nutshell, this is the procedure:

  1. Prepare the existing environment
    1. Download the latest binaries
    2. Remove Interim Updates
  2. Perform Upgrade
    1. Enable Exchange Server maintenance mode (High Availability)
    2. Upgrade Schema
    3. Install Exchange Binaries
    4. Enable Exchange Server services

Prepare the existing environment

Download Exchange Server SP/RU/CU binaries

When you are performing an Upgrade, you can go ahead and install the latest Cumulative Update (CU). You DO NOT, need to install all previous CU’s one by one in an incremental way. This is because a CU is a full installation of Exchange Server plus a collection of all the updates, patches and changes that has been made available so far.

For an Example, if you are on Exchange Server 2016 RTM, you can straight away install CU4. Because CU4 will contain all changes made in each of the CU’s previously released.

Exchange 2016

Version Blog post
Exchange 2016 CU4 Released: December 2016 Quarterly Exchange Updates
Exchange 2016 CU3 Released: September 2016 Quarterly Exchange Updates
Exchange 2016 CU2 Released: June 2016 Quarterly Exchange Updates
Exchange 2016 CU1 Released: March 2016 Quarterly Exchange Updates
Exchange 2016 RTM Exchange Server 2016: Forged in the cloud. Now available on-premises

Exchange 2013

Version Blog post
Exchange 2013 CU15 Released: December 2016 Quarterly Exchange Updates
Exchange 2013 CU14 Released: September 2016 Quarterly Exchange Updates
Exchange 2013 CU13 Released: June 2016 Quarterly Exchange Updates
Exchange 2013 CU12 Released: March 2016 Quarterly Exchange Updates
Exchange 2013 CU11 Released: December 2015 Quarterly Exchange Updates
Exchange 2013 CU10 Released: September 2015 Quarterly Exchange Updates
Exchange 2013 CU9 Released: June 2015 Exchange Cumulative Update and Update Rollups
Exchange 2013 CU8 Released: Exchange Server 2013 Cumulative Update 8
Exchange 2013 CU7 Released: Exchange Server 2013 Cumulative Update 7
Exchange 2013 CU6 Released: Exchange Server 2013 Cumulative Update 6
Exchange 2013 CU5 Released: Exchange Server 2013 Cumulative Update 5
Exchange 2013 SP1 Released: Exchange Server 2013 Service Pack 1
Exchange 2013 CU3 Released: Exchange Server 2013 Cumulative Update 3
Exchange 2013 CU2 Released: Exchange Server 2013 Cumulative Update 2
Exchange 2013 CU1 Released: Exchange Server 2013 Cumulative Update 1
Exchange 2013 RTM Exchange Server 2013 Reaches General Availability

Remove Interim Updates

Now that you have downloaded the binaries, you need to get rid of the Interim Updates that may have been installed in your environment. In some cases, if an Interim Update is installed, Microsoft Exchange Server CU’s or SP’s cannot be installed. Therefore, before installing the binaries, read the release notes for any information on removal of interim updates. Below steps will guide for that;

  1. In Control Panel, double-click Programs and Features.
  2. In the Currently installed programs list, click Interim Update for Exchange Server 201X (KBxxxxxx), where xxxxxx is the Knowledge Base article number that is associated with the IU.
  3. Click Remove.
  4. At a command prompt, run sn.exe -Vu * to enable strong name verification.
  5. Run sn.exe –Vl to verify that strong name verification is enabled.

Perform Upgrade

Enable Exchange Server maintenance mode (High Availability)

If you have standalone servers in your environment, then an upgrade will require you to go in for a downtime. Why? Because the services needs to be stopped during the upgrade process. In such a scenario, skip to the Upgrade section.

However, if your environment consists multiple servers in terms of 2xCAS or 2xMBX or 2x(CAS+MBX) or any of a combined scenario, it is highly advised that you perform upgrading of the servers one by one. This will ensure that your environment will stay online using the rest of the high available servers taking the downtime away.

There’s a catch in here. Let’s say we have a 2 Mailbox Server scenario: Server A and B. You decide to mount all databases into ServerA and upgrade Server B. Although this seems an okay way to do, it is not the case. Reason is that the Active Manager and other Exchange backend workers doesn’t know that you are doing a planned maintenance. Thus, the workers will think that the server is in a failed state. Can we fix this? Absolutely. You have to take the services on Server B or the server that you are about to do the upgrade into ‘Maintenance Mode’.

Below steps will guide you to do so;

  1. Run Exchange PowerShell as Administrator.
  2. Run below command to drain connections on the Hub Transport service
    Set-ServerComponentState <SERVER> -Component HubTransport -State Draining -Requester Maintenance

  3. Run below command to disable cluster services
    Suspend-ClusterNode <SERVER>

  4. Run below command to prevent databases from being mounted on the server. It will also immediately move any mounted databases on the server to other servers if copies exist and are healthy.
    Set-MailboxServer <server> -DatabaseCopyActivationDisabledAndMoveNow $true

  5. Verify that no mailbox databases are mounted on the server.

    Get-MailboxDatabaseCopyStatus

  6. Run below command to block databases being automatically activated on the specified Mailbox server.
    Set-MailboxServer <server> -DatabaseCopyAutoActivationPolicy Blocked

  7. Run below command to put the state of all components together to Inactive.
    Set-ServerComponentState <server> -Component ServerWideOffline -State Inactive -Requester Maintenance

  8. Close PowerShell.

Upgrade Schema

Now that the server has been put to maintenance mode and you are ready to proceed with the Upgrade process. First thing we need to do is, update the schema. This is critical because without the required schema updates, the setup will not proceed.

  1. Run command prompt as Administrator.
  2. Run below command
    Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

  3. Run below command
    Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms

  4. Run below command
    Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

  5. Navigate to the Exchange installation binaries folder.
  6. Run setup.exe.
  7. On the Check for Updates window, select your choice and click OK.
  8. On the Upgrade window, click Next.
  9. On the License Agreement page select “I accept ….” and click Next.
  10. On the Readiness Check page, ensure no prerequisites are pending. Click Next to proceed.
  11. Now the installation will proceed.
  12. Once the installation is finished, click Finish.
  13. Restart the server.

 

Now that the server has been upgraded successfully we can resume Exchange services back to normal mode. Follow the below steps.

  1. Run Exchange PowerShell as Administrator.
  2. Run below command to enable connections on the Hub Transport service.
    Set-ServerComponentState <SERVER> -Component HubTransport -State Active -Requester Maintenance

  3. Run below command to enable cluster services
    Resume-ClusterNode <SERVER>

  4. Run below command to enable databases from being mounted on the server.
    Set-MailboxServer <server> -DatabaseCopyActivationDisabledAndMoveNow $false

  5. Run below command to enable databases being automatically activated on the specified Mailbox server.
    Set-MailboxServer <server> -DatabaseCopyAutoActivationPolicy Unrestricted

  6. Run below command to put the state of all components together to Activate.
    Set-ServerComponentState <server> -Component ServerWideOffline -State Active -Requester Maintenance

  7. Close PowerShell.

Now that your server is back in the game. It’s time to upgrade the second server. Perform the steps above from the beginning to do so.

Happy Upgrading!! J

Written by judeperera

January 6, 2017 at 11:45 am

Field Report: Cisco and Exchange woes. 451 5.7.3 Cannot achieve Exchange Server authentication. Telnet fails with 200*****

leave a comment »

So this was a strange case that i came across with. Here’s my setup.

env

 

In this scenario, strangely it was noticed that the mail is getting queued in our Exchange 2013 environment which is bound to Exchange 2010 mailboxes. Upon the error notification, we identified the below error.

451 4.4.0 Primary target IP address responded with: “451 5.7.3 Cannot achieve Exchange Server authentication.” Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts.

Additionally, upon doing a telnet from the Server B range to Server A range, we noticed that we are not getting the expected banner while telnet’ing port 25 as below.

Also, EHLO command fails with 500 5.3.3 Unrecognized command

2

If i try to telnet the relevant server from the same subnet, I get the correct HELO banner and the EHLO reply as below.

3

If you are getting the above error, it could be due to two reasons.

  1. The sending Exchange Server do not have authentication permission on the receiving Exchange Server(s).
    If this is the case, that means the Exchange server which we cannot send email to doesn’t have it’s Exchange Server Authentication enabled. Because of this, when the Sending Exchange server is trying to authenticate against the receiving servers, it fails. To resolve this we need to set Exchange Server Authentication enabled, on the recipient servers receive connector.

    1. Login to your Exchange Hub Transport Server where you cant send the mail to.
    2. In the Exchange Management Console, expand Server Configuration in the console tree, and select Hub Transport.
    3. In the work pane, select the Receive Connectors tab.
    4. Double-click the Default Receive connector.
    5. Ensure that the Exchange Server authentication option is selected.
      4
    6. Click OK to save.
    7. Restart the Microsoft Exchange Transport Service.
  2. There could be a Cisco firewall sitting between the Exchange Servers.
    This could be a trike situation. If you have placed a firewall between servers due to whatever reason and if your firewall is a Cisco FWSM,  Cisco PIX or Cisco ASA then the Mailguard feature might be the cause. (In my case, this was the culprit)
    If this is your issue. you need to disable ESMTP inspection. Microsoft has a KB article written which you can check out so the network team can set it up accordingly.

    https://support.microsoft.com/en-us/kb/320027

Did it work? Anything else you want to add? let us know..

Field Report: Cant Move/Delete emails with attachments

with one comment

Issue:


Users were complaining that they are unable to delete mails from outlook. The environment was Exchange Server 2010 with SP1. Users were on Outlook 2010. Strangely, the issue was not seen for all the users. However there were multiple complaints from a few users that they are getting an error message while trying to delete old emails.

Upon checking up with the users, here’s what was noticed:

  • While using outlook:
    • User cannot delete email with attachments.
    • User cannot move email with attachments.
    • Upon deleting and moving the below error message is thrown.
    • User can move/delete emails that does not have any attachments.
Some items cannot be deleted. They were either moved or already deleted, or
access was denied.

More Details>>

This error may be the result of trying to delete more than 4,000 messages at 
one time.  Outlook can delete no more than 4,000 messages when it is working 
with a server message store.
To avoid this error, delete fewer than 4,000 messages in a single operation.
It is also possible that you do not have the appropriate permissions to 
delete messages. If you need to delete content from a folder owned by someone 
else, contact the owner of the folder to obtain the necessary permissions, 
or have the owner delete the content for you.

das

Now this was quite strange. After looking up for a bit, I came up with a similar issue. However it was related to Exchange Server 2010 RU6 or SP3 environments. If you have such an environment, you might want to look over here: https://support.microsoft.com/en-us/kb/2822208

In our case, it was Exchange Server 2010 SP1. So it had to do with something else.

Cause:


One gotcha out of the symptoms was that it had to do with something related to attachments. So while going through message sizing parameters for user, rules, database, connectors and Transport Services, it was identified that the there was a change done to the Exchange Transport Settings on the Hub Transport Server.  The Maximum Send/Receive sizes was reduced. And that my friend, was the culprit! Here’s what has happened.

Initially, the Maximum Send/Receive sizes were at 10MB. Life was so good. People could send and receive attachments, they were able to delete, move and so much more. Suddenly the administration decided to reduce the sizes to 3MB. This seemed to be the main cause for our issue. Due to some reason (I’m yet to get deep into what relationship it has) mails that’s got attachments greater than the Maximum Send/Receive size (3MB) are not getting deleted.

Resolution:


  1. Open Exchange Management Console.
  2. Navigate to Organization Configuration, Hub Transport , Global Settings , Transport Settings , General tab.
  3. Change the below values to a reasonable value (in my case it was 10MB, the previous value)
    1. Maximum receive message size (MB)
    2. Maximum send message size (MB)
  4. Restart Microsoft Exchange Transport service.
  5. Ask your users to close Outlook and check if the mail items can be deleted now.

That was it. Things worked and they were able to delete the emails.

Bit more information regarding why this happens, I’ll look into more and post over here.

If you have any inputs do comment over here.

 

Written by judeperera

October 27, 2016 at 11:52 am