Well, Well, Well Microsoft gifted us a new year gift this year as a bug in FIP-FS scan engine came in Exchange 2016 and Exchange 2019 Servers due to which queue size keeps on growing, resulting in stopping the mail flow.

Cause

The problem relates to date check failure with the change of new year and does not imply to failure of AV Engine. FIP-FS scan engine is probably the anti-malware virus scanner which is being used since Exchange 2013 times. Purpose of this engine is to scan the on-premises Exchange Server installation for malicious content. However, Microsoft claims that its not a Malware Scanning or Malware engine issue and therefore would not impact security in any prospect.

Problem Identification

To identify if the mail flow in the Exchange Servers is stopped due to this issue, we need to first check Event Viewer and look for Event Id 5300

Description would look like “The FIP-FS” Microsoft Scan Engine failed to load. PID:3676. Error Code 0x80004005. Error Description: Can’t convert “2201010005” to long.

Resolution:

1. Workaround provided by Microsoft
2. Permanent Solution provided by Microsoft
3. Verify Engine Update Information
4. Re-enabling Anti Malware Scanning
5. Check mail flow and look for FIP-FS Events in Event Viewer with event id 6027

1 – Workaround

• Disable Malware filter:

Open Exchange Management Shell as administrator and run the following:

Get-TransportAgent *Malware*| Disable-TransportAgent

Restart Transport Service

Perform the steps on all Exchange Servers starting with the server having the largest queue size.

Verify if it worked:

Get-ExchangeServer  |% {Get-TransportAgent "Malware Agent"}

[PS] C:\>Get-ExchangeServer | % {Get-TransportAgent "Malware Agent"}

Output

• Bypass Malware Filtering Server

Open Exchange Management Shell as administrator and run the following:

Set-MalwareFilterServer - BypassFiltering $True -Identity <Server-Name>

Restart Transport Service

Verify if it worked:

Get-ExchangeServer|% Get-MalwareFilteringServer

[PS] C:\>Get-ExchangeServer | % {Get-MalwareFilteringServer}

http://amupdatedl.microsoft.com/server/amupdate

2 – Permanent Solution provided by Microsoft

Microsoft released a script that can help mitigate the issue permanently. There are ideally 2 ways to perform the fix viz Automated Solution and Manual Solution.

Key points:

1. Whether you try Manual Way or Automated Solution the steps should be performed on each Exchange 2016 and Exchange 2019 Servers in order to get the desired results.
   
2. If you have disabled or bypassed anti-malware scanning as you might have performed the workaround in order to get rid of the issue, then we recommend you to re-enable antimalware scanning after performing the permanent fix provided by Microsoft.
   
3. Automated Solution can be performed on multiple Exchange Servers in parallel as well. But we should ideally start with the Exchange Server having the largest queue size.
   
   Automated Solution: These actions can be automated with the scan engine reset script from https://aka.ms/ResetScanEngineVersion 
   
   Steps to perform:
   
   – Run Exchange Management Shell as administrator
   – Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
   – Change the path to the scripts folder
   – Run the script .\ResetScanEngineVersion.ps1

[PS] C:\>Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
[PS] C:\>cd C:\scripts
[PS] C:\scripts>.\Reset-ScanEngineVersion.ps1
EX01-2019 Stopping services...
EX01-2019 Removing Microsoft engine folder...
EX01-2019 Emptying metadata folder...
EX01-2019 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EX01-2019 Starting engine update...
Running as EXOIP\administrator.
--------
Connecting to EX01-2019.exoip.local.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate

Manual Solution: In order to resolve the issue without an automated script you can follow the below mentioned steps:

A – Remove existing engine and metadata

Steps to perform:

– Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.

– Use Task Manager to ensure that updateservice.exe is not running.

– Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.

– Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

B – Update to latest engine

Steps to perform:

– Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.

– Open the Exchange Management Shell in admin mode.

– Navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts).

– Run .\Update-MalwareFilteringServer.ps1.

– [PS] C:>cd "$env:ProgramFiles\Microsoft\Exchange Server

Output

[PS] C:\>cd "$env:ProgramFiles\Microsoft\Exchange Server\V15\Scripts"
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Update-MalwareFilteringServer.ps1 EX01-2019

Running as EXOIP\administrator.
--------
Connecting to EX01-2019.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate

3 – Verify Engine Update Information

Whether you have performed Manual Solution or an Automated one you need to check if our fixed actually worked. In order to do that you need to follow the below mentioned steps:

• Start Exchange Management Shell in Admin Mode.

• Run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.

• Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 or higher.

Output

[PS] C:\>Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

[PS] C:\>Get-EngineUpdateInformation


Engine            : Microsoft
LastChecked       : 01/02/2022 08:22:33 AM +01:00
LastUpdated       : 01/02/2022 08:22:41 AM +01:00
EngineVersion     : 1.1.18800.4
SignatureVersion  : 1.355.1227.0
SignatureDateTime : 01/01/2022 12:29:06 PM +01:00
UpdateVersion     : 2112330001
UpdateStatus      : UpdateAttemptSuccessful

4 – Re-Enable Anti Malware Scanning

As already mentioned above you need to perform these steps only if you have performed the workaround steps which included disabling and bypassing Malware Filter.

1. If you have disabled Anti-MalwareScanning then,
   
   Steps to perform:
   
   – Open Exchange Management Shell and run Enable-AntiMalwareScanning.ps1
   
   – Restart Microsoft Exchange Transport Service

Output

             [PS] C:\>& $env:ExchangeInstallPath\Scripts\Enable-AntimalwareScanning.ps1

             Anti-malware engines are updating. This may take a few minutes.
              Checking for engines updated after 12/26/2021 8:31:11 AM.
               Updating Microsoft. Last updated : 1/2/2022 8:22:41 AM
               WARNING: The following service restart is required for the change(s) to take effect:  MSExchangeTransport. Anti-malware scanning is successfully enabled. Please restart MSExchangeTransport  for the changes to take effect.

2. If you bypassed Anti-MalwareScanning then,

Steps to perform:

– Open Exchange Management Shell and run

[PS] C:>Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $false -Identity $_.Name}

– Restart Microsoft Exchange Transport Service

5 – Check mail flow and look for FIP-FS Events in Event Viewer

We can check the mail flow by simply running Get-queue cmdlet in Exchange Management Shell in Admin mode and can check if there are emails still stuck in queues or not.  

You need to check mail flow in all scenarios possible:

• Inbound External Mail flow
• Outbound External Mail flow
• Internal mail flow

Key Point:

It does take some time in order to get the queue to size 0 as these emails were stuck initially and now have to be released so we request you to grab a cup of coffee and monitor the situation. The queue size would definitely go down but the pace would be slow.

If you encounter Event Id 6027 FIP-FS Events in Event Viewer and face mail flow issues we would request you to restart Exchange Servers!