Skip to content

If the emails remain on the Exchange server and cannot be forwarded to the smarthost for sending, it may be because the certificate bound to the corresponding connector no longer exists or has been expired. Of course, it is also possible that the expected subject alternate name (SAN) is missing or incorrect. In that case you may receive an error stating:

454 4.7.5 The certificate specified in TlsCertificateName of the SendConnector could not be found

You can verify whether you have such an issue by checking the mail queue:

Get-Queue

In case you have a lot of mails stuck in one of your mail queues you can further investigate the affected queue by running:

Get-Queue <queue name>

e.g. Get-Queue "SERV-MAIL\3" | fl

Having a look at the LastError property reveals the aforementioned error.

In my case the outbound Office 365 Send Connector was involved. In order to fix this I had to issue the following commands:

$TLSCert = Get-ExchangeCertificate -Thumbprint <thumbprint of valid certificate>
$TLSCertName = “$($TLSCert.Issuer)$($TLSCert.Subject)”
Get-SendConnector -identity “<send connector name>” | Set-SendConnector -TlsCertificateName $TLSCertName
Restart-Service MSExchangeTransport

You have to replace the thumbprint accordingly, i.e. matching your own certificate's thumbprint.

Run Get-ExchangeCertificate cmdlet

The procedure would be the same for all other Send Connectors or Receive Connectors.

By the time you go back to Queue viewer the queues should have started to empty.

Further reading:

https://docs.microsoft.com/en-us/exchange/troubleshoot/email-delivery/mails-issues-occur-after-april-15-2016

https://www.petenetlive.com/KB/Article/0001631

Just recently I came across an interesting support case which involved an Exchange 2010 Offline Address Book (OAB) and Outlook 2010 clients trying to download it. The affected users received  an error stating: Error 0x80190194 - The operation failed. Problem is, this is a very common error when downloading the OAB and there are many server side problems which can generate this error:


This error can be caused by many a issues, e.g.

  • missing IIS and/or folder permissions (server-side)
  • IIS authentication issues (server-side)
  • IIS misconfigured http redirection (server-side)
  • file replication service issues (server-side)
  • missing or misconfigured OAB distribution settings (server-side)
  • OAB generating mailbox issues (server-side)
  • DAG replication issues and arbitration mailbox (server-side)
  • missing or misconfigured proxy settings (client-side)
  • download issues in terms of BITS (client-side)
  • et al

Troubleshooting methodology

In order to identify the affected OAB that my users tried to download I first had to get hands on the corresponding OAB GUID by running:

Get-OfflineAddressBook | ft Name,GUID

With the OAB GUID identified I started testing with a browser by navigating to the corresponding URL https://<ServerFQDN>/OAB/<GUID>/oab.xml, checking each server separately. I thereby received an Error 404 - Not Found on one of my servers, which in turn resulted in the aforementioned Outlook error 0x80190194 for my users. This error (which is basically a 404) appeared sporadically depending on the Exchange 2010 Server they were redirected to through our Load Balancer:

Further research on the server showing the Error 404 in particular showed that OAB with GUID 72189f79-62fa-4bcf-82ea-56dc45cfdeb0 is missing in IIS under the OAB node, thus assuming that this particular OAB has not been replicated between my Exchange 2010 Servers:

Missing OABs in the file system ... check the Microsoft Exchange File Distribution Services (MSExchangeFDS) on the affected server

The following location on the Client Access Server can be checked to see if the OAB files have been replicated:

  • %ExchangeInstallPath%ClientAccess\OAB\<GUID>

Permissions should also be checked. If any of the default permissions are locked or are missing, the OAB files might not be replicated. Have a look at How to solve Internal Error 500 when testing connection to the OAB or Exchange 2010: OAB download error 80190194.

I copied the missing web.config file as well an restarted the MSExchangeFDS service, to no prevail. Then I executed Get-OfflineAddressBook "<OAB Name>" | Update-OfflineAddressBook to no prevail. Finally I execute iisreset to no prevail.

Well, checking the distribution methods for the affected OAB revealed that the 2nd Exchange 2010 Server was missing in the Distribution tab of the OAB's Properties. I should have done this in the first place 🙂

After adding the missing Exchange 2010 Server and restarting the Microsoft Exchange File Distribution service by executing Restart-Service MSExchangeFDS the replication started immediately. The missing OAB showed up in IIS as well as in the folder structure.

Alternatively you could run the following cmdlet to initiate the OAB replication:

Update-FileDistributionService <ServerFQDN> -type oab

But even after a successful OAB replication I still received an http error 500 - Internal Server Error when acessing the OAB via browser in order to verify that everything's fine:

So, back to the web.config file, that was presumably missing the first time around. What does a web.config file actually do when placed in the OAB file directory:

When you configure Http Redirection a web.config file is created in the OAB directory. This file has incorrect permissions. Assign Read and Read & Execute permission to Autheticated Users group then restart IIS using iisreset /noforce.
Now you can try to download the OAB using Outlook. It may be required to download it twice because sometimes the name of the OAB doesn't appear at first try.

Well, that didn't count for me so I checked the web.config and IIS settings to verify whether any settings have been adjusted in the past that I didn't know of. As that wasn't the case I deleted it, but the issue still persisted.

Verify OfflineAddressBook property on the affected user's mailbox and adjust the missing OAB association:

Get-Mailbox <User>| fl Name,OfflineAddressBook
Get-Mailbox <User> | Set-Mailbox -OfflineAddressBook "<OAB Name>"

Further troubleshooting on the client-side:

With a corresponding Event Log entry on the client computer, Event ID 27, Source: Outlook, the issue can be further analyzed by following Microsoft's KB843483 article:

Protokollname: Application
Quelle:        Outlook
Datum:         05.12.2018 12:28:31
Ereignis-ID:   27
Aufgabenkategorie:Keine
Ebene:         Informationen
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      <Computername>
Beschreibung:
Starting OAB download. (See event data).
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Outlook" />
    <EventID Qualifiers="16384">27</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-12-05T11:28:31.000000000Z" />
    <EventRecordID>18716</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Computername</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Starting OAB download. (See event data).</Data>
    <Binary>0200000000000000730100007301000073010000000000000000000000000000E9FD00000000000000000000D05B00003C600000306A0000286B0000F8670000601B2E13520B0000000000005C0000002000000005000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000005C0063006F006E0073006F0072007300660069006E0061006E007A002E006400650020002D00200041006C006C006500000031623662323737382D643765332D343732612D616234622D64356335343264343836333000</Binary>
  </EventData>
</Event>
Protokollname: Application
Quelle:        Outlook
Datum:         05.12.2018 12:28:45
Ereignis-ID:   27
Aufgabenkategorie:Keine
Ebene:         Warnung
Schlüsselwörter:Klassisch
Benutzer:      Nicht zutreffend
Computer:      <Computername>
Beschreibung:
OAB Download Failed. (Result code in event data).
Ereignis-XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Outlook" />
    <EventID Qualifiers="16384">27</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-12-05T11:28:45.000000000Z" />
    <EventRecordID>18717</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Computername</Computer>
    <Security />
  </System>
  <EventData>
    <Data>OAB Download Failed. (Result code in event data).</Data>
    <Binary>0200000094011980730100007301000073010000000000000000000000000000E9FD00000000000000000000D05B00003C600000306A0000286B0000F8670000601B2E13520B0000000000005C000000200000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000</Binary>
  </EventData>
</Event>

In my case the error messages stated exactly what I tried to achieve:

  • Result Code 2, i.e. You forced a full .oab file download manually.

Furthermore I checked some other client-side settings and known issues that could cause an OAB download error, such as whether BITS has some kind of problem when trying to download the OAB. An error 0x80200049 is often caused from the BITS job list being full. To fix this, you must clear/reset the BITS job list. Microsoft outlook uses BITS to download the OAB and if the BITS queue goes full it simply stops:

bitsadmin /list /verbose

bitsadmin /reset

The client-side proxy settings, i.e. the client should have  unobstructed access to Exchange via https:

netsh winhttp show proxy

netsh winhttp reset proxy

netsh winhttp set proxy <proxy>:<port>

In the end the last thing I had to ensure and configure was to enable GlobalWebDistribution for all my Exchange 2010 OABs in order to prevent server-specific connections when clients try to download the OAB:

Get-OfflineAddressBook | Where {$_.ExchangeVersion.ExchangeBuild.Major -Eq 14} | Set-OfflineAddressBook -GlobalWebDistributionEnabled $True -VirtualDirectories $Null

In my particular case that did it: the issue was resolved and setting the VirtualDirectories property to $Null reverted my solution attempt that I suggested previously 🙂 And I can tell you why: because my environment consisted of Exchange 2010 and Exchange 2016 Servers due to being in the middle of a migration.

Update:

With Exchange 2013 onward the CAS role proxies the OAB download request to an appropriate Mailbox role server. The CAS role maintains a log of each request it handles in the log files, present in folder %ExchangeInstallPath%\Logging\HttpProxy\OAB\. These log files are an excellent tool to identify which mailbox server the CAS chose to serve the request. Download issues can be analyzed with log files found in %ExchangeInstall%\Logging\OABDownload. The OAB generation logs can be found in the \Logging\OABGeneratorLog folder.

Maybe one of the steps outlined above will help you, too, to get rid of issues with downloading OABs for good.

Further reading:

Another Exchange 2016 migration, another error while trying to migrate custom Receiver Connectors from Exchange 2010 to 2016 ...  an analysis.

...continue reading "Exchange 2016 – Migration of existing, custom Receive Connectors from Exchange 2010 to 2016 results in multiple errors"

While migrating from Microsoft Exchange 2010 to Exchange 2016 we came upon a typical issue in which Outlook keeps giving the message: "The Microsoft Exchange Administrator has made a change that requires you quit and restart Outlook". 

This means that users were totally unable to connect to Exchange during the migration and co-existence phase:

The reason for that was quite obvious (at least after we did some research and thinking) as we switched the Client Access from the old Exchange 2010 servers to the newly implemented Exchange 2016 servers, i.e. all client connections have been directed to Exchange 2016 only. As no mailboxes have been migrated at this point in time client connections need then to be down proxied to Exchange 2010 by Exchange 2016. All things considered it could only have been a question of the protocol being used by our Outlook 2010 clients. Assuming that our Outlook 2010 clients try to connect via RPC-over-TCP (as is the default setting configured in all internal Outlook 2010 profiles) this is an expected behaviour as Exchange 2013 and later no longer support RPC-over-TCP connections from client applications. They only support RPC-over-HTTP, in later versions MAPI-over-HTTP (MAPIhttp). An automatic switch in protocol to RPC-over-HTTP does not occur, thus Outlook 2010 won't find a valid connection end point and cannot be down proxied to Exchange 2010. The importance of this consideration is that if Outlook clients are using RPC-over-TCP prior to being migrated, it becomes even more important to have a process to update those profiles properly.

Required Outlook 2010 version:

In an Outlook profile under Exchange Proxy Settings you’ll find two options regarding fast and slow networks:

  • On fast networks, connect using HTTP first, then connect using TCP/IP
  • On slow networks, connect using HTTP first, then connect using TCP/IP

In order to understand what this setting really means have a look at the article Outlook Anywhere: Fast vs. Slow network connection over HTTP or TCP/IP:

Outlook Anywhere; Emphasis on “Anywhere”
First of all, lets take a closer look at the terms being used here:

  • TCP/IP connection
    This is the traditional (internal) direct-to-Exchange connection also known as a “RPC over TCP” connection or as a (not entirely technical correct) MAPI connection.
  • HTTP or HTTPS connection
    This is the over-the-Internet connection introduced in Outlook 2003 also known as a “RPC over HTTP” connection and nowadays knows as “Outlook Anywhere”.
    As of Outlook 2013 SP1 in combination with Exchange 2013 SP1, this is a MAPI over HTTP connection or simply: MAPI/HTTP.

The description for the HTTP connection doesn’t really hold true anymore as the HTTP connection can also be used internally. In fact, over the past few Exchange versions, the trend was to move away from the direct RCP connections and towards HTTP connections, even internally. Actually, as of Exchange 2013, all Outlook connectivity is taking place via Outlook Anywhere.

So, with having the option On fast networks, connect using HTTP first, then connect using TCP/IP being disabled we receive the aforementioned error:

With having the option On fast networks, connect using HTTP first, then connect using TCP/IP being enabled the connection can be established successfully:

Now, in order to rollout such a configuration change on a large scale (after careful testing of course!) you could utilize GPOs by either configuring the corresponding Outlook setting or by using GPP and manipulating the user's registry settings directly.

The required Group Policy Setting can be found in the corresponding Microsoft Office ADMX Templates under:

  • User Configuration | Policies | Administrative Templates | Microsoft Outlook 2013 | Account Settings | Exchange
  • User Configuration| Policies | Administrative Templates | Microsoft Outlook 2016 | Account Settings | Exchange

Please keep in mind that this setting is unavailable with Office 2010 ADMX.

Alternatively you could add the setting manually via GPP (Group Policy Preferences) under:

  • User Configuration| Preferences | Windows Settings | Registry

This XML based code looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<Registry uid="{ADF10FBF-F683-4549-A19E-885D5A58D23E}" changed="2018-11-26 11:04:59" image="0" status="RPC" name="RPC" clsid="{9CD4B2F4-923D-47f5-A062-E897DD1DAD50}">
<Properties name="" value="" type="REG_SZ" key="Software\Policies\Microsoft\office\14.0\outlook\RPC" hive="HKEY_CURRENT_USER" default="0" displayDecimal="1" action="C"/>
</Registry>
<?xml version="1.0" encoding="UTF-8"?>
<Registry uid="{0C4FA337-45F4-4A16-965F-48155F45186F}" changed="2018-11-26 11:05:56" image="10" status="ProxyServerFlags" name="ProxyServerFlags" clsid="{9CD4B2F4-923D-47f5-A062-E897DD1DAD50}">
<Properties name="ProxyServerFlags" value="0000002F" type="REG_DWORD" key="Software\Policies\Microsoft\office\14.0\outlook\RPC" hive="HKEY_CURRENT_USER" default="0" displayDecimal="1" action="C"/>
</Registry>
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Policies\Microsoft\office\14.0\outlook\rpc]
"ProxyServerFlags"=dword:0000002f

With having the option On fast networks, connect using HTTP first, then connect using TCP/IP being enabled by GPO or GPP, the settings are greyed out, cannot be changed by the user, and the connection can be established successfully:

The Outlook Connection Status now shows: RPC/HTTP

Prior to switching Client Access from Exchange 2010 to Exchange 2016 all Outlook 2010 clients connected via RPC-over-TCP, which shows as: RPC/TCP

With this configuration adjustment we were able to proceed with the Exchange 2016 migration without having any Outlook 2010 connectivity issues.

Further reading:

As part of my Security Best Practices regarding Microsoft Exchange and Microsoft IIS I always implement a couple of configuration settings to harden the underlying IIS, e.g.

  • disabling the "X-AspNet-Version" header,
  • disabling deprecated and/or unsecure protocols,
  • disabling deprecated and/or unsecure ciphers,
  • setting up for SSL Perfect Forward Secrecy,
  • enabling TLS 1.2,
  • et al

In order to tighten your security on Exchange 2016's IIS you should at least start with enabling HTTP Strict Transport Security (HSTS) which I'm going to describe here. As per Microsoft:

HTTP Strict Transport Security (HSTS), specified in RFC 6797, allows a website to declare itself as a secure host and to inform browsers that it should be contacted only through HTTPS connections. HSTS is an opt-in security enhancement that enforces HTTPS and significantly reduces the ability of man-in-the-middle type attacks to intercept requests and responses between servers and clients.

There are a couple of recommendations and Best Practices available that give further information on how to harden Windows Server 2012 R2/2016 as well as Exchange 2013/2016, respectively. In this article I will focus on HSTS only as I didn't find any particular articles outlining this issue [...]. Please check the Further reading section at the bottom of thids article for more information, e.g. IIS Crypto by Nartac Software:

IIS Crypto is a free tool that gives administrators the ability to enable or disable protocols, ciphers, hashes and key exchange algorithms on Windows Server 2008, 2012 and 2016. It also lets you reorder SSL/TLS cipher suites offered by IIS, implement best practices with a single click, create custom templates and test your website.

You could always have a look at Center for Internet Security's (CIS) Benchmark Documents as well, e .g. for Microsoft Exchange Server 2016, which provides a "prescriptive guidance for establishing a secure configuration posture for Microsoft Exchange Server 2016" and "was tested against Microsoft Exchange Server 2016."

Windows Server 2012 R2 (IIS 8.5)

1. Open the Internet Information Services (IIS) Manager console, and click your server. Then click HTTP Response Headers in the IIS section of the middle pane:

2. Click Add in the Actions pane on the right, enter the following values in the Add Custom HTTP Response Header dialogue window, then click OK:

  • Name: Strict-Transport-Security
  • Value: max-age=31536000

3. The newly added Custom HTTP Response Header will be added to the list of configured HTTP Response Headers in the middle pane:

4. The new setting will become effective immediately. You don't have to iisreset your Exchange server.

Windows Server 2016 (IIS 10)

With IIS 10.0 version 1709 onwards Microsoft has implemented native HSTS support. Have a look at IIS 10.0 Version 1709 Native HSTS Support on how to configure HSTS in Windows Server 2016 version 1709+ via Powershell:

Import-Module IISAdministration
Reset-IISServerManager -Confirm:$false
Start-IISCommitDelay

$sitesCollection = Get-IISConfigSection -SectionPath "system.applicationHost/sites" | Get-IISConfigCollection
$siteElement = Get-IISConfigCollectionElement -ConfigCollection $sitesCollection -ConfigAttribute @{"name"="Contoso"}
$hstsElement = Get-IISConfigElement -ConfigElement $siteElement -ChildElementName "hsts"
Set-IISConfigAttributeValue -ConfigElement $hstsElement -AttributeName "enabled" -AttributeValue $true
Set-IISConfigAttributeValue -ConfigElement $hstsElement -AttributeName "max-age" -AttributeValue 31536000
Set-IISConfigAttributeValue -ConfigElement $hstsElement -AttributeName "redirectHttpToHttps" -AttributeValue $true

Stop-IISCommitDelay
Remove-Module IISAdministration

The new setting will become effective immediately. You don't have to iisreset your Exchange server.

Verification

You can check whether HSTS has been successfully implemented by browsing to SSLLabs' SSL Server Test page and enter the server's corresponding hostname (in case it is publicly resolvable and directly reachable from the internet, which often is the case with SMBs):

That's it! You have successfully enabled HSTS.

Further reading:

Just recently I ran into an error during initial setup of Exchange 2016 on a newly installed Windows Server 2012 R2 stating: "Service 'MpsSvc' failed to reach status 'Running' on this server". Further down the troubleshooting road I found out that this quite common error of not being able to start the Windows Firewall service is not Exchange Server specific.

...continue reading "Exchange 2016 – Setup fails with error “Service ‘MpsSvc’ failed to reach status ‘Running’ on this server”"

As I did quite some Exchange migrations (including single as well as multi tenancy environments) I was always wondering whether there are any additional post migration tasks that need to be addressed before you start removing your legacy Exchange server(s) from your organization. For instance, as with migrating from Exchange 2003 to, say, Exchange 2010 you had to update Global Address Lists, Address Lists, Offline Address Books, et al to the new Exchange 2010 version by executing the corresponding cmdlts, e.g. Move-OfflineAddressBook. Now when migrating from Exchange 2010 to Exchange 2016 (or even just from Exchange 2013 to Exchange 2016 with both having an almost identical architecture) what remains of those post migration tasks? Which ones are a necessity? What kind of architectural changes require you to think different and adjust certain settings prior to removing your legacy Exchange server(s)?

By talking about post migration tasks, what do I mean? Well, after you've successfully migrated all your databases, mailboxes, public folders, send/receive connectors, et al, what's still left? Maybe the not so obvious parts of your Exchange infrastructure that need addressing as well in order to avoid any unwanted behaviors and/or errors after you removed the last legacy Exchange server from your organization.

...continue reading "Exchange 2016 Migration – Finishing Touches – Post Migration Tasks"

After logging into Exchange 2016's ECP you receive an HTTP Error 500 (same goes with OWA):

06-12-_2016_12-23-00

Searching the internet ends up with several possible solutions to this issue, ranging from missing System Attendant Mailboxes, to bogus ADSI settings regarding the Exchange CAS Service. I tried several of them to no prevail.

...continue reading "Exchange 2016 – HTTP Error 500 after logging into ECP/OWA"

Exchange 2016 - Poor Outlook 2016 Performance - Troubleshooting - Server-side or Client-Side?

Just recently I came across a newly installed Exchange 2016 environment and had to analyze a "poor performance issue". I started my investigation by asking ... Google! Of course, everybody does it, but is not willing to commit it. This left me with the idea of creating another blog article on this matter. On the bottom of my blog article you'll find all relevant URLs which gave me (more or less) useful information during my time debugging the problem. As it turned out, there are multiple aspects to consider.

...continue reading "Exchange 2016 – Poor Outlook 2016 Performance – Troubleshooting"

After migrating to Exchange 2013 and/or 2016, and still having a couple of Microsoft Outlook 2007 installations left, the following issue started popping up: as soon as users launch their Outlook 2007 (while already being migrated to Exchange 2013/2016), they were always prompted for their Logon Credentials, though Remember my password has been checked:

outlook_2007_logon_02

...continue reading "Outlook 2007 and Exchange 2013/2016 – Keeps Prompting for Logon Credentials"