OPNsense causing Exchange TLS connections & some websites to intermittently fail

[sc0ttjm]

Hi,

We have a specific email domain we send to on a regular basis that will only accept TLS SMTP connections and as such has it's own SMTP SEND and RECEIVE Connectors.

Our Exchange Server has been communicating with this domain perfectly for many years.

We have recently moved the Exchange Server (Virtual Machine) from the office into a Data Center and it is now behind an OPNsense firewall.

This is the only change we have made, the server was simply shutdown, moved and powered up again, it even has the same IP settings.

Since the move, we are getting intermittent rejections as the server has failed to negotiate a TLS encrypted session.

I could really do with some help on where to start trying to diagnose this as I can't see anything wrong and nothing else has changed. 

The recipient's IT team have told me this:

"This NDR usually occurs when the connecting mail servers fail to offer a certificate for the TLS handshake and attempts to communicate in plain text. We requires and force the use of TLS encryption, and any connecting mail server that is not capable of using TLS encryption will be rejected in this manner.

The external influence of proxies can also produce a similar issue. Cisco Firewalls with ‘Mail Inspect” enabled are a good example of this."

Can anybody offer any assistance with this?

Thanks in advance
Scott

[franco]

Hi Scott,

What version is this? And is it LibreSSL flavour?


Cheers,
Franco

[sc0ttjm]

I'm using Exchange 2010 (until we can migrate to MS365) and the firewall is OPNsense Business 21.4.1.
The recipient supplies the SSL Cert and we are using Mutual Auth TLS.

[franco]

Hi Scott,

Thanks. I know of LibreSSL fails in certain versions but with business being OpenSSL with a recent version I do not suspect any actual errors in the connections made from the software. That leaves connection state aborts or another component between the OPNsense and one of the Exchange servers that messes with the connection.

The ActiveSync protocol over HTTP(S) has some severe oddities that have been known to break proxies as well... I would try to make sure that no connections are being dropped by the default rule. If that is the case you need to add a pass rule for ActiveSync source IP/port to disable state tracking selectively.


Cheers,
Franco

[Fright]

Hi
any clue in the transport logs on Exchange?
maybe the receiving server wants tls1.2 only but it is not allowed on exch, for example?
and the connection log from the receiving server would be very helpful

[sc0ttjm]

This seems to have been an issue with the setting "IP Do-Not-Fragment" under Firewall > Settings > Normalization.

This setting was turned on to address an issue we were having with some websites that use SSL and some remote management tools we use like splashtop, that did not like fragmentation.

We also had to change the MTU to match the Data Centre's MTU size and I think that's what broke exchange connections.

I turned off "IP Do-Not-Fragment", applied the settings, turned it back on, applied the settings again and then everything started working straight away again.

It seems if you make any changes to MTU, you have to undo the "IP Do-Not-Fragment" setting then re-apply it afterwards.

[Fright]

glad it works
maybe @franco will correct me, but I don't see any indication in the filter.inc script why this might be happening. is it possible that you specified mss in the interface settings?

[sc0ttjm]

Thanks @Fright, I haven't ever specified any MSS settings.

[sc0ttjm]

glad it works
maybe @franco will correct me, but I don't see any indication in the filter.inc script why this might be happening. is it possible that you specified mss in the interface settings?

Hi @Fright, strange thing is it happened again this morning with a secure website that doesn't like fragmented packets, I turned off the "IP Do-Not-Fragment" setting, applied it, turned it back on, applied it and the website loads again.
It seems I have to keep doing this for it to work.
Any ideas?

[Fright]

Hi
I'm still not sure if it is about applying the "IP Do-Not-Fragment" setting.
you can always check if the setting is applied in the scrub rules in Firewall: Diagnostics: pfInfo: Rules
there should be a 'no-df' directives in scrub rules

can you try to just restart the PF when the problems with webserver connection starts?

I'm afraid that deeper diagnostics will be required (packet capture etc)

[sc0ttjm]

Thanks @Fright, I'm new to OPNsense, so thanks for pointing that out.
It does indeed say "@0 scrub on hn0 all no-df fragment reassemble"

All I know is that when the users complain of issues reaching certain HTTPS sites, I turn off "IP Do-Not-Fragment" then turn it back on and the problem has gone for a while.

[Fright]

hi!
I read your github ticket and  captured traffic from the https://integrated.infotrack.co.uk/
this host sets the ID Field to zero with DF flag
so can you try to use random-id directive as the man says?
Unfortunately some operating   systems   also generate their
      dont-fragment packets with a   zero IP   identification field.  Clear-
      ing the dont-fragment bit on   packets   with a zero IP ID may cause
      deleterious results if an upstream router later fragments the
      packet.  Using the random-id   modifier (see below) is   recommended in
      combination with the   no-df modifier to ensure unique   IP identi-
      fiers.

[sc0ttjm]

Hi @Fright, Thanks for the suggestion.
Just to update you, it happened again today, I tried restarting PF from the "lobby" page as previously suggested to see if that also cured it, but that didn't work.
I did the usual, "IP Do-Not-Fragment" off, apply, on, apply and it started working again.
Other sites are effected but the infotrack page is used daily in production so is the one noticed most often and causes the most disruption.

[Fright]

hi
sorry for delay.
I still think it's not about opnsense and not the no-df directive.
can you try to clear the counters

Code: [Select]

pfctl -Finfo and show the counter values (Counters and Limit Counters from Firewall: Diagnostics: pfInfo)when the problem starts again?
maybe a message like "PF frag entries limit reached" appears in the logs when the problems start?