Problem with MailServer behind opnsense (OpenVPN+fixed IP+Port Forwarding)

[mbach]

Dear all,

first of all I want to say, that opnsense is a really nice and reliable firewall solution.
After having a steep learning curve we now got very reliable results in comparison to pfsense.
With pfsense (a nice firewall solution as well) we did many times crash our configuration without any clue, why this happened.
With opnsense everything works as it should - besides a nasty problem with our mailserver.

Problem (configuration details see below):
Mailserver with fixed IP over OpenVPN-Client-DialIn to an IP-Provider. NAT of ports 25 (SMTP), 143 (IMAP with StartSSL), 587 (Submission with StartSSL). NAT-Outbound rules are set.
The mailserver is working well. Mail can be received and sent by SMTP (25) to the outside world.
Clients can fetch and send email by IMAP (143)and SMTP (587) from all LAN interfaces.
But #1: clients cannot fetch and send email from outside our LAN.
But #2: Sometimes it works from outside our LAN...

Detailed problem description:
External IP of mailserver: 42.8.5.2 (changed for example)
Internal IP of mailserver: 10.2.1.100 (changed for example)
DNS-name of mailserver (split DNS): mail.domain.it
That means: ping domain from outside our LAN = IP is 42.8.5.2
ping domain from inside our LAN = IP is 10.2.1.100
We are testing the mailserver with: telnet mail.domain.it 143 and telnet 42.8.5.2 143

The problem is: it works - sometimes...
Firewall tells us: connection (TCP) established
But if we test with telnet from different WAN-line or by LTE we cannot always connect with telnet.
Two lines do work, other lines do not work...
Our log does not tell us any blocked situation.

Our configuration:
OpenVPN-client to IP-Provider (Portunity) to get a fixed IP.
WAN-DialIn with Fritzbox-Routers. Opnsense ? exposed host.
IP of Routers: 10.3.201.1 | 10.3.202.1 | 10.3.203.1 - mmh, one LAN interface does have the same IP range, but different Mask.... opnsense has fixed IPs according to those IPs: 10.3.201.2 | 10.3.202.2 | 10.3.203.2
Mask: 255.255.255.0
Could that be part of the problem?

On the Portunity-Interface we have the following firewall rules:

Rule on "WAN-Portunity":
Proto                Source          Port   Destination  Port   Gateway            Schedule  Description
IPv4 TCP          *                  *       10.2.1.100  25      *                                         NAT
IPv4 TCP/UDP   *                *       10.2.1.100  587    *                                         NAT
IPv4 TCP/UDP   *                *       10.2.1.100  143    *                                         NAT
_____________

Rule on "DMZ" (where the mailserver is located):
Proto                Source          Port   Destination  Port   Gateway            Schedule  Description
IPv4 *               *                  *        DMZ net       *       *
...
IPv4 *               10.2.1.100 *        *                   *       WAN_Portunity
...
_____________

NAT Port Forwarding
                                               Source                        Destination                 NAT
If                         Proto          Address          Ports   Address          Ports   IP                   Ports
WAN_Portunity  TCP            *                     *          *                      25       10.2.1.100    25
WAN_Portunity  TCP/UDP   *                     *          *                      587     10.2.1.100    587
WAN_Portunity  TCP/UDP   *                     *          *                      143     10.2.1.100    143

Comment: we put also our external IP (42.8.5.2) as destination address in those rules - no difference. We used TCP/UDP for Ports 587+143 just for testing, whether StartSSL could cause our problem.
_____________

NAT Outbound manual rule
                                                       Source                        Destination                   NAT    Static
If                         Source                Port    Destination     Port   NAT Address      Port    Port
WAN_Portunity  10.2.1.100/32  *         *                      *       WAN_Port_addr.  *        NO


Does anybody have a clue, what we made wrong???

Thanks for your support!


Regards, Markus

[fabian]

You may have a multi wan issue. Can you try to mark the connection as "sticky".

[mbach]

Dear Fabian,

thanks for your hint.

We did the following:

1. We changed the IP range of the WAN-Routers and Gateways:
192.168.1.1/24 | 192.168.2.1/24 | 192.168.3.1/24

IP of opnsense-Interfaces now:
192.168.1.2 | 192.168.2.2 | 192.168.3.2

Gateways do work.

2. We marked the connection as "sticky". No real change in behaviour...

I just try to sketch the principal routing:
incoming:
Internet -> Portunity (fixed IP) -> OpenVPN client -> NAT to internal IP -> internal IP

outgoing:
internal IP -> Gateway: Portunity-VPN -> Internet
If we check the outgoing route (tracert), everything seems to work.
And more amazing: Port 25 (SMTP) with no SSL seems not to make any troubles at all.
All emails can be received and sent...

We also use the NAT outbound rule to get the packets back on the same way.

We did an external port testing - all ports are open.
We tried to connect with telnet - no answer.

In the meantime we made a fresh installation on a second place and everything is working out of the box.

Could it be, that SSL-certificates (self-issued) are filtered?
We recognized that with another site, that opnsense is filtering mailconnections and give out a "password error", although we did not activate any squid or IDS/IPS filtering.... (we just activated IDS once and deavtivated it for further testing).

Thanks, Markus

[fabian]

We did an external port testing - all ports are open.
We tried to connect with telnet - no answer.

this can happen because there is no nat reflection.

[franco]

Hi Markus,

It's highly unlikely that this is a SSL filtering issue, traffic is not touched without IPS in place.

In most cases this revolves around rules that drop parts of the traffic, one direction not getting the packets back as expected (no NAT) or routes pointing in the wrong direction locally.

Since you also have OpenVPN this is probably where the biggest problems can arise--where MTU might matter, the protocol type used, tun vs. tap usage, etc. Sorry for not being able to be more specific here. It might help to experiment with these settings.

One questions: why steep learning curve?

Cheers,
Franco

[mbach]

Hi Franco,

thanks for your suggestions.
We just reinstalled the firewall and stumbled over a problem with one internet line.
We will change router and are resetting the line right now. It is a cable internet line (not ADSL/VDSL) and cable carrier are not very reliable in Germany...
The problem with this configuration (cable internet + OpenVPN DialIn + fixed IP) is: it worked for years with another software firewall. That does not mean, you cannot run into troubles... 

We will keep you informed about our procedures to solve that nasty problem.

Steep learning curve? Mmh, many new settings with every update, compared to pfsense a working framework.
Maybe it would have been more correct to write: a steep, but very short learning curve...

Many things just work, as they should. Sometimes it is too simple, to believe, that it just works.
But sometimes you wonder, why something simple (like DynDNS updating) does not work out of the box.

But we just started with opnsense and we are interested in more...

Thanks for your suggestions and help!

Cheers, Markus

[franco]

Hi Markus,

If we can pinpoint the problem I see no reason to not have it fixed for at least the same number of years to come.

Thanks for explaining. I think people get used to this weekly thing. Upgrades are fun and easy. There are some ideas about providing a fixed version with bugfixes only, but we're still weighing options.

DynDNS may reveal additional info in the system log file (just filter for e.g. "dns" there) or running the scripts manually from the console. Mostly typos in domains or missing connectivity, sometimes an unmaintained DynDNS provider that needs a bit of work to adapt to a new API or parameter rename.


Grüße,
Franco

PS: Although I'm stuck on 50 Mbit ADSL now, I do not ever want to go back to the previous 100 Mbit cable experience...

[mbach]

Hi all,

after it seemed, that we solved the problem, we are still stuck with the same problem as described in 1).

That means: it works and in the next second it does not work any more.
Port 25 does not have any problems, only 143 and 587 with StartSSL-TLS encryption.

We are running out of any further ideas....

We checked all further sources: dyndns-logs, firewall-logs, ids-logs - no failures/blocks.
As I have written before it seams, that the answer is missing.

We tried the sticky connections - no success. And that should not be the problem, as we do not use Multi-WAN, but just an DynDNS-Client as one WAN-Connection for a fixed IP.

Would be great, if anybody would have further ideas...

Thanks, Markus