NAT not working with MQTT

[endophilius]

Hello everyone,

I'm running OPNsense on an amd64-machine with version 25.1.5_5. About half a year ago I set up NAT Port Forwarding for mosquitto on port 1883 (standard MQTT port). It worked until lately I noticed that my MQTT client outside of my network can't reach mosquitto anymore.

My network structure is:

• OpenDTU (192.168.178.42) as the mqtt client which is set up to connect to 192.168.178.69:1883
• FRITZ!Box (192.168.178.1 with range 192.168.178.0/24) which is my primary gateway and is configured to not block any connections. This is shared with other trusted users and the only network which provides WiFi access for the OpenDTU to connect to.
• OPNsense (WAN: 192.168.178.69, LAN: 10.1.3.1 with range 10.1.3.0/24) is my router for my private network. It has a NAT Port Forward with the following options:
  • TPC/IP Version: IPv4
  • Protocol: TCP
  • Source: *
  • Destination: 192.168.178.69/32 (Port: 1883)
  • Redirect target IP: 10.1.3.100 (Port: 1883)
    All my deny-rules in my firewall are nonquick and should not block this traffic.
• Server (10.1.3.100) which runns mosquitto and accepts connections on port 1883.

What I tried:

• Check wether I am able to configure Port Forwardings
    - I have 26 Port Forwardings of which 25 are working happily with HTTP and SSH traffic. I cloned the MQTT rule from one of these working rules.
  
• Test wether the Server accepts connections
    - With MQTTX and from within my 10.1.3.0/24 network I can connect to mosquitto with my credentials.
• Test wether I can reach the Server from other networks
    - I tried the same thing from another network managed by my OPNsense (10.1.1.0/24) and it worked.
• Test if OpenDTU makes requests or if the FRITZ!Box blocks them
    - I can see the requests im my firewall log so they must have been sent and are passed by my primary gateway.
• Test if the firewall blocks the traffic
    - I switched on logging on all of my deny-rules and got correct deny-results (e.g. for trying to access port 1884), but there was no deny for port 1883.
    - I also saw the redirect happening in the firewall logs.
• Test wether this is a problem with the OpenDTU
    - I tried connecting with the exact same PC as I used earlier but this time from my 192.168.178/24 network. It didn't work, so the problem seems to be somewhere inside or near the OPNsense.
• Test wether the traffic is blocked by my IDS/IPS
    - I switched of intrusion detection and prevention, it didn't work either.
• Is the routing between the FRITZ!Box and my OPNsense correct?
    - Yes, even though I'm pointing to 192.168.178.69 in OpenDTU, I also set my OPNsense as the gateway for 10.0.0.0/8 in the 192.168.178.0/24 network.
    - This also works, traffic is correctly routed.

There seem to be a few connectivity issues between the FRITZ!Box and my OPNsense as I also can't reach some outer services from inside my OPNsense. Thinking of some kind of disk issue or so I reinstalled the OS to another disk and restored a backup which was made at a time when the MQTT rule worked. But even this didn't do what it' supposed to, but restored everything correctly.

I'm running out of ideas what the problem could be and what I can do to find it. Do you have any ideas what could cause the NAT connection to be blocked or not correctly forwarded/routet?

Thanks for every reply :-)

[EricPerl]

Given your setup (OPN behind another stateful FW), I assume you've disabled reply-to (in Firewall > Settings > Advanced).

[endophilius]

Thanks for the reply. I checked the option and I havn't checked 'Disable reply-to' so the reply-to on WAN rules should be enabled.

I also enabled NAT reflection. With that I can access mosquitto from within my local network when going through 192.168.178.69 so the NAT is correctly reflected, but it depends on where the traffic comes from. If it comes from LAN, it goes through, if it comes from WAN it is blocked.

[Patrick M. Hausen]

If the client is in the same network as the WAN interface you must check that "Disable reply-to" box.

[EricPerl]

https://docs.opnsense.org/manual/firewall_settings.html#disable-reply-to

Per the first part of the warning, with the setting unchecked, replies out from OPN WAN will be directed to the GW for that network (your fritzbox) where they likely get blocked/ignored because the requests did not flow this way.

[endophilius]

I now checked this option and disabled the reply-to. I also waited to mitigate a possible blacklist/graylist on my FRITZ!Box. Until now the connection can't be established and the symptoms are the same as with the option unchecked.

[EricPerl]

The Fritzbox shouldn't be used at all in this scenario (potential exception if it acts as a switch in between the hosts).
Communication between the client and OPN.WAN is on the same subnet.
In other words, this should no longer be the case:

Test if OpenDTU makes requests or if the FRITZ!Box blocks them
  - I can see the requests im my firewall log so they must have been sent and are passed by my primary gateway.

There should be no 10.xxx traffic on the Fritzbox networks (OPN.WAN side) so this is useless:

Is the routing between the FRITZ!Box and my OPNsense correct?
  - Yes, even though I'm pointing to 192.168.178.69 in OpenDTU, I also set my OPNsense as the gateway for 10.0.0.0/8 in the 192.168.178.0/24 network.
  - This also works, traffic is correctly routed.

Anyway, at this point, it's probably time to do a packet capture (WAN + LAN) to see where the flow stops.
You can filter by port and IP (client or server) based on usage in your network.

That's how I figured out something was wrong on my internal router.
Packets out of WAN (reply back to client) had the MAC of my edge router LAN... They should have had the MAC of the client.

[endophilius]

Hi everyone,

sorry that I didn't respond for so long, I had some other things intercepting.

Now I did the packet capture and it seems like the OPNsense is not responding the the SYN the OpenDTU sends. I have attached the .pcap below (and I am aware of the privacy risk it brings with it but I'm okay with that).

[EricPerl]

That's only the WAN side. It establishes that OpenDTU tries to open a connection.

If the LAN side was included, we'd see if the frame is properly forwarded (from OPN.LAN.Address to the server).
And we'd also see if we were getting a reply (assuming the filtering is appropriate).

As of now, I can't tell where the traffic stops...

[endophilius]

The packets also go through at the LAN side. They even can reach the server-host but not the docker-container that is running on the server. But this is not an OPNsense problem, so I'll ask at other places that are more specific to my issue.

Thank you for your help!