[WORKAROUND FOUND] Trouble with Firewall Rules on return communication

[kyferez]

So I have OPNsense as a firewall between 2 VLANs. I have 2 Win 2012 R2 servers that since upgrading to 17.1 cannot communicate over Kereros, LDAP, etc. However SMB currently works fine, which is weird because they are all the same firewall rule (using aliases). I did packet traces on both servers and checked the logs on the firewall. Screenshot attached. You see the firewall logs passed traffic, which the receiving server gets, but the SYN ACK reply is lost.

In this test, I initiate port 88 traffic from server s1 to server lic1. lic1 receives the SYN, sends a SYN ACK. s1 does not receive the SYN ACK.

Any ideas? How do I verify if the firewall received the SYN ACK from lic1? Do you have a GUI tcpdump utility?


Here is the network details:

VLAN1: 192.168.1.0/24
OPNsense Interface: 192.168.1.1
Server s1: 192.168.1.220
Firewall disabled.
s1 default gateway: 192.168.1.254 but has a permanent route to 192.168.2.0/24 via 192.168.1.1, verified working with tracert.

VLAN2: 192.168.2.0/24
OPNSense Interface: 192.168.2.1
Server lic1: 192.168.2.230
Firewall disabled.
lic1 default route is 192.168.2.1

Thanks!

PS the image max attachment of 192kb is way too small; makes it hard to post multiple screenshots.

[bartjsmit]

The dumps taken by Interfaces->Diagnostics->Packet capture can be downloaded for analysis by GUI tools, most commonly Wireshark.

Bart...

[djGrrr]

Make sure you are on 17.1.1 first, then run this command in shell / console as root:
sysctl net.pf.share_forward=0

[kyferez]

djGrr, what does that command do? Nevermind, I found it. Waiting for the latest updates to apply and will report back.

Thanks!

[kyferez]

No change, setting it to 0 or to 1. Either way on 17.1.1 I get the same results.

[kyferez]

So  OPNsense is sending the SYN ACK packet back to VLAN1, but instead of directly to the s1 server, it sends it to the MAC address of the default gateway. See image. Is this expected behavior?

Franco, can you comment on this?

[JeGr]

did you configure both default gateways on the corresponding interfaces of opnsense?

[djGrrr]

You may need to use the "disable reply-to" advanced option on one of your rules, or add a higher priority firewall rule before any other rule with the gateway set for traffic between the 2 vlans; for example, if you have a "default route" firewall rule for traffic with source of vlan 2 net, destination any, add a rule above it: source vlan 2: destination vlan1 net (or an alias of destination networks), without a gateway set.

A screenshot of firewall rules would help to debug this.

[kyferez]

There is no default gateway for VLAN2. Only VLAN1.
None of the rules have a Default Gateway set.
I do have a Allow VLAN2 any to VLAN1 rule... see attachment for Gateway and rules.

[djGrrr]

Try editing the "Allow VLAN2 any to VLAN1" rule, under advanced options, check "disable reply-to", then save and apply and see if anything changes

[kyferez]

djGrrr,

Done. No change. (yes I applied the change)

[kyferez]

Any other ideas?

[djGrrr]

try disabling your floating rules temporarily and see what happens.

[kyferez]

So it appears we have found a bug...? I saw a number of posts reading the forums yesterday that looks like this may be related to issues others are having...

Anyway, thanks for the idea djGrrr, that lead me to a workaround.

I noticed that port 445 stopped working when I disabled my floating rules - I had that port in my Remote Management Alias on one of the Floating rules.

Sooooo.... I added a new floating rule with Destination AD_DNS_Servers and Destination Ports AD_Services and applied the changes. BOOM it now works! YAA servers can sync again!

[djGrrr]

Keep In mind that floating rules are processed first, and if you have the quick option selected (the default), no further rules will be processed once it is matched. Also, if you do not select an interface, or perhaps a direction for the rule, it can override rules on interface tabs. Generally you should put all rules under specific interfaces, and only use floating rules when they are required (like for a direction other than "in")