What to set in firewall for special purpose LAN<->LAN routing

[uucico]

Hi,

Update:
I wrote hours (clock, not just feeling!) on this posting and configured everything again to be sure to describe correctly.
And then it so happened that I found my problem myself! However, in case someone else also facing such an issue, I share my problem and my solution nevertheless.


TL;DR:
I have a non-standard setup (routing LAN to LAN just to offer clients a single default route for every target) and fail to configure my firewall rules (only turning off helps, but I don't want this).

Solution:
Ensure to have all flows allowed by State Type: None rules (only visible after enabling Advanced option)

Now in detail, in case it helps anyone in future.

Background:
Let me state why I think my setup actually may make sense.
I operate a few VMs and containers that communicate via some VPNs to other sites. The WAN and VPN stuff is out of my scope (some IT network team cares). They are migrating to new firewalls (I think) and come every day or so with a new static routing table. Essentially one after the other destination "moves" from one gateway to another.
I did not want to reconfigure all my VMs every day in heaps of maintenance windows with avoidable downtime, so I setup two OPNsense appliances as VMs on my very little cluster, to be used as default route by every VM and container. The OPNsense knows the routes, so I have just 1 place to maintain.

The setup:
I setup the (few) gateways for each (of the few more) destinations. Please note that the gateways are in the same subnet, so my configuration is exceptional: client VMs route via OPNsense, altough they can reach the "next hop" themselves. This makes packets from the gateways skipping OPNsense, so I added an "allow all RFC1918 IPs, stateless" rule (yeah, this one I noticed...).

In OPNsense, following a (carefully picked / random) guide from the internet, which stated to go to "System / Gateways / Single, IP-Adresse", add the gateway, and mark it as down; and then in "System /
Routes / Configuration" set the route. Then a floating firewall rule PASS any src/dest RFC1918 state type: none, a rule "PASS: LAN interface IN lan to any" and "PASS: LAN interface IN any to LAN". I configured CARP and a slave/backup and found everything working fine. I tested connectivity, ping and surfed the web. I even installed ookla-speedtest on the OPNsense and at night was able to find the advertised link speeds.
Yeah!

What I observed:
What I did not test was browser speed test from the VMs, and indeed unfortunately this looked horrible in the uploads. Google found 0.05 Mbps, Cloudflare even failed to report any uploads (download speeds for both were fine). A debian GNU/Linux based container with speedtest-cli showed 1.5Mpbs, however this can be explained. As said, surfing the web works fine, and more importantly, I tried iperf (default options, each side once server and once client) and found it fast (in both directions), too. Just speed tests failed in upload.
Since iperf worked I failed to see that I may have an issue with stateful packet filtering (yes, maybe it was late so I didn't see the issue yet, I was irritated because it was working, just be slow).

NB: Here, no V*LAN is involved whatsoever, anywhere. There are unrelated issues with slow download rates after upgrading.

When I disabled firewalling, it works. So I had a clue at least.

What I tried first:
I tried emulation of E1000 and VirtIO on my Proxmox host (Linux KVM based virtualisation platform), with queue=8, Hardware CRC|TSO|LRO disable, hw.ix.flow_control=0 boot (via Tunable+), various OPNsense versions (24.1, 24.1.4, 23.7, 23.1), floating rules "IPv4 * LAN net * * * * *" and "IPv4 * * * LAN net * * *",
nothing helped.
However, as said, Firewall>Settings>Advance>Disable Firewall - Disable all Packet filtering DOES help. So I had a packet filter issue.

Solution:
Surely obvious for the BSD masters, the rules  "PASS: LAN interface IN lan to any" and "PASS: LAN interface IN any to LAN" need to have "State type: None" be set in "Advanced Options". I missed that since friday (unfortunaltey 4 working days here).

Well, I'm not sure why omitting this does not lead to an unconditional communication issue (i.e. 0.00 Mbps upload or so) but to slowly but go apparently working situtations (I have to admit I don't know how modern speed tests work, tcpdump strongly suggest HTTPS).

I hope it helps (and even if a single pure dude finds this and saves the day it was worth writing it)

[ifontana]

Hello UUCICO, you know that I have the same problem as you. I'm starting to use opnsense as a gateway to use zenarmor, also installed on vm in proxmox, and I have the same problem with upload speed. If I deactivate the firewall I have 700/700 mb, if I activate it I have 700/0.1 mb. I have disabled the nat function because it is performed by another mikrotik router. I have also tried enabling nat, and the speed is 160/160 mb. Beyond that, I think the problem comes from the firewall side. In fact, I have noticed that opnsense also limits the download and upload bandwidth. With the fw disabled, I have 500/500 mb, and if I go through another router (as a gateway) I have 900/900 mb.
I have tried creating the rules you suggested but they have not worked for me. Could you give me a little more detail about the rules you created?
I would really appreciate it!
greetings