Messages

Hi,

I have two pfsync'd CARP routers and unfortunately it happend that I made the instance on a virutalisation environment labelled "SECONDARY" the master (and thus, the backup on "PRIMARY") and this heavily confuses people (trying to configure on the backup device).
Do I understand correctly that the configuration changes are applied directly on the backup device, so that I can simply swap the side who has the pfsync IP of the other side? Or are there traps I should watch out for?

Thanks for reading and any hints appreciated!

[and my solution]

Hi,

Update:
I wrote hours (clock, not just feeling!) on this posting and configured everything again to be sure to describe correctly.
And then it so happened that I found my problem myself! :) However, in case someone else also facing such an issue, I share my problem and my solution nevertheless.


TL;DR:
I have a non-standard setup (routing LAN to LAN just to offer clients a single default route for every target) and fail to configure my firewall rules (only turning off helps, but I don't want this).

Solution:
Ensure to have all flows allowed by State Type: None rules (only visible after enabling Advanced option)

Now in detail, in case it helps anyone in future.

Background:
Let me state why I think my setup actually may make sense.
I operate a few VMs and containers that communicate via some VPNs to other sites. The WAN and VPN stuff is out of my scope (some IT network team cares). They are migrating to new firewalls (I think) and come every day or so with a new static routing table. Essentially one after the other destination "moves" from one gateway to another.
I did not want to reconfigure all my VMs every day in heaps of maintenance windows with avoidable downtime, so I setup two OPNsense appliances as VMs on my very little cluster, to be used as default route by every VM and container. The OPNsense knows the routes, so I have just 1 place to maintain.

The setup:
I setup the (few) gateways for each (of the few more) destinations. Please note that the gateways are in the same subnet, so my configuration is exceptional: client VMs route via OPNsense, altough they can reach the "next hop" themselves. This makes packets from the gateways skipping OPNsense, so I added an "allow all RFC1918 IPs, stateless" rule (yeah, this one I noticed...).

In OPNsense, following a (carefully picked / random) guide from the internet, which stated to go to "System / Gateways / Single, IP-Adresse", add the gateway, and mark it as down; and then in "System /
Routes / Configuration" set the route. Then a floating firewall rule PASS any src/dest RFC1918 state type: none, a rule "PASS: LAN interface IN lan to any" and "PASS: LAN interface IN any to LAN". I configured CARP and a slave/backup and found everything working fine. I tested connectivity, ping and surfed the web. I even installed ookla-speedtest on the OPNsense and at night was able to find the advertised link speeds.
Yeah!

What I observed:
What I did not test was browser speed test from the VMs, and indeed unfortunately this looked horrible in the uploads. Google found 0.05 Mbps, Cloudflare even failed to report any uploads (download speeds for both were fine). A debian GNU/Linux based container with speedtest-cli showed 1.5Mpbs, however this can be explained. As said, surfing the web works fine, and more importantly, I tried iperf (default options, each side once server and once client) and found it fast (in both directions), too. Just speed tests failed in upload.
Since iperf worked I failed to see that I may have an issue with stateful packet filtering (yes, maybe it was late so I didn't see the issue yet, I was irritated because it was working, just be slow).

NB: Here, no V*LAN is involved whatsoever, anywhere. There are unrelated issues with slow download rates after upgrading.

When I disabled firewalling, it works. So I had a clue at least.

What I tried first:
I tried emulation of E1000 and VirtIO on my Proxmox host (Linux KVM based virtualisation platform), with queue=8, Hardware CRC|TSO|LRO disable, hw.ix.flow_control=0 boot (via Tunable+), various OPNsense versions (24.1, 24.1.4, 23.7, 23.1), floating rules "IPv4 * LAN net * * * * *" and "IPv4 * * * LAN net * * *",
nothing helped.
However, as said, Firewall>Settings>Advance>Disable Firewall - Disable all Packet filtering DOES help. So I had a packet filter issue.

Solution:
Surely obvious for the BSD masters, the rules  "PASS: LAN interface IN lan to any" and "PASS: LAN interface IN any to LAN" need to have "State type: None" be set in "Advanced Options". I missed that since friday (unfortunaltey 4 working days here).

Well, I'm not sure why omitting this does not lead to an unconditional communication issue (i.e. 0.00 Mbps upload or so) but to slowly but go apparently working situtations (I have to admit I don't know how modern speed tests work, tcpdump strongly suggest HTTPS).

I hope it helps (and even if a single pure dude finds this and saves the day it was worth writing it)

my first €25, thank you for your great efforts and keeping it free as in freedom