Messages

Have a look under "System: Settings: Miscellaneous" - "Periodic Backups". I back up RRD data on power off, and it can take a bit of time. Periodic backup might increase writes, if SSD endurance is a concern. Of course this is only relevant if you have the RRD enabled ("Reporting: Settings" - "Reporting Database Options"). There may have been changes that slow down the backups - hard to say.

Also, I have AMD Zen 4 (Ryzen 7000) machines (including my firewall) which periodically retrain memory on reboot, which can take several minutes (particularly since I use ECC RAM). Generally cold boots only, but not always. BIOS dependent.

[...]
I noticed an error regarding the ice interfaces while running sysctl command, do you think it's related?

I do not, but of course I could be wrong. I don't recall the error on my machine (offline, so I can't look), but I believe I had had both virtualization and SR-IOV disabled in the BIOS (regular PC, bare metal FreeBSD install).

Speaking of that, I quoted the wrong pciconf command: should be "-lvc [interface]".

How about some more "netstat"s: "-m", "-i", "-sp ip" - anything look odd? (I have an anomalous Idrop stat on ixl interfaces and lots of unaccountable Oerrs on one bridge.)

[...] Is 1Gbit/s the best we can expect from this device for a single flow?

It shouldn't be, but I don't have similar hardware to test.

[...]
I tried multiple tunable settings found on this forum [...]

So long as you're not terminating iperf sessions on the firewall, about the only tunables/sysctls that'll help you are RSS: net.isr.bindthreads, net.isr.maxthreads, net.inet.rss.bits, net.inet.rss.enabled. Have a look at "netstat -Q" (IIRC) to check. I believe OPNsense sets most other necessary sysctls reasonably. But I believe RSS is mainly good for throughput.  How does CPU utilization look while running the tests?

If you weren't using a Deciso device, I'd recommend looking at the ice devices, e.g. "dmesg | grep ice" and "pciconf -lvV ice0" for any oddities (mine really wanted to set up as x4 v3; you want x4 v4 or x8 v3 at a minimum).

That speed does look suspicious, like a per-flow/session shaper.

How transparent is your bridge? That is, do you want to manage your firewall through the bridge interface? (You may not have much choice if you are limited to two interfaces and don't care to use VLANs.)

You can find the bridge interface settings (to configure an IP address) under "Interfaces" ("Interfaces: [Name] -> Generic configuration -> IPv4/6 Configuration Type" and additional config as needed); the default route (if not configured automatically via e.g. DHCP) would be under "System: Gateways: Configuration". I have static IPs, so my firewall public address and gateway are static. If you're performing NAT on your firewall (a bit odd for a transparent bridge)... it may not require additional setup (beyond what you have).

[...]This is fine[...]

I'll disagree with you there. But thanks for the note - I'd done exactly as you said: created composite aliases with private and bogon networks. The composite (I have one for v4 and one for v6, but bogonsv6 is, as always, broken) preferred the negated addresses. I've split up the policies to handle the goofiness; now I just have to remember why.

Automatic configuration is like autocorrect: Don't try to be too helpful.

I was going to say "Running tagged and untagged traffic on an interface will be a disappointing experience", but I can't say that with certainty unless there's a bridge involved. Still, it's not recommended - it appears that some of the interface hooks (in the FreeBSD kernel) were poorly designed/placed, and you'll get packets showing up where they shouldn't. Documentation link: VLAN and LAGG setup (the "Attention" sections).

I'd recommend tagging all traffic on trunk ports simply out of an abundance of paranoia.

You'll need to buy an X550-T2 [...]

I believe the X550 always supported multi-gig... whereas the early X710s did not. I tend to buy NOS/surplus, so I occasionally run across an old one. Something to watch out for.

I'd stay away from the new E610 unless one falls into your lap - it seems to offer no process/power improvement over the X550. It does support PCI-e v4, so it might be worthwhile if you have slot limits (unlikely at 2.5Gb).

tcpflags on the dropped packets are 'SA'.

From my firewall, I would expect "S". Yours looks kinda like a new state setting up halfway into the process. I haven't seen this in my own setup. You wouldn't happen to have synproxy enabled on the pass rule for these sessions? It's an option under "State Type" near the bottom of the "Advanced features" in the rule definition. I can't see this affecting ICMP, and synproxy should only break under certain conditions.

[...] If I click on the rule ID in the info popup, it opens a tab which then immediately closes itself.

I get that if the linked rule is automatic, I believe. I mask the automatic default block with my own, so I only see logs for the former when a packet somehow escapes my rules.

[...] the only oddity is that the matching rule is alleged to be "let anything out of the firewall itself" [...]

Yep, that's normal.

[...]My guess here is that the firewall knows the matching state for these rules, otherwise it wouldn't know the private IP they're for.[...]

Reasonable. Hit the "i" on the right for one of those live logs and see if there's anything unusual (e.g. TCP flags), and look at "Firewall: Diagnostics: States" to make certain the bidirectional state is set up properly. I've seen pf drop TCP sessions that violate the handshaking rules, and they log as default denies (in my ruleset).

[...]I'm just wondering if there's a change recently in default network stack on some OSes or in CSP or something like that, that causes an impact on TCP session management...

Sounds like a misconfiguration to me. But it'd be an odd one.

OPNsense does have a bunch of per-rule advanced settings, but I don't see one that might work for you. pf appears to allow a complete set of timeout settings per-rule, which might help you out if they were exposed in the GUI. I tried a specific option for "State timeout" ("tcp.finwait 5"), but the GUI expects an integer. Perhaps the setting is only for "tcp.established", which is unhelpful. I may request an enhancement for this, as per-rule control can occasionally be handy.

I discounted the path issue, as both paths seem to pass through OPNsense, and "Firewall: Settings: Advanced", down under "Miscellaneous", "Bind states to interface" is unchecked by default. Do you have it checked?

How about that ix0? Unassigned/unconfigured?

Interesting. I've never fixed an issue through more aggressive state retirement. Always the opposite. It'd be nice if there were better diagnostics and control (per-rule or per-protocol state lifetime). My old Fortigate had this, and it was useful to modify DNS session lifetime (to 15s for clients, 90s for servers, default 30s) as some remote servers would send a few packets after a minute or so idle. Discarding these stray packets seemed to have no functional impact, but you never can tell and the logs were annoying.

I see ~6000 state mismatches over 32 days. I don't notice an impact... but that's no surprise. Log view on OPNsense is pretty terrifyingly bad - I may dig into specifics, assuming mismatches are logged (I've never seen one offhand). (Searching the logs from the GUI gets me an occasional result after several minutes. Not helpful.)

Can you check/alter the client source port range?

Sounds like you have no convenient GUI access when the problem occurs, which makes troubleshooting less convenient. Also, the consistency of the issue you describe doesn't match typical issues. Still, have you checked "arp -a" for correct mappings? How about counters, say "netstat -i" - do they increment as you expect?

I see a VLAN in there. Do you have the main interface assigned/in use? If so, that'll cause wackiness. (I forget the precise nature - it's deterministic, but I don't see a useful application for it. Best avoid it.)

This depends on whether you have either [...]

Heh. As I've said, in my experience they have no effect. I'd guess that the new bridge code (by Kristof Provost, who also appears to maintain the pf port) hooks the interface before pf. I haven't bothered to dig through the code, as it's not likely an easy issue to patch. Of course, I'd always recommend setting the sysctls/tunables appropriately to avoid wackiness if the behavior is ever changed/corrected. UserSN's "pass all" rule should be safe in any case. (If I was even more paranoid than I am I'd configure a pass rule on each bridge member interface with logging enabled and a... distinguishable description.)

[...]
From your reply I gather my current config is correct and that moving my block rules from [BRIDGE] to [WAN] won't work or bring any benefits?

Yup, unless you remove WAN from the bridge and configure alternative forwarding (e.g. proxy ARP). Feel free to try it, but if your current WAN rule shows no evaluations in the stats, it's not likely to work.

In my case I was purely testing, so I had only one test machine behind the firewall (no operational risk). I set up a few different pass rule schemes on the bridge and member interfaces, with no block rules. Basically, for bridge/member: pass all/pass all, pass all/pass specific, pass all/none, pass specific/pass all, none/pass all. The only rules that functioned were those on the bridge, regardless of the "net.link.bridge.pfil_*" sysctl settings. I rebooted and verified them from the console after each change. To be clear, though, I did not attempt to characterize the operation of the default floating rules, and I may not have been comprehensive in the rule variations vs sysctls (my patience is limited, and my purpose was to test combinations that I would actually use).

The only real disadvantage of the bridge from a rule standpoint is that you cannot configure a rule based purely on input interface (e.g. my earlier example of an anti-source-spoof rule). I can't think of any other limitations offhand.