pf routing rules randomly start being ignored till service restart

[bryanvaz]

Every couple of months, pf randomly starts ignoring firewall rules that force traffic from our untrusted lab DMZs to use our wireguard VPN gateway, and starts routing traffic out via the our default underlying gateway. If the pf service, and ONLY the pf service, is restarted, without touching anything else, they DMZ starts routing over the VPN again. This is obviously a gigantic problem, because it means that other pf rules might be failing as well and letting in traffic that shouldn't be coming in, and letting out traffic that shouldn't be leaving.

We have opnsense configured to:
1) Group the DMZ subnet and non-DMZ canary into an alias
2) pf rule (first match) for the LAN interface to pass traffic from the DMZ alias with destination '!LAN net' to use Gateway WAN_WG_IPv4 - rule at the top, so is above the default rule for "pass LAN net -> * (any)"
3) pf rule (first match) for WAN interface to block traffic for 'DMZ alias -> !LAN net' on * (any) gateway

What commands or logs can I dump from to figure out exactly what routing and firewalls are being used/ignored when it enters this failed state, or if pf is even still alive?

The above configuration passed the following tests:
1) As normal, `curl icanhazip.com` reports the wireguard gateway ip, not the default underlying gateway ip
2) When wireguard connection is offline (via a forced configuration error and disabling in webui), no traffic passes from DMZ and no external routes are reachable from the DMZ machines

(Note: the non-DMZ canary performs a check on the route to ensure that the route is protected, if not it will notify, but doesn't have the ability to kill, since the first few times it happened we thought it was a configuration error.)

[jeremy.mcmillan]

I have set up vxlan interfaces to link a VM host's guest (lab) VMs directly to my OPNsense firewall to create a DMZ. I'm finding traffic is blocked by the default rule despite rules on the interface which plainly match. Some of those packets make it through though, because software with robust retries kinda sorta still works.

Is there a way to trace rule evaluation?

[sja1440]

These are the settings that I use on my firewall to prevent the leakage bryanvaz describes.

(1) Firewall: Settings: Advanced->Gateway Monitoring->Skip rules = selected.
(2) I put an explicit blocking rule immediately under the DMZ routing rule on the LAN interface - just in case the DMZ routing rule doesnt fire.

Notes on (1) The GUI Help says for this setting:
Skip rules when gateway is down
By default, when a rule has a specific gateway set, and this gateway is down, rule is created and traffic is sent to default gateway. This option overrides that behavior and the rule is not created when gateway is down

Notes on (2) this rule does no harm, but might be unnecessary. I find the FreeBSD pf documentation ambiguous in places.

I don't know if that helps.