1/4 VLAN External connectivity issue.

[ajeffco]

Hello,

TL;DR:  I have a VLAN (1 out of 4) that connects to the internet without issue for up to 6 minutes at most, then suddenly stops connecting past the first firewall, but conectivity to/from all other internal VLAN connectivity works.

First, the background.
This setup is being driven by multiple people in my household working and schooling from home.  Spectrum has had more than a few outages since we started working from home, some quick, some not so quick.  For me it's usually no big deal, for my wife outages are very bad, even short ones.  And for the kid schooling from home the longer ones are annoying.

I'm running two Proxmox rigs which in turn are currently running 3 OPNsense VMs.  Proxmox node 1 is running two firewall VMs, outer-fw1 (Spectrum ISP) and inner-fw1.  Proxmox node 2 is running a single firewall, inner-fw2.  In the future Proxmox node 2 will also have outer-fw2 VM (TBD ISP).  My goal is to get to redundant internet links with redundant critical components (DHCP, DNS, etc).  This has so far been working well, the two inner firewalls are setup in HA mode, I can fail back and forth without any problem whatsoever.

I'm also running other critical services on the Proxmox nodes, they are working as well.
The problem:
After at most 6 minutes, the server I'm testing with in VLAN 20 stops communicating to the internet and the external firewall.  It can ping all interfaces on the inner firewall, on all VLANs.  It can ping other devices on other VLANs.  It just can't get past the inner firewall itself in any form.

The time to stop working is intermittent, it ranges from 30 seconds up to @6 minutes.  All other VLANs are able to connect without issue when VLAN 20 stops working.

I'm  not sure where to start to even look for this. 

My first thought when I found the issue was firewall rules, but then I realized by accident in testing that when inner-fw1 rebooted, VLAN 20 started working again when inner-fw2 CARP kicked in, and continued working when inner-fw1 came back up and took back the VIPs.  To me this eliminates firewall rules.  I have left inner-fw1 down for a while, it took a little longer to happen on inner-fw2, but it happens there as well.

Hopefully someone can give me some advice as to where to troubleshoot.  I have attached a rather ugly drawing but hopefully it will help illustrate my environment.  I'm not even what logs to provide, as the logs I've looked at don't show any "abnormalities" when the issue occurs.
Thanks in advance for any help.