Network suddenly crashed: "watchdog timeout on queue 0"

[senseivita]

I was designing a UI for a local system and playing random stuff of YouTube as whitenoise on another computer when both suddenly lost their connection. The one playing YouTube was the odd one since usually YouTube/Netflix/… can keep on going for a while on their buffer when the network is lost. I had an IP address, but I couldn't ping my immediate gateway, not anything above it. I could still connect to another computer on the same subnet which is doubled homed and from there see what's up, I connected to the upstream router, pfSense, which basically does most of the networking, it connect via a transit network to OPNsense, and that to the first core switch, at least logically--it's all virtualized.

I logged into vCenter to get the consoles from both firewall systems, the upstream could ping out, OPNsense on the other hand showed the network interface reading "watchdog timeout on queue 0" filling the whole screen. It would keep printing that. I restarted it and as soon as it got back it wasn't finished booting when it was printing that again. Like I said, the edge firewall was doing most of the work anyway, OPNsense I was just starting to deploy to play with the things it has pfSense lacks, I just made the switch take its IP address of the /30 transit network and I got back where I was.

I'm quitting OPNsense for now, I've tried to deploy it several times but I just can't get it to be stable--and it's not like mundane things, it's a little more serious with potential for data loss, like not starting up on the update to 19.1; or connectivity problems on multi-WAN with the same ISP (hence same gateway)--I was able to spoof the gateway with another router in between, BTW, but it sort of defeat the purpose of such an advanced system. :/

Anyway, I thought I should report it. I hope it helps somebody. Also found out that to boot a failed EFI-based 19.1 upgrade, you can just change it to BIOS without reinstalling! I reported that on Github, though.

[senseivita]

I've been playing with the firewall now that's not doing anything in the network (and there can't cause any damage) and discovered this issue is triggered my enabling monitors on gateways. It will lose all connectivity on the parent interface with the monitored gateway--so, if you add a single interface like I've been doing, you'd lose ALL of it. If you want to troubleshoot from console, the console menu lacks the option to configure these values unless you're a wiz with FreeBSD's shell (I'm not) but if you happen to lose connectivity quick enough (it's not always fast, it may take a big big while) you probably didn't make enough modifications so you can still roll back changes in console.

OPNsense has the ability of increasing the rollback points history thankfully. It must be done from the web GUI, and obviously beforehand, preferably shortly after installing set that thing to something bigger, I went with 200, then you can go back to the new install wizard on the column menus anytime.

OPNsense has issues, a lot of them, regarding gateways. I'm starting to understand how it handles things. In my case, I have a multi-WAN setup, 3x PPPoE connections to the same ISP: this creates an unbelievably large amount of issues with random connectivity issues because I have the same gateway on the 3 links. It's fixed this by putting a pfSense edge appliance in front of OPNsense, have the edge dial the PPPoE and create 3 new PPPoE servers in the edge, so from OPNsense's point of view it would have three different gateways. pfSense doesn't have problem with this, and pfSense was (and again is for the time being) what I'm using to connect so naturally I expected OPNsense to behave the same way since both very similar to one another.

The confusion is further amplified by the fact that both pfSense and OPNsense in their documentation are ambiguous about gateways issue and what is shown are actually screenshots for the DNS servers for the firewall, how to designate a different DNS server per gateway, not how to designate the gateway. What I have not tried is creating a gateway a virtual IP to mask the same gateway. I don't think it'd work but so far what I'm expecting to do for things to work is never what needs to be done, so...yeah.