weird issues with network

[curiousgeorge]

I need some help with figuring out what is going on.
I'll start with my setup:
I am running opnsense on proxmox. I have a 8 core, 16 thread AMD ryzen 5 with 32GB RAM. Of this, 14 threads are assigned to the opnsense along with 28GB of RAM. The system has a 500GB drive for Proxmox and a 256GB drive for opnsense. I have a 2.5GB port for internet and a 2.5GB port for LAN along with a 1GB port for proxmox admin controller. The traffic from NICs for opnsense is blocked by proxmox firewall for security. I have VLANs for internet connected IOT devices, work computers, adult devices, kids devices, and non-internet connected IOT devices. I am using Omada OC300 controller with 3 wireless APs and 3 wired smart switches for WiFi and ethernet connections. I have a total of over 50 devices connected. i also have a server for local AI with Home assistant and NextCloud exposed through a cloudflare tunnel with Geo restrictions blocking access from anywhere outside the US and a bunch of other restrictions. Everything that supports 2-factor has 2-factor and the server can only be logged in with public key and 2-factor.

On Opnsense, I have zenarmor with policies set to block unknown devices so each device must be manually approved. I also have multiple policies with a Home level subscription. I am running surricata with snort rules, Maltrail, monit, and CrowdSec for security. I also use KEA dhcp and unbound DNS, and have a Wazuh agent running for general security. I have disabled root access to the GUI, which was set up with 2-factor TOTP and created a separate user which also has 2-factor enabled. Omada also has 2-factor. I use let encrypt certs for my domain on both. I also have static ip addresses set for all devices. I know, its a lot.

For the past year, I have been having weird issues with Opnsense.

Issue 1:
I have a car that is connected to my network and it seems to be connected fine. I can get updates for the car through wifi. However, when it is connected to my network, the API for the car does not work and the car mobile app shows that it is disconnected. I have tried to disable surricata, zenarmor, Maltrail, monit, corwsec, and Wazuh and it still shows as not connected on the app. It also wont connect through the API I use with Home Assistant when connected to my network. I dont see any firewall states that are have an issue for the device. I see nothing blocked in zenarmor. It all looks fine. Now for the odd part. I reinstalled the entire system including the proxmox, which was updated to 9.2 (latest), and reinstalled Opnsense 26.7. I used my backup configs for Opnsense and Zenarmor to restore everything. Everything in the car worked fine for a day or so and then it was showing up as not connected in the app and API. I disabled everything after it showed up as not connected and it changed nothing. The logs for Opnsense show no updates or changes to configs. Wazuh shows no changes either. I have no idea where to look for more details on why this is happening.

Issue 2:
I upgraded my daughters computer and updated the motherboard and cpu, so the wireless card changed. I trusted the device in zenarmor, updated the mac address, and it all looked like it was fine except the wifi would not connect to the internet. I could not find anything being blocked in any of the security software.

Issue 3:
When I used to connect my phone to the network, the browser would randomly have issues like pages not loading css (plain html only) and my VPN would randomly drop. It would also stop working at times and clearing app cache force restarting it would return it normal. This would never happen on the 5g data.

I work extensively with Linux, software dev, AWS, and a secured system at work, so I have some decent knowledge, but BSD is not something I have done much work with. I have considered wifi interference and lowered the power on the APs and one of the APs is located right in my bedroom which is near my work desk.

I will also give a little bit of background in to why I am so paranoid about security. Years ago, I informed the FBI of a person who was involved in hacking ISP like Comcast and many financial institutions and that person went to prison for years with over a $1M confiscated along with expensive cars and homes. He found out that I informed the FBI before he went to prison possibly through the court case, and he was someone I knew. He was involved with foreign hackers from Russia and many hackers from around the world including another one in the US who was of Russian origin. I'm not saying I'm getting hacked, but it certainly is in the back of my mind since he has all the information needed to track me.

I would like determine why this odd stuff is happening to my network despite no logs indicating any issues. I need someone who is an expert in Opnsense to help me diagnose the issue since I admittedly have limited knowledge of BSD and mostly interact with Opnsense through the GUI. There are other details I may be leaving out about the setup and can share information after I de-ID the output. Can someone with extensive knowledge help me diagnose this and find root cause, please?

I am not a bot. I am not making any of this up. I am a slightly paranoid human looking for some help. I have already tried to use Gemini and my local qwen 3.8 and it was mostly things I had already looked at like logs and firewall rules and states. I dont understand how something can change without me making a change to it and no logs or alerts indicating any issues.

[nero355]

I have a 8 core, 16 thread AMD ryzen 5 with 32GB RAM. Of this, 14 threads are assigned to the opnsense along with 28GB of RAM.

Why use Proxmox at all when almost all of it is OPNsense ?!


About your issue itself :
How do you know it's OPNsense and not something else ?!

I can imagine WiFi compatibility issues between the Omada Accesspoints and the car and the new PC you mentioned for example.

Another very easy thing to check is WiFi Channels or complete Accesspoints overlapping because 2.4 GHz is very annoying to tune down :
Even the 4dBi on Ubiquiti UniFi products and 7 dBi with the antenna gain included on TP-Link Omada products is sometimes not low enough !!
I ended up completely disabling it on one Accesspoint to avoid connection issues ;)

Have you tried a different Router instead of OPNsense ?
Did you capture the traffic of the Clients having issues and analyse it in Wireshark ?
You can do that both at your Switches and OPNsense to compare/check different stages of communication.

[curiousgeorge]

I have a 8 core, 16 thread AMD ryzen 5 with 32GB RAM. Of this, 14 threads are assigned to the opnsense along with 28GB of RAM.

Why use Proxmox at all when almost all of it is OPNsense ?!


About your issue itself :
How do you know it's OPNsense and not something else ?!

I can imagine WiFi compatibility issues between the Omada Accesspoints and the car and the new PC you mentioned for example.

Another very easy thing to check is WiFi Channels or complete Accesspoints overlapping because 2.4 GHz is very annoying to tune down :
Even the 4dBi on Ubiquiti UniFi products and 7 dBi with the antenna gain included on TP-Link Omada products is sometimes not low enough !!
I ended up completely disabling it on one Accesspoint to avoid connection issues ;)

Have you tried a different Router instead of OPNsense ?
Did you capture the traffic of the Clients having issues and analyse it in Wireshark ?
You can do that both at your Switches and OPNsense to compare/check different stages of communication.

I'm using proxmox to fully backup the initial install after updates and config restore. It's much easier than reinstall and restore. Since proxmox can't be accessed from anything except a separate NIC which remains disconnected, it's about as safe as a vm host can get.

I can set a port to tee all traffic to a port and connect to that port. Great suggestion.

[meyergru]

From your description, the main obstacle is the complexity of your setup and the lack of a controlled baseline for troubleshooting. You describe several different symptoms without having established where each failure occurs. Systematic fault isolation comes before BSD-specific expertise.

A crucial distinction here is between configuration and runtime state. An unchanged configuration does not imply unchanged behaviour. DNS answers, DHCP leases, ARP caches and firewall connection states change during normal operation. Security products can update signatures, reputation data and blocklists automatically. The same configured policy can therefore produce different results as its inputs and state change. Looking for manual configuration changes will not explain those differences, and the absence of configuration-change logs is entirely unsurprising.

You have also combined several access-control mechanisms whose interactions need to be understood. Some combinations have documented restrictions: Zenarmor and Suricata in IPS mode, when using netmap, require separate interfaces because they cannot both own the same netmap interface. That is one concrete example of why individually useful security products cannot simply be combined without checking interoperability.

Running the Proxmox firewall alongside the OPNsense firewall introduces another independent filtering layer. Traffic dropped before reaching the VM cannot appear in the guest firewall's logs. Each layer needs to be investigated at the point where it acts.

Wireless networking - being unreliable at best - adds further variables. Access points may perform ARP broadcast-to-unicast conversion or multicast-to-unicast conversion, introducing additional processing and learned state. These features can require disabling for particular applications; TP-Link's own Omada guidance for Sonos is an example. A connected Wi-Fi client therefore does not establish that every relevant type of traffic is being delivered correctly.

These examples illustrate the diagnostic problem; they do not establish suspected causes of your particular failures.

The important point is that these variables create combinations. The diagnostic search space grows multiplicatively as configuration choices, runtime states and interactions are combined. Understanding each component individually is insufficient if you cannot also account for their combined behaviour. Thus, you need a minimal, known-working baseline, one reproducible symptom at a time, and tests that locate where communication breaks down. Reinstalling everything and restoring the same configuration also restores any configuration problems it contains, so Proxmox does not help you there, either.

Realistically, you are unlikely to find volunteers willing to reconstruct and untangle this entire environment free of charge. A forum can help with specific, sufficiently isolated problems. You need to do the groundwork yourself: reduce the complexity of your setup and present one clearly isolated, reproducible failure at a time.