Very mixed WAN performance

[thex]

I have recently switched from a simple mikrotik hex router running opwenWRT to a fanless mini PC (Intel n5105 with 5 intel i225V3 nics)

Setup was pretty straight forward but since the switch I have very fluctuating WAN speeds.
Sure as I'm on cable they were far from stable before and in the evening when many people in the neighborhood started streaming my 400 Mbit/s connection would drop to 200 something but the rest of the day (especially during the night it was pretty stable between 300 and 415. (sadly I currently cant access the graph easily)

However this is what the last 7 days looked like:


I have already followed some guides regarding optimizations for gigabit speeds but they did not do much, got about 10 mbps more but that could also have been fluctuations. (builed down to multithreading isr tuneables mostly)

Do you have any idea what could cause such wild fluctuations?
I would blame it on the ISP but I have seen very clearly in my speedtest graphs that these problems started exactly when switching to opnsense.

Setup: Cable ---> Modem (Bridge Mode) ---> opnsense box (sam MAC as old openwrt) ---> Test PC

* Opnsense latest version is virtualized on ProxMox but nics are passed through directly.
* CPU Load is 50% while doing a speed test, 6-8% when idle
* I have currently enabled shaping but with a 420 Mbit/s limit and have also tasted with it deactivated getting similar fluctuating speeds
* Very few firewall rules, mostly a few port forwards.

Let me know if you need any mor information to support me.

Thanks

[thex]

Some additional info:

Iperf3 locally to opnsense works with 2.35 Gbit/s both directions

CPU load on VM according to proxmox goes up to 50% but opnsense dashboard shows max of 19%

[lilsense]

Couple of things.

1st run opnsense barebones (no proxmox)
2nd check your bufferbloat
https://www.waveform.com/tools/bufferbloat
and tune it using : https://docs.ibracorp.io/opnsense/

[thex]

1st will be hard to do as this is "just" a home setup and to do this it would be quite some effort and downtime to go to bare metal.

I did some bufferbloat investigations yesterday and got "OK" values (but they were also were better with the small openwrt box before)
I will do some more investigations/optimization looking at the documentation you shared.

Any additional tips how to investigate?
It really looks like the firewall to WAN can't do more than about 370Mbit/s although CPU is not loaded >50%
I also checked if the shaper works and if I set it for example to 100Mit I get about 90 mbits which I guess is expected but if I set it to 1000 I still never get over 370

[lilsense]

Needle in a haystack comes to mind when dealing with virtual firewalls as there are so many variables involved which would hard to predict.

What else is running on the proxmox besides OPNsesne?

[thex]

Currently nothing. Plan is reverse proxy and pihole and maybe some other light weight containers/services like mailrise which shall always be available.