Messages

Any suggestions on what to do to fix or isolate the problem?

How is it holding up?

I am not sure if this is related to upgrading to 25.7 or something that has changed by my mobile carrier.  I have a multi-WAN configuration setup to use Google Fiber as the primary connection and a mobile broadband connection for the secondary.  The mobile broadband connection is using a router that is providing a 129.168.12.x IP address.  The failover WAN is getting the proper IP and is able to ping that gateway.  When I SSH into the OPNSense box, I am able to ping the monitor IP (8.8.4.4) and have it go over the mobile WAN.  Then when I ping the other monitor IP (8.8.8.8) it uses Google Fiber.  I have let the ping run and I don't see any spikes in latency or dropped packets.  The Gateway monitoring is showing something totally different:

IPv4 250 192.168.12.1 8.8.4.4 20635.6 ms 129.2 ms 28.0 %

Has anyone experienced something similar?  Are there settings that I should update to correct this?

I am thinking that something might have changed.  I have a similar setup with GFiber as my primary and Mobile Internet for my secondary.  The GW monitoring shows a lot of latency/packet loss on the Mobile Internet port.  When I go into the CLI on OPNSense and ping the monitoring IP, I don't see the same issues.

217 packets transmitted, 217 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 34.208/51.097/83.731/9.542 ms

I also know that it is using the Mobile Internet because when I ping the other monitoring IP (8.8.8.8) the latency is far less.  I am wondering if it might be something related to the multiple NAT.

Possibly just a heck of a lot of load. Netflow is an incredibly expensive operation and might even be made even worse by virtual hardware. Just a guess, not verified in any setup, yet.

I believe that you are 100% correct in that.  I am debating if I should take the interfaces and move them from pass through to virtualized to see if it still works.

I have narrowed down a few things on the speed issue.  I tried doing a full reset on the OPNSense configuration and everything worked perfect in terms of speed.  I then started trying to disable some of the running services.  I narrowed the issue down to Netflow Distributor.  If I disable that service, my speeds go all the way up.  I am wondering what is up with that service that could be causing the problem.

I was able to get the VM to now do a 5Gb up and down on the LAN side.  The WAN side is similar.  This is after I got my configuration moved over from my Baremetal machine.  The Server is running Xeon Gold 5115.  I wonder if they are just to slow to handle much more than that.

Did you follow the tips here?

You can also try using multiqueue in the NIC settings in Proxmox and hopefully, you have enabled RSS (IDK if your walkthroughs cover that).

I switched to ESXi to see if that would help resolve some of the issues.  I was able to get the WAN side of the network to show full bandwidth (8/8).  I Did find something strange in the settings.  When I went in and checked the boxes for hardware offloading, it seemed to remove it.  When I had them unchecked, they seemed to work correctly.  So that is the good part of the testing so far.  When I check on a Windows VM on the LAN side, I get very slow download speeds, but my upload speeds seem on par to what I would think to see.

I have OPN virtualised on Proxmox. What is 8/8 ?

8gb/8gb Fiber Connection

I just installed ESXI to see if it had the same issues.  Here is something VERY strange:

Windows VM - full speed 8/8
FreeBSD 14.2 - full speed 8/8
OPNSense - reduced speed to around half

Does anyone have a VM running on an internet connection that is 8/8 or higher?  I think there is just a limitation with the virtualized interfaces not being able to handle the traffic.

I am currently running OPNSense configured like multiple of the walkthroughs show online.  After configuration, my speeds on my WAN interface (VirtIO) is almost identical to what I had with my bare metal install.  The speeds on the LAN (VirtIO) interface are horrible.  I have tried everything that I could think of to fix the issue.  The one strange thing that I saw was:



ethtool --offload ens4f1 (WAN interface) generic-receive-offload off - fixing my LAN speeds, but cuts my WAN speeds in half.  If I go and turn it back on, the WAN speed goes back to normal, but the download on the LAN download goes to almost nothing, but the upload is almost spot on.  I would rather not pass my interfaces through to the VM if I don't have to.

Does anyone else have an issue with the passthru device causing the VM to lockup?

I just recently setup OPNSense and attempted to install Adguard.  It keeps failing to start and create the yaml file on the current version of OPN.  Any suggestions on what I could possibly do to work around that?

I am wondering if it is an issue with the version of the pkg command that I am running.  I installed bhyve and just realized that it upgraded the pkg version (or I believe it did).  Would that cause this issue?