Slow connection on PPPoE via OPNsense

[HunvHunv]

Hi,

I am new to OPNsense and I have a problem with my PPPoE connection using OPNsense.
My Environment:
I have Supermicro X11SBA-LN4F (https://www.supermicro.com/en/products/motherboard/X11SBA-LN4F), that has 4 network adapter onboard and a Intel Pentium N3710. I added the maximum of 8GB RAM and a 512GB SSD (yes, oversized).
I installed VMware ESXi 7.0u2 on that and created a VM with 2 virtual network adapters, 4GB RAM and 2 vCPUs.  Currently there is just this one VM. In this VM I installed OPNsense, updated it to 21.1.4 and installed the VMware Tools Plugin. I configured 2 Interfaces: 1 for internal network and another one for the modem connection. I set up the information in the wizard at the start. I have a FibreChannel connection to my provider and should have 600MBit down and 100MBit up. My current Firewall can just handle 100MBit. Thats the reason for the change.
The PPPoE connection connects to the internet provider and a connection is established. But it is very unstable but works somehow. i.e. enough to update OPNsense.
I have pings up to 4000ms. Not constant. It is more or less random between 10ms and the 4000ms at every ping. This is from different devices in the network so it is not my PC. If I ping directly from OPNsense, I have the same result.
When I plug the cable to my modem back to my old firewall and set this firewall up as a gateway in OPNsense at System => Gateways, I have a constant low ping from my PC via the OPNsense and the old Firewall down to 3ms (DNS Server of my Provider).

From my understanding this means: The physical infrastructure is OK because the same cables and devices are working perfectly, when not using OPNsense as the uplink device. Also the hardware is not overloaded. No metrics show anything beside the base load when this happens. It seems like an issue with the software or drivers for the network adapter used for the PPPoE connection from OPNsense. The network adapter used on the motherboard is officially support by the ESXi Server. I also configured the E1000 network adapter for the VM like it is recommended in the OPNsense documentation.
I already tried to enable the disable-options at Interfaces => Settings. I also played around with the security policies on the ESXi server (i.e. promiscuous mode). But nothing changed anything.

Does anyone has an idea what the problem can be or where I can start deeper investigation?

[HunvHunv]

Hi, since I really need this, I continue troubleshooting this weekend.

I tried (without changes):
- because of a YouTube video regarding pfsense using just 1 vCPU
- uninstalling the igbn driver hoping another one will be used after. Failed => no network driver anymore
- setting the MTU to different values
- using VMXNET3 adapter instead of E1000
- Turn of power/performance optimization on the Motherboard and on the ESXi Server.
- Reinstalling ESXi and OPNsense
- Using Sophos UTM instead of OPNsense but with the same issues(!)

Beause of the last I think it is something about the network drivers on the ESXi Server. In the past at my job I already had issues with the igbn drivers as well, so this may be a path to follow.
The problem is that with ESXi 7.0 the old legacy igb drivers are not supported anymore. the igbn driver, that is currently used, is at the latest version 1.4.11.2 at my ESXi Host.
Does anyone that is virtualizing an OPNsense had a similar issue with the igbn driver on ESXi 7.0 in the past and was able to solve it?

[HunvHunv]

I tried one more thing. Instead of ESXi 7.0, which does not support the legacy igb driver anymore, I installed ESXi 6.7 instead. After install, I ran the following command via Shell Console (or via SSH) "esxcli software vib remove -n igbn" and restarted after. After the restart the igbn driver is not present anymore but the legacy igb driver will take over running the network adapters.
The big issue is fixed now. BUT the ping is still between ~50ms and ~300ms. Which is kind of stable compared to the values before but still unusable. But The igb driver also fixed something else. Before I had to configure at LAN default allow rule the "State Type" in the Firewall Rule setting always as "none". Otherwise most traffic was dropped by the firewall. This is also fixed by using the igb driver of ESXi 6.7.
Never the less. Even with testing different settings, hardware acceleration etc. the issue stays the same.
My next try was to use Microsoft Hyper-V 2019 instead of VMware ESXi. This is using 100% complete different drivers and virtualization engine.
After I installed and setup Hyper-V, I installed OPNsense the same way as I did on VMware. And finally: No latency issues on the pppoe connection any more.  Also the advanced firewall setting option is not required anymore.
As a summary my opinion about this is, that VMware ESXi is not supported if Intel network adapters that are using the igbn or the igb drivers are in use.
My last option, which I do not have to use now because Hyper-V works, would be to use my FritzBox router, that is still running for my DECT phones, as a router in between the Internet and the OPNsense setup. This setup was already kind of tested when I used my old firewall as the gateway instead the pppoe line.

[ashish14]

I had the same setup earlier and didn't face any issues.

My link was 400Mbps Up/Down and didn't face any latency or bandwidth issue.

ESXi 6.7 and igb driver Intel 82576 Dual Port Gigabit Card.

i310th gen 4c/8t, 8gb RAM, 240GB SSD.

I have re-purposed my Setup so unable to test anything now.

What I can remember about my setup within ESXi

vSwitch0 -

WAN Portgroup Name - igb0 as active and igb1 unused adapter.

LAN Portgroup Name -  igb1 as active and igb0 unused adapter.


Good Luck.

[inc10521]

Maybe a stupid thing to ask, but, did you load the Os-VMWare plugin in your OPNSense?

[danielm]

As a side note, why are you even running your main firewall on a VM when you could just use a physical machine? It is much more reliable in many ways and you will get better latencies as well.

[ashish14]

Maybe a stupid thing to ask, but, did you load the Os-VMWare plugin in your OPNSense?

Naa, wasn't facing any performance issues and wanted to keep it simple.

[HunvHunv]

Sorry for late answers. I lost focus on this, after I had a solution. But for others that may face the same issues:

Maybe a stupid thing to ask, but, did you load the Os-VMWare plugin in your OPNSense?

Yes, I did.

As a side note, why are you even running your main firewall on a VM when you could just use a physical machine? It is much more reliable in many ways and you will get better latencies as well.

Because I want to run a second (NextCloud) VM on the same hardware. Beside of this, a VM can be fully backuped and restored on other hardware in case of any issues. If I have physics, I have to setup everything from scratch again and in best case some application backups just works - but because of other hardware it may also not.

In general a status from the Hyper-V setup I run now:
It just works. Only thing is, that Hyper-V need more memory than ESXi Server. So I cannot use as much memory for my NextCloud VM as I like to use. But beside of this. No performance issue, everything runs stable.