EDIT: The CARP appears to be the problem. As soon as I stopped using CARP addresses, the entire thing is super fast, I am getting wirespeed.
I have a couple firewalls virtualized on proxmox, and when we moved from 100mbit to 1Gbit, I found that I was getting 150mbps down, and I tried migrating to PCI Passthrough.
It turns out that *download* is slow, but, *upload* is fast.
This is consistent. I have tried every form of tweaking that I can - does anyone have any idea how to solve this problem? I see that this is not entirely unusual from the forum history, but, there are no real "resolutions".
I have two firewalls, one is using and Intel 553 10GB nic, with one port passed through, and VLANs
The other is using 2 I210 1GB nics, one for LAN, and one for WAN.
I have tried many things, including general purpose guides on FreeBSD tuning - nothing makes an impact. I did successfully discover a way to reduce performance to 10mbit, but, I have not been able to move past 150mbps down.
Upload is consistent, almost 1gbit.
I have insight turned on, but, the performance remains the same with it disabled. No Suricata or anything else, as of yet.
There was a problem with some Intel cards, specifically the drivers not being available within the last couple of years. Once they received a driver, there still was an issue with the multi-NIC versions. I ran into this as well and referenced https://docs.netgate.com/pfsense/en/latest/hardware/tune.html (https://docs.netgate.com/pfsense/en/latest/hardware/tune.html)
Also, have you thought about using a different hypervisor? ESXi is free when not requiring the additional functionality. For multiple VM, no frills (basic) use, it may be a better choice and from my experience, no head-ache. If you use ESXi, make sure you give a valid email address as you will be emailed a key (for basic usage). When you install, it will be in "evaluation mode" for a few months. You should entered the free key you received as it will prevent the server from automatically shutting down when not in eval / licensed mode.
I am desperate enough to give it a try. Pretty sure that I can run ESXi on this platform.
I am not thrilled with VMware, but, It might be a better solution, for my use.
The only reason why I recommended it is the issues I have had with opensouce hypervisors. They have all been difficult to setup / configure, and I was constantly researching for information to resolve errors / issues I end up with. Like microsoft, I don't care for them much; but one thing these company are great at is ease of use. I'm not speaking of "clicking", rather, setting up NICs does not take 1 day to search how to change the MTU because the way I found in the forum does not work anymore.
OK, I have gone to hardware, and I am still going slower than hell.
160mbps down, 850mbps up.
This is a supermicro X10SDV-TP8F.
dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac class=0x020000
dev.ix.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 handle=\_SB_.PCI0.BR2C.H000
dev.ix.0.%driver: ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver
dev.ix.%parent:
I don't even know what to tune on this at this point.
Did you test performance on the hardware with another platform? i.e. boot a rescue Linux or so and to some iperf testing
maybe it's simply a physical problem (clean your connections, maybe use a different cable etc.)?
Which NIC is affected? Both or just the X553 or I210?
Maybe related to this...
https://forum.opnsense.org/index.php?topic=18754.msg109387#msg109387
I was first running under Proxmox 6. I have a "control" VM running on another proxmox hardware node. It is stable at 150mbps down, and ~800mbps up.
I have tried proxmox with IOMMU, and now, I have OPNsense running on the bare metal.
I have tried VLANs + the x553, I have tried access mode with the I210s and the I350.
I have a Linux VM, small in stature, using Fedora 33 under proxmox, I can saturate the 1GB link in both directions without incident. This is running Wireguard, and I can saturate the link, through the VPN, targeting a remote iperf3.
Linux on baremetal is wirespeed as well.
I have tried some tuning, no luck. I was able to create a situation where it was only get 10mbps of performance. Genuinely, I have no idea what to try.
I have tried a fresh install, etc. If I could get a linux-based firewall as nice as OPNsense, I would use it in a heartbeat for this role.
This has to be a timing or a driver problem, but, I am too n00b to the FreeBSD to troubleshoot this.
Quote from: glasi on May 08, 2021, 10:02:39 AM
Which NIC is affected? Both or just the X553 or I210?
Maybe related to this...
https://forum.opnsense.org/index.php?topic=18754.msg109387#msg109387
I am pretty sure that this is what I am hitting - I am working to build a kernel with this fix and test.
..... yeah, Unless someone ports the patches to the Hardened Kernel, that ain't going to happen.
I am going to check out IPFire for the moment.
Raise it as an issue on Github, the Core team is usually open to pulling in bugfixes like that
OH
MY
GOD
It is CARP.
I have an HA pair, the CARP is 100% the problem. As soon as I disable using a CARP address for internal and external routing, everything is immediately fast.
I am pursuing this with my ISP Per other links
https://forum.netgate.com/topic/143547/slow-connection-using-carp-interface/9