Slow Download, Fast Upload (CARP is Slow)

[joshland]

EDIT:  The CARP appears to be the problem.  As soon as I stopped using CARP addresses, the entire thing is super fast, I am getting wirespeed.

I have a couple firewalls virtualized on proxmox, and when we moved from 100mbit to 1Gbit, I found that I was getting 150mbps down, and I tried migrating to PCI Passthrough.

It turns out that *download* is slow, but, *upload* is fast.

This is consistent.   I have tried every form of tweaking that I can - does anyone have any idea how to solve this problem?  I see that this is not entirely unusual from the forum history, but, there are no real "resolutions".

I have two firewalls, one is using and Intel 553 10GB nic, with one port passed through, and VLANs

The other is using 2 I210 1GB nics, one for LAN, and one for WAN.

I have tried many things, including general purpose guides on FreeBSD tuning - nothing makes an impact.  I did successfully discover a way to reduce performance to 10mbit, but, I have not been able to move past 150mbps down.

Upload is consistent, almost 1gbit.

I have insight turned on, but, the performance remains the same with it disabled.  No Suricata or anything else, as of yet.

[errored out]

There was a problem with some Intel cards, specifically the drivers not being available within the last couple of years.  Once they received a driver, there still was an issue with the multi-NIC versions.  I ran into this as well and referenced https://docs.netgate.com/pfsense/en/latest/hardware/tune.html

Also, have you thought about using a different hypervisor?  ESXi is free when not requiring the additional functionality.  For multiple VM, no frills (basic) use, it may be a better choice and from my experience, no head-ache.  If you use ESXi, make sure you give a valid email address as you will be emailed a key (for basic usage). When you install, it will be in "evaluation mode" for a few months.  You should entered the free key you received as it will prevent the server from automatically shutting down when not in eval / licensed mode.

[joshland]

I am desperate enough to give it a try.  Pretty sure that I can run ESXi on this platform.

I am not thrilled with VMware, but, It might be a better solution, for my use.

[errored out]

The only reason why I recommended it is the issues I have had with opensouce hypervisors.  They have all been difficult to setup / configure, and I was constantly researching for information to resolve errors / issues I end up with.  Like microsoft, I don't care for them much;  but one thing these company are great at is ease of use.  I'm not speaking of "clicking", rather, setting up NICs does not take 1 day to search how to change the MTU because the way I found in the forum does not work anymore.

[joshland]

OK, I have gone to hardware, and I am still going slower than hell.

160mbps down, 850mbps up.

This is a supermicro X10SDV-TP8F.

dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac class=0x020000
dev.ix.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 handle=\_SB_.PCI0.BR2C.H000
dev.ix.0.%driver: ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver
dev.ix.%parent:

I don't even know what to tune on this at this point.

[marcquark]

Did you test performance on the hardware with another platform? i.e. boot a rescue Linux or so and to some iperf testing
maybe it's simply a physical problem (clean your connections, maybe use a different cable etc.)?

[glasi]

Which NIC is affected? Both or just the X553 or I210?

Maybe related to this...
https://forum.opnsense.org/index.php?topic=18754.msg109387#msg109387

[joshland]

I was first running under Proxmox 6.  I have a "control" VM running on another proxmox hardware node. It is stable at 150mbps down, and ~800mbps up.

I have tried proxmox with IOMMU, and now, I have OPNsense running on the bare metal.

I have tried VLANs + the x553, I have tried access mode with the I210s and the I350.

I have a Linux VM, small in stature, using Fedora 33 under proxmox, I can saturate the 1GB link in both directions without incident.  This is running Wireguard, and I can saturate the link, through the VPN, targeting  a remote iperf3.

Linux on baremetal is wirespeed as well.

I have tried some tuning, no luck. I was able to create a situation where it was only get 10mbps of performance.  Genuinely, I have no idea what to try.

I have tried a fresh install, etc. If I could get a linux-based firewall as nice as OPNsense, I would use it in a heartbeat for this role.

This has to be a timing or a driver problem, but, I am too n00b to the FreeBSD to troubleshoot this.

[joshland]

Which NIC is affected? Both or just the X553 or I210?

Maybe related to this...
https://forum.opnsense.org/index.php?topic=18754.msg109387#msg109387

I am pretty sure that this is what I am hitting - I am working to build a kernel with this fix and test.

[joshland]

..... yeah, Unless someone ports the patches to the Hardened Kernel, that ain't going to happen.

I am going to check out IPFire for the moment.

[marcquark]

Raise it as an issue on Github, the Core team is usually open to pulling in bugfixes like that

[joshland]

OH
MY
GOD

It is CARP.

I have an HA pair, the CARP is 100% the problem. As soon as I disable using a CARP address for internal and external routing, everything is immediately fast.

[joshland]

I am pursuing this with my ISP Per other links

https://forum.netgate.com/topic/143547/slow-connection-using-carp-interface/9