Messages

I am pursuing this with my ISP Per other links

https://forum.netgate.com/topic/143547/slow-connection-using-carp-interface/9

OH
MY
GOD

It is CARP.

I have an HA pair, the CARP is 100% the problem. As soon as I disable using a CARP address for internal and external routing, everything is immediately fast.

..... yeah, Unless someone ports the patches to the Hardened Kernel, that ain't going to happen.

I am going to check out IPFire for the moment.

Which NIC is affected? Both or just the X553 or I210?

Maybe related to this...
https://forum.opnsense.org/index.php?topic=18754.msg109387#msg109387

I am pretty sure that this is what I am hitting - I am working to build a kernel with this fix and test.

I was first running under Proxmox 6.  I have a "control" VM running on another proxmox hardware node. It is stable at 150mbps down, and ~800mbps up.

I have tried proxmox with IOMMU, and now, I have OPNsense running on the bare metal.

I have tried VLANs + the x553, I have tried access mode with the I210s and the I350.

I have a Linux VM, small in stature, using Fedora 33 under proxmox, I can saturate the 1GB link in both directions without incident.  This is running Wireguard, and I can saturate the link, through the VPN, targeting  a remote iperf3.

Linux on baremetal is wirespeed as well.

I have tried some tuning, no luck. I was able to create a situation where it was only get 10mbps of performance.  Genuinely, I have no idea what to try.

I have tried a fresh install, etc. If I could get a linux-based firewall as nice as OPNsense, I would use it in a heartbeat for this role.

This has to be a timing or a driver problem, but, I am too n00b to the FreeBSD to troubleshoot this.

OK, I have gone to hardware, and I am still going slower than hell.

160mbps down, 850mbps up.

This is a supermicro X10SDV-TP8F.

dev.ix.0.%pnpinfo: vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac class=0x020000
dev.ix.0.%location: slot=0 function=0 dbsf=pci0:4:0:0 handle=\_SB_.PCI0.BR2C.H000
dev.ix.0.%driver: ix
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver
dev.ix.%parent:

I don't even know what to tune on this at this point.

I am desperate enough to give it a try.  Pretty sure that I can run ESXi on this platform.

I am not thrilled with VMware, but, It might be a better solution, for my use.

EDIT:  The CARP appears to be the problem.  As soon as I stopped using CARP addresses, the entire thing is super fast, I am getting wirespeed.

I have a couple firewalls virtualized on proxmox, and when we moved from 100mbit to 1Gbit, I found that I was getting 150mbps down, and I tried migrating to PCI Passthrough.

It turns out that *download* is slow, but, *upload* is fast.

This is consistent.   I have tried every form of tweaking that I can - does anyone have any idea how to solve this problem?  I see that this is not entirely unusual from the forum history, but, there are no real "resolutions".

I have two firewalls, one is using and Intel 553 10GB nic, with one port passed through, and VLANs

The other is using 2 I210 1GB nics, one for LAN, and one for WAN.

I have tried many things, including general purpose guides on FreeBSD tuning - nothing makes an impact.  I did successfully discover a way to reduce performance to 10mbit, but, I have not been able to move past 150mbps down.

Upload is consistent, almost 1gbit.

I have insight turned on, but, the performance remains the same with it disabled.  No Suricata or anything else, as of yet.