[SOLVED] NAT VIP - Bandwidth issues after upgrade

[machados]

Hi Guys!
I'm experiencing some NAT VIP bandwidth issues after updating my opnsense cluster from 20.7 to 22.1.

In simple terms, the outbound NAT traffic is "capping" at about ~2 Mbps.
Doing simple iperf tests:
- before (20.7 ): 520Mbps
- after (22.1.8 ): 2.6Mbps

I've just reverted a node to double check, and I'm getting the same values.

My setup is similar to this one: https://docs.opnsense.org/manual/how-tos/carp.html .
 
In the meantime I'm testing the bandwidth of the firewall itself:
- from a desktop in the LAN to the firewall: 1 Gbps (normal LAN)
- from the firewall to the internet: 1 Gbps (normal ISP)
- from a desktop in the LAN, using firewall proxy (squid): 1Gbps (normal ISP)

The issue seems to be related with the NAT. In our setup, we have Manual NAT with the WAN VIP address.

Has anyone experienced a similar issue?

Thank you guys!

[franco]

I moved your post and removed the duplicate.

Are you using VLANs? There are a number of things to consider during migration for 22.1, but 21.1 and 21.7 migration notes might also play a role here. It's been quite a few changes over the last 18 months...


Cheers,
Franco

[machados]

Thank you for clearing the duplicate. I'll review the migration notes.

My setup is on PVE, two hosts. One opnsense VM instance per PVE host. VLANs are implemented/declared on the hypervisor, VM hw definition. So I'm not using VLANs on the firewall. The firewall itself has multiple NICs, and I'm not using the VLAN  interface feature in opnsense.

The migration procedure was via the GUI, the full process from 20.7 to 22.1.8 (a lot of reboots), but at all times the HA/VIP/CARP was working properly. So it seemed to be fine.
The bandwidth measurements were done with iperf and fast.com . The firewall nodes, both of them, don't seem to have a bandwidth issue. Desktop on the LAN to/from the firewall is ok. Firewall to/from the internet is ok. Desktop on the LAN to the internet via firewall proxy (squid) is ok.
The issue happens when using NAT. In my scenario, I have Manual Rules only, and I'm NATing with a WAN VIP address.

@franco thank you for your help, I'll review the migration notes. I'll prepare a test bench to test the setup, once I have news I'll share them here.
If someone has experienced a similar issue, or has a working scenario, it would be nice to hear from you.

Thank you!

[franco]

Hmm, is this a KVM setup? This might be relevant on FreeBSD 13... https://forum.opnsense.org/index.php?topic=28622.0


Cheers,
Franco

[machados]

Hi Franco!

Update - this seems to be related to VLANs.

First, sorry for the previous description, in my setup I do use the VLAN feature in the firewall itself, but only for the internal (LAN) side.
Second, I did the exact same setup, but with less complexity - single server, no cluster, no VIP, 22.1.2 (iso image), with VLAN setup on the LAN side - performance is ok.
Third, after upgrading to the latest version (22.1.9), immediately notice the NAT upload bandwidth capping at around 2~5Mbps.
Forth, following the "VLAN hint" I changed the interface setup. Now the VLAN is set on the PVE (VM hardware), I'm not using the VLAN feature on the opnsense. Upload performance is back to normal values.

So, it seems it is something related with the VLAN feature between versions 22.1.3 and 22.1.9.

My next test would be to try the 22.7.b update. In the meantime, once you hinted the VLAN setup, do you by chance have more info on this matter?

Thank you!

PS: I went through the release notes 22.1.3 til 22.1.8, there are a lot of changes in the interface management, but I didn't find any "caveat warning".

[franco]

22.1 migration notes mention that for VLANs you now need to assign and enable the parent interface as well in order to disable hardware acceleration features which get in the way of performance a lot in FreeBSD.


Cheers,
Franco

[machados]

Please mark this as SOLVED.

We reverted (backup) to the latest update, 22.1.9, adjusted the interface assignment settings. The VLANs parent interface is now assigned and enabled.
Everything is working well.

Thank you for your quick guidance and insight.

[franco]

ok great to hear