[solved] Massive performance problems floating rules vs interface rules

[Gunni]

Hi there I have many performance issues with OPNsense. Here is one of them:
I have GBit Ethernet connected with about 20 VLANs.
I can iPerf between two VLANs at about 930 MBits/sec
When running the iPerf3 server on the OPNsense I get the following behaviour:

Firewall disabled: about 930 MBits/sec

Firewall enabled, Port 5201 opened in floating rules (tried different rule positions, top, middle, bottom): about 930 MBits/sec

Firewall enabled, Port 5201 opened in interrface rules (first rule): about 25 MBits/sec

I got more, but maybe this one is an easier one

System information:
OPNsense 21.7.2
OPNsense Cluster:
Master: 24 core Xeon @ 1,9GHz 192GB RAM
Backup: VM on ESX, 8 cores @2,2 GHz 16GB RAM

[mimugmail]

Just dont use the implementation on OPN itself ... it performs more than bad

[Gunni]

So I should not use OPNsense? Well that is a bold answer in an OPNsense forum.

[Greelan]

Lol. He was just referring to iperf

[Gunni]

But as you can see in my post iPerf works fine if using the floating rule or disabling the firewall.
So iperf is not the problem but OPNsense is.

[opnfwb]

I think the point being made here is, can you replicate the low throughput when using two iperf clients on either side of the VLANs.

It is bad practice to host an iperf session on the firewall itself. The firewall's priority is to push traffic through the interfaces, not source and host traffic on one interface. So you may not get representative results when the iperf instance is hosted directly on the firewall.

Whenever I'm using iperf to test throughput, I always host the client and server sessions on separate systems that exist on the networks I am testing. The firewall is only used to route traffic between the server and client iperf sessions.

Your listed configuration mentions ESX. It should be trivial to setup a client/server VM pair and drop them on the VLANs you want to test for throughput (for instance, put the client on VLAN 20, the server on VLAN 19, etc. etc.). That would easily remove the firewall hosted iperf as a potential variable.

[Gunni]

@openfwb:
The problem is i have massive problems all over the place.
First I tried a cluster with two OPNsense instances on the ESX. But the moment I get little bit of load on the OPNsense (about 50 Mbit ) I got packet loss of about 30%.
So I thought maybe it is the virtualization, and so I put the Master on the Hardware I mention in my post.
Still the same problems.
During testing I found the problem with the floating vs interface rules, and thought maybe that is my problem.
As it is reproducible very easy I thought that might be solvable.
I do not want to buy a hardware appliance for our company if I can not be sure that those problems do not occur there.
Another problem: When trying to download through a web proxy behind an IPFire firewall I get 20Mbit from a Linux Server, but only 2Mbit from the OPNsense. When I use the IPFire as child web proxy I get the 20Mbit from the OPNsense as well as the Linux server.
So there are problems.
And you can not tell me, that there is no problem with the firewall, when the performance degrades 50 times by just moving a firewall rule from the floating rules to the interface rules.

[mimugmail]

It's something in the virtualisation. Did you enable hardware offloading in OPNsense?

[Gunni]

It's something in the virtualisation. Did you enable hardware offloading in OPNsense?

No and no.
It is the same problem on hardware. And no I did not enable hardware offloading.

[opnfwb]

And you can not tell me, that there is no problem with the firewall, when the performance degrades 50 times by just moving a firewall rule from the floating rules to the interface rules.

There's obviously an issue that you have found. But we don't know what causes it and you seem unwilling to try a very simple method to rule out a potential variable (the firewall rule sorting) by just spinning up a client/server VM and pushing traffic that way.

Your lack of information about your environment also means most of us are shooting in the dark trying to help you. What is your ESXi version? Are you running openvmtools on all of the firewall appliances? Which NICs have you tried (vmx3, e1000)? What VM hardware version are you running for the OPNsense appliance?

You also haven't given us information on the networks. Are we talking about purely virtual routing where OPNsense is pushing traffic from all of your VLANs to various vSwitches or vDS managed port groups? Or is OPNsense pushing traffic out of the VM back on to a physical layer? That can be a huge variable too.

If you are pushing traffic back out to a physical layer, is a port over subscribed or used in another vSwitch that is causing the bandwidth variables?

[Gunni]

Sorry for not giving enough information. I am quite new to OPNsense and do not know what information is needed, as I had so many problems, I did not know where to start.
Neverthelesse, I found out the problem myself.

For now I use a WAN gateway in my LAN until I can fully switch to OPNsense.
I found out how to look at the raw FW rules and found the difference for the LAN interface in the rules with "reply-to"
So disabling that behaviour seems to make this problem go away.

Firewall -> Settings -> Advanced -> Disable reply-to -> (x) Disable Reply-to on WAN rules