Firewall config can't support 10Gb/s bandwidth ?

[longfsilver]

Hello,

I am currently working on setting up an OPNsense firewall on a 10Gb network and I am having some difficulties with bandwidth.
To be clear during my tests I find that my bandwidth is constrained when my network flow goes through the firewall. Instead of having average at 10Gb/s bandwidth, I've only 3Gb/s.

My configuration is as follows:

Operation:

• Server A sends requests to a server B through the firewall.
• Redirections are done in NAT (Queries from Server A to Firewall> NAT Translation on Firewall> Returning the Request to Server B).
• The 3 servers have the same hardware configuration : Dell PowerEdge C6320 Server, Intel Xeon, 16 Gb RAM with 2 NICs Intel 82599ES 10-Gigabit SFI / SFP +
• For the OS, server A and B are in Debian 7.9 and 7.8 and the Firewall in OPNsense 19.1.
• The 3 servers are interconnected with two switches configured for 10Gb / s per port (SFP, full-duplex, lacp).
• The 3 machines are on the same site.
• No traffic on the firewall except server A and B.

Problem:

• During a direct transfer between server A and B, I reach 10Gb/s bandwidth.
• As soon as I redirect requests to the firewall such as: Server A> Firewall> Server B, my bandwidth is divided by 3, never exceeding 3Gb/s.

Tests carried out:
On the firewall:

• LAGG> LACP
• LAGG> RoundRobin
• LAGG> LoadBalancing
• Simple VLAN
• VLAN priority test from 0 to 7.
  
• Modification of MTU, tests with values ​​1500, 1400, 1300, 1250.
• Same for MSS
• NIC configuration (10Gb full-duplex, auto select, default)
• Firewall> Settings> Miscellaneous> tests in normal aggressive, conservative
• Interfaces> Settings> Hardware CRC, Hardware TSO, Hardware LRO disabled
• Compilation of the kernel with driver provided by Intel for the network card.
• Switching from the version of OPNsense 18.7 to 19.1.

On the switch:

• Active / active LACP
• LACP active / passive

Measurement tools:

• Tests with iPerf
• Tests with SCP
• Tests with RSYNC
• Tests with netcat
• Opnsense Dashboard

Questions :

• Have you ever encountered such cases? If so, can you tell me what configuration you used.
• Is the hardware part problematic for you (especially the network card) ? If so what do you recommend.

Thank you in advance,

[mimugmail]

Did you try disabling NAT?

[longfsilver]

Hello mimugmail,

Yes I've disable NAT but nothing change.

Actually, I use this config

On the switches :
2 ports 10Gb full-duplex using port channel active/active (I've tried passive/active).

On the server A & B :
I've added route on each to force the traffic through the Firewall

On the Firewall :
I use oneVLAN for the test.
No restrictive rule.
No NAT.
2 interfaces in different VLAN using LAGG on LACP mode.

The only change wich increase my bandwidth is when I've disabled my Firewall (Router mode) from 3Gb/s to 6-7Gb/s.

[mimugmail]

What happens without using LAGG

[longfsilver]

Hello,

> What happens without using LAGG
Nothing, unfortunately.
I've also seen some errors on my NICs :

<5>ix1: link state changed to DOWN
<5>ix0: link state changed to DOWN
<5>lagg0: link state changed to DOWN
<5>lagg0_vlan20: link state changed to DOWN
<5>lagg0_vlan40: link state changed to DOWN
<5>ix0: link state changed to UP
<5>lagg0: link state changed to UP
<5>lagg0_vlan20: link state changed to UP
<5>lagg0_vlan40: link state changed to UP
<5>ix0: link state changed to DOWN
<5>lagg0: link state changed to DOWN
<5>lagg0_vlan20: link state changed to DOWN
<5>lagg0_vlan40: link state changed to DOWN
<5>ix0: link state changed to UP
<5>lagg0: link state changed to UP
<5>lagg0_vlan20: link state changed to UP
<5>lagg0_vlan40: link state changed to UP

Now, I've tested without LACP on my switches and shutdown one of my switch port to test if the LACP can be the problem.
And even if I have no more flaps on my NIC, my bandwidth still stay to 4Gb/s, almost 5Gb/s without NAT.

[pylox]

Hi longfsilver,

maybe there is some extra tuning for FreeBSD neccessary. You will find some very useful informations here:

https://people.freebsd.org/~olivier/talks/2018_AsiaBSDCon_Tuning_FreeBSD_for_routing_and_firewalling-Paper.pdf

btw.: How many sockets/cpu cores and how many queues per (10gb-) nic do you have ?

regards pylox

[newsense]

Hello,

> What happens without using LAGG
Nothing, unfortunately.
I've also seen some errors on my NICs :

<5>ix1: link state changed to DOWN
<5>ix0: link state changed to DOWN
<5>lagg0: link state changed to DOWN
<5>lagg0_vlan20: link state changed to DOWN
<5>lagg0_vlan40: link state changed to DOWN
<5>ix0: link state changed to UP
<5>lagg0: link state changed to UP
<5>lagg0_vlan20: link state changed to UP
<5>lagg0_vlan40: link state changed to UP
<5>ix0: link state changed to DOWN
<5>lagg0: link state changed to DOWN
<5>lagg0_vlan20: link state changed to DOWN
<5>lagg0_vlan40: link state changed to DOWN
<5>ix0: link state changed to UP
<5>lagg0: link state changed to UP
<5>lagg0_vlan20: link state changed to UP
<5>lagg0_vlan40: link state changed to UP

Now, I've tested without LACP on my switches and shutdown one of my switch port to test if the LACP can be the problem.
And even if I have no more flaps on my NIC, my bandwidth still stay to 4Gb/s, almost 5Gb/s without NAT.

Are you on 19.1.2 ? Those flip flops should have been fixed.