Slow routing performance for SMB/Samba traffic

[glasi]

Hi all,

I am using OPNsense in router-on-a-stick VLAN configuration and experiencing performance issues when routing SMB traffic between different VLANs.

The technical setup looks as follows:

Code: [Select]

                             LACP                                                 
+----------------------+     Trunk         +----------------------+               
| OPNsense 20.7.3      |     VLAN          | Cisco SG250-18       |               
| Intel Atom C3558     |-------------------| Switch               |               
| 8 GB RAM             |     2x 1 Gb/s     |                      |               
+----------------------+                   +----------------------+               
                                                 /          \                     
                                                /            \                   
                                        1 Gb/s /              \ 1 Gb/s           
                                              /                \                 
                                             /                  \                 
                            +----------------------+      +----------------------+
                            | File server          |      | Client               |
                            | VLAN 10              |      | VLAN 70              |
                            |                      |      |                      |
                            +----------------------+      +----------------------+
OPNsense will route all SMB traffic between the two VLANs. Disappointingly, I only get a transfer speed of 65-68 MB/s.

During transfer CPU is idling between 45% and 60%.

Code: [Select]

last pid: 80859;  load averages:  1.12,  0.61,  0.48
48 processes:  1 running, 47 sleeping
CPU:  0.1% user,  0.0% nice, 40.9% system,  0.1% interrupt, 58.9% idle
Mem: 202M Active, 314M Inact, 577M Wired, 175M Buf, 6758M Free
Swap: 8192M Total, 8192M Free

Interestingly, OPNsense is able to route at a speed of 112 MB/s between the two VLANs when performing speed tests via iPerf3.

Furthermore, I achieve full speed (112 MB/s) for my SMB traffic when the file server and the client are member of the same VLAN and traffic does not need to get routed by OPNsense.

Disabling interface scrubbing slightly improves the situation (approx. +8 MB/s). However, then my VoIP does not work anylonger.

For the sake of clarity, I do not use IDS/IPS.

Any ideas how to improve SMB traffic routing?

[glasi]

I did some further reading and testing. And it looks like my problem is not limited to SMB/Samba alone, but that it is more of a general routing problem with OPNsense.

Measure: In a first step, I disabled all packet filtering on the OPNsense device.
Result: No improvement.

Measure: In a second step and in order to rule out sources of error, I have removed the LAGG/LACP configuration in my setup.
Result: No improvement.

In the next step, I made some performance comparisons. I did tests with the following two setups:

a) Client (Ubuntu 20.04.1 LTS)   <-->   OPNsense (20.7.4)       <-->   File Server (Debian 10.6)
b) Client (Ubuntu 20.04.1 LTS)   <-->   Ubuntu (20.04.1 LTS)   <-->   File Server (Debian 10.6)

In both setups the client is a member of VLAN 70 and the file server is a member of VLAN 10. In setup b) I have enabled packet forwarding for IPv4.

The test results were as follows:

Samba transfer speeds (MB/sec)

Routing device	Client --> Server	Server --> Client
a) OPNsense	67,3	71,2
b) Ubuntu	108,7	113,8

iPerf3 UDP transfer speeds (MBit/sec)

Routing device	Client --> Server	Server --> Client
a) OPNsense	948 23% packet loss	945 25% packet loss
b) Ubuntu	948 1% packet loss	938 0% packet loss

Packet loss leads to approx. 25% reduced throughput on the receiving device.

As I have seen there is also another thread with regards to poor throughput with OPNsense. You can find it here: https://forum.opnsense.org/index.php?topic=18754.0

[glasi]

Back with some more test results.

I did a rollback to OPNsense 20.1 for testing purposes.


Samba transfer speeds (MB/sec)

Routing device	Client --> Server	Server --> Client
OPNsense 20.1	109,3	102,6

iPerf3 UDP transfer speeds (MBit/sec)

Routing device	Client --> Server	Server --> Client
OPNsense 20.1	948 0% packet loss	949 0% packet loss

As you can see OPNsense 20.1 gives me full wire speed.

[packetmaster]

Hi,

What is the full model of the NICs are used in your firewall / box?  Feels like some of the driver changes made to the Intel NICs in the recent FBSDs might have something to do with what you're seeing.

[glasi]

Intel Atom C3000 Series features Intel X553 NICs.

A couple of weeks ago I've compiled version 3.3.14 of the NIC driver from Intel sources. The self compiled driver did not bring any improvement.

[mimugmail]

Intel Atom C3000 Series features Intel X553 NICs.

A couple of weeks ago I've compiled version 3.3.14 of the NIC driver from Intel sources. The self compiled driver did not bring any improvement.

No, because FreeBSD12 is now using iflib which does not perform very well compared to FreeBSD11.

[glasi]

Hence, we have to wait for changes/improvements in FreeBSD12?

[mimugmail]

Correct