OPNsense Forum

English Forums => General Discussion => Topic started by: glasi on October 03, 2020, 02:59:55 pm

Title: Slow routing performance for SMB/Samba traffic
Post by: glasi on October 03, 2020, 02:59:55 pm
Hi all,

I am using OPNsense in router-on-a-stick VLAN configuration and experiencing performance issues when routing SMB traffic between different VLANs.

The technical setup looks as follows:

Code: [Select]
                             LACP                                                 
+----------------------+     Trunk         +----------------------+               
| OPNsense 20.7.3      |     VLAN          | Cisco SG250-18       |               
| Intel Atom C3558     |-------------------| Switch               |               
| 8 GB RAM             |     2x 1 Gb/s     |                      |               
+----------------------+                   +----------------------+               
                                                 /          \                     
                                                /            \                   
                                        1 Gb/s /              \ 1 Gb/s           
                                              /                \                 
                                             /                  \                 
                            +----------------------+      +----------------------+
                            | File server          |      | Client               |
                            | VLAN 10              |      | VLAN 70              |
                            |                      |      |                      |
                            +----------------------+      +----------------------+
OPNsense will route all SMB traffic between the two VLANs. Disappointingly, I only get a transfer speed of 65-68 MB/s.

During transfer CPU is idling between 45% and 60%.

Code: [Select]
last pid: 80859;  load averages:  1.12,  0.61,  0.48
48 processes:  1 running, 47 sleeping
CPU:  0.1% user,  0.0% nice, 40.9% system,  0.1% interrupt, 58.9% idle
Mem: 202M Active, 314M Inact, 577M Wired, 175M Buf, 6758M Free
Swap: 8192M Total, 8192M Free

Interestingly, OPNsense is able to route at a speed of 112 MB/s between the two VLANs when performing speed tests via iPerf3.

Furthermore, I achieve full speed (112 MB/s) for my SMB traffic when the file server and the client are member of the same VLAN and traffic does not need to get routed by OPNsense.

Disabling interface scrubbing slightly improves the situation (approx. +8 MB/s). However, then my VoIP does not work anylonger.

For the sake of clarity, I do not use IDS/IPS.

Any ideas how to improve SMB traffic routing?
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: glasi on October 29, 2020, 12:49:22 pm
I did some further reading and testing. And it looks like my problem is not limited to SMB/Samba alone, but that it is more of a general routing problem with OPNsense.

Measure: In a first step, I disabled all packet filtering on the OPNsense device.
Result: No improvement.

Measure: In a second step and in order to rule out sources of error, I have removed the LAGG/LACP configuration in my setup.
Result: No improvement.

In the next step, I made some performance comparisons. I did tests with the following two setups:

a) Client (Ubuntu 20.04.1 LTS)   <-->   OPNsense (20.7.4)       <-->   File Server (Debian 10.6)
b) Client (Ubuntu 20.04.1 LTS)   <-->   Ubuntu (20.04.1 LTS)   <-->   File Server (Debian 10.6)

In both setups the client is a member of VLAN 70 and the file server is a member of VLAN 10. In setup b) I have enabled packet forwarding for IPv4.

The test results were as follows:

Samba transfer speeds (MB/sec)

Routing device        Client --> Server        Server --> Client
a) OPNsense67,371,2
b) Ubuntu108,7113,8


iPerf3 UDP transfer speeds (MBit/sec)

Routing device        Client --> Server        Server --> Client
a) OPNsense
948
23% packet loss
945
25% packet loss
b) Ubuntu
948
1% packet loss
938
0% packet loss

Packet loss leads to approx. 25% reduced throughput on the receiving device.

As I have seen there is also another thread with regards to poor throughput with OPNsense. You can find it here: https://forum.opnsense.org/index.php?topic=18754.0 (https://forum.opnsense.org/index.php?topic=18754.0)
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: glasi on October 29, 2020, 04:47:22 pm
Back with some more test results.

I did a rollback to OPNsense 20.1 for testing purposes.


Samba transfer speeds (MB/sec)

Routing device        Client --> Server        Server --> Client
OPNsense 20.1109,3102,6


iPerf3 UDP transfer speeds (MBit/sec)

Routing device        Client --> Server        Server --> Client
OPNsense 20.1
948
0% packet loss
949
0% packet loss


As you can see OPNsense 20.1 gives me full wire speed.
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: packetmaster on October 30, 2020, 10:54:09 am
Hi,

What is the full model of the NICs are used in your firewall / box?  Feels like some of the driver changes made to the Intel NICs in the recent FBSDs might have something to do with what you're seeing.
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: glasi on October 30, 2020, 03:09:25 pm
Intel Atom C3000 Series features Intel X553 NICs.

A couple of weeks ago I've compiled version 3.3.14 of the NIC driver from Intel sources. The self compiled driver did not bring any improvement.
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: mimugmail on October 30, 2020, 04:45:41 pm
Quote from: glasi on October 30, 2020, 03:09:25 pm
Intel Atom C3000 Series features Intel X553 NICs.

A couple of weeks ago I've compiled version 3.3.14 of the NIC driver from Intel sources. The self compiled driver did not bring any improvement.

No, because FreeBSD12 is now using iflib which does not perform very well compared to FreeBSD11.
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: glasi on October 30, 2020, 05:24:03 pm
Hence, we have to wait for changes/improvements in FreeBSD12?
Title: Re: Slow routing performance for SMB/Samba traffic
Post by: mimugmail on October 30, 2020, 07:36:06 pm
Correct