Messages

Nope it's not, Im not using the WAN interface at all, I'm just testing the speed of the LAN port hard wired to a Gigabit switch, like this:

[ OPNSesnse  running iperf3 in server mode ] LAN <=== [ Gitgabit Swith ] <=== [Macbook laptop with a 1Gb NIC]

I don't doubt that I'll be able to get more using both interfaces (WAN/LAN) flowing traffic that way but it will still not equate to full 1Gbps as a single interface is only able to route about 1/4 of that currently. Most importantly, the setup is the same for PfSense but its performance is much higher.

Thanks but the original numbers (at the very top of the thread) were using VirtIO without IPS enabled.

Hey folks, thanks for all the replies and apologies for the slow response, I did not know that I had to ask to be notified as the creator of a thread. Gonna try to answers all the questions on the thread

1. unbound doesn't seem to be the issue, see attached screenshot for a run. As a matter of fact, the box seems to be snoozing with reasonably high idle time.

2. Run with 2 CPUs (1 socket 2 cores). No marked difference

Code Select Expand


% iperf3 -c 172.16.160.204
Connecting to host 172.16.160.204, port 5201
[  5] local 172.16.160.144 port 53463 connected to 172.16.160.204 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  31.0 MBytes   260 Mbits/sec                 
[  5]   1.00-2.00   sec  29.7 MBytes   249 Mbits/sec                 
[  5]   2.00-3.00   sec  27.6 MBytes   231 Mbits/sec                 
[  5]   3.00-4.00   sec  26.1 MBytes   219 Mbits/sec                 
[  5]   4.00-5.00   sec  25.9 MBytes   217 Mbits/sec                 
[  5]   5.00-6.00   sec  25.8 MBytes   216 Mbits/sec                 
[  5]   6.00-7.00   sec  24.8 MBytes   208 Mbits/sec                 
[  5]   7.00-8.00   sec  24.6 MBytes   206 Mbits/sec                 
[  5]   8.00-9.00   sec  25.9 MBytes   218 Mbits/sec                 
[  5]   9.00-10.00  sec  25.2 MBytes   211 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   267 MBytes   224 Mbits/sec                  sender
[  5]   0.00-10.00  sec   266 MBytes   223 Mbits/sec                  receiver

iperf Done.


Code Select Expand

3. Run with 1 CPU core and epro1000 NIC
% iperf3 -c 172.16.160.204
Connecting to host 172.16.160.204, port 5201
[  5] local 172.16.160.144 port 53902 connected to 172.16.160.204 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.47 MBytes  45.9 Mbits/sec                 
[  5]   1.00-2.00   sec  27.0 MBytes   227 Mbits/sec                 
[  5]   2.00-3.00   sec  22.5 MBytes   189 Mbits/sec                 
[  5]   3.00-4.00   sec  28.3 MBytes   237 Mbits/sec                 
[  5]   4.00-5.00   sec  28.0 MBytes   235 Mbits/sec                 
[  5]   5.00-6.00   sec  28.2 MBytes   236 Mbits/sec                 
[  5]   6.00-7.00   sec  27.9 MBytes   234 Mbits/sec                 
[  5]   7.00-8.00   sec  27.9 MBytes   234 Mbits/sec                 
[  5]   8.00-9.00   sec  28.5 MBytes   239 Mbits/sec                 
[  5]   9.00-10.00  sec  28.1 MBytes   236 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   252 MBytes   211 Mbits/sec                  sender
[  5]   0.00-10.00  sec   251 MBytes   210 Mbits/sec                  receiver

iperf Done.


4. I didn't install the speedtestcli since I'm running all the tests locally (on the same switch) not really trying to test out the IP circuit.

5. I disabled IPv6 on the WAN (I saw some of the same "promiscuous" log entries) and here are the results:

Code Select Expand

% iperf3 -c 172.16.160.204
Connecting to host 172.16.160.204, port 5201
[  5] local 172.16.160.144 port 53960 connected to 172.16.160.204 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  18.6 MBytes   156 Mbits/sec                 
[  5]   1.00-2.00   sec  28.6 MBytes   240 Mbits/sec                 
[  5]   2.00-3.00   sec  28.7 MBytes   240 Mbits/sec                 
[  5]   3.00-4.00   sec  28.7 MBytes   241 Mbits/sec                 
[  5]   4.00-5.00   sec  29.0 MBytes   243 Mbits/sec                 
[  5]   5.00-6.00   sec  28.8 MBytes   242 Mbits/sec                 
[  5]   6.00-7.00   sec  28.2 MBytes   237 Mbits/sec                 
[  5]   7.00-8.00   sec  29.1 MBytes   244 Mbits/sec                 
[  5]   8.00-9.00   sec  28.5 MBytes   239 Mbits/sec                 
[  5]   9.00-10.00  sec  27.7 MBytes   232 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   276 MBytes   231 Mbits/sec                  sender
[  5]   0.00-10.00  sec   275 MBytes   231 Mbits/sec                  receiver

iperf Done.


TLDR: things are still strangely slow, I really think this is a kernel or tuning issue but I'm unsure where to even being to look.

and the second one

VM settings

[OPNSense 19.7.9_1-amd64]

Howdy folks, I'm running some tests on OPNSense 19.7.9_1-amd64  vs PFSense 2.4.4-RELEASE-p3 (amd64)  - both of them are running as virtual machines on the same host with no tuning but all patches applied. All I'm doing is installing iperf3 and running it in server mode for the tests, here are the results:

OPNsense:

% iperf3 -c 172.16.160.204
Connecting to host 172.16.160.204, port 5201
[  5] local 172.16.160.144 port 50482 connected to 172.16.160.204 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.99 MBytes  58.6 Mbits/sec                 
[  5]   1.00-2.00   sec  32.9 MBytes   276 Mbits/sec                 
[  5]   2.00-3.00   sec  33.0 MBytes   277 Mbits/sec                 
[  5]   3.00-4.00   sec  32.4 MBytes   272 Mbits/sec                 
[  5]   4.00-5.00   sec  31.9 MBytes   268 Mbits/sec                 
[  5]   5.00-6.00   sec  31.0 MBytes   260 Mbits/sec                 
[  5]   6.00-7.00   sec  31.1 MBytes   261 Mbits/sec                 
[  5]   7.00-8.00   sec  30.8 MBytes   259 Mbits/sec                 
[  5]   8.00-9.00   sec  31.2 MBytes   261 Mbits/sec                 
[  5]   9.00-10.00  sec  31.0 MBytes   260 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   292 MBytes   245 Mbits/sec                  sender
[  5]   0.00-10.00  sec   292 MBytes   245 Mbits/sec                  receiver

PFsense:
% iperf3 -c 172.16.160.190
Connecting to host 172.16.160.190, port 5201
[  5] local 172.16.160.144 port 49663 connected to 172.16.160.190 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  96.7 MBytes   811 Mbits/sec                 
[  5]   1.00-2.00   sec   112 MBytes   935 Mbits/sec                 
[  5]   2.00-3.00   sec   111 MBytes   935 Mbits/sec                 
[  5]   3.00-4.00   sec   112 MBytes   935 Mbits/sec                 
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec                 
[  5]   5.00-6.00   sec   112 MBytes   939 Mbits/sec                 
[  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec                 
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec                 
[  5]   8.00-9.00   sec   109 MBytes   914 Mbits/sec                 
[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.07 GBytes   923 Mbits/sec                  sender
[  5]   0.00-10.01  sec  1.07 GBytes   922 Mbits/sec                  receiver

The virtual machines are running under proxmoxve (linux/kvm) with same hardware settings (see screenhots). Needless to say I can get full gigabit performance through pfsense but about 4x lower using pfsense - is this expected?

Big thanks for a great product and great community