Messages

Thanks a lot man :) appreciate your patience.

Thanks for your patience.
I have set values according to your suggestions:

Code Select Expand

root@OPNsense:~ # ifconfig vtnet0 | grep mtu
vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1512
root@OPNsense:~ #

Code Select Expand

root@OPNsense:~ # ifconfig vtnet0_vlan835 | grep mtu
vtnet0_vlan835: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
root@OPNsense:~ #

Code Select Expand

root@OPNsense:~ # ifconfig pppoe0 | grep mtu
pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1508
root@OPNsense:~ #
Proxmox side:

Code Select Expand

[root@pve-02]: ~ # ifconfig enp1s0 | grep mtu
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1512
[root@pve-02]: ~ #

Code Select Expand

[root@pve-02]: ~ # ifconfig vmbr2 | grep mtu
vmbr2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1512
[root@pve-02]: ~ #
Then i tried with the pings from within the firewall:

Code Select Expand

root@OPNsense:~ # ping -4 -c4 -D -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1472 data bytes
1480 bytes from 8.8.8.8: icmp_seq=0 ttl=117 time=5.410 ms
1480 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=5.976 ms
1480 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=4.952 ms
1480 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=5.074 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.952/5.353/5.976/0.397 ms
root@OPNsense:~ #

Code Select Expand

root@OPNsense:~ # ping -4 -c4 -s 1480 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1480 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # My understanding is that the above ping should result differently according to the first post?

Code Select Expand

root@OPNsense:~ # ping -4 -c4 -D -s 1480 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1480 data bytes
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long
ping: sendto: Message too long

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ #

Main confusion comes from:

or the worst-case scenario with both PPPoE and VLAN involved, you would theoretically need an MTU of 1512 on the ethernet port, 1508 on the VLAN created on it and then 1500 for PPPoE. This is the third case in the appended image.

which makes me thinking that both enp1s0 and vmbr1 shall be set on 1512. This would reflect on vtnet0 inside the vm with the same value.
Then I can config via /etc/rc.conf.local (didn't found any other way from the UI) the MTU for the vtnet0_vlan835 to 1508 and finally to configure 1500 in pppoe0. So far, clear.
But then:

1. The OpnSense settings here are somewhat "wrong". If you have a WAN over PPPoE over VLAN, you "should" have to set WAN MTU = 1500, pppoe0 = 1508, ethernet port = 1512

which, to me, is not clear. "WAN MTU" is intended to be the vlan? becase if it's vtnet0 it's in contrast to what said before...
Lastly

PPPOEVLAN MTU: 1508 (if needed in your setup).

again, before, was suggested to set to 1500.

Sorry for the confusion, i'm quite a newbie on the field and trying to understand better but need some guidances :).
What I have pasted is my actual configuration, searching for suggestions as my ISP doesn't publish MTU guidances. My goal is to optimize my network to use the best combination of MTUs across the "different interfaces"..
Please do let me know if any other clarification is required.

[network device]

Hi, i read this entire topic but still have some doubts about how shall I configure my MTUs across the entire stack.
My setup is:
Proxmox with enp1s0 as WAN phisical port.
From here, a VirtIO bridge is created (vmbr2). This is passed to the OPNSense vm as WAN.
Inside OPNSense, i see this as vtnet0. From here, I need to create a VLAN 835 to comply with my ISP requirements (VLAN Ethernet 802.1q).
Finally, a PPPoE connection (RFC 2516) is created.

So, long story short, can anyone clarify which values must be set for
Proxmox side: enp1s0, vmbr2
OPNSense side: vtnet0, vtnet0_vlan835, pppoe0

Atm i have:

network device	mtu size
enp1s0	1512
vmbr2	1512
vtnet0	1512
vtnet0_vlan835	1508
pppoe0	1500

and have specified 1452 as MSS field under Interfaces > WAN.
Ping:

Code Select Expand

root@OPNsense:~ # ping -c4 -s 1464 -D 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 1464 data bytes
1472 bytes from 8.8.8.8: icmp_seq=0 ttl=117 time=3.625 ms
1472 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=4.102 ms
1472 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=4.334 ms
1472 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=3.938 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.625/4.000/4.334/0.258 ms
root@OPNsense:~ #

I am kinda limited by the host cpu. Being a N5105 i only have 4 cores available, hence, i've given firewall vm 2 cores.
However, I did bump the queues to 4, but it's actually the same. Cpu spikes near 100% and same speed...

OPNSense:

Code Select Expand

-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.2.204, port 54384
[  5] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54390
[  8] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54398
[ 10] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54412
[ 12] local 192.168.2.1 port 5201 connected to 192.168.2.204 port 54428
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  4.12 MBytes  34.5 Mbits/sec                 
[  8]   0.00-1.00   sec  3.88 MBytes  32.4 Mbits/sec                 
[ 10]   0.00-1.00   sec  36.0 MBytes   301 Mbits/sec                 
[ 12]   0.00-1.00   sec  3.75 MBytes  31.4 Mbits/sec                 
[SUM]   0.00-1.00   sec  47.8 MBytes   400 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.01   sec  10.8 MBytes  89.2 Mbits/sec                 
[  8]   1.00-2.01   sec  9.88 MBytes  81.9 Mbits/sec                 
[ 10]   1.00-2.01   sec  3.25 MBytes  27.0 Mbits/sec                 
[ 12]   1.00-2.01   sec  10.1 MBytes  84.0 Mbits/sec                 
[SUM]   1.00-2.01   sec  34.0 MBytes   282 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.01-3.01   sec  13.5 MBytes   113 Mbits/sec                 
[  8]   2.01-3.01   sec  3.38 MBytes  28.3 Mbits/sec                 
[ 10]   2.01-3.01   sec  25.9 MBytes   217 Mbits/sec                 
[ 12]   2.01-3.01   sec  2.75 MBytes  23.1 Mbits/sec                 
[SUM]   2.01-3.01   sec  45.5 MBytes   382 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.01-4.01   sec  15.8 MBytes   132 Mbits/sec                 
[  8]   3.01-4.01   sec  14.2 MBytes   120 Mbits/sec                 
[ 10]   3.01-4.01   sec  24.1 MBytes   202 Mbits/sec                 
[ 12]   3.01-4.01   sec  4.00 MBytes  33.6 Mbits/sec                 
[SUM]   3.01-4.01   sec  58.1 MBytes   488 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.01-5.01   sec  10.6 MBytes  89.1 Mbits/sec                 
[  8]   4.01-5.01   sec  22.2 MBytes   187 Mbits/sec                 
[ 10]   4.01-5.01   sec   896 KBytes  7.34 Mbits/sec                 
[ 12]   4.01-5.01   sec  3.25 MBytes  27.3 Mbits/sec                 
[SUM]   4.01-5.01   sec  37.0 MBytes   310 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.01-6.20   sec  1.75 MBytes  12.3 Mbits/sec                 
[  8]   5.01-6.26   sec  31.4 MBytes   211 Mbits/sec                 
[ 10]   5.01-6.26   sec   384 KBytes  2.53 Mbits/sec                 
[ 12]   5.01-6.26   sec  9.38 MBytes  63.2 Mbits/sec                 
[SUM]   5.01-6.20   sec  42.9 MBytes   302 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.20-7.01   sec  6.00 MBytes  62.6 Mbits/sec                 
[  8]   6.26-7.01   sec  21.4 MBytes   239 Mbits/sec                 
[ 10]   6.26-7.01   sec  0.00 Bytes  0.00 bits/sec                 
[ 12]   6.26-7.01   sec  2.50 MBytes  28.0 Mbits/sec                 
[SUM]   6.20-7.01   sec  29.9 MBytes   312 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.01-8.05   sec  12.6 MBytes   101 Mbits/sec                 
[  8]   7.01-8.05   sec  10.2 MBytes  82.1 Mbits/sec                 
[ 10]   7.01-8.05   sec  25.5 MBytes   204 Mbits/sec                 
[ 12]   7.01-8.06   sec  9.00 MBytes  72.0 Mbits/sec                 
[SUM]   7.01-8.05   sec  57.4 MBytes   460 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.05-9.01   sec  13.0 MBytes   114 Mbits/sec                 
[  8]   8.05-9.01   sec  6.38 MBytes  55.8 Mbits/sec                 
[ 10]   8.05-9.01   sec  1.00 MBytes  8.75 Mbits/sec                 
[ 12]   8.06-9.01   sec  12.2 MBytes   107 Mbits/sec                 
[SUM]   8.05-9.01   sec  32.6 MBytes   286 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.01-10.01  sec  21.1 MBytes   177 Mbits/sec                 
[  8]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec                 
[ 10]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec                 
[ 12]   9.01-10.01  sec  16.5 MBytes   138 Mbits/sec                 
[SUM]   9.01-10.01  sec  37.6 MBytes   316 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.01-10.02  sec   128 KBytes   136 Mbits/sec                 
[  8]  10.01-10.02  sec  0.00 Bytes  0.00 bits/sec                 
[ 10]  10.01-10.02  sec  0.00 Bytes  0.00 bits/sec                 
[ 12]  10.01-10.02  sec   128 KBytes   134 Mbits/sec                 
[SUM]  10.01-10.02  sec   256 KBytes   272 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec   109 MBytes  91.6 Mbits/sec                  receiver
[  8]   0.00-10.02  sec   123 MBytes   103 Mbits/sec                  receiver
[ 10]   0.00-10.02  sec   117 MBytes  97.9 Mbits/sec                  receiver
[ 12]   0.00-10.02  sec  73.6 MBytes  61.6 Mbits/sec                  receiver
[SUM]   0.00-10.02  sec   423 MBytes   354 Mbits/sec                  receiver

host proxmox:

Code Select Expand

[root@pve-02]: ~ $ iperf3 -c 192.168.2.1 -P4
Connecting to host 192.168.2.1, port 5201
[  5] local 192.168.2.204 port 54390 connected to 192.168.2.1 port 5201
[  7] local 192.168.2.204 port 54398 connected to 192.168.2.1 port 5201
[  9] local 192.168.2.204 port 54412 connected to 192.168.2.1 port 5201
[ 11] local 192.168.2.204 port 54428 connected to 192.168.2.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.23 MBytes  43.8 Mbits/sec    1   1.41 KBytes       
[  7]   0.00-1.00   sec  5.23 MBytes  43.8 Mbits/sec    1   1.41 KBytes       
[  9]   0.00-1.00   sec  38.9 MBytes   326 Mbits/sec  197    782 KBytes       
[ 11]   0.00-1.00   sec  4.95 MBytes  41.4 Mbits/sec    1   1.41 KBytes       
[SUM]   0.00-1.00   sec  54.3 MBytes   455 Mbits/sec  200             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  12.2 MBytes   103 Mbits/sec    1    400 KBytes       
[  7]   1.00-2.00   sec  10.9 MBytes  91.9 Mbits/sec    7    376 KBytes       
[  9]   1.00-2.00   sec  3.75 MBytes  31.5 Mbits/sec    1   1.41 KBytes       
[ 11]   1.00-2.00   sec  11.4 MBytes  95.5 Mbits/sec   17    373 KBytes       
[SUM]   1.00-2.00   sec  38.3 MBytes   322 Mbits/sec   26             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  13.3 MBytes   111 Mbits/sec   11    529 KBytes       
[  7]   2.00-3.00   sec  4.23 MBytes  35.4 Mbits/sec   11    379 KBytes       
[  9]   2.00-3.00   sec  27.5 MBytes   231 Mbits/sec  157    803 KBytes       
[ 11]   2.00-3.00   sec  3.11 MBytes  26.1 Mbits/sec   10    366 KBytes       
[SUM]   2.00-3.00   sec  48.1 MBytes   404 Mbits/sec  189             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  16.2 MBytes   136 Mbits/sec    5    293 KBytes       
[  7]   3.00-4.00   sec  13.4 MBytes   113 Mbits/sec    1    427 KBytes       
[  9]   3.00-4.00   sec  22.5 MBytes   189 Mbits/sec  232    416 KBytes       
[ 11]   3.00-4.00   sec  4.35 MBytes  36.5 Mbits/sec    7    192 KBytes       
[SUM]   3.00-4.00   sec  56.5 MBytes   474 Mbits/sec  245             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  12.5 MBytes   105 Mbits/sec    0    324 KBytes       
[  7]   4.00-5.00   sec  22.4 MBytes   188 Mbits/sec    0    478 KBytes       
[  9]   4.00-5.00   sec  2.50 MBytes  21.0 Mbits/sec    0    419 KBytes       
[ 11]   4.00-5.00   sec  3.17 MBytes  26.6 Mbits/sec    1    204 KBytes       
[SUM]   4.00-5.00   sec  40.5 MBytes   340 Mbits/sec    1             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  2.50 MBytes  21.0 Mbits/sec    0    328 KBytes       
[  7]   5.00-6.00   sec  25.5 MBytes   214 Mbits/sec    0    510 KBytes       
[  9]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec    0    416 KBytes       
[ 11]   5.00-6.00   sec  6.40 MBytes  53.7 Mbits/sec    0    225 KBytes       
[SUM]   5.00-6.00   sec  35.6 MBytes   299 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  3.75 MBytes  31.5 Mbits/sec    0    338 KBytes       
[  7]   6.00-7.00   sec  29.0 MBytes   243 Mbits/sec    0    530 KBytes       
[  9]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0    416 KBytes       
[ 11]   6.00-7.00   sec  5.34 MBytes  44.8 Mbits/sec    0    242 KBytes       
[SUM]   6.00-7.00   sec  38.1 MBytes   319 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  12.5 MBytes   105 Mbits/sec    1    365 KBytes       
[  7]   7.00-8.00   sec  10.1 MBytes  85.0 Mbits/sec    0    537 KBytes       
[  9]   7.00-8.00   sec  22.5 MBytes   189 Mbits/sec    0    460 KBytes       
[ 11]   7.00-8.00   sec  8.76 MBytes  73.5 Mbits/sec    0    267 KBytes       
[SUM]   7.00-8.00   sec  53.9 MBytes   452 Mbits/sec    1             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  12.5 MBytes   105 Mbits/sec    0    390 KBytes       
[  7]   8.00-9.00   sec  6.71 MBytes  56.3 Mbits/sec    0    547 KBytes       
[  9]   8.00-9.00   sec  3.75 MBytes  31.5 Mbits/sec    0    464 KBytes       
[ 11]   8.00-9.00   sec  11.7 MBytes  98.0 Mbits/sec    0    301 KBytes       
[SUM]   8.00-9.00   sec  34.6 MBytes   291 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  21.2 MBytes   178 Mbits/sec    0    431 KBytes       
[  7]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0    547 KBytes       
[  9]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0    464 KBytes       
[ 11]   9.00-10.00  sec  16.9 MBytes   142 Mbits/sec    0    341 KBytes       
[SUM]   9.00-10.00  sec  38.2 MBytes   320 Mbits/sec    0             
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   112 MBytes  94.0 Mbits/sec   19             sender
[  5]   0.00-10.02  sec   109 MBytes  91.6 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   127 MBytes   107 Mbits/sec   20             sender
[  7]   0.00-10.02  sec   123 MBytes   103 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   123 MBytes   103 Mbits/sec  587             sender
[  9]   0.00-10.02  sec   117 MBytes  97.9 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  76.0 MBytes  63.8 Mbits/sec   36             sender
[ 11]   0.00-10.02  sec  73.6 MBytes  61.6 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   438 MBytes   368 Mbits/sec  662             sender
[SUM]   0.00-10.02  sec   423 MBytes   354 Mbits/sec                  receiver

iperf Done.


Did you enable multiqueue on the VM NIC interfaces in Proxmox? The throughput you are getting suggests, you did not.

If you refer to this settings, I have: https://imgur.com/a/K3upFP1

Hi everyone, thanks for this howto. I am having some troubles on the following hardware:
- Intel N5105
- Intel I226-V rev. 4

I have installed proxmox (latest version) and virtualised OPNSense (last version as well).
I am using VirtIO method, so both WAN and LAN are a linux bridge, not passed trough.
My ISP gives me 2.5/1G down/up plan, but this is the result of my speedtest:

From pve host itself:

Code Select Expand

Retrieving speedtest.net configuration...
Testing from Telecom Italia (79.17.151.204)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Cloudfire Srl (Milano) [0.90 km]: 19.152 ms
Testing download speed................................................................................
Download: 281.67 Mbit/s
Testing upload speed...
Upload: 205.35 Mbit/s
From OPNSense:

Code Select Expand

root@opnsense:~ # speedtest

   Speedtest by Ookla

      Server: Sky Wifi - Milano (id: 50954)
         ISP: TIM
Idle Latency:     5.46 ms   (jitter: 0.17ms, low: 5.31ms, high: 5.55ms)
    Download:   539.83 Mbps (data used: 915.9 MB)                                                   
                 46.37 ms   (jitter: 73.23ms, low: 3.21ms, high: 433.53ms)
      Upload:   623.16 Mbps (data used: 1.0 GB)                                                   
                 48.06 ms   (jitter: 35.79ms, low: 3.95ms, high: 465.33ms)
 Packet Loss:     0.0%
Tried as well an iperf3 test between a LXC container on the same LAN bridge and OPNSense:

Code Select Expand

Accepted connection from 192.168.2.8, port 40784
[  5] local 192.168.2.1 port 5201 connected to 192.168.2.8 port 40786
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  65.8 MBytes   549 Mbits/sec                 
[  5]   1.00-2.00   sec  65.6 MBytes   552 Mbits/sec                 
[  5]   2.00-3.00   sec  75.4 MBytes   632 Mbits/sec                 
[  5]   3.00-4.01   sec  68.5 MBytes   568 Mbits/sec                 
[  5]   4.01-5.00   sec  72.8 MBytes   618 Mbits/sec                 
[  5]   5.00-6.01   sec  68.6 MBytes   571 Mbits/sec                 
[  5]   6.01-7.00   sec  67.1 MBytes   567 Mbits/sec                 
[  5]   7.00-8.00   sec  76.1 MBytes   639 Mbits/sec                 
[  5]   8.00-9.00   sec  71.4 MBytes   599 Mbits/sec                 
[  5]   9.00-10.00  sec  77.0 MBytes   647 Mbits/sec                 
[  5]  10.00-10.01  sec  1.12 MBytes   759 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   709 MBytes   594 Mbits/sec                  receiver
Does anyone know what could be the issue?

Tunables: https://imgur.com/a/tdpPeWr
Offloading: https://imgur.com/2fMhQQW

Thanks in advance.