Messages

thx
but this didn't help
all the latency is under 1ms and just sometimes during the tst it spikes up to cca 500ms an gets therefore packet lose

first thing i have tried

there was a limit and the maximum speed for bufferbloat is 4Gbps (i have networks 10-25Gbps...)
and it doesn't remove the problem with packet loss completely (sometimes is the latency so high -> packet loss)

there is no problem with VyOS, Debian, Redhat, OpenWRT etc.

one month spend in trying different "tunables"
and other configuration options...

The problem itself is on using any Proxmox NIC emulator and PPPoE + VLAN

If a network card is used directly the problem disappears.

Hi,
I'm trying to replace some (cca. 10+) OpenWRT firewalls with OPNsense.

Internet connection:
optical network 10/10 Gbps
Hardware:
Proxmox virtualization on AMD EPYC 9745 256 threads and lot of RAM
(doesn't matter, if i use Proxmox on a completely different HW)
Network cards:
Broadcom BCM57414 through VirtIO or PCIe passthrough
(doesn't matter, if virtualized or directly using HW)

They all report high jitter, latency and packet loss on download,
if i use Ookla speedtest via the CLI directly on the firewall itself
(there should be no NAT involved if used this way).

https://www.speedtest.net/apps/cli

Code Select Expand

pkg add --force "https://install.speedtest.net/app/cli/ookla-speedtest-1.2.0-freebsd13-x86_64.pkg"

Code Select Expand


   Speedtest by Ookla

      Server: 10G-CZNET.CZ - Prague (id: 21429)
         ISP: CESNET z.s.p.o.
Idle Latency:     0.79 ms   (jitter: 0.03ms, low: 0.78ms, high: 0.84ms)
    Download:  7473.87 Mbps (data used: 9.4 GB)                                                   
                 56.42 ms   (jitter: 54.22ms, low: 0.87ms, high: 495.29ms)
      Upload:  5101.07 Mbps (data used: 8.8 GB)                                                   
                  1.15 ms   (jitter: 0.17ms, low: 0.89ms, high: 3.10ms)
Packet Loss:    29.6%
  Result URL: https://www.speedtest.net/result/c/ce4f06d7-db26-42f3-b207-6bb2a5abe15a


I've tried almost all the advice available here.
(do not check gateway etc...)
The OPNsense firewall is set on default values.
Any advice?

PS the same environment with OpenWRT:

Code Select Expand


   Speedtest by Ookla

      Server: ISP Alliance a.s. - Prague (id: 4162)
         ISP: CESNET z.s.p.o.
Idle Latency:     0.77 ms   (jitter: 0.03ms, low: 0.73ms, high: 0.83ms)
    Download:  9043.70 Mbps (data used: 9.5 GB)                                                   
                  4.93 ms   (jitter: 0.24ms, low: 0.80ms, high: 6.77ms)
      Upload:  6626.55 Mbps (data used: 7.1 GB)                                                   
                  1.47 ms   (jitter: 0.48ms, low: 0.86ms, high: 3.43ms)
Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/92f73887-737d-411d-95e4-5ae75ac6cce

(the different servers Ookla speedtest connects to in these examples do not matter)

OK, problem solved using PCIe passthrough.

It looks like a problem in the FreeBSD network VirtIO driver... 

I'm using PCIe passthrough a lot and I can give it a try.

The reason to use virtualized network cards was,
that I didn't have to plug the LAN from the firewall into the switch
and afterwards plug the Proxmox server itself into the switch... 

I have tried to delete the vlan01 and to set the PPPoE Parent interface to vtnet1
and at the same time set the Virtual machine net1 VLAN tag to 848,
which is correctly working, but has the same upload speed issue.

I didn't have this problem with upload speed using OpenWRT.

Hi,

I'm a long OpenWRT user (routing a backbone network 5x 25Gbps).
After an OpenWRT upgrade at home, which caused a complete wipe,
I have decided to give OPNsense a try.

My home network parameters (GPON Cetin, Czech Republic):
PPPoE - user name and password
VLAN - 848
MTU - 1492
download - 2000Mbps
upload - 1000Mbps

I have force-installed the FreeBSD 13 package of Ookla Speedtest
and i get 2020Mbps for download and 150-350Mbps for upload
on the OPNsense firewall itself (means no NAT involved etc.).
I believe that this is caused by a wrong MTU somewhere,
but I'm unable to fix it (and I have tried many options).

Configuration of Proxmox:
Linux bridge vmbr0 - IP:192.168.1.2 GW:192.168.1.1 Bridge ports:enp2s0f0
Linux bridge vmbr1 - Bridge ports:enp2s0f1
Virtual machine net0 - Model: VirtIO Multiqueue: 8 Bridge: vmbr0
Virtual machine net1 - Model: VirtIO Multiqueue: 8 Bridge: vmbr1

Configuration of OPNsense:
Interfaces, Other Types, VLAN add vlan01 - Vlan tag: 848 Parent: vtnet1
Interfaces, Point-to-Point, Devices add pppoe0 - Link Type: PPPoE Parent interface: vlan01
(+ Username and Password)
Interfaces, Asignments interface [WAN] change Device to pppoe0

I have nothing else changed, all is left at the default values.
On the [WAN] Interface is shown - Calculated PPP MTU: 1492
The maximum ping size I can send from the firewall is ping -D -s 1464 1.1.1.1
which seems OK to me.

Can anybody help me please?