Issue with proxmox / opnsense / NAT / BRR / CUBIC

[amapi]

Hello all (and happy new year)

i'm asking expert after lots of time trying and searching.

My lab is

Fiber Freebox Delta bridge mode (Free / France) 10Gb/s Download / 700 MB Upload

Storage
  Truenas Scale = 1x10GB SFP+

A cluster of 3 proxmox
  Proxmox 1 = 2x10Gb SFP+
  Proxmox 2 = 1x1Gb ethernet
  Proxmox 3 = 1x2.5Gb Ethernet



On proxmox 1:
  linux brige (vmbr0) link with nic0 LAN
  linux brige (vmbr1) link with nic1 WAN

  First VM (opnsense) with 2 NIC
    NIC0 LAN (on vmbr0)
    NIC1 WAN (on vmbr1)

  Second VM (alpine linux) with one NIC
    NIC0 LAN (on vmbr0)

On proxmox 2:
  linux brige (vmbr0) link with nic0 LAN

  First VM (alpine linux) with one NIC
    NIC0 LAN (on vmbr0)

Truenas Scale
  NIC0 = LAN
 
 
VM Opnsense = Gateway (firewall/NAT)

Everything work fine (even IPV6 with Freebox TV player IPV6) Except one "little" thing, NAT outside proxmox 1 (so, all my lan)



LAN
Trafic performance (done with iperf3)

  proxmox 1 -> truenas = 9Gb/s
  truenas -> proxmox 1   = 9Gb/s
 
  proxmox 2 -> truenas = 2.5Gb/s
  truenas -> proxmox 2 = 2.5Gb/s
 
  vm(proxmox 1) -> truenas = 9Gb/s
  truenas -> vm (proxmox 1) = 9Gb/s
 
  vm(proxmox 2) -> truenas = 2.5Gb/s
  truenas -> vm (proxmox 2) = 2.5Gb/s 



INTERNET (NAT Thrue opnsense)
Trafic performance (done with speedtest)

  Proxmox 1 -> Internet = 1.5 Gb/s : OK for me
  Internet -> Proxmox 1 = 700MB/s : OK for me

  VM Opensense -> Internet = 1.5 Gb/s : OK for me
  Internet -> VM Opensense = 700MB/s : OK for me

  vm(proxmox 1) -> Internet = 1.5 Gb/s : OK for me
  Internet -> vm(proxmox 1) = 700MB/s : OK for me

Anything else not directly INSIDE proxmox 1 (where opnsense is hosted)

=============== BBR
      Server: LaFibre.info BBR-IPv4 - Palaiseau (id: 45446)
    Download:  1427.95 Mbps (data used: 957.4 MB)
      Upload:   590.06 Mbps (data used: 1.1 GB)


      Server: BOUYGUES TELECOM BBR - Meudon (id: 47318)
    Download:  1762.61 Mbps (data used: 2.5 GB)
      Upload:   614.28 Mbps (data used: 641.0 MB)


      Server: LaFibre.info BBR - Massy (id: 2231)
    Download:  1513.09 Mbps (data used: 1.4 GB)
      Upload:   618.12 Mbps (data used: 586.8 MB)




=============== CUBIC
      Server: LaFibre.info CUBIC - Orly (id: 45454)
    Download:    34.60 Mbps (data used: 55.2 MB)
      Upload:   575.48 Mbps (data used: 806.2 MB)


      Server: BOUYGUES TELECOM CUBIC - Vélizy (id: 47424)
    Download:    32.60 Mbps (data used: 49.4 MB)
      Upload:   597.55 Mbps (data used: 856.8 MB)


      Server: Hivane NetWork Cubic - Ivry-sur-Seine (id: 49781)
    Download:    30.76 Mbps (data used: 46.6 MB)
      Upload:   120.26 Mbps (data used: 190.6 MB)
    
    
So, any VM, computer or whatever (not directrly inside proxmox 1) downloading internet things thrue opnsense got worst download speed ever.

Speedtest is the best i can do. but when i try do download alpine iso (exemple)

im 700Mb/s from proxmox 1, (or any vm hosted by proxmox one)
im 1Mbs/s from anything not hosted inside proxmox one itself.


If someone have any idea

Thx a lot









 

[elcocoloco]

Disabled firewall on the Linux bridges in Proxmox? By default they are enabled

[amapi]

Hello

firewall in proxmox (on pve itself, bridge, nic) is disabled.

In fact, i have switched to untangle with lots of performance improvement.

But i will try to find why this problem with opnsense

[FraLem]

Hi there,
What sort of 10G NIC's on Proxmox 1?
Can you just try to run Iperf3 from the the Opnsense VM?
Rgds

[amapi]

hello , the nic is
SFP+ 2 Ports LC - CHIPSET BROADCOM BCM57810 - 10GbE Ethernet Fiber Network Adapter

I will try iperf3 later