Hello all (and happy new year)
i'm asking expert after lots of time trying and searching.
My lab is
Fiber Freebox Delta bridge mode (Free / France) 10Gb/s Download / 700 MB Upload
Storage
Truenas Scale = 1x10GB SFP+
A cluster of 3 proxmox
Proxmox 1 = 2x10Gb SFP+
Proxmox 2 = 1x1Gb ethernet
Proxmox 3 = 1x2.5Gb Ethernet
On proxmox 1:
linux brige (vmbr0) link with nic0 LAN
linux brige (vmbr1) link with nic1 WAN
First VM (opnsense) with 2 NIC
NIC0 LAN (on vmbr0)
NIC1 WAN (on vmbr1)
Second VM (alpine linux) with one NIC
NIC0 LAN (on vmbr0)
On proxmox 2:
linux brige (vmbr0) link with nic0 LAN
First VM (alpine linux) with one NIC
NIC0 LAN (on vmbr0)
Truenas Scale
NIC0 = LAN
VM Opnsense = Gateway (firewall/NAT)
Everything work fine (even IPV6 with Freebox TV player IPV6) Except one "little" thing, NAT outside proxmox 1 (so, all my lan)
LAN
Trafic performance (done with iperf3)
proxmox 1 -> truenas = 9Gb/s
truenas -> proxmox 1 = 9Gb/s
proxmox 2 -> truenas = 2.5Gb/s
truenas -> proxmox 2 = 2.5Gb/s
vm(proxmox 1) -> truenas = 9Gb/s
truenas -> vm (proxmox 1) = 9Gb/s
vm(proxmox 2) -> truenas = 2.5Gb/s
truenas -> vm (proxmox 2) = 2.5Gb/s
INTERNET (NAT Thrue opnsense)
Trafic performance (done with speedtest)
Proxmox 1 -> Internet = 1.5 Gb/s : OK for me
Internet -> Proxmox 1 = 700MB/s : OK for me
VM Opensense -> Internet = 1.5 Gb/s : OK for me
Internet -> VM Opensense = 700MB/s : OK for me
vm(proxmox 1) -> Internet = 1.5 Gb/s : OK for me
Internet -> vm(proxmox 1) = 700MB/s : OK for me
Anything else not directly INSIDE proxmox 1 (where opnsense is hosted)
=============== BBR
Server: LaFibre.info BBR-IPv4 - Palaiseau (id: 45446)
Download: 1427.95 Mbps (data used: 957.4 MB)
Upload: 590.06 Mbps (data used: 1.1 GB)
Server: BOUYGUES TELECOM BBR - Meudon (id: 47318)
Download: 1762.61 Mbps (data used: 2.5 GB)
Upload: 614.28 Mbps (data used: 641.0 MB)
Server: LaFibre.info BBR - Massy (id: 2231)
Download: 1513.09 Mbps (data used: 1.4 GB)
Upload: 618.12 Mbps (data used: 586.8 MB)
=============== CUBIC
Server: LaFibre.info CUBIC - Orly (id: 45454)
Download: 34.60 Mbps (data used: 55.2 MB)
Upload: 575.48 Mbps (data used: 806.2 MB)
Server: BOUYGUES TELECOM CUBIC - Vélizy (id: 47424)
Download: 32.60 Mbps (data used: 49.4 MB)
Upload: 597.55 Mbps (data used: 856.8 MB)
Server: Hivane NetWork Cubic - Ivry-sur-Seine (id: 49781)
Download: 30.76 Mbps (data used: 46.6 MB)
Upload: 120.26 Mbps (data used: 190.6 MB)
So, any VM, computer or whatever (not directrly inside proxmox 1) downloading internet things thrue opnsense got worst download speed ever.
Speedtest is the best i can do. but when i try do download alpine iso (exemple)
im 700Mb/s from proxmox 1, (or any vm hosted by proxmox one)
im 1Mbs/s from anything not hosted inside proxmox one itself.
If someone have any idea ???
Thx a lot
Disabled firewall on the Linux bridges in Proxmox? By default they are enabled
Hello
firewall in proxmox (on pve itself, bridge, nic) is disabled.
In fact, i have switched to untangle with lots of performance improvement.
But i will try to find why this problem with opnsense
Hi there,
What sort of 10G NIC's on Proxmox 1?
Can you just try to run Iperf3 from the the Opnsense VM?
Rgds
hello , the nic is
SFP+ 2 Ports LC - CHIPSET BROADCOM BCM57810 - 10GbE Ethernet Fiber Network Adapter
I will try iperf3 later