Messages

I put the unit into production a week ago with the following settings.

Code Select Expand

Checksum offloading : enabled (unchecked)
TSO : disabled (checked)
LRO : disabled (checked)
VLAN hardware filtering : enabled
Hardware acceleration : QAT
PowerD : enabled (Hiadaptive)
hw.ibrs_disable : 1
hw.intr_storm_threshold : 10000
hw.ix.enable_aim : 1
hw.ix.flow_control : 3
hw.ix.unsupported_sfp : 1 (but my 4 DAC worked without that as well)
net.inet.ip.intr_queue_maxlen : 4096
kern.ipc.nmbclusters : 2000000
kern.ipc.nmbjumbop : 524288
machdep.hyperthreading_intr_allowed : 1
net.inet.rss.bits : 3
net.inet.rss.enabled : 1
net.isr.bindthreads : 1
net.isr.dispatch : hybrid
net.isr.maxthreads : -1
vm.pmap.pti : 0

It's been running good so far.

What's your experience with the SFP+ cages ?!?!

I have all 4 SFP+ ports up and running without issues so far.

I ordered 4 DAC from 10Gtek on Amazon:

• 2 Intel-coded (because I feared I may run into issues with my usual DAC)
• 2 Cisco-coded (the same i usually have for all my Ubiquiti stuff).

All 4 SFP+ ports are connected via those DAC to 2 Unifi US-16-XG switches.
No recognition or bandwidth issues.

[net.isr.bindthreads=1net.isr.dispatch=hybridnet.isr.bindthreads=1net.isr.maxthreads=-1net.inet.rss.enabled=1net.inet.rss.bits=3]

Test 4:
Checksum offloading : enabled
LRO/TSO : disabled  (=default)
VLAN offloading : enabled
vm.pmap.pti= 1 -> 0
hw.ibrs_disable= 0 -> 1
net.isr.bindthreads=1
net.isr.dispatch=hybrid
net.isr.bindthreads=1
net.isr.maxthreads=-1
net.inet.rss.enabled=1
net.inet.rss.bits=3


--> Load <= 10.0 (~20% system ~50% interrupt ~25% idle) & ~9 Gb/s

Throughput stable and the system definitly seems more usable  :)

Test 3 :
Checksum offloading : enabled
LRO/TSO : disabled  (=default)
VLAN offloading : enabled
vm.pmap.pti= 1 -> 0
hw.ibrs_disable= 0 -> 1
--> Load ~ 13.0 (~95% system) & ~9 Gb/s

No change compared to test #2 where hardware offloading seems to have helped quite a bit.

Hello,

I decided to go back physical from VMware VM to a Qotom Q20331G9.
I reloaded/validated my configuration bit by bit on the new box and I'm now in the process of benchmarking performance before putting it into production.

My concern is about CPU usage when trying to push to 10 Gb/s using iperf3.

Source VM / Vlan A : iperf3 -c xxx.xxx.xxx.xxx -P 40 -t 600
Destination VM  / Vlan B : iperf3 -s
Opnsense 24.7.9_1

Through my Opnsense VM I can reach 8-9 Gbit/s and the CPU won't move up (load average stays <= 1.0).
It's running 4 vCPU out of a Ryzen 5 3600, 4 GB of RAM, 2 VMXNET3 vNIC . ESXi 8 host iwth intel x520 10 Gb/s uplinks.

Same test but through the Qotom box, I reach the same speed (which is good news) however the CPU easily reaches a load average over 30.0  making webUI and SSH hardly usable  :-[

Test 1 :
Checksum offloading : disabled (=default)
LRO/TSO : disabled  (=default)
VLAN offloading : disabled (=default)
--> Load >= 30.0 (~95% system) & ~9 Gb/s

Test 2 :
Checksum offloading : enabled
LRO/TSO : disabled  (=default)
VLAN offloading : enabled
--> Load ~ 13.0 (~95% system) & ~9 Gb/s

I'm not running any IDS/IPS (yet), and just about a dozen FW rules on the test interfaces.
For the test the flow is going in and out of the same NIC (trunk port with both tagged VLANs)
I tried to disable stuff like netflow local collector but no improvement.

Any game changing tunables i should look into?
Or is the C3758R just too slow to sustain 10Gb/s without choking?

Any Qotom Q20331G9 owners who may share their experience with this unit ?

Thank you.

Hello,

I have the same issue since 24.1.6 (I just upgraded to 24.1.7).
Everything works for a while and I get 100% CPU usage from one of the dhcp_relay processes which blocks the whole DHCP service.

My setup :

Edge sites (x2) :
- ESXi 8
- OpenSense VM as main gateway
- OpenSense VM as "helper" with DHCP relay for multiple VLANs
- Multiple VLANs
- Unifi switches
- Windows Server VM with DHCP server (as standby)

Central site :
- ESXi 8
- OpenSense VM as main gateway
- OpenSense VM as "helper" with DHCP relay for multiple VLANs
- Multiple VLANs
- Unifi switches
- Windows Server VM with DHCP server (as standby)

Site-to-site Wireguard VPN

No DHCP guarding whatsoever on Unifi side.

Opnsense VMs (router and helper) all have an interface in each VLAN.
Target DHCP servers on edge sites are both the local and the central Windows DHCP server.

This setup worked flawlessly for months (if not years) before 24.1.6.

My Windows DHCP servers also serve the very VLAN where my Opnsense VM and my Windows servers have their management interface.
I tried deactivating the DHCP relay for this management VLAN as per https://forum.opnsense.org/index.php?topic=40126.0 thinking it would solve it (I could live with that workaround even if not ideal and degraded compared to before 24.1.6).

But the issue still occurs now and then, i have to restart dhcp relay for some other VLANs to have CPU come down to normal.
My latest workaround is to make a daily reboot of the helper VM. Definitly not a bulletproof approach.

1°/ Would you know if the developer team is aware of the situation and working on it ?
2°/ Would you know where i could find useful logs for the new dhcp relay service ?

I've been very happy with the tremendous work around OpnSense.
It's the first time in years that I encounter such a blocking issue after an upgrade.

Thank you in advance.

PS : I submitted a bug report : https://github.com/opnsense/core/issues/7471

Hello,

I made the switch to OPNsense a few weeks ago and setup NordVPN as well.

Regarding DNS, i'm using OpenDNS servers instead of my ISP's.
When it comes to Unbound, I left "all" as the outgoing interface.
So basically I guess DNS traffic doesn't go through the tunnel, but everything else (based on firewall rules) does.

If I remember correctly NordVPN offers DNS service as well, but maybe not inside the tunnel.
Also maybe a firewall rule such as "This firewall -> UDP/53 -> GW NordVPN" could work (outside unbound configuration). Just a guess.

Hello,

First of all, I've been a pfSense user for over 10 years now and I must say I'm very pleased discovering and  making the switch to OPNsense. :)

Now to the problem I'm facing.

I'm running OPNsense 20.1.2 inside a VMware VM and using VLAN to access two WAN routers (xDSL + 4G).
When I create a single OpenVPN tunnel to NordVPN using either WAN access, it works.

However I'd like to have some failover to my VPN access.
Unlike pfSense, OPNsense doesn't offer to use a gateway group as the OpenvPN interface.

As a workaround I tried to  :
- create two NordVPN tunnels, one using WAN1 and one using WAN2
- create 2 interfaces
- create a gateway group

On the paper, it's supposed to work :
- both tunnels are up
- both OpenVPN tunnels are not overlapping (one is 10.8.x.0/24, the other is 10.7.x.0/24)
- I use firewall rules to route the traffic through the gateway group of my choosing
- routing table looks fine, eg:

Proto   Destination    Gateway    Flags    Use   MTU       Netif    Netif (name)
ipv4      10.8.2.0/24   10.8.2.1   UGS      0   1500      ovpnc1   NORDVPN_1       
ipv4      10.8.2.1      link#20   UH      0   1500      ovpnc1   NORDVPN_1       
ipv4      10.8.2.22      link#20   UHS      0   16384  lo0   
ipv4      10.7.1.0/24   10.7.1.1   UGS      0   1500      ovpnc2   NORDVPN_2       
ipv4      10.7.1.1      link#21   UH      0   1500      ovpnc2   NORDVPN_2       
ipv4      10.7.1.10      link#21   UHS      0   16384  lo0   


However routing gets messed up, and both tunnels are unreachable (traceroute KO).
Looks like when the second tunnel goes up it conflicts/breaks the first one.

Any idea what I could be missing ?