Messages

if you set

Code Select Expand

dev.netmap.buf_num > 1000000 does that even work? i tried that and got a message like

Code Select Expand

081.363854 [1363] netmap_config_obj_allocator requested objtotal 2000000 out of range [4, 1000000] which seem to set the actual buf_num to the default, not even 1000000...

i am using ixl driver on X710-DA2 with native netmap driver.

Apparently, you need a business license or higher for multi core support. This is laughable. In this day and age, charging for multi core support, seriously? Modern CPU's have had multiple cores for years. What software company charges for multi core support? NONE, except for Zenarmor you need to revisit your strategy. Multi core support is nothing novel or innovative. All modern software applications support it.

what. the whole reason i want multi-core support is so i can downgrade my home hardware to lower TDP cpu... ugh

The multicore support is in developmentstage and we plan to ship it with version 2.1 in July.

thanks for the update!

I use Vagrant - need to urgently update my project for 25.1. OPNsense runs in a virtual machine, source tree(s) are on the host system. OPNsense mounts project directory via NFS.

For 24.7 see https://github.com/punktDe/vagrant-opnsense

My current large show stopper is that I need to configure a Debian VM on a powerful hypervisor for nested virtualisation and Vagrant so I can use VScode over SSH to that machine - because my Intel Mac is being replaced by Apple silicon.

oh yea, that is certainly doable with the proxmox setup i have, just have the code on a linux vm and nfs export and have opnsense nfs mount it. i'll give this a try. seems promising.

thanks!

curious as to hear how people are working with opnsense developement.. git clone / git branch / make mount / edit / git commit / git push / create pull request, is great and all. currently doing that on a Proxmox VM and its fairly simple.

but using vim/emacs on the terminal is so-so. vscode via remote SSH to freebsd doesn't really work without linux binary support, which the opnsense repo seems to be missing essential packages

Code Select Expand

devel/linux-rl9-libsigsegv
emulators/linux_base-rl9
i don't necessarily have to have vscode, but largely gotten use to it at this point. neovim seems like its possible with some effort to get it working on freebsd.

so i'd like to hear what other solutions people are using before i spend too much time in one direction.

thanks

wireguard app on mobile, at least on iOS, you can exclude WiFi names so it will on demand connect to wireguard always, except when one of the SSIDs is is in the exclusion list. so you don't need to be going through the wireguard interface at home.

Shouldn't the peer addresses be /32 rather than /24?

yea the /24 on the peer is likely clobbering the entire subnet

if you are just looking for the CLI tool, ookla doesn't have a FreeBSD 14 pkg, but the 13 pkg works fine, only you have to force install it.

Code Select Expand

pkg --force add "https://install.speedtest.net/app/cli/ookla-speedtest-1.2.0-freebsd13-x86_64.pkg"

i turned off the firewall normalization rule the Road Warrior Docs say to use and now i consistently get

Up:

Code Select Expand

[SUM]   0.00-5.00   sec  5.23 GBytes  8.98 Gbits/sec   34             sender
[SUM]   0.00-5.00   sec  5.23 GBytes  8.98 Gbits/sec                  receiver
Down:

Code Select Expand

[SUM]   0.00-5.00   sec  4.62 GBytes  7.93 Gbits/sec  5610             sender
[SUM]   0.00-5.00   sec  4.62 GBytes  7.93 Gbits/sec                  receiver

[The Intel X E-2278G with X710-DA2]

O-M-G ** SOLVED ** THANK YOU eric!!

it was the reply-to... changed it to

and immediately all OPN installs worked in both directions... from 23.1 to 25.1.1 on all my hardware setups...

The Intel X E-2278G with X710-DA2

Up: 8.60 Gbits/sec
Down: 7.27 Gbits/sec

Code Select Expand

iperf3 --client 192.168.1.20 --omit 1 --time 5 --parallel 16 --format g
...
[SUM]   0.00-5.00   sec  5.00 GBytes  8.59 Gbits/sec  3327             sender
[SUM]   0.00-5.00   sec  5.00 GBytes  8.60 Gbits/sec                  receiver

iperf3 --client 192.168.1.20 --omit 1 --time 5 --parallel 16 --format g --reverse
Reverse mode, remote host 192.168.1.20 is sending
...
[SUM]   0.00-5.00   sec  4.23 GBytes  7.27 Gbits/sec  7211             sender
[SUM]   0.00-5.00   sec  4.23 GBytes  7.27 Gbits/sec                  receiver


yea i followed the documentation, verified mtu, with or without the firewall normalization rule, tried messing with MTUs on both sides. there is no difference.

i also verified the out-of-box MTUs are the same as pfSense and OpenWRT according to ifconfig

outdated documentation, go out to the user list and one of the icon/actions is to create a cert for user

yea i have run with an actual gateway as well, adding the WAN to my normal network, i do this initially so that i can install any needed packages and say update 25.1 to 25.1.1.

in either case, there is no difference in behavior. i tried eliminating the external network after a few days to try and isolate it more, but the results are exactly the same either way, unfortunately

you can check with

Code Select Expand

pciconf -lcv whether ASPM is even enabled for NICs. i found that in a few aliexpress type machines i have, ASPM is just straight up disabled and there is no option to enable it.

linux i226 driver works fine with ASPM. tested OpenWRT on Odroid H4 ultra and no speed issues. load OPNsense on and immediate speed issues. disabled ASPM in BIOS, thankfully Odroid is a good brand with regular BIOS updates and support, and it immediately fixed all my speed test issues i was seeing.

[wireguard results]

my basic setup looks like this:



3 computers are

• completely isolated, directly connected
• in my 10g setup, SFP+ OM3 fiber or UniFi DAC cables, results are the same
• fresh vanilla installs for ubuntu 24.04, pfSense 2.7.2 CE, OPNsense 24.10.2, 25.1.1
• router software is changed by just swapping the SSD, all other hardware stays exactly the same

wireguard is setups using the official pfSense documentation and OPNsense road warrior documentation.

iperf3 commands on the client:

Code Select Expand

iperf3 --client 192.168.1.100 --no-delay --omit 1 --time 5 --parallel 4 --format m
iperf3 --client 192.168.1.100 --no-delay --omit 1 --time 5 --parallel 4 --format m --reverse
to verify the setup, i setup a NAT port forward on port 5201 to 192.168.1.100 and run iperf3 against the WAN IP from client

Code Select Expand

iperf3 --client 192.168.160.10 --no-delay --omit 1 --time 5 --parallel 4 --format m
iperf3 --client 192.168.160.10 --no-delay --omit 1 --time 5 --parallel 4 --format m --reverse
pfSense, OPN 24.10.2, OPN 25.1.1 all showed ~9.45 Gbit/sec in both directions.

wireguard results

• pfSense
  
  • upload: 7.6 Gbit/sec
  • download: 6.6 Git/sec
• OPN 24.10.2
  
  • upload: 8.0 Gbit/sec
  • download: 543 Mbit/sec
• OPN 25.1.1
  
  • upload: 8.0 Gbit/sec
  • download: 538 Mbit/sec

if i disable pf via

Code Select Expand

pfctl -d and re-run the iperf3 commands but still going through the wireguard interfaces

• OPN 25.1.1
  
  • upload: 8.1 Gbit/sec
  • download: 7.5 Gbit/sec

For this test the CPUs are nearly 100% in use with the kernel threads and wireguard threads. This is basically the CPU bound max of the 2278g setup.


pfSense and OPN are setup with a wireguard interface and have a single firewall rule: allow from wg0 net to any.

i have tried various MTUs, outbound NAT rules, and firewall normalization rules suggested in the documentation or on these forums. those extra rules made no difference whatsoever to the results. the reverse iperf3 direction is always stuck ~ 500 Mbits and the CPUs are nearly 100% idle during transmission.

no matter which CPU/nic i used in the router, e-2278g, e-2414, G7400, n305, i3-14000t, i5-13400t, --reverse direction is always ~ 500 Mbit/sec. some kernel level delay blocking CPU from working as hard as it can.


the installs and setups are as out-of-box as possible. pfSense i had to install the wireguard package. OPNsense i install the cpu-microcode-intel package.

i have gone through various tunables, but none make any real impact. 8 Gbps vs 500 Mbps isn't going to be tweaked, unless its some sort of bug that can be worked around

i have also tried many other machines as noted earlier for client, router, server. i also put in my AMD 7950X3D windows 11 desktop into the mix as client and server. there is no difference in behavior whatsoever.


i have setup a OpenVPN tunnel via the OPNsense OpenVPN road warrior documentation and i get the same behavior. so its not wireguard. i did not try an IPsec tunnel.

i also diff'd the output of

Code Select Expand

sysctl -a of pfSense 2.7.2 and OPNsense 25.1.1 and saw no real meaningful differences.