Recent posts

#1

26.1 Series / Re: How to keep gateway on Gat...

Last post by gnsinfo - Today at 07:13:29 AM

- How can I ensure that traffic entering via a specific WAN interface is always forced to exit through that same interface's gateway,

This has been a hot question lately.  The 'reply-to' option is supposed to handle this, or so I think, but you are not the only one seeing that the default route is always chosen for reply traffic.

https://forum.opnsense.org/index.php?topic=50882.0
https://github.com/opnsense/core/issues/9702

OPNsense 26.1 has a new flow in the setup Wizard which asks whether or not to optimize the system for Multi-WAN (source-based policy routing).  I did a comparison of two config files with and without this setting and the difference was only two things.  In the Multi-WAN optimized setup, the global firewall options 'Disable force gateway' and 'Disable reply-to' were both disabled (unchecked).  They were the exact opposite in the setup for single gateway, which favors the default system route.

There appears to be some mysterious X factor affecting Multi-WAN that we haven't discovered yet.

(Or, it's above my skillset and no one is patient enough to point it out.)

It works!!!
I can't find Disable reply-to, anyway, in the Reply-to option, I selected WAN interface.
After that, it is no longer blocked email traffic and etc.
Thanks for your effort. (",)(__)...

Have good times all in all.

#2

General Discussion / Re: From DEC850 to Minisforum ...

Last post by pfry - Today at 05:21:35 AM

[...]I don't know which Intel driver is preinstalled in OPNsense.[...]

It'd be the kernel driver, most likely. Version numbering may be different - I haven't looked at the driver on Intel's web site. I wouldn't expect the X710 to be your issue, but I suppose you never can tell. I haven't really tested mine yet - I need to get a couple new servers running and upgrade my Internet link. You might want to look at PCI-e speed and width, just to be sure.

Code Select Expand

root@fw:/home/user # dmesg | grep ixl
[1] ixl0: <Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.3-k> mem 0xf5000000-0xf57fffff,0xf5a18000-0xf5a1ffff at device 0.0 on pci1
[1] ixl0: fw 9.152.77998 api 1.15 nvm 9.50 etid 8000f4ab oem 1.270.0
[1] ixl0: PF-ID[0]: VFs 32, MSI-X 129, VF MSI-X 5, QPs 384, I2C
[1] ixl0: Using 1024 TX descriptors and 1024 RX descriptors
[1] ixl0: Using 8 RX queues 8 TX queues
[1] ixl0: Using MSI-X interrupts with 9 vectors
[1] ixl0: Ethernet address: 3c:fd:fe:e7:2d:88
[1] ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
[1] ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
[1] ixl0: SR-IOV ready
[1] ixl0: netmap queues/slots: TX 8/1024, RX 8/1024


#3

General Discussion / Re: From DEC850 to Minisforum ...

Last post by virtualdimension - Today at 04:10:02 AM

The network card in the Minisforum is an Intel X710-DA2.
I don't know which Intel driver is preinstalled in OPNsense. The latest driver available for FreeBSD and my network card is from late December 2025 on the Intel website.
FreeBSD driver for Intel X710
But I don't know how to install it...

#4

General Discussion / Re: From DEC850 to Minisforum ...

Last post by virtualdimension - Today at 03:56:17 AM

My internet provider's modem (which also acts as an ONT) is connected via RJ45 cat.7 network cable

I am not sure what the current status is, but if you want 10 Gbps speed via RJ45 then CAT6a is actually the only Certified and Officially Acknowledged cable type, while everything above it still awaits official recognition so to speak.

from the 10Gbit output to the 10Gbit input of the DEC850 with the 10Gtek ASF-10G-T80 SFP+ module (I also used an Ubiquiti UACC-CM-RJ45-MG) and I created a DMZ for the IP address provided by the modem to the DEC850 firewall.

Since you are using a DMZ IP Address my first question would be : Private or Public range ?

Also please note that most modules that convert SFP+ to RJ45 or the other way around work up to 30 meters of cable !!

From the 10 Gbit output of the DEC850 I enter the 10 Gbit WAN port of a Ubiquiti UDM-PRO MAX with the 10G Direct Attach Cable (UACC-DAC-SFP10-1M).

Why ?!

The whole Ubiquiti UniFi UDM range is a mess and especially when messing around with 10 Gbps connections and even worse when there is buffering to 1 Gbps connections involved and the WAN uses PPPoE on top of that...
If you need the UniFi Controller part of it then just connect it as a Client to your Management VLAN and leave it like that in the future or replace it with a simple dedicated UniFi Controller device.

If I connect directly to the 10 Gbit LAN port of the internet provider's modem, I can reach the maximum speed (about 7.8-8.5 Gbit download and 1.8-2 Gbit upload.

This is with a PC connected to the ISP's ONT/Router combo device you mentioned in the DMZ setup part above I assume ?

Which OS and what kind of hardware does it have ?


Whatever you do from this point on please stick to testing with :
- DEC850
- Minisforum MS-A2
- The PC you used for the full speed 10 Gbps test.

There is no point in involving the Ubiquiti UniFi UDM Pro in this whole thing!

- iperf3 is by default single core. Look into the multithread option

Larger Window Sizes can improve the speed too! :)

- I've tried also with cat.6A, but doesn't change nothing.

- The DMZ IP address is private. I've removed all Bogons network blocks in OPNsense.

- The connection cables between the modem/firewall/Ubiquiti are 1 meter long

- I know that PPPoE connections (especially with Ubiquiti) can cause problems, but that's not the case with me, since the UDM receives the connection already established by the modem through the firewall. However, if that doesn't solve the problem, I'll try what you suggested with the controller.

- Yes, you're correct. I used both a desktop computer (and a mobile workstation) with Windows 11 and 10 Gbit network cards (both internal and external).

- The Ubiquiti UniFi UDM PRO MAX is the device to which all the devices are connected, so it is the one that must ultimately guarantee the final performance, but to try to understand who is the bottleneck I have to do the tests between the UDM and OPNsense and between OPNsense and the main modem.

Modem IP: 192.168.10.1
Minisforum MS-A2 IP: 192.168.1.1
Ubiquiti IP: 10.0.0.1

I've tried iperf3 -c (and -s) from and to Minisforum and Ubiquiti, with -P 1 (and 8) and -t 30
I've tried disabling Zenarmor, Crowdsec, and Suricata, but it makes virtually no difference in terms of throughput. The speed is still around ~3.5-4 Gbps, and the processor is practically idle. I'll have to try connecting directly to the Minisforum, bypassing the UDM, and see what results I get.

#5

General Discussion / Re: From DEC850 to Minisforum ...

Last post by virtualdimension - Today at 03:31:23 AM

You did not say anything apart from the Zenarmor part about multithreading.

See this, point 10:

a. With speeds > 1 GBit, you need to enable multithreading for unimpeded measurements.
b. You also need to enable RSS to use all cores. It may also depend on how the FreeBSD NIC drivers are optimized, there may be special tuneables for yours.

That being said: With Zenarmor, you can only utilize one thread at this time. Period. Pinning a core does only make sure that this one core does not get utilized by anything else, but you will still be limited by it.

Yes, I also activated RSS through tunables.

- net.inet.rss.enabled = 1
- net.isr.bindthreads = 1
- net.isr.maxthreads = -1
- net.inet.rss.bits = 4
- dev.ixl.0.rss_enabled = 1
- dev.ixl.1.rss_enabled = 1
- net.isr.dispatch = deferred (also tried hybrid)
- kern.ipc.maxsockbuf = 16777216

I tried using powerD (both with the Hyadaptive and Maximum options), but then I disabled it because it didn't make any difference. In the BIOS, I also disabled all power-saving options for both the CPU and network cards.
The promiscuous mode is on.
The "pin core" on Zenarmor is off.

#6

26.1 Series / Re: Options to stabilize prefi...

Last post by Javier® - Today at 02:55:12 AM

My ISP delegates a dynamic 56 prefix, and it hasn't changed since I started using OpenBSD with dhcp6leased.

#7

General Discussion / Re: From DEC850 to Minisforum ...

Last post by virtualdimension - Today at 02:54:58 AM

I am not quiet sure if I understand what you iperf or why you even have UDM and OPNsense. I would go with either one of them.
I also don't quiet get you setup or network topology, nor what speed exactly is your problem. And I can't give you good advise on why your Minisforum performs that bad. So I can just give you some general advise that applies to anyone. Maybe that helps.

- iperf3 is by default single core. Look into the multithread option
- make sure you have power savings disabled. I had mine on hiadaptive, got 5GB/s for the first thest, and when I ran the second test shortly after, I got my 9GBit/s because the CPU could not enter power saving yet. Disabling PowerD got me always 9GBit/s.
- Even my old 4-core i3-8100 is fast enough for 9GBit/s. But I don't run Zenarmor or Suricata, only Crowdsec.

I use OPNsense as a firewall because it has many more features and is much more secure than Ubiquiti, but I prefer Ubiquiti for its ecosystem (switches, access points, cameras, graphical interface, secure remote access, etc.), so I use both.
So, from the modem where the ONT is directly connected, I enter the OPNsense WAN port, and then the filtered connection passes from there to the Ubiquiti.

Yes, I am aware that the default iperf test is single core, in fact I have done several tests with the -P 4 (8, 16) commands

#8

26.1 Series / Re: Options to stabilize prefi...

Last post by OPNenthu - Today at 02:48:06 AM

Forgive me if I've been overly enthusiastic.

Not at all.  I welcome the suggestions, and thank you.

And yes people here tell me all the time how my ISP is terrible for not following RIPE recommendations for a static /48.  I think this forum is heavily skewed toward non-U.S. countries where ISPs are maybe more consumer friendly and less profiteering :)  My ISP is one of the larger ones here and they commonly do this, unless you pay for a business plan.

#9

26.1 Series / Re: Options to stabilize prefi...

Last post by Javier® - Today at 02:30:25 AM

One of the biggest problems with IPv6 is ISPs, in theory, they should assign a fixed prefix.
The only thing I've noticed with this configuration is that IPv4 DNS is used more. But the faster of the two still works, and it does depend a lot on the client.
NAT in IPv6 isn't from ULA to global, it's global to global, I think.
Forgive me if I've been overly enthusiastic.
Thank you OPNenthu

#10

26.1 Series / Re: Options to stabilize prefi...

Last post by OPNenthu - Today at 02:18:42 AM

This seems fuzzy.  I asked ChatGPT (but I don't trust it):

1. ULA vs Global IPv6

If the client only has a ULA (fc00::/7) and the destination is a global IPv6 address, RFC 6724 technically prefers matching scopes. That can:

- De-prioritize ULA for global destinations
- But many real-world stacks still try IPv6 first if it appears usable

So I guess it depends a lot on the client.

I know that Happy Eyeballs will choose the faster of the two, but I didn't think that clients would even try the ULA route if there is no GUA and IPv4 was available.

(And of course, if the destination is IPv6 only then it can't use IPv4.)

---

Anyway, another good alternative method to keep in my toolbox. Thanks again.

To reiterate though the only problem I'm seeing with DHCPv6-PD so far is the premature deprecation of the prefix when the modem reboots and gives me the same prefix again.  I think there is a bad assumption built into the stack that dynamic IPv6 means a new prefix every time.  That's just not true.