Messages

Not to be too far off topic, I can't even get gigabit fiber to my house, and cable is not reliable because they haven't upgraded their plant in 20 years.

And then the question of a static IP... Generally no or lots of money.

I need to remember not to take fiber for granted. I'm not even with the best fiber provider in Switzerland, like @patient0, but it's cheap, fast, very reliable, and lets you bring your own hardware (as long as it's on their list of approved hardware). I don't have static IPv4 but I'm using godns to update a DNS record from one of my domains, which is good enough for me.

@ou1 - have you come across these?

https://medium.com/@truvis.thornton/opnsense-firewall-configuration-performance-tuning-for-multi-gigabit-internet-and-better-speeds-in-cfc80c49c544
https://calomel.org/freebsd_network_tuning.html

One of the standout tunables is "kern.ipc.maxsockbuf".  On my system (2.5GbE interfaces, 1Gbps ISP) the default 4MB size seems to be OK (the mbuf denied/delayed counters are at zero) but maybe yours needs more if you can spare the memory?

Thanks for the links.

I haven't see that specific guide, but I see that I already have almost all of those settings. My kern.ipc.maxsockbuf was already at 16MB and my deny/delay counts are also at 0/0/0. I've tried setting some more of the options from this guide but it didn't seem to make any difference. I'm maxing out at around 6.2Gbps. If I enable Netflow, it drops to under 5.

I think given I have VLAN tagging on both LAN and WAN interfaces, 15W fan-less box, the performance is already really good.

I've come across this blog post, which also confirms that the DEC750 wouldn't be capable of reaching 10Gbps in my use case

https://blog.shade.sh/index.php/archive/2116

I've turned on hyperthreading in BIOS and applied these settings which allow all 8 cores to process interrupts as described in the post. This got my internet iperf3 performance up to 6.2Gbps, speedtest.net to 5Gbps, which is a small improvement, but I guess this is as fast as it will get.

Also from the GUI if you go to Interfaces->Overview->[WAN/LAN]->Details and look for the same (Media) as well as Line Rate.

Sorry I don't have more helpful hints.  Hopefully someone with one of these devices has some more thoughts, or you might also e-mail Deciso support.  Will be good for us to know the outcome for future upgrade decisions.

regarding #2 Thanks for the tip, I didn't know so much information was available in the GUI.

Media+Link Rate looks correct, 10GBase-SFI <full-duplex rxpause txpause>. And Media (Raw) Ethernet autoselect (10GBase-SFI <full-duplex rxpause txpause>).

Line Rate: 10.00 Gbit/s

This applies to both ax0 and the VLAN 10 interface which has ax0 as the parent.

Unfortunately I can't really test iperf3 from the firewall itself because it's maxing out the CPU. It hits 5.3Gbps, but with CPU at 100%. Testing with iperf3 -s on the DEC750, I can reach 6.3Gbps from the LAN, but again, it's maxing out the DEC750 CPU so it doesn't mean much.

I will probably contact Deciso to ask is this performance is to be expected, or if I should keep looking for possible optimizations or misconfiguration.

Thanks for the reply

It seems I've gone a bit down the rabbit hole and decided to try something else, to remove the XGS-PON SFP+ adapter from the equation.

I've connected my old ONT (XGS-PON, 10Gbe) to the ISP fiber and connected my PC directly to the ONT.
set up VLAN 10 on my PC, got a provider non-routable IP, provisioned the new ONT, restarted it, got a public IP.

speed test: 8Gbps
https://www.speedtest.net/result/18844302123

disconnected PC from ONT, removed VLAN 10 from PC, plugged my DEC750 into the ONT using a 10Gbe adapter, plugged my PC into another 10GBe adapter. Repeated the same test (this time through the DEC750 and the ONT)
speed test: 4.3Gbps
https://www.speedtest.net/result/18844315297

These tests were 5min apart, there wasn't much chance for hitting thermal limits on the SFP+ modules, they were only both connected for about 2 minutes.

Answer to your questions:

1) I now removed the module completely.

2) I am assuming so, it's definitely going over 5Gbps sometimes, but just barely. That's with protocol overheads too, I'm only looking at data throughput.

3) VLAN 10 on WAN interface could play a role, but impossible to test without. I don't have any other 10G routers that I could use for internal testing. I've followed all the multi-gig guides, including all tunables. Hardware offloading is all disabled, but enabling made no performance improvements, only corrupted UDP packets.

4) I don't have a VLAN id 1, I just meant that ISP needed VLAN tag 10, and on the LAN interface I didn't set up any VLAN.

The last thing to test will be turning off the firewall on the DEC750 and using it as a router. I'm holding back a bit on that because I'm worried of locking myself out, hate the thought of restoring a config through the console. But this will be a last-resort test.

I will look through your links later today, thank you.

I don't own one but the published port-to-port throughput for DEC750v1 and v2 (as tested with TCP, full-duplex, unknown # of streams) is 8.5Gbps, so for a pure firewall application without IDS/IDP I don't see why you couldn't get similar results.

Aside from that, I have doubts about the SFP+ modules and VLAN setup you mentioned.  A lot of variables there.

1) Is it reasonable to compare your XGS-PON module to the ISP modem and expect similar results?  Have you been able to separately test and verify the module on another router box and confirm that it could hit 8Gbps with your ISP?  (I'm not saying it doesn't- just that we can't assume.)

2) Do both of the transceivers negotiate properly?

3) Do you need any specific NIC and VLAN offload settings or tunables (e.g. RSS) that are recommended for the DEC750?  Maybe your WAN connection is also impacted by some overhead due to VLAN tagging.

4) AFAIK, there's no "default" VLAN in OPNsense.  Did you explicitly create one with VLAN ID 1?  Not that it matters really, but could be some VLAN filtering overhead issue as well (?)

--

Update: Just in case...

https://docs.opnsense.org/hardware/defaults.html

Looking trough the default config, I don't see any interface offload settings or tunables that aren't already default in OPNsense, except for maybe the ones related to Meltdown & Spectre mitigation.  However there have been threads here recommending to at least enable RSS.

https://forum.opnsense.org/index.php?topic=24409.0
https://docs.opnsense.org/troubleshooting/performance.html#receive-side-scaling

--

UPDATE 2:

Interesting recent thread here: https://forum.opnsense.org/index.php?topic=49030.0

Presumably both of your SFP+ modules are using the same speed and not mixing (you'll need to check) but I wonder if maybe you are hitting a thermal limit with XGS-PON on the DEC750.

Actually, I may have just found the issue. For some reason, I had unchecked "Disable hardware checksum offload" some time ago. Turning it back on fixes the SIP issues with NetFlow disabled.

Update: that's definitely it, I feel kinda stupid now for playing with settings like that and not trusting Deciso's default recommended settings.

I'm experiencing a very strange issue on OPNSense Business 25.10.2, running on a DEC750. I believe this was a problem also on previous versions, but I only disabled NetFlow just before upgrading to 25.10.2.

If I disable NetFlow (clear all interfaces, disable Capture Local, reboot), I can no longer make outgoing calls from my SIP phone. Incoming calls work fine. It remains this way until I re-enable NetFlow. I don't even need to enable it on my VOIP interface, it just needs to be enabled.

Looking at captured traffic, the client is sending large INVITE packets which are being fragmented. This happens both with NetFlow enabled and disabled. The only difference is that when it doesn't work (when NetFlow is disabled), there is no response from the server. It seems that the server is either silently dropping the packets, or they are not being delivered.

With NetFlow enabled, I get responses 100 Trying, 183 Session Progress, 180 Ringing.
With Netflow disabled, I get no responses, then client re-sends the INVITE, over and over until the call fails.

I have no static NAT rules, just Hybrid Outbound NAT, no SIP-specific OPNSense configuration whatsoever. I don't see any dropped packets in the firewall logs.

Any insight into this would be very appreciated.

Hello community

I have a DEC750, a 2-year old model with 2.5G ports.

On the WAN side, I have a 10G XGS-PON module, 10G internet plan from my provider. The WAN interface requires a non-default VLAN tag.

For testing, I plugged my PC directly into a 10GBe transceiver in the DEC750 using default VLAN 1.

Doing internet speedtests, the highest speed I can achieve is 5.2Gbps (iperf3 with parallel streams, speedtest net, cnlab). Is this to be expected? I don't want to spend too much time trying to increase this if it's already maxing out the firewall throughput of the DEC750. What I haven't tried is disabling the firewall on the DEC750, but this falls into the "spending too much time" category since I'd never use it in that configuration.

Using the router provided by my ISP, I can reach just over 8Gbps in speed tests.

The acme-client plugin version 4.7 has a bug where sftp upload automation can fail. This was fixed in 4.8. Luckily I caught this today, with many of my certs expiring tomorrow.

More info:
bug report from january 2025
fix before version bump to 4.8

I've been going back and forth trying to decide if I should renew the business license which came with my hardware, but it's really hard to swallow the price for a home user. Well, the time has come, and my license expires in a couple of weeks. I was surprised to see that the 3y license price went up from 359 to 399EUR. I was on the fence at 359, never got the chance to renew even once, and the price is already up.

This price increase definitely seals it for me - I will be switching to community edition. I would sign up in a heartbeat for something around 75EUR/year so I can stay on a slower, more stable release cycle and so I could support the project more than I already have. I understand that that costs are going up, but consumers are also starting to watch their spending and I think Deciso is pricing out home users. I'm sure I'm not the only one that will end up contributing 0EUR to the project only because of the lack of distinction between home and business.

Although I don't really have any suggestions on how to solve your issue, do you mind if I ask what kind of issues you ran into by enabling RSS? I'm playing with it now and it would be nice to know what to watch out for, not just assume it's working.

thanks

We have been discussing additional home and enterprise versions, but nothing concrete was decided at this point as both come with additional challenges and require further infrastructure improvements.

Hi Franco, have you guys had a chance to consider some other Home subscription options for people with OPNsense hardware? I am dreading the day that my business subscription will run out and it's really hard to justify the 150eur fee for home use.

I'd be more than happy to pay for access to a stable release channel for home use, but the current price is around 3x my pain threshold.

If I am forced to switch to community edition, do you have any tips on how to stay on a stable release schedule, closely tracking the business edition version? Is there a way to install a specific release using the GUI? For example, Business edition 24.4.1 is based on 24.1.8, but 24.1.9 has since been released. If I were on community edition, I'd like to install 24.1.8 as of today.