Picked up a DEC3920 from OPNsense Shop. US customs held it for 7+ days, after two docusign attempts with Fedex international customer service, they finally released it.
The hardware itself is very well made. Almost a work of art. There is fan noise, though its minor, not like a 1U Dell / Supermicro. You can hear the fan noise go up and down. AMD V3C18 CPU, comes with hyperthreads disabled.
Not sure how much more lab testing I will do before it replaces my Protectli VP2440. I only have 1 Gbps fiber, upgrading to 2 Gbps, so kept the testing simple to WAN on 2.5g interface and LAN on 10g. I didn't have a chance to setup and test wireguard, but since the VP2440 with N150 can handle ~2Gbps wireguard, this should have no problem with my use case.
Pleasantly surprised this thing idles less than the Protectli VP2440 (which is 12w with Coreboot 0.9.1-rc3 with ASPM enabled). Out of the box without having to mess with any settings or tuneables.
Total Cost
DEC3920: $2,721.69
Shipping: $81.24
Tariff fee: $33.58
Fedex fee: $15.00
Total: $2,851.51
CPU
AMD Ryzen Embedded V3C18 8-Core Processor
BIOS
Version: 025.05.01
Release Date: 03/13/2026
Memory
1x 16gb DDR5 Micron Technology CT16G56C46U5.C8H
Speed: 5600 MT/s
NVMe
Transcend TS256GMTE712A 256gb
NVMe PCIe Gen 4 x4
NAND Flash Type,112-layer 3D NAND (TLC)
Sequential Read Speed "Up to 3,800 MB/s"
Sequential Write Speed "Up to 3,200 MB/s"
Random Read / Write "Up to 350,000 IOPS / Up to 330,000 IOPS"
Endurance "Up to 4,000 TBW (Terabytes Written) / 1.84 DWPD"
MTBF "3,000,000 hours"
Network
igc i226-V nics ship with v2.25 firmware
ax 10g nics with 19.118.48
Power and iperf tests
All power measured at wall outlet of 110v using PN2000 power meter.
reboot - 57.64s
idle - wan igc1, lan igc0 (first boot) - 8.78w
iperf upload - 18.75w - 2.35 Gbit/s
iperf download - 18.60w - 2.35 Gbit/s
idle - zenarmor, wan igc0, lan ax0 - 9.92w
iperf upload - 21.53w - 2.35 Gbit/s
iperf download - 22.04w - 2.35 Gbit/s
Zenarmor with SQLite database with default policy with more than 50% policy blocks turned on. adds about 1w to idle. ~5w to routing 2.5g.
Real World
Connected to home network on 1 Gbps fiber, ZenArmor home, wireguard, AdGuard Home DNS, vlans. No problems maxing out 946 Mbps in both directions with plenty of spare CPU. ZenArmor's single core engine takes 1 of the 8 CPUs to about 59%. Given that, if using ZenArmor, DEC3920 can probably do 2.5 gbps, but not much more. 842 Mbps iperf with over wireguard + zenarmor. The client is on a network that requires MTU = 1280 for wireguard to work, so 842 Mbps is basically line speed for 1 Gbps link with 1280 mtu + wireguard.
Photos
(https://cdn.imgchest.com/files/178a348a7fa3.png)
(https://cdn.imgchest.com/files/cf7d8d3ec5ce.jpeg)
(https://cdn.imgchest.com/files/5cc10dc04224.jpeg)
(https://cdn.imgchest.com/files/14c719b8c0f9.jpeg)
(https://cdn.imgchest.com/files/806829b7cd1a.jpeg)
(https://cdn.imgchest.com/files/419c2f367d40.jpeg)
This chassis design really is beautiful.
Quote from: dirtyfreebooter on April 03, 2026, 10:25:45 PMUS customs held it for 7+ days, after two docusign attempts with Fedex international customer service, they finally released it.
Maybe they were trying to figure out if it's a prohibited item under the new policy :P
https://protectli.com/news/statement-fcc-covered-list-update-to-ban-foreign-routers-in-the-us/
Quote from: dirtyfreebooter on April 03, 2026, 10:25:45 PMNot sure how much more lab testing I will do before it replaces my Protectli VP2440. I only have 1 Gbps fiber, upgrading to 2 Gbps, so kept the testing simple to WAN on 2.5g interface and LAN on 10g. I didn't have a chance to setup and test wireguard, but since the VP2440 with N150 can handle ~2Gbps wireguard, this should have no problem with my use case.
I'm just curious: if the VP2440 already does 2Gbps on WAN, where did it fall short? Is it just ZenArmor, or did you see bottlenecks in VLAN routing as well?
Quote from: dirtyfreebooter on April 03, 2026, 10:25:45 PMPleasantly surprised this thing idles less than the Protectli VP2440 (which is 12w with Coreboot 0.9.1-rc3 with ASPM enabled).
I wish coreboot had an option to disable the iGPU as that could be contributing a little to the power draw.
the vp2440 i think is pretty great. x710 is rock solid. the fanless design of the vp2440 is also well engineered.
i always wanted to support this project more. i do pay for business license for example. i always wanted to get some opnsense hardware but frankly the previous gen cpus were kinda lame. especially with zenarmor.
i just really appreciate this project and want it to stay around and prosper.
Would you consider using another image hoster? imgur is not available in the UK. I would have liked to see those.
https://help.imgur.com/hc/en-us/articles/41592665292443-Imgur-access-in-the-United-Kingdom
Quote from: cookiemonster on April 04, 2026, 09:07:07 PMWould you consider using another image hoster? imgur is not available in the UK. I would have liked to see those.
https://help.imgur.com/hc/en-us/articles/41592665292443-Imgur-access-in-the-United-Kingdom
oh didn't realize that. i switched them over to imgchest.com
Great review. Thanks for the benchmarks, it provides a good baseline.
It looks beautiful, I can only imagine how it will be a cherry in a rack.
In regards of noise.
Does it contribute to the ovetall noise in the room?
Or does it blend in?
Or does it stick out?
Basicaly how do you percieve the noise generated by this DEC in your normal enviroment.
Regards,
S.
Quote from: Seimus on April 04, 2026, 09:39:35 PMIn regards of noise.
Does it contribute to the ovetall noise in the room?
Or does it blend in?
Or does it stick out?
Basicaly how do you percieve the noise generated by this DEC in your normal enviroment.
its quiet for 1U. its certainly not like supermicro 40mm x 25mm @ 15,000 RPM fan. its not as quiet as Noctua 40mm x 20mm. if i run stress-ng cpu test, it pushes a decent amount of air, but doesn't have the super high pitched sound. to me, its similar to my unifi unvr or unifi poe 48 switch. if you are standing right in front of it, you can hear a faint sound and you can hear the fan change speed. network rack is in a utility room in the basement, so noise not too much of a problem, but i certainly wouldn't want a bunch of enterprise servers running. this is certainly not that.
Perfect, thank you!
Regards,
S.
having a bit of trouble with the 2.5g interfaces. seems like after a decent amount of traffic passes through, i lose the WAN connection on igc0. there are no messages in dmesg or system log.
# ping google.com
PING google.com (142.251.46.142): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^C
--- google.com ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss
if i pull the ethernet cable and plug it back in, instant fix.
ping google.com
PING google.com (142.251.46.142): 56 data bytes
64 bytes from 142.251.46.142: icmp_seq=0 ttl=119 time=3.047 ms
64 bytes from 142.251.46.142: icmp_seq=1 ttl=119 time=2.780 ms
64 bytes from 142.251.46.142: icmp_seq=2 ttl=119 time=2.550 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.550/2.793/3.047/0.203 ms
all i did was swap out the VP2440 for the DEC3920. otherwise my network has been stable for years. i guess i might have to pull it out and do longer stress tests? seems to occur once WAN on igc0 hits around 100GB of traffic.
the confusing part is there is zero feedback from kernel/dmesg.
this now happened twice since moving DEC3920 from lab test to my main firewall
A1 seems have some ideas.
QuoteYour issue is a well-documented hardware/driver bug with the Intel I226-V and the FreeBSD igc driver. The root cause is a TX hang quirk in the I226 hardware that the Linux driver handles via automatic recovery, but the FreeBSD driver does not — causing the NIC to silently freeze while the OS thinks the link is still up.
Looking at your specific sysctl output, there are several clear indicators and actionable fixes:
What Your sysctl Reveals
dev.igc.0.eee_control: 1 — Energy Efficient Ethernet (EEE) is enabled. This is a primary known trigger for I226-V link hangs.
dev.igc.0.fc: 3 — Flow control is set to bidirectional (TX+RX both on). This has been associated with link drops under traffic on this NIC family.
dev.igc.0.watchdog_timeouts: 0 — The FreeBSD driver's watchdog never fires, confirming it lacks the TX hang detection/reset logic that Linux's igc_main.c has at line ~3150. Traffic stops silently.
dev.igc.0.link_irq: 8 — 8 link state interrupts over the session; the hang doesn't necessarily show as a link-down event.
dev.igc.0.mac_stats.missed_packets: 297 — Minor but consistent with the NIC entering a degraded state over time.
fw_version: EEPROM V2.25-0 — You're on the latest
A1 seems to be stupid here and dev.igc.0.eee_control=1 means energy efficient ethernet is DISABLED already...
i226 on FreeBSD is annoying.
ASPM L1 is enabled for igc NICs from pciconf, Disabled for ax (10g) NICs. I am going to try some of these, as i tried to leave everything as it shipped / defaults.
seems like all the sysctls around energy where set correctly already. setting hw.pci.enable_aspm=0 and rebooting did not do anything, igc still has ASPM enabled. there seems to be a global ASPM on/off in the BIOS. this happens so quickly and frequently, i am questioning how Deciso did not already discover this.. in their testing...
Quote from: dirtyfreebooter on April 04, 2026, 09:19:40 PMQuote from: cookiemonster on April 04, 2026, 09:07:07 PMWould you consider using another image hoster? imgur is not available in the UK. I would have liked to see those.
https://help.imgur.com/hc/en-us/articles/41592665292443-Imgur-access-in-the-United-Kingdom
oh didn't realize that. i switched them over to imgchest.com
Thank you
Please attach images to your posts directly on this forum and do not use "image hosters" at all. These are not image hosting services but personal data harvesting services. Or why do you think they're free to use as in "no cost"?
Quote from: Patrick M. Hausen on April 06, 2026, 12:55:56 AMPlease attach images to your posts directly on this forum and do not use "image hosters" at all. These are not image hosting services but personal data harvesting services. Or why do you think they're free to use as in "no cost"?
i tried attaching them, the limit is 256KiB for the entire post. so that doesn't really work.
In some posts I've started exporting screenshots in WebP format as it compresses really well and I can squeeze in an extra image or two compared to PNG or JPEG. But yes, it's sometimes required to split images across multiple posts.
Quote from: dirtyfreebooter on April 05, 2026, 06:36:38 PMASPM L1 is enabled for igc NICs from pciconf, Disabled for ax (10g) NICs.
For what it's worth, ASPM L1 is enabled on my V1410's I226-V NICs:
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
max read 512
link x1(x1) speed 5.0(5.0) ASPM L1(L1)
This might not be the best for latency but I haven't seen any stability issues or drop outs at all. I suspect your VP2440 might also have ASPM L1 on the I226-V ports, but disabled on the X710 ports with the latest coreboot update. So this configuration seems common.
In all likelihood this passes Deciso's validaton testing so I'm wondering if something else could be at play.
Quote from: OPNenthu on April 06, 2026, 02:34:15 AMIn some posts I've started exporting screenshots in WebP format as it compresses really well and I can squeeze in an extra image or two compared to PNG or JPEG. But yes, it's sometimes required to split images across multiple posts.
Quote from: dirtyfreebooter on April 05, 2026, 06:36:38 PMASPM L1 is enabled for igc NICs from pciconf, Disabled for ax (10g) NICs.
For what it's worth, ASPM L1 is enabled on my V1410's I226-V NICs:
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
max read 512
link x1(x1) speed 5.0(5.0) ASPM L1(L1)
This might not be the best for latency but I haven't seen any stability issues or drop outs at all. I suspect your VP2440 might also have ASPM L1 on the I226-V ports, but disabled on the X710 ports with the latest coreboot update. So this configuration seems common.
In all likelihood this passes Deciso's validaton testing so I'm wondering if something else could be at play.
VP2440, AMI bios ASPM is completely disabled for all devices. Coreboot 0.9.0, ASPM is enabled for igc and ixl, but igc did this same behavior. Protectli released 0.9.1-rc3 Coreboot and i run that for weeks with full ASPM and no issues. Protectli support says they addressed the ASPM issue with igc in that Coreboot BIOS, FWIW.
i never once had an issue with X710 and ASPM, on the VP2440 or supermicro boards with X710-DA2 pci nic.
Yes, you're right. They disabled it on the VP2440's I226-V interfaces, but none of the V-series ones were affected.
ugh, the drop out happened 2x times tonight. this is turning into a disaster. i've emailed decsio for support. hopefully they have some answers.
Hm,
Sorry to hear you see such problems. If there is no logs present I would suspect its in firmware related problem.
Hopefully Deciso can provide some solid feedback.
But what you describe sounds like the Issue ppl had on i226 around firmware 2.17 which was related to ASPM & Power savings.
https://forum.opnsense.org/index.php?msg=247197
Regards,
S.
response from deciso
QuoteThere's a lot of misinformation about the Intel nics unfortunately, on our end these have proven to be highly stable for years, currently we use the Intel 226, but the 225 versions we had before have been solid as well (using the right firmware and powered properly, poor hardware design choices in our experience cause quite some issues as well).
When the machine reports "host down" the first question you need to ask yourself is if the port is properly linked, our documentation contains some pointers about where to look in these cases, you can find it here https://docs.opnsense.org/hardware/support.html#network-connectivity-issues
Please make sure to test using default settings, no specific tunables whatsoever to prevent these causing unforeseen issues. You can always share the output of the commands in our documentation for us to have a small look, if you need an engineer to assist, that's also possible, but comes at additional cost.
i know they can't test everything, but i have used this config in so many varieties of OPNsense and it worked fine. Supermicro X11SCL-iF, Protectli VP2440, Odroid H4 Ultra, Lenovo P3 tiny, Aliexpress N200 topton and no issues.
its not a complex config and i didn't use any tuneables except from the shipping config and the 2 zenarmor adds. since friday i have had now 4 cases of networking goes down on i226, no logs, no output from kernel. NOTHING. unplug/plug the WAN cable and instant connection again.
i plugged my VP2440 back in last night and its now its been 12+ hours of working fine.
prior, my OPNsense uptimes are basically the OPNsense business release dates, and my internet, fiber, has also only went out 1 time is over 5 years and that was due to construction company cutting lines in the neighborhood.
my take at this point is that, Decsio $3000 firewall doesn't work and i have to pay to maybe figure out the issue with their firmware is sus. at this point, i guess my only other option is to send it back and pay for return shipping.
frankly what seems amazing, turned quickly in a horrible experience
You started the thread on Easter Friday (bank holiday) and it's now Easter Monday (bank holiday), give Deciso some time to react.
Quote from: patient0 on April 06, 2026, 05:29:17 PMYou started the thread on Easter Friday (bank holiday) and it's now Easter Monday (bank holiday), give Deciso some time to react.
i am aware of the holiday and i asked them to not reply during their break. i also have 30 days to return and US customs held the device for more than a week. i think i am entitled to some support of their $3000 firewall and basic DHCP WAN dropping connection silently. also to let others know my experience, since there is very little out there on what its like deal with an issue with official opnsense hardware.
--
i am more concerned with the initial response itself, basically telling me i226 is perfectly stable and should check my network cables (switch i already verified multiple machines (vp2440, supermicro x11scl-if) and multiple cables before i even bothered contacting them). i am not asking for help setting up my network, behavior points to some sort of hardware/firmware subtle bug is happening.
a better response, if any during holiday, was "its holiday in EU, we will be back on tuesday" vs gaslighting me on i226 issues and my network cable.
also, based on the response, quoted above, it seems unclear they really are going to even follow up. they basically told me i can pay for support or return it, paying for shipping back to netherlands.
What device is WAN connected to, an ONT? For testing purposes, can you add a small switch between the OPNsense WAN and whatever your ISP provided?
Quote from: patient0 on April 06, 2026, 05:42:33 PMWhat device is WAN connected to, an ONT? For testing purposes, can you add a small switch between the OPNsense WAN and whatever your ISP provided?
ONT - Calix 711GX (outdoor, not a ONT/Router combo)
The OPNsense systems i've used over the past 5 years with the same ISP / same ONT:
- Supermicro X11SCL-iF with igb WAN and ixl LAN (X710-DA2 nic)
- Odroid H4 Ulta with igc WAN and LAN (i226v)
- Protectli VP2440 with igc WAN and ixl LAN (X710-BM2)
- Lenovo P3 tiny igb WAN and igb LAN (i350-t4)
none of these system exhibit the behavior of the DEC3920 with the WAN going out with nothing in dmesg or console. my ISP has only ever gone out when a construction company cut wires while putting in a new bridge in the neighborhood. i consider the ISP/ONT rock solid by itself, though understand it could be some issue with combination, specific to the DEC3920
i have some unmanaged 5-port switches. Netgear 1Gbps model and Netgear 2.5Gbps model. so i can do this test. What specifically am i looking to test by adding in the switch?
Quote from: patient0 on April 06, 2026, 05:42:33 PMi have some unmanaged 5-port switches. Netgear 1Gbps model and Netgear 2.5Gbps model. so i can do this test. What specifically am i looking to test by adding in the switch?
Quite a few issues that come up here or on the other sense forum are between the upstream device and the router WAN. E.g. the upstream device is picky or is not as flexible as switch ports or a port flaps. Adding a switch would show if it is a specific issue of the two devices or not. Generally the routes WAN (and LAN) ports don't like the connected device ports flapping at all.
I'm aware that you have/had a number of devices that run well with the same upstream device and as a result the 3900 should behave well. The test would only be to narrow it down. It's of course also sensible to wait for help from Deciso.
Quote from: dirtyfreebooter on April 06, 2026, 06:23:24 PMi consider the ISP/ONT rock solid by itself
Can you rule out Firmware Updates done by the ISP ?
Has there been any recent downtime due to Network Maintenance during the night ?
QuoteThough understand it could be some issue with combination, specific to the DEC3920
To be honest I think it's simply the Intel NIC having issues since the whole i22x range had some seriously weird bugs over the years...
Quote from: patient0 on April 06, 2026, 06:42:07 PMI'm aware that you have/had a number of devices that run well with the same upstream device and as a result the 3900 should behave well. The test would only be to narrow it down. It's of course also sensible to wait for help from Deciso.
yea, i will certainly see what deciso says this week, then i have to make a decision before the 30 days is up, with 10 days in US customs. so i am willing to gather as much information beforehand as i can and hopefully deciso doesn't ignore the effort.
Quote from: nero355 on April 06, 2026, 06:50:32 PMCan you rule out Firmware Updates done by the ISP ?
Has there been any recent downtime due to Network Maintenance during the night ?
yea, this ONT is ancient 1 Gbps, i doubt it gets many updates. its been flawless for 5 years, literally, not joking or exaggerating. its impressive how stable this ISP is compared to other friends in town how only have Comcast cable and how much their connection goes down.
another note too, when the connection stops working, nothing in the logs / dmesg, but when i pull the cable and put it back in, connection comes back, but i also see the up/down event in the logs/dmesg, so nots 100% proof the ONT connection is stable, but some evidence it is, since i see the up/down events if i manually pull the cable.
i agree, i226, especially on FreeBSD with ASPM/power stuff seems sus. all the driver mailing lists posts i see seem to imply that that linux driver has specific code to prevent this TX hang / stall. any linux server, proxmox, truenas, i've run with i226 has never seen these types of network issues seen in FreeBSD/OPNsense.
Any difference if you spoof the WAN MAC?
Is the issue reproducible on other ports?
Can you swap igc0 with another port to see if the same drops happen?
If you can confirm the issue on two ports then the next step I'd try after confirming with Deciso would be to update the firmware on one of the ports to 2.31 which is the latest we have publicly available and see if the issue persists.
Quote from: newsense on April 07, 2026, 10:13:20 AMupdate the firmware on one of the ports to 2.31 which is the latest we have publicly available
I thought it was 2.32 for i226 at the moment ?!
yea i did upgrade the i226 firmware on my VP2440 to 2.32. i would not upgrade the DEC3920 without Deciso blessing or if i decide to keep the hardware. i am currently trying to run a bunch of iperf3 / packet-sender tests in an isolated setup, see if i can reproduce or if its only the ONT connection.
i put the VP2440 back in use sunday night and of course it has worked flawlessly since then. 100% uptime on connection and ONT, etc
Quote from: newsense on April 07, 2026, 10:13:20 AMIs the issue reproducible on other ports?
Can you swap igc0 with another port to see if the same drops happen?
If you can confirm the issue on two ports then the next step I'd try after confirming with Deciso would be to update the firmware on one of the ports to 2.31 which is the latest we have publicly available and see if the issue persists.
yea, in my latest tests, i am trying WAN on igc0 and igc1
https://www.reddit.com/r/opnsense/comments/1sesnc1/opnsense_randomly_loses_lan_wan_connection_with/
lol, i226 is such hot garbage, and even more so in freebsd
Another thing I didn't see mentioned here:
Did you try disabling the auto negotiation on the ports ?
What switch is it connected to then?
igc (on other hardware) and Netgear was a bad mix on my end and we replaced the switch with a Cisco for that reason.
Cheers,
Franco
On my end I have two DEC750 with igc connected to a Netgear MS510TXPP stack and its rock solid, even with auto negotiation (2.5gig is negotiated).
Router is a Fritzbox and both igc are also directly connected there, no link drops.
I guess it really depends...
Quote from: franco on April 08, 2026, 11:16:32 AMWhat switch is it connected to then?
igc (on other hardware) and Netgear was a bad mix on my end and we replaced the switch with a Cisco for that reason.
Cheers,
Franco
it was connected directly to my Calix 711 GE ONT. which is the same setup i have had for > 5 years. and other igc based firewalls, odroid h4 ultra, aliexpress mini pcs, protectli vp2440, etc. but of course DEC3920 is different and has a different i226 firmware then all the others.
so i put the DEC3920 back in lab mode, isolated with a linux server on both sides. i put over 10TB of traffic through, and not a single issue. i even put a 1gbe switch on the WAN side so that the 2.5g wan would autonegotiate @ 1 gbps.
so last night i pulled the power from the ONT for 5 minutes, i know, its dumb. put the DEC3920 back, powered back the ONT. going to see if that helps, as i was not able to reproduce the behavior. if it goes out again, i will try putting that dumb switch in between the ONT and WAN.
maybe some progress, but this is all just certainly --> try this, try that. after running a bunch of tests in the lab, i made some minor tweaks by iterating with AI over sysctls by feeding AI netstat -Q and sysctl dev.igc and retesting with iperf3 and packet-sender.
i put the DEC3920 back on my WAN, and at this point, i have had the longest uptime since i got the device.
# uptime
10:24AM up 2 days, 1 hr, 5 users, load averages: 1.17, 0.68, 0.52- i pulled the power from the ONT for 5 minutes, before connecting the ONT to DEC3920.
- tested all cables with Klein Tools VDV526-200 Cable Tester
i switched the WAN to igc1, which interrupts are mapped to cpu 4,5,6,7
changed ax0 cpu offset to 1, it only has 3 interrupt queues, mapping it to cpus 1,2,3
leaving cpu 0 for system interrupts.
this was based off looking at the interrupt counters.
now, i kinda think this is all ridiculous for 1 Gbps, when this system was 10g and 2.5g interfaces. I would think the CPU could just plow through 1 Gbps, but it doesn't. especially when you add ZenArmor and Wireguard into the mix.
Tunableschange the cpu offset from 0 to 1 for ax0, mapping tx queues to cpu 1, 2, 3
dev.ax.0.iflib.core_offset=1
igc tweaks
these come from looking at r_drops, r_stalls on the tx queues from sysctl dev.igc.1.iflib while running iperf3 at 1 Gbps
dev.igc.0.fc=0
dev.igc.1.enable_aim=0
dev.igc.1.fc=0
dev.igc.1.iflib.override_nrxds=4096
dev.igc.1.iflib.override_ntxds=4096
dev.igc.1.iflib.rx_budget=512
dev.igc.2.fc=0
dev.igc.3.fc=0
hw.igc.max_interrupt_rate=64000
added to help spread the packets over the igc queues, otherwise 1 cpu processes everything (from netstat -Q)
kern.ipc.nmbclusters=2000000
net.inet.ip.intr_queue_maxlen=4096
net.inet.rss.bits=3
net.inet.rss.enabled=1
net.isr.bindthreads=1
net.isr.maxthreads=-1
added by zenarmor
dev.netmap.buf_num=1000000
dev.netmap.ring_num=1024
dev.netmap.admode=2
things i added to prevent netmap queue full messages in dmesg
dev.netmap.generic_rings=4
dev.netmap.generic_ringsize=2048
factory defaults
hw.ibrs_disable=1
vm.pmap.pti=0
ice_ddp_load=YES
i wish could just turn off ASPM for the igc NICs in the BIOS/firmware.
another alternative i thought was to just use ax1 for WAN using a SFP to RJ45 adapter and ignore the i226 ports.
Any idea what "errors out" indicates? (Looks like "oerrs" from "netstat -i".) Does it increment during use? Speculating: Could be something as innocuous as cosmetic buffer evictions. I don't know of much that Ethernet hardware can detect on transmit, so I would tend to suspect a driver-level issue. If you want to look at it, a managed switch (inserted into the link) might show useful data... or not.
Quote from: pfry on April 10, 2026, 07:49:11 PMAny idea what "errors out" indicates? (Looks like "oerrs" from "netstat -i".) Does it increment during use? Speculating: Could be something as innocuous as cosmetic buffer evictions. I don't know of much that Ethernet hardware can detect on transmit, so I would tend to suspect a driver-level issue. If you want to look at it, a managed switch (inserted into the link) might show useful data... or not.
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
...
vlan0.201 1500 <Link#16> f4:90:ea:01:ef:cd 444345350 0 0 277320308 767 0
though, 767 errors out of 277,320,308 packets.. that is probably on the range of normal i would think...
i haven't been monitoring that, because well, before, my WAN would just completely out without any message, until i unplugged the WAN cable physically (no amount ifdown/ifup ever fixed it), so i just been waiting for the WAN to go out and not watching things extremely close, since prior i got 5x WAN outages in 1.5 days.
Quote from: dirtyfreebooter on April 10, 2026, 09:55:10 PMthough, 767 errors out of 277,320,308 packets.. that is probably on the range of normal i would think...
Yet, any number significantly greater than zero in production would spark my curiosity. With Gigabit and above full duplex, flow control, etc. layer 2 errors should just not happen.
If you reboot the connected switch for an update or unplug and replug the cable(s) while the firewall is in full operation - yes. But you should be able to name the cause easily because it was something like that. As soon as the interface goes
down no output errors should occur, either. So in these reboot/unplug cases they come from the fraction of a second when the other side is disconnected and OPNsense has not yet noticed that.
On my system I have single digit numbers probably from my last Mikrotik update. If 767 errors out of 277,320,308 fits that bill in your scenario is yours to judge.
Quote from: Patrick M. Hausen on April 10, 2026, 10:16:40 PMQuote from: dirtyfreebooter on April 10, 2026, 09:55:10 PMthough, 767 errors out of 277,320,308 packets.. that is probably on the range of normal i would think...
Yet, any number significantly greater than zero in production would spark my curiosity. With Gigabit and above full duplex, flow control, etc. layer 2 errors should just not happen.
If you reboot the connected switch for an update or unplug and replug the cable(s) while the firewall is in full operation - yes. But you should be able to name the cause easily because it was something like that. As soon as the interface goes down no output errors should occur, either. So in these reboot/unplug cases they come from the fraction of a second when the other side is disconnected and OPNsense has not yet noticed that.
On my system I have single digit numbers probably from my last Mikrotik update. If 767 errors out of 277,320,308 fits that bill in your scenario is yours to judge.
yea, i am going to watch these values from now on, i had the ONT powered off when i booted the DEC3920, then after it was booted, i restored power, so those could have all been from that initial power on of the ONT, maybe the port flaps while initializing, who knows how this cheapo ONT reacts when initializing.
IMO, i still think this is related to i226 firmware/driver and ASPM. looking at my VMs i do some opnsense development on, they all have zero errors, but much less data going through and they are not connected to the cheapest hardware i own, which is the ONT. i wish intel would bring the freebsd igc driver up to par with linux, as i have never had a single issue with i226 and linux.
i will give this about a week, then i might try and use the other SFP port with an UniFi 1G SFP to RJ45 adaptor (it listed on the supported SFP transceivers), just to compare.
This is a commercial device with great performance especially in relation to power consumption, well worth the money, IMHO, but still a commercial solution. So if ASPM is an issue, it's Deciso's job to provide a BIOS that either categorically disables it or at least gives you the option to.
Just saying ...
Not a big fan of 2.5 Gbit/s anyway. 1 G is enough for all connected desktop systems and 10 G fibre is just better than anything else up to that speed.
ok it already increased...
vlan0.201 1500 <Link#16> f4:90:ea:01:ef:cd 446025589 0 0 278611733 1609 0
it did seem to happen on the WAN DHCP renewal.. i have quantum fiber, the DHCP leases are for 30 min, so it RENEWs every 15 minutes...
2026-04-10T14:55:01-06:00 Notice dhclientdhclient-script: Reason RENEW on vlan0.201 executing
so i am going to see it happen there is anything going on there. i guess i could also put my VP2440 back in service, i never looked at this values before, so i have no clue if this is normal for my setup or was happening all along. DEC3920 WAN going completely out was new though..
Then again an ONT is not a switch - so as long as the numbers are zero or close to it for everything internal, it's quite plausible it's not Deciso's fault. I would insist on getting a statement on that ASPM setting, though. I don't have their latest generation of devices, never had a problem with the 1 G versions.
Quote from: Patrick M. Hausen on April 10, 2026, 11:05:45 PMThen again an ONT is not a switch - so as long as the numbers are zero or close to it for everything internal, it's quite plausible it's not Deciso's fault. I would insist on getting a statement on that ASPM setting, though. I don't have their latest generation of devices, never had a problem with the 1 G versions.
well i have 3 other systems with igc that never had issues with the same ONT in over 5 years.. so while it might be that the DEC3920 BIOS and nic firmware is a bad combination with my ONT, no other system i had hard dropped the WAN 5x and < 2 days in over 5 years.
i figured it out, i have a speedtest scheduled twice a day, 5:55a and 2:55p. the 2:55p just ran (i am in america mountain time zone)
vlan0.201 1500 <Link#16> f4:90:ea:01:ef:cd 447245078 0 0 279796907 1731 0
manually running speedtest...
$ speedtest --server-id=8862
Speedtest by Ookla
Server: CenturyLink - Denver, CO (id: 8862)
ISP: CenturyLink
Idle Latency: 2.97 ms (jitter: 0.15ms, low: 2.62ms, high: 3.02ms)
Download: 940.40 Mbps (data used: 458.4 MB)
7.86 ms (jitter: 28.55ms, low: 2.09ms, high: 283.75ms)
Upload: 940.56 Mbps (data used: 423.2 MB)
2.58 ms (jitter: 0.10ms, low: 2.41ms, high: 2.85ms)
Packet Loss: Not available.
more errors
vlan0.201 1500 <Link#16> f4:90:ea:01:ef:cd 447610515 0 0 280159181 1932 0
I'm not sure those error counts are even statistically significant (?)
For what it's worth I only see similar kinds of low-count errors on a Linux workstation where I'm using a Realtek NIC to do some layering (VLAN tagging+bridging) for some VM clients that share the interface. Handles it fine, but not without an occasional drop.
enp6s0 is the Realtek RTL8125.
enp10s0 is an I225-V rev.02 (one of famously "bad" ones), but it is for host access on an untagged switch port so it's not really burdened.
$ netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
br1020 1500 244331264 0 0 0 209686600 0 0 0 BMRU
br1030 1500 44951 0 0 0 36076 0 0 0 BMRU
enp10s0 1500 50737101 0 0 0 22372021 0 0 0 BMRU
enp6s0 1500 471593652 0 16078 0 210160120 0 0 0 BMRU
enp6s0.1020 1500 244331324 0 0 0 209883174 0 44298 0 BMRU
enp6s0.1030 1500 45152 0 0 0 276948 0 0 0 BMRU
lo 65536 119592 0 0 0 119592 0 0 0 LRU
virbr0 1500 0 0 0 0 0 0 0 0 BMU
On my OPNsense the error counts for all the igc parent interfaces is exactly zero. The only errors at all are on the VLAN interfaces, but they are extremely tiny (on the range of 5-6 packets since last boot).
igc1 is WAN.
root@firewall:~ # netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
igc0 1500 <Link#1> <redacted> 60787 0 0 45417 0 0
...
igc1 1500 <Link#2> <redacted> 182874157 0 0 55760928 0 0
...
igc2 1500 <Link#3> <redacted> 33646094 0 0 134920988 0 0
igc3 1500 <Link#4> <redacted> 21343776 0 0 47165695 0 0
...
vlan0.1010 1500 <Link#9> <redacted> 2224 0 0 2229 5 0
...
vlan0.1020 1500 <Link#10> <redacted> 32241487 0 0 129964335 5 0
...
vlan0.1030 1500 <Link#11> <redacted> 1409212 0 0 4954579 6 0
...
vlan0.1040 1500 <Link#12> <redacted> 7473 0 0 6580 5 0
...
vlan0.1050 1500 <Link#13> <redacted> 20174816 0 0 39480063 6 0
...
vlan0.1060 1500 <Link#14> <redacted> 2249 0 0 2350 4 0
...
vlan0.1070 1500 <Link#15> <redacted> 1176925 0 0 7676822 6 0
...
Maybe I've just been lucky but my experience with 2.5GbE so far has been OK. As I showed earlier I even have ASPM L1 permanently enabled as per coreboot on the OPNsense, and I never actually went through with updating the NVM firmware on the NICs because I couldn't identify a need. YMMV. I've seen some anecdotal reports online of people claiming that things got worse for them after NVM updates.
I wonder if your output error count is just sign that your VLAN filtering is maybe struggling in some rare situations, but it doesn't seem like it's enough to matter (at least to my untrained eye).
One thing I did do is I put a randomly generated MAC on the WAN interface. I did it as a security/privacy enhancement, but the reason I suggested you to try it is because I know that some, especially residential, ISPs keep lists of approved customer CPE. It's not likely the case but I was thinking they might be kicking you off after some time because they detect the Deciso OUI. I don't know... just a crazy thought.
Quote from: OPNenthu on April 11, 2026, 12:18:39 AMI'm not sure those error counts are even statistically significant (?)
One thing I did do is I put a randomly generated MAC on the WAN interface. I did it as a security/privacy enhancement, but the reason I suggested you to try it is because I know that some, especially residential, ISPs keep lists of approved customer CPE. It's not likely the case but I was thinking they might be kicking you off after some time because they detect the Deciso OUI. I don't know... just a crazy thought.
yea i dont think its an actual problem, its 0.00027% and it only happens when the link is at max, so maybe its just my shitty old ONT. i like it because its outside and only an ONT, if i ask for a new one, they only give out indoor ones that are router/ONT combos that need to be in bridge mode.
nah, quantum fiber does not have any mac address restrictions on WAN device, the provisioning is done on the ONT device somehow. i swapped 3 different OPNsense boxes with 3 different macs and never had an issue in 5 years
Quote from: dirtyfreebooter on April 10, 2026, 06:41:52 PMTunables
- removed some SysCtl related stuff -
I wish could just turn off ASPM for the igc NICs in the BIOS/firmware.
If I am honest :
I feel like you are doing way too much stuff with an expensive device that should just work out of the box!
Quote from: Patrick M. Hausen on April 10, 2026, 10:41:33 PMSo if ASPM is an issue, it's Deciso's job to provide a BIOS that either categorically disables it or at least gives you the option to.
Just saying ...
+1 Fully agree! :)
QuoteNot a big fan of 2.5 Gbit/s anyway.
I seriously hate it together with 5 Gbps since they were introduced :
It's just milking the market instead of pushing for affordable 10 Gbps NICs that came earlier on the market than the 2,5 Gbps and 5 Gbps ones did !!!
But I guess we will have to learn to live with this situation for now...
i know this is AI slop, but i put the data, all the output of sysctl -a and netstat -Q and netstat -i into AI and this is what it said
QuoteThis is a very specific and telling symptom. In FreeBSD (the foundation of OPNsense), Oerrs (Output Errors) on a VLAN interface while the parent physical interface (igc1) shows 0 Oerrs almost always points to a software buffer overflow or a MTU/tagging bottleneck rather than a hardware failure.
The fact that it jumps by ~200 every time you run a speedtest suggests that during the "Upload" phase (or high-concurrency TCP ACK bursts), the system is dropping packets as it tries to push them into the VLAN tagging engine.
1. The Root Cause: The "VLAN Tagging Bottleneck"
When you send a packet out of vlan0.201, the OS has to take the raw packet, wrap it in an 802.1Q tag, and then hand it to igc1.
If igc1 is busy or the CPU core responsible for that VLAN is saturated, the VLAN output queue fills up. When it hits its limit, it drops the packet before it even touches the physical NIC. This increments Oerrs on the VLAN but keeps the physical NIC's stats clean.
2. The Solution: Increase the "Send Shovel"
To fix this, we need to increase the software transmit queue and ensure the "offloading" is actually helping rather than hurting.
then it suggested
Quote# Critical. This increases the global interface output queue from the default (often 50-128) to match your 4096 descriptors.
net.link.ifqmaxlen=4096 # default is 50
but yea, all this for 1 Gbps.. how in the world could this ever handle 2.5g or 10g on the WAN? hah.
FWIW, i ran speedtest 3x, and 3x the Oerrs increased. i just set net.link.ifqmaxlen=4096 and rebooted (its a boot-time tunable). i ran speedtest 3x and Oerrs increased only once. i just don't think this matters.
i'll try WAN on ax1 with UniFi 1G SFP to RJ45 adapter later this weekend.
ok, i couldn't wait, i swapped WAN to ax1 with UniFi 1G SFP to RJ45 adapter.
run speedtest
speedtest --server-id=8862
Speedtest by Ookla
Server: CenturyLink - Denver, CO (id: 8862)
ISP: CenturyLink
Idle Latency: 2.97 ms (jitter: 0.04ms, low: 2.92ms, high: 3.00ms)
Download: 939.80 Mbps (data used: 655.7 MB)
4.18 ms (jitter: 23.82ms, low: 2.22ms, high: 438.52ms)
Upload: 937.74 Mbps (data used: 434.3 MB)
37.86 ms (jitter: 3.14ms, low: 3.09ms, high: 50.13ms)
Packet Loss: Not available.
and on first try, i got Oerrs on WAN. it must be related to the ONT.
vlan0.201 1500 <Link#16> f4:90:ea:01:ef:d1 1049994 0 0 801383 238 0
also, as expected, using SFP to RJ45 adapter, the idle power went from 10.1w to 12.6w
Quote from: dirtyfreebooter on April 11, 2026, 12:23:30 AM[...]yea i dont think its an actual problem, its 0.00027%[...]
I just noticed I have some, a much higher percentage:
root@fw:/home/user # netstat -i
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
[...]
bridge0 1500 <Link#20> 58:9c:fc:10:85:67 156005318 0 0 68605805 4905211 0
[...]
QuoteThis is a very specific and telling symptom.[...]the system is dropping packets[...]
Uh... that looks like an AI hallucination. It's looking to me like buffer evictions or similar. Possibly less than ideal, but I don't see any performance issues (I only have a 500Mb service). Oh, and there's all sorts of stuff (as in devices; ixl and ix at least) attached to that bridge. One of these days I'll upgrade (service and servers), and see if I have issues then.
yea i don't think its an actual issue. before, my connection was dropping off completely and the only fix to physically unplug and plug the cable back in, nothing i did on CLI like if down / if up, brought the connection back, so at least i feel i am at a better spot.
i stumbled across this bug discussing the i226 TX hang on FreeBSD: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279245
there is shell script, aspm_disable attached at the bottom, you can easily disable ASPM on PCI device.
# pciconf -l | grep igc
igc0@pci0:1:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc1@pci0:2:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc2@pci0:3:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
igc3@pci0:4:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000
# aspm_disable 02:00.0
PCIe capability found at offset 0xa0
Link Control offset: 0xb0
Current Link Control: 0x0042
New Link Control (ASPM disabled): 0x0040
setpci -s 02:00.0 b0.w=0040
ASPM disabled for 02:00.0.
interestingly enough, disabling ASPM on igc1 had no noticeable effect on the idle power consumption.
i was able to completely eliminate the Oerrs by using traffic shaper. my bufferbloat grade was a C, so i did the traffic shaping from the OPNsense docs and got an A+, slightly lower max bandwidth, but now the WAN interface is overwhelming the ONT.
and also dialed back on the tunables, once traffic shaping seem to eliminate all interface errors.
# make sure ax0 and igc1 don't overlap cpus, leave cpu 0 for system interrupts
dev.ax.0.iflib.core_offset=1
# igc tweaks
dev.igc.1.fc=0
# enable RSS
net.inet.rss.bits=3
net.inet.rss.enabled=1
net.isr.bindthreads=1
net.isr.maxthreads=-1
# enabled zenarmor, increased buffers to prevent netmap full errors in dmesg
dev.netmap.admode=2 # added by zenarmor
dev.netmap.buf_num=1000000 # added by zenarmor
dev.netmap.generic_rings=4
dev.netmap.generic_ringsize=2048
dev.netmap.ring_num=1024 # added by zenarmor
dev.netmap.ring_size=131072
# factory defaults
hw.ibrs_disable=1
vm.pmap.pti=0
ice_ddp_load=YES
I think that 'igc' devices don't depend on RSS being enabled in kernel settings. They use it implicitly.
You can check that by leaving 'net.inet.rss.enabled' to its default (0/off) and the output of 'vmstat -i' will show that igc devices have interrupts distributed across RX queues. On my machine the number of queues matches the number of cpus (4).
I don't think it hurts to leave the setting enabled though, and it probably benefits your SFP+ interfaces. But as others said, this kind of thing should either already be the default on your expensive firewall platform if it's needed, or it shouldn't be needed at all.
It looks like Deciso have not yet provided a default config for the DEC3920 but when they do, you could check it to see if RSS is explicitly enabled: https://docs.opnsense.org/hardware/defaults.html
The problem is now that you've made multiple changes, we don't know which one(s) affected your original issue. :-/
yea i saved the config.xml from first boot. it only had 3 tunables.
hw.ibrs_disable=1
vm.pmap.pti=0
ice_ddp_load=YES
yea, without bindthreads/maxthreads/rss_enabled, the IRQs are mapped to separate CPUs, but the netstat stats show all the packets being queued on 1 cpu. honestly for 1 Gbps, i find it strange that this CPU can't do it with just 1 core, but yea.
with RSS disabled, watching sysctl
# sysctl dev.igc.1.iflib.{txq0,txq1,txq2,txq3}.r_enqueues
dev.igc.1.iflib.txq0.r_enqueues: 3928608
dev.igc.1.iflib.txq1.r_enqueues: 1070
dev.igc.1.iflib.txq2.r_enqueues: 1219
dev.igc.1.iflib.txq3.r_enqueues: 235
all the packets seem to go to txq0, but maybe its because 1 Gbps isn't enough to stress even one core.
strange is that i never really root cause my connection issue. was it because i powered off the ONT for 5 minutes and it really let it reset? was it because i moved from igc0 to igc1 for WAN?
I see a similar imbalance on mine where one queue is preferred. If your network is not very diverse (for example if you are a single client doing a bunch of speed tests) then this might explain the skew toward a single core, perhaps?
root@firewall:~ # vmstat -i | grep igc
irq130: igc0:rxq0 91154 0
irq131: igc0:rxq1 27129 0
irq132: igc0:rxq2 27424 0
irq133: igc0:rxq3 35187 0
irq134: igc0:aq 2 0
irq135: igc1:rxq0 24060289 111
irq136: igc1:rxq1 48661715 224
irq137: igc1:rxq2 43006681 198
irq138: igc1:rxq3 155563356 715
irq139: igc1:aq 2 0
irq145: igc2:rxq0 140748349 647
irq146: igc2:rxq1 13709652 63
irq147: igc2:rxq2 13733157 63
irq148: igc2:rxq3 13726372 63
irq149: igc2:aq 2 0
irq150: igc3:rxq0 23630091 109
irq151: igc3:rxq1 26275878 121
irq152: igc3:rxq2 23673100 109
irq153: igc3:rxq3 23192884 107
irq154: igc3:aq 2 0
In my case igc2 is the VLAN parent for my workstation and igc1 is WAN. The skew makes sense, since the majority of the web traffic is igc2->igc1 and it's mostly the same exact kinds of flows (speed tests, YouTube...) from the same host. I've been doing a lot of shaper related testing lately.
igc3 is iot/mobiles and other background traffic... family browsing/streaming, etc.
===
Quote from: OPNenthu on April 12, 2026, 04:19:34 AMI see a similar imbalance on mine where one queue is preferred. If your network is not very diverse (for example if you are a single client doing a bunch of speed tests) then this might explain the skew toward a single core, perhaps?
Basically you are right, it depends on how the traffic is hashed which impacts what queue will be used. Usually the traffic of the same tuple will be hashed into the same queue. Thats why you may see an imbalance.
Regards,
S.
@dirtyfreebooter
If I may ask you, we have a topic on the forum in regards of "Intel i225/i226 2.5G NIC Information/Issue Tracking Thread".
I do not ask you to disassemble the new DEC, but could you put your Description of the issue and the Disabling of ASPM (via script) in there?
https://forum.opnsense.org/index.php?topic=38055.30
Regards,
S.
just wanted to post a follow up. after my last changes (those few tunables), i have had very stable experience. i updated to 26.4 and migrated my rules, but i have had 100% uptime on OS and WAN since i updated and rebooted for the update.
haven't had a change to redo some of my cable management since i replaced the unifi cloud gateway fiber, but i like the pop of red :) lol