Messages

I had set this issue aside while I was dealing with bigger things, but with OPNSense 26.7.2 this is now resolved via Endpoint-independent NAT.

• Gaming device IPs added to a firewall alias ("GamingHosts").
• Source NAT set to hybrid mode.
• Add a Source NAT rule:
  
  • Interface: WAN/IPv4/UDP
  • Source address: GamingHosts
  • Endpoint independent: Checked
• I had previously set up a Static Port rule. Not sure if it's still necessary but I kept it for TCP and just removed UDP to avoid conflict:
  
  • Interface: WAN/IPv4/TCP
  • Source address: GamingHosts
  • Static-port: Checked

That's it, no UPnP necessary.

Ok, it's been 2 weeks without incident, so I'm going to (cautiously) call this resolved by the switch from coreboot BIOS to AMI. I did update Protectli; they indicated there's no plan for a patch to coreboot for the VP3200 series at this time. The support person mentioned they wanted to talk to the dev team about it, but I haven't heard anything in the week since, so I'm assuming nothing further is forthcoming. I'll update this if that changes.

Thanks to everyone who pitched in with troubleshooting help and suggestions!

@fornax did you get any statement from them about this?  Are they even working with you to diagnose and root cause, or just pushing you to AMI?

The only time I've reached out to them so far was specifically about the NVM update, but I did summarize the issue, so they did respond to that as well. And I'm glad you mentioned that, because looking back at the email I realize I misread it initially and probably should have tried this sooner. The relevant part is:

At this point we have seen similar behavior in the VP3200 series' coreboot implementation compared to the VP2440's. The VP2440 had a NIC throughput degradation issue that was related to ASPM. (https://kb.protectli.com/wp-content/uploads/sites/9/2026/02/TSB-2025-001_VP2440-ASPM-Network-Interface-Performance-Issue-v2_0.pdf) Our other products don't seem to display this behavior.

I missed that they've actually seen "similar behavior" in my model as in the VP2440, which had a coreboot issue that required an update. So that's entirely on me, I should have pursued that at the time. I was initially going to wait until I considered this resolved and then reach back out to let them know, but I'm accelerating that now.

Wanted to drop in a status update. I'm not ready to declare victory quite yet, but this is the longest I've gone without an incident in quite a while. Possibly it was an issue with coreboot? Fingers crossed. I'll give it another week and then consider this resolved.

@fornax, it has been seen in past that auto-negotiate for 2.5gbps was an issue between certain devices that can do 2.5Gbps. If the WAN shows problem again, try:

sudo ifconfig igc1 down
sudo ifconfig igc1 media 1000baseTX
sudo ifconfig igc1 up

Yeah, I actually had both interfaces hardcoded to 1000 Full before the rebuild, thanks for the reminder.

I also flashed the BIOS from coreboot to AMI, which was a suggestion Protectli threw out there when I contacted them previously.

So the WAN "died", and then you installed fresh copy of 26.7? That's not really a good troubleshooting tactic, because now the issue (if it still exists) is reset.

I wanted to rule out corruption in the install. Also, as this was my first experience with OPNSense, I did a fair amount of poking around and playing with settings, so I wanted a fresh start to rule out any possibility of lingering config weirdness. And those are ruled out now, as I had another incident last night.

Is WAN router or LAN switch 2.5G capable?

WAN router yes, LAN switch no.

Even on 15.1 (26.7), verify EEE settings and look for events.

Code Select Expand

root@scutum:~ # sysctl -a | grep igc | grep eee
hw.igc.eee_setting: 1
dev.igc.1.eee_control: 1
dev.igc.0.eee_control: 1
dmesg and syslog have never shown any random events. Everything is associated with bootup, manual interface bounces, or other actions on my part (eg config changes).

And you are running latest NVM for the 226's ?

As of post #29, yes.

First time for everything, I guess... yesterday I had an issue that was only resolved after bouncing the WAN interface instead of LAN like every other time. In a fit of frustration I reinstalled the OS today, so it's on a fresh 26.7. That should rule out OS corruption or weird config issues from me tinkering with things. I also installed the -igc2 kernel. Will continue to monitor.

I did have another incident with the -igc kernel the other day (attempted to post about it with logs but it looks like it didn't go through?). There were no syslog events associated with anything other than the reboot after installing the kernel or the manual interface bounce.

Try disabling vlan tagging from offloading, this brings it back to kernel.

Code Select Expand

ifconfig igc0 -vlanhwtag -vlanhwtso -vlanhwcsum

Interesting find and definitely worth a try, doing that now.

A kernel patch related to your issue has been developed, assuming you're on 26.1.11_6, you can install it with:

# opnsense-update -zkr 26.1.11-igc

Sure, I'll install that now. What's the change(s) in this one?

May I suggest not involving this game into these issues ?

Yeah, I know the game is buggy as hell. If it was the only issue I was having I wouldn't be here. But it's a useful canary; as I usually check into the game fairly early, it's often my first indication that something's going wrong. And the fact that bouncing the LAN interface on the firewall makes it immediately start working again (and the fact that I never had these issues before switching to the new firewall) leads me to believe this particular issue isn't the game's fault.

No indicators if your issue is WAN or LAN side.

I haven't dug too much into the WAN side because only bouncing the LAN interface has any effect. I've tried bouncing WAN first on a few occasions and it never changes anything.

DHCP stops working, so just for the LAN side?

Yes, the firewall's DHCP client works just fine, it's the LAN-side clients on one or more VLANs that stop working.

VLANs? How do you mean? Two i226's and VLAN's on LAN side, so you run .1q or your VLAN's are L3 ?

Yes, 802.1q VLANs on the LAN side only.

Can you SSH into the FW using a non i226 iface? I suggest you do that and start monitoring stuff. Run a ping to the FW LAN IP address, then wait for the no-traffic issue to arise, what does the ping show?

Start a 2nd ssh session, then
dmesg -w | grep -E 'igc'

see if any kernel messages arrive when your issue happens.

The firewall has just the two i226-Vs. I could set it up to run in the background like the previous one, though. FreeBSD's dmesg doesn't appear to have a follow option, but neither dmesg nor the syslog have never shown any indication that they see anything wrong anyway.

Well, to complicate matters it looks like there's a separate issue with dnsmasq. Although in the past I've definitely run into VLANs losing DHCP which was solved with an interface bounce, this particular incident appears to be something else. It was not solved by bouncing the interface. (dnsmasq is throwing "no address range available for DHCP request via vlan01" despite vlan01 being in the interfaces list and having a range assigned.)

Still open to further suggestions, but I'm now leaning toward starting from scratch with a reinstall of OPNSense.

Ok, turning off flow control and swapping the LAN-side cable didn't change anything. As we're 5 pages in and have multiple setups being talked about, I'm going to sort of "reboot" the OP:

Symptoms are one or more of the following:

• Games (particularly GTA Online, but I suspect I also had an issue with the online features of Death Stranding at one point) unable to contact servers.
• Sites (most or all) not loading on first attempt, but 10-20 seconds later retrying is successful.
• DHCP stops working. Something I've found since I started this thread is that it seems to happen on some VLANs but not all at once, and the VLANs that do/don't work aren't the same in every incident.
• There are NO link flaps noted in dmesg/syslog other than the ones I force to fix the issue.

The only thing that ties these issues together is that they're all fixed by bouncing the firewall's LAN interface (ifconfig igc0 down && ifconfig igc0 up).

Current state:
OPNSense was last updated 7/12 (version 26.1.11_6). Kernel is the -igc_aspm kernel first mentioned in post #17. i226-V NVM version on both interfaces is 2.32. LAN link is statically set to 1000Mbps full duplex, FC off on both ends. The only tunables set are dev.igc.0.fc=0 and dev.igc.1.fc=0.

Things I've tried so far in this thread:

• Various tunables mentioned in the OP.
• The -iflib kernel mentioned in post #2 and the -igc_aspm kernel.
• Swapping the LAN-side cable.
• Updating the NVM from stock 2.17 to current 2.32.

Currently the system is experiencing the issue in that I'm getting the usual error preventing me from playing GTA Online and DHCP is not working on one VLAN. I'm going to leave it in this state for a while to troubleshoot since most everything else is still working. If anyone has any other suggestions, I'm all ears. Thanks for the guidance so far.

Oh, and I've had continuous pings running from the firewall both to the LAN and WAN, and both are still happily plugging away with 100% return.

One more thing to try before I'm going to suggest a more complicated troubleshooting setup to help classify the issue:

Disable flow control on all igc ports through System -> Settings -> Tunables. Tunable: "dev.igc.X.fc", Value: "0" (where X is the port). For 4 ports I'd expect 4 entries.

Ok, I had those in when I was tinkering with tunables initially, I'll try adding them back into the current state (just 2 interfaces here).

Yeah, I've never been entirely convinced that my issue is the same as the "usual" issue people seem to have with the I226-V, in part because of the lack of visible link flapping. I'm not a network engineer though... part of my motivation for setting this up was educational. I didn't expect to get hit with such an esoteric issue up front though, heh.

I'll try the ping test and see where that takes me.

Cables?, and what devices are connected to these 226's?

WAN connects to the ISP's modem/router device. LAN connects to a GrandStream GWN7801P switch. Cables are mostly CAT6. I can try swapping the LAN cable.

Here's what the switch has to say about the port:

Code Select Expand

Ethernet1/0/8 is up
  Hardware is Gigabit Ethernet
  Full-duplex, 1000Mb/s, media type is Copper
  flow-control is off
  back-pressure is enabled
  selfloop-detection is disabled
  selfloop-detection running is false
  real time input rate  11986 bits/sec, 4 packets/sec
  real time output rate  10192 bits/sec, 6 packets/sec
     55595737 packets input, 73006841505 bytes, 0 throttles
     Received 35574 broadcasts (69151 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame
     69151 multicast, 0 pause input
     27367707 input packets with dribble condition detected
     32114150 packets output, 9685006492 bytes, 0 underrun
     0 output errors, 0 collisions
     0 babbles, 0 late collision, 0 deferred
     0 PAUSE output

And OPNSense:

Code Select Expand

igc0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN (lan)
        options=48420b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO,HWSTATS,MEXTPG>
        ether 64:62:66:24:f8:27
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        groups: InternalIFs TrustedIFs
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: WAN (wan)
        options=48420b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO,HWSTATS,MEXTPG>
        ether 64:62:66:24:f8:28
        inet 10.0.0.166 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>