Messages

Thanks for confirming 2.32 is vendor approved.

"Vendor approved" is probably overstating it. I'll just say that the Protectli support rep working with me indicated that he tested the upgrade to that version and used it in his guidance to me for the procedure.

At any rate, I successfully updated:

Code Select Expand

[1] igc0: <Intel(R) Ethernet Controller I226-V> mem 0x80400000-0x804fffff,0x80600000-0x80603fff at device 0.0 on pci1                                       
[1] igc0: EEPROM V2.32-0 eTrack 0x80000422                                                                                                                         
[1] igc0: Using 1024 TX descriptors and 1024 RX descriptors                                                                                                       
[1] igc0: Using 4 RX queues 4 TX queues                                                                                                                           
[1] igc0: Using MSI-X interrupts with 5 vectors                                                                                                                   
[1] igc0: netmap queues/slots: TX 4/1024, RX 4/1024                                                                                                               
[1] igc1: <Intel(R) Ethernet Controller I226-V> mem 0x80700000-0x807fffff,0x80900000-0x80903fff at device 0.0 on pci2                                             
[1] igc1: EEPROM V2.32-0 eTrack 0x80000422                                                                                                                         
[1] igc1: Using 1024 TX descriptors and 1024 RX descriptors                                                                                                       
[1] igc1: Using 4 RX queues 4 TX queues
[1] igc1: Using MSI-X interrupts with 5 vectors
[1] igc1: netmap queues/slots: TX 4/1024, RX 4/1024
I haven't changed anything else yet. Still on the -iflib kernel, no tunables, still on coreboot. I'm going to monitor for a bit, and if/when it pops up again I'll go from there.

BTW, Protectli have a statement about why they can't distribute i226 NVM updates at the bottom of this document: https://kb.protectli.com/kb/how-to-update-intel-nic-nvm-firmware-on-protectli-vaults/.  They link to the thread here with the BillyCurtis images, for those needing to do it.

Thanks, I missed that bit. Sounds like I just need to hold my breath and do it. They specifically mention they've validated an update to 2.27. Is there any compelling reason to consider 2.32? (Edit: Scratch that, I got a response from Protectli pointing at 2.32, so I'll go ahead and try that. They also mentioned I may have better luck with AMI BIOS vs. coreboot. I need to get some clarification, but will proceed with one or both and report back.)

Ok, thanks for the additional comments. As mentioned in the original post, I haven't updated the NVM because frankly it's a bit intimidating. As far as I can tell Protectli only provides an update for one model (that isn't mine), so I'm going third-party for something that can brick the interface. Sounds like that may be what I need to do anyway though. I'll try a plea to Protectli while I prepare.

While having a look at this issue, I noticed a potential bug in the iflib code making an automatic reset in case of a TX hang impossible, a custom kernel has been published which resolves this (though likely not the final patch version). Would you mind installing this kernel to see if this changes anything about the issue?

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Code Select Expand

dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 2544406
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 3606676
dev.igc.0.mac_stats.tx_frames_512_1023: 14052
dev.igc.0.mac_stats.tx_frames_256_511: 17180
dev.igc.0.mac_stats.tx_frames_128_255: 26586
dev.igc.0.mac_stats.tx_frames_65_127: 105743
dev.igc.0.mac_stats.tx_frames_64: 5542
dev.igc.0.mac_stats.mcast_pkts_txd: 2184
dev.igc.0.mac_stats.bcast_pkts_txd: 83
dev.igc.0.mac_stats.good_pkts_txd: 3775779
dev.igc.0.mac_stats.total_pkts_txd: 3775779
dev.igc.0.mac_stats.good_octets_txd: 5499410065
dev.igc.0.mac_stats.good_octets_recvd: 179427410
dev.igc.0.mac_stats.rx_frames_1024_1522: 28848
dev.igc.0.mac_stats.rx_frames_512_1023: 11786
dev.igc.0.mac_stats.rx_frames_256_511: 10339
dev.igc.0.mac_stats.rx_frames_128_255: 22011
dev.igc.0.mac_stats.rx_frames_65_127: 1596460
dev.igc.0.mac_stats.rx_frames_64: 26780
dev.igc.0.mac_stats.mcast_pkts_recvd: 3350
dev.igc.0.mac_stats.bcast_pkts_recvd: 44496
dev.igc.0.mac_stats.good_pkts_recvd: 1696224
dev.igc.0.mac_stats.total_pkts_recvd: 1705605
dev.igc.0.mac_stats.mgmt_pkts_txd: 0
dev.igc.0.mac_stats.mgmt_pkts_drop: 0
dev.igc.0.mac_stats.mgmt_pkts_recvd: 0
dev.igc.0.mac_stats.unsupported_fc_recvd: 0
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.recv_length_errors: 0
dev.igc.0.mac_stats.missed_packets: 197
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 612
dev.igc.0.queue_rx_3.rxd_head: 613
dev.igc.0.queue_rx_3.interrupt_rate: 20832
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 368
dev.igc.0.queue_rx_2.rxd_head: 369
dev.igc.0.queue_rx_2.interrupt_rate: 20832
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 830
dev.igc.0.queue_rx_1.rxd_head: 830
dev.igc.0.queue_rx_1.interrupt_rate: 4032
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 605
dev.igc.0.queue_rx_0.rxd_head: 606
dev.igc.0.queue_rx_0.interrupt_rate: 20832
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 173
dev.igc.0.queue_tx_3.txd_head: 173
dev.igc.0.queue_tx_3.interrupt_rate: 20832
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 1000
dev.igc.0.queue_tx_2.txd_head: 1000
dev.igc.0.queue_tx_2.interrupt_rate: 20832
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 46
dev.igc.0.queue_tx_1.txd_head: 46
dev.igc.0.queue_tx_1.interrupt_rate: 4032
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 735
dev.igc.0.queue_tx_0.txd_head: 735
dev.igc.0.queue_tx_0.interrupt_rate: 20832
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335938
dev.igc.0.device_control: 1075578433
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 3
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.tso_tcp_flags_mask_last_segment: 3967
dev.igc.0.tso_tcp_flags_mask_middle_segment: 3958
dev.igc.0.tso_tcp_flags_mask_first_segment: 4086
dev.igc.0.dmac: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
        CTRL     401c0641
        STATUS   00380683
        CTRL_EXIT        100000c0

Interrupt Registers
        ICR      00000000

RX Registers
        RCTL     04408002
        RDLEN    00004000
        RDH      0000025e
        RDT      0000025d
        RXDCTL   02040808
        RDBAL    75d88000
        RDBAH    00000000

TX Registers
        TCTL     a503f0fa
        TDBAL    750f8000
        TDBAH    00000000
        TDLEN    00004000
        TDH      000002df
        TDT      000002df
        TXDCTL   0201011f
        TDFH     00000000
        TDFT     00000000
        TDFHS    00000000
        TDFPC    00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.fw_version: EEPROM V2.17-0 eTrack 0x80000303
dev.igc.0.enable_aim: 1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 613
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 612
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 369
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 368
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 831
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 830
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 606
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 605
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 0
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 256058
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 256060
dev.igc.0.iflib.txq3.ring_state: pidx_head: 0060 pidx_tail: 0060 cidx: 0060 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 512129
dev.igc.0.iflib.txq3.txq_processed: 512169
dev.igc.0.iflib.txq3.txq_in_use: 44
dev.igc.0.iflib.txq3.txq_cidx_processed: 169
dev.igc.0.iflib.txq3.txq_cidx: 129
dev.igc.0.iflib.txq3.txq_pidx: 173
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 0
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 195065
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 195065
dev.igc.0.iflib.txq2.ring_state: pidx_head: 0505 pidx_tail: 0505 cidx: 0505 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 390078
dev.igc.0.iflib.txq2.txq_processed: 390118
dev.igc.0.iflib.txq2.txq_in_use: 42
dev.igc.0.iflib.txq2.txq_cidx_processed: 998
dev.igc.0.iflib.txq2.txq_cidx: 958
dev.igc.0.iflib.txq2.txq_pidx: 1000
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 0
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 3023378
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 3023383
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0535 pidx_tail: 0535 cidx: 0535 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 6046724
dev.igc.0.iflib.txq1.txq_processed: 6046764
dev.igc.0.iflib.txq1.txq_in_use: 42
dev.igc.0.iflib.txq1.txq_cidx_processed: 44
dev.igc.0.iflib.txq1.txq_cidx: 4
dev.igc.0.iflib.txq1.txq_pidx: 46
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 301485
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 301488
dev.igc.0.iflib.txq0.ring_state: pidx_head: 0432 pidx_tail: 0432 cidx: 0432 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 602805
dev.igc.0.iflib.txq0.txq_processed: 602845
dev.igc.0.iflib.txq0.txq_in_use: 42
dev.igc.0.iflib.txq0.txq_cidx_processed: 733
dev.igc.0.iflib.txq0.txq_cidx: 693
dev.igc.0.iflib.txq0.txq_pidx: 735
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 3
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.allocated_msix_vectors: 5
dev.igc.0.iflib.use_extra_msix_vectors: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%iommu: rid=0x100
dev.igc.0.%parent: pci1
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V


I wasn't referring to instances of "watchdog" or "timeout" in your historic logs, these messages should pop up as a result of installing the custom kernel. If this is the case, I'd like to know if you still have to intervene manually.

Ok, I misunderstood. I'll try installing that kernel tomorrow and we'll see what happens. For what it's worth, when the DNS-like issue popped up earlier I found that ifconfig down/up on WAN didn't do anything, but doing it on LAN did resolve it.

Have you configured Hybrid NAT and a Static-Port Rule for the specific Client ?

I have, yes. Most of the time it works. Just occasionally I have to reset the NIC as described above.

All I know is that ASPM can be disabled globally or on a per-device basis. The latter normally needs BIOS or driver support, the former, AFAIK, does not.

Yeah, unfortunately it doesn't look like the coreboot BIOS offers any configurability here, and I'm on the latest (only?) version for my hardware. May try switching to AMI BIOS at some point, but I'm not there yet.

I don't know that ASPM can be reliably disabled on Protectli units, at least those running coreboot.  The 'hw.pci.enable_aspm=0' sysctl has not worked for me.  I spoke to them several months ago in a support ticket and was told that this is a common request on Reddit and they are looking into adding ASPM controls in coreboot but was not given any timeframe.  In the same conversation I was told that EEE gets a bad rep and I should not worry about it. 🤷

I'm on coreboot, so that's interesting, thanks. And yeah, I noticed that disabling ASPM via the tunable didn't appear to have any affect on temperatures or anything, so I guess that tracks.

Having said that I have also seen the DNS timeout on rare occasion on my V1410 but never thought twice about it (assumed it was a normal network glitch).  I don't see any watchdog related messages in dmesg as of now.

When it happens for me, it's basically every DNS resolution that hasn't been made recently. It's hard to miss.

If it's this issue, you should see "watchdog timeout" messages appearing in your dmesg/system log. After this, an automatic reset should recover connectivity. If this happens, can you share these logs?

Yeah, a lot of the stuff I've read indicates I should see interfaces flapping in the logs, but I haven't actually seen any evidence of that. There's no incidence of "watchdog" or "timeout" anywhere in dmesg or the month or so of system logs that have built up, and the only link state changes I see appear to correspond to when I uncheck/check the box and apply. So it's possible this is something else entirely.

Your description of the issue sounds similar to others, however, there are still a lot of gaps to fill. Most notably, do you always need manual intervention to fix the issue? or does it recover on its own? Is it always the same igc interface? What is the auto-negotiated link state at the time of failure (# ifconfig igcX)?  If there's no auto-negotiation, what link speed did you set it to?

So far the issue hasn't ever sorted itself out without my intervention, but since I can't tell when it's happening other than by the symptoms, I can't say for sure that it doesn't happen more often and occasionally fix itself. For most of the life of this issue speeds were on auto-negotiate (typically WAN 2500 Full, LAN 1000 Full). A few days ago I switched everything to static 1000 Full; the issue has popped up again once or twice since then.

I'll remove the tunables since they don't appear to be doing anything for me anyway, and I'll do a bit more digging next time this happens and see if I can come up with something more concrete to provide. In particular, I'm also curious if just ifconfig down/up will resolve it.

I'm working on troubleshooting an issue that's been popping up irregularly since deploying OPNSense on a Protectli VP3210 (both new to me). The device is set up to perform all DHCP, DNS, firewall, and routing duties for the home network behind it.

Approximately every 1-7 days the network starts acting up. The symptoms aren't always consistent, but so far have tended to fall into one of three categories:

1. Something that looks like a DNS issue. Attempts to resolve an address will usually time out first try, but then succeed immediately a few seconds later. If I connect to the upstream router and use the same resolver, everything is normal.

2. DHCP will stop working for some/all devices.

3. An online game I play regularly has trouble connecting to the game servers.

Regardless of the symptom, the workaround that resolves it (temporarily) is the same. Go to Interfaces -> Settings, uncheck "Disable hardware checksum offload", Apply, recheck the box, Apply again. Everything immediately starts working as it should. (This is why I assume this is a NIC issue.)

Doing some research, I see that it's not uncommon for people to have issues with the Intel i226-V NICs, something I missed when I chose the hardware. Based on what I read I've been playing with various tunables, rebooting as necessary:

dev.igc.0.fc=0
dev.igc.1.fc=0
dev.igc.0.eee_control=0
dev.igc.1.eee_control=0
net.isr.bindthreads=1
net.isr.maxthreads=-1
net.isr.dispatch=deferred
net.inet.ip.intr_queue_maxlen=3000
hw.pci.enable_aspm=0

So far nothing has made a difference. The other thing that seems to be done commonly with these NICs is to upgrade the NVM firmware, which I'll try if I have to but that's a bit intimidating. Anyone have any other ideas before I go that route?

Thanks for the response. I'm aware of the security implications of UPnP. It's not ideal, but there are a few things it seems to be necessary (or at least the lesser evil) for. But I've tried so many things over the last few days that it's indeed possible I can get away with disabling it now. I'll test that; thanks for mentioning it.

Does your network have Managed Switches that can handle VLAN Tagging ?

If so, then just create a VLAN in which you connect the ISP Router as a Untagged Device and forward it also Untagged to the LAN Ports of the two Gaming PC's and you are DONE! :)

It does. Our current workaround is actually for one of us to connect to the old router which now sits outside the OPNSense network; similar concept. Ultimately though I'd like to get everything under OPNSense. I wouldn't have spent so much time on this if I didn't know it's possible (as the old router handled it without any specific configuration). But it seems to be tricky to get two machines on the same network to connect to each other via a P2P network like Rockstar uses for GTA Online. Still plugging away at it.

Hi all. I recently replaced my basic home wifi router with an OPNSense box. Things went mostly smoothly, but I quickly found out that certain online multiplayer games that use a P2P model (particularly GTA Online) were not happy. After a bit of research, I made a static port NAT rule for our two gaming computers and set up os-upnp, and that seemed to work. Both are able to play online simultaneously with other players, but the one thing we can't do is actually play together. When either of us attempts to join the other, it fails with a generic "Unable to connect to session".

In our setup, our ISP's provided router gives out private addresses (10.0.0.x), so that's what OPNSense's WAN interface gets. This is the same situation as the previous wifi router, which worked fine, so I know this is possible without changing the ISP router. I'm still fairly new getting into the nuts and bolts of NAT (and UPnP), so I'm not really sure how to troubleshoot this. I can say though that if one of us connects directly to the ISP router (bypassing OPNSense), we're able to play together successfully. Obviously that's not ideal.

The first of the two things I've changed is switching Outbound NAT to Hybrid and adding a rule for each machine:
Interface: WAN
Source: 192.168.1.x/32
NAT Address: WAN address
Static Port: Yes

The second thing was installing and configuring os-upnp (miniupnpd). Here's /var/etc/miniupnpd.conf. Note that the ext_allow_private_ipv4 line was added manually (and the service restarted) since it's not available in the UI. I think this was the last change that made online play work initially.

ext_ifname=igc1
ext_allow_private_ipv4=yes
http_port=2189
listening_ip=vlan06
ext_perform_stun=allow-filtered
ext_stun_host=stun.l.google.com
ext_stun_port=19302
secure_mode=yes
pcp_allow_thirdparty=no
ipv6_disable=yes
presentation_url=https://192.168.1.1/
friendly_name=OPNsense UPnP IGD & PCP
uuid=<uuid>
serial=60732055
model_number=26.1.8_5
allow 1024-65535 192.168.1.<x>/32 1024-65535
allow 1024-65535 192.168.1.<y>/32 1024-65535
deny 1-65535 0.0.0.0/0 1-65535
enable_upnp=yes
enable_pcp_pmp=yes
force_igd_desc_v1=yes
lease_file=/var/run/miniupnpd.leases
lease_file6=/var/run/miniupnpd.leases-ipv6

I haven't manually added any firewall rules for this. The only rules outside of the built-in rules for this internal interface are to drop IPv6, allow full access to the internet, and allow DNS, DHCP, and NTP to the OPNSense interface. If anyone can point me in a direction, it would be greatly appreciated.