Messages

If finally works.

Seems to have been something wrong with the interface config.
I unchecked block privates and block bogons, and it's working now.
I'll probably reenable to bogons block at some point if it doesn't break it, but at least it's up and gateway ping is finally responding.

I'm guessing it uses some private local address for something to do with RA and SLAAC from the LM1200.

The final interface config for "VWAN2" is:
Enable [X]
Block private networks [_]
Block bogon networks [_]
IPv4 Configuration Type [DHCP]
IPv6 Configuration Type [SLAAC]
Reject Leases From [192.168.5.1]
Override MTU [X]

Everything else is unset or left to defaults.

I'm sure that I can get NPT working at this point.

Thanks for the support Maurice.

Yes, I'm in the US. I'm not sure if it's using CGNAT, but it probably is. My IPv6 test results show a different public IPv4 than the modem.

I just tested plugging directly into a windows 10 laptop.

Client was assigned both an IPv6 address and a temporary in the same prefix.

I was able to successfully ping 2620:fe::fe

I can try and get a screenshot and include the ipconfig info. I'm not certain yet why opnsense isn't working.

Another interesting observation is that the DNS servers assigned by the modem are
fd00:976a::9,10

I'm not sure why an ISP would use public ULA address. I'm guessing it's the Netgear providing these.

Let me know what other data I should collect.

Thanks!

I didn't. I will try that.

One thing I also failed to disclose is that the WAN interface is heading into a switch for VLANs. So actually, it's VWAN1 and VWAN2. I'm not sure how this is impacting broadcasts, but as far as the upstream modem devices go, they are untagged. The Opnsense side is a "trunk" port with tags for the 2 WAN VLANs.

I did assign the regular WAN interface, but it is not configured due to the VWANs being what I'm interested in.

I'll try directly connecting my old "networking" laptop that I use for this kind of thing.

Again, thank you.

Thanks for the reply.

The Monitor IP on the Gateway is set to quad 9 (2620:fe::fe) which does respond on WAN1. It doesn't seem to matter what I set this to, I get no ping response outside of the actual opnsense interface address at 2607::

I'm not sure if it has something to do with the service/ISP side, or if it's the hardware, but it doesn't seem to want to work at all. It could be that the "data sim" in the LTE modem is restricted by T-Mobile somehow. Which is funny because IPv4 will NAT through it all day long.

I really appreciate your help and insight. I wouldn't have known about ndproxy otherwise.

Hopefully this post helps others in the future.

If I solve this in the future, I'll update with my findings, but I'm not sure I can fix it, and it will require much deeper troubleshooting.

Maurice thank you for the recommendation to use the gateway monitor to try and determine if the interface config is valid. It's a great time saver.

Unfortunately I've not been able to figure this out. The gateway just will not respond to IPv6 ICMP, no matter what I do.

The WAN2 side is a Netgear LM1200 with t-mobile data sim, in bridge mode. It get's an address in the same /64 as the router interface, but it is a different address. I attempted to configure ndproxy, but that hasn't enabled the interface to work. I will note OPNsense is grabbing a /64 over SLAAC, and the ndproxy config recommends getting a /128.

I'm not sure I can make this work with the current hardware. I may need a different LTE modem.

Some extra info, which may not help:
LTE Modem IPv6
2607:XXXX:YYYY:9f26:70d6:0665:7552:90fd
OPNsense IPv6
2607:XXXX:YYYY:9f26:20d:b9ff:fe45:c4b8/64

WAN2 Interface configuation
IPv4 DHCP
IPv6 SLAAC
Promiscuous mode CHECKED

I've also tried DHCPv6, which picks up the same address as SLAAC.

ndproxy config:
uplink = WAN2
Downlink MAC = WAN2 MAC
uplink address = Link-Local WAN2 GW address


I appreciate the help. I just can't figure it out. I may have to abandon this and try making a 6to4 tunnel on the working v4 address instead.


elyl, your issue of intermittent connection seems more related to location and signal strength, not necessarily OPNsense. I would recommend configuring your WAN interfaces under DHCP client configuration to reject leases from the service IP, for my netgear modem that's 192.168.5.1. This will help avoid picking up a private 192.168.0.0/16 address on the WAN side, instead of a real address. Install the unifi wifiman app on a phone, and start surveying where the best cell signal is. Put your LTE modem in that location. If you can't put it there, run LTE TS9 antennae.

Attempting to verify things, I tried to have opnsense ping from the WAN2 interface. No luck. I tried to ping both the address obtained by the LTE modem (netgear lm1200) and the opnsense interface address. Neither work.

I must not have the correct IPv6 config on the WAN2 interface. I'll try switching it from DHCPv6 to SLAAC, and maybe figure out what t-mobile or the mvno is using for IPv6.

I'm not sure what addresses the modem provides. The status page suggests it obtains a /64, but I don't know for sure. The opnsense interface address and the obtained modem address have the same first 64 bits. So it seems to at least be picking up the network.

I'm still trying to figure out why I get a 6to4 stf interface. Reading up on it here:
https://man.freebsd.org/cgi/man.cgi?query=if_stf&sektion=4&format=html

EDIT: The address acquired by the modem appears to be dynamic. The first 2 quartets are the same, but the third is different.

I'm trying to figure out how to get IPv6 failover working on OPNsense.
My network is configured for dual stack 4 and 6.

WAN1 is Comcast, set to DHCP6 with working prefix deligation, LAN is set to track interface, and IPv6 is working fine.
Clients using SLAAC, and RA is configured "Unmanaged" for the same. I run pi-hole for DNS and local name resolution. All works fine.

WAN2 is t-mobile on an LTE modem, IPv6 on the modem works. The WAN2 interface is set to DHCP6 and getting assigned a /64. Interestingly this connection is also creating a opt5_stf interface, which I've left unassigned, because I don't know what it is.

None of the clients seem to pick up the WAN2 address space. I assume that is because track interface will only use RA for the default gateway. Do I need to set up more RA for another route/subnet?

I have a gateway group for both IPv4 gateways, and IPv6 gateways. The IPv4 failover works fine (no surprise). IPv6 is unable to route once WAN1 goes down.
I can't figure out NPT, the documentation is hazy to me after having configured a few things I thought would work.
pfsense docs mention the target should be /64 (but not the delegated address). How do I know which address that is?
https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
opnsense docs mention nothing of the sort.
https://docs.opnsense.org/manual/nptv6.html#nptv6

What am I supposed to put in NPT config to get this to work? Where do I find the correct information if the interface address isn't correct? Do I get it from upstream on the modem?

Thanks

It really bothers me the answer is "Don't do this" when it could just as easily be "Let's fix it."

In the spirit of open source, I've decided to take it to the FreeBSD kernel people and see if we can get something going.

I guess I'll update this topic if there ever is a fix. My experience with open source projects has been hit or miss though. So we will see if I can get any traction.

Seems like it IS a FreeBSD issue. And seems like all USB Ethernet devices seem to be broken on a kernel level.

So much for my devd fencing idea.

I just found this bug report via reddit:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252165

The reporter outlines the exact symptom.

https://www.reddit.com/r/OPNsenseFirewall/comments/12kv3ns/random_ue0_link_state_change_to_down_ue_link/

And with the duration of this bug, and the current status of the bug report, I have no hope that this will ever be fixed. Time to find a different (more expensive) solution. *sigh*

So I think I've figured out what the problem is, but I have no idea how to actually fix it.

netwait didn't seem to matter or make any difference. I tried adding it to the boot config:

Code Select Expand


root@OPNsense:~ # cat /usr/local/etc/rc.syshook.d/early/21-netwait

echo "NETWAIT"
netwait_enable="YES"
netwait_timeout="10"
netwait_ip="${defaultrouter}"
netwait_if="ue0"


On to my diagnosis/findings

Steps to reproduce:
Plug in USB NIC and uplink.
Configure/Enable interface in the GUI.
Watch the flapping in /var/log/system/latest.log

What I think is actually going on:
I think devd is somehow "fencing" on the USB NIC device/category, being both an ethernet NIC, and a USB. This will happen when the interface is configured in the GUI, but may be subverted by booting with the interface config already enabled. However hotplugging the USB also seems to cause the issue. (I actually had it resolve the issue once as well, the interface was "ON", and I unplugged and replugged it, but this doesn't always work.)

This appears to be an issue with OPNsense, and not FreeBSD. I was able to get into the CLI and manually run

Code Select Expand


root@OPNsense:~ # ifconfig ue0 down
root@OPNsense:~ # ifconfig ue0 up
root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 48.X.X.X
bound to 48.X.X.X -- renewal in 21600 seconds.
^C
root@OPNsense:~ # ifconfig ue0
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN2 (opt3)
        options=80008<VLAN_MTU,LINKSTATE>
        ether 20:7b:d2:d8:29:56
        inet 48.X.X.X netmask 0xffffff00 broadcast 48.255.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Clearly manually configuring the interface from the shell works. FreeBSD seems to be happy about the NIC.

This also does not cause the flapping the the logs. Something about OPNsense devd is causing it to constantly up/down/configure/reconfigure the interface. It will do this in an endless loop.

Not sure where to go next. Maybe I'll get on github with a bug report?

Thanks.

Thanks, but it's not heat. Both of these devices have aluminum casings and feel fine to the touch.

I did find this on the FreeBSD forums. I'll try it out when I have time later.
https://forums.freebsd.org/threads/kernel-upgrade-from-12-2-stable-to-12-4-release-sshd-could-not-bind.90114/#post-620622

Is there a recommended method for settings? Is it appropriate to modify /etc/rc.conf ?

Just as an update. There appears to be issues with these realtek USB NICs, and FreeBSD.

I thought by turning off HW features (Interfaces -> Hardware settings -> Overwrite global settings) for CRC, TSO, and LRO, there was an improvement. But it was shortlived.

I noticed these devices tend to flap UP and DOWN a lot. Perhaps there are kernel driver issues. I thought this would be a cheap and easy way to add a NIC to my platform which is already fully utilized. Unfortunately it doesn't seem to work.

If anyone has any suggestions for getting these USB NICs to function properly, please let me know.

Code Select Expand


<13>1 2024-03-11T23:07:50-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="510"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:53-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="511"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:53-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="512"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="513"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="514"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:55-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="515"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:55-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="516"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:56-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="517"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:56-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="518"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:57-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="519"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:57-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="520"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:58-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="521"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:58-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="522"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:07:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="523"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:07:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="524"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:08:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="525"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:08:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="526"] <6>ue0: link state changed to UP
<13>1 2024-03-11T23:08:08-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="527"] <6>ue0: link state changed to DOWN
<13>1 2024-03-11T23:08:08-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="528"] <6>ue0: link state changed to UP


I tried both an Anker and a Ugreen NIC. Same results.

Code Select Expand


<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="9"] ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0 (disconnected)
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="10"] ure0: at uhub1, port 1, addr 1 (disconnected)
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="11"] rgephy0: detached
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="12"] miibus0: detached
<13>1 2024-03-11T20:23:49-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="13"] ure0: detached
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="14"] ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="15"] ure0 on uhub1
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="16"] ure0: <Realtek USB 10/100/1000 LAN, class 0/0, rev 3.00/31.00, addr 1> on usbus0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="17"] miibus0: <MII bus> on ure0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="18"] rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus0
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="19"] rgephy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
<13>1 2024-03-11T20:23:54-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="20"] <6>ue0: <USB Ethernet> on ure0

<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="1"] ugen0.2: <Realtek USB 10/100/1000 LAN> at usbus0 (disconnected)
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="2"] ure0: at uhub1, port 2, addr 1 (disconnected)
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="3"] rgephy0: detached
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="4"] miibus0: detached
<13>1 2024-03-11T23:03:46-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="5"] ure0: detached
<13>1 2024-03-11T23:03:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="6"] ugen0.2: <ASIX AX88179A> at usbus0
<13>1 2024-03-11T23:03:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="7"] axge0 on uhub1
<13>1 2024-03-11T23:03:59-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="8"] axge0: <NetworkInterface> on usbus0
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="9"] miibus0: <MII bus> on axge0
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="10"] ukphy0: <Generic IEEE 802.3u media interface> PHY 3 on miibus0
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="11"] ukphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
<13>1 2024-03-11T23:04:00-06:00 OPNsense.home.arpa kernel - - [meta sequenceId="12"] <6>ue0: <USB Ethernet> on axge0


I'm having an issue with the second WAN. Seems like the device and NIC work fine. It seems to be able to pick up a DHCP config when the router boots. But if I reconnect the cable while opnsense is running I get continuous dhclient attempts, but no actual response. It's almost as if something about the firewall is preventing DHCP on the "wan2" interface.

Code Select Expand


<13>1 2024-03-10T20:56:08-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)
<13>1 2024-03-10T20:56:09-06:00 OPNsense.home.arpa dhclient 69737 - [meta sequenceId="28"] dhclient-script: Reason PREINIT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 76044 - [meta sequenceId="1"] dhclient-script: Reason TIMEOUT on ue0 executing
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 77695 - [meta sequenceId="2"] dhclient-script: New IP Address (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 78719 - [meta sequenceId="3"] dhclient-script: New Subnet Mask (ue0): 255.255.255.0
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 80979 - [meta sequenceId="4"] dhclient-script: New Broadcast Address (ue0): 48.255.255.255
<13>1 2024-03-10T20:57:27-06:00 OPNsense.home.arpa dhclient 82814 - [meta sequenceId="5"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:28-06:00 OPNsense.home.arpa dhclient 86364 - [meta sequenceId="6"] dhclient-script: New Routers (ue0): 48.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa dhclient 15693 - [meta sequenceId="7"] dhclient-script: Reason FAIL on ue0 executing
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="8"] /usr/local/etc/rc.linkup: ROUTING: entering configure using 'opt3'
<11>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="9"] /usr/local/etc/rc.linkup: ROUTING: not a valid opt3 interface gateway address: 'missing'
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="10"] /usr/local/etc/rc.linkup: ROUTING: configuring inet default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="11"] /usr/local/etc/rc.linkup: ROUTING: keeping inet default route to 76.X.X.X
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="12"] /usr/local/etc/rc.linkup: ROUTING: configuring inet6 default gateway on wan
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="13"] /usr/local/etc/rc.linkup: ROUTING: keeping inet6 default route to fe80::XXXXXX%igb0
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="14"] /usr/local/etc/rc.linkup: plugins_configure monitor (,WAN2_DHCP)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="15"] /usr/local/etc/rc.linkup: plugins_configure monitor (execute task : dpinger_configure_do(,WAN2_DHCP))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="16"] /usr/local/etc/rc.linkup: plugins_configure ipsec (,opt3)
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="17"] /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,opt3))
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="18"] /usr/local/etc/rc.linkup: plugins_configure dhcp ()
<13>1 2024-03-10T20:57:30-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="19"] /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
<13>1 2024-03-10T20:57:31-06:00 OPNsense.home.arpa opnsense 13518 - [meta sequenceId="20"] /usr/local/etc/rc.newwanip: Failed to detect IP for interface opt3
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="21"] /usr/local/etc/rc.linkup: plugins_configure dns ()
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="22"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
<13>1 2024-03-10T20:57:40-06:00 OPNsense.home.arpa opnsense 63307 - [meta sequenceId="23"] /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
<13>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa opnsense 34870 - [meta sequenceId="24"] /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for opt3(ue0)
<27>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="25"] connection closed
<26>1 2024-03-10T20:57:49-06:00 OPNsense.home.arpa dhclient 68039 - [meta sequenceId="26"] exiting.
<13>1 2024-03-10T20:57:50-06:00 OPNsense.home.arpa opnsense 39292 - [meta sequenceId="27"] /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt3(ue0)


It just repeats these attempts in an endless loop until I disable the interface or reboot.

EDIT: Manually running dhclient on the interface gives the same results.

Code Select Expand


root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 14
DHCPDISCOVER on ue0 to 255.255.255.255 port 67 interval 16
My address (48.X.X.X) was re-added
My address (48.X.X.X) was deleted, dhclient exiting


Thank you for your time.

I wanted to put this online somewhere, because my search results were almost entirely unhelpful. I hope this save someone some pain.

If you have a PC Engines APU 2, and you've had it for a while, you MUST update the firmware/BIOS on it to install OPNsense 22.1.

=== Some History (because I want to tell the story) ===
Long ago, in 2015, I tried OPNsense when it first forked, it wasn't for me. I had issues with my multiple VPN setup. I don't know if it was me or the software, but we weren't getting along. I decided to save my self some hours tinkering every week and go back to pfsense.

In December, I was unable to upgrade my existing pfSense from 2.4.5, it seems I encountered a pretty bad (yet common) bug, and the entire package system was corrupted with the wrong versions of things. I tried upgrading it "manually" via USB stick, and it left me with a non-functioning system. Thankfully a downgrade worked, and I had a router again. The corruption left me with a system I couldn't update, and would need to again manually re-install.

I finally had time again, and I've been wanting to try OPNsense again (thanks reddit!), what better time than now? I'm going to have to do a fresh install anyway.

=== Install Problems ===
My hardware is a PC Engines APU 2C4 I got quite some time ago. I think the firmware on it was dated 2016. I may have upgraded it once since I got it.

I downloaded the latest OPNsense. Used belenaEtcher to create a serial USB installer. Took screenshots and a manual backup of my current pfsense config.xml. And planned my migration over a day. I slept on it and felt good to go forward.

I popped the USB into my APU 2C4. Fired up TeraTerm using a spare windows craptop and my trusty USB RS232-Serial cable, and went to town. Powered off the unit, and booted the USB.

This is where the fun starts! After a few minutes of trying to boot and getting a bunch of "Waiting for CAM" messages, it gives up and says "Mounting from ufs:/dev/sdb failed with error 19". And dumps me out to a "mountroot>" command prompt.

It turns out that older versions of the APU firmware don't like using GPT, and the USB 3.0 drivers exacerbate the issue.

=== The Wrong Solution ===
At this point there was a lot of "advice" online to boot setting kern.cam.boot_delay="10000"
This is bad advice! It will make you wait forever for your device to figure out why it can't negotiate a USB 3.0 connection, and it doesn't solve the underlying cause/issue.

=== Problem Solving (or how I stubbornly used Windows) ===
After a bunch of research and looking into BSD and other help forums, I learned about the GPT issue. I finally figured out I really should upgrade the APU firmware, and that would (hopefully) solve the problem.

This is a very good resource on how to use flashrom via software you probably already have installed on your APU!
https://teklager.se/en/knowledge-base/apu-bios-upgrade/

The easiest option would have been to boot back into pfsense, and run flashrom via an SSH session. This didn't work for me because of the corrupt package versioning. I couldn't get the correct versions of packages, and this left me unable to run flashrom. I kept getting 'Shared object "libarchive.so.7" not found, required by "pkg"'. And after using the pkg-static to install flashrom, running it I would get a 'Undefined symbol "fstat@FBSD_1.5"'. These are due to the corrupt packages and versions in pfsense.

Another trip on the merry-go-round! I tried to create a USB 2.0 stick with MBR, TinyCore, and the APU firmware to flash. I couldn't figure out how to make a proper TinyCore USB using belenaEtcher. I'll spare you the details, I just used the PC Engines provided imager.

=== (my) SOLUTION ===
Ultimately, I created a USB 2.0 stick with MBR using Windows. I had to "undo" the GPT partition from a prior OPNsense attempt.
There was a bit of trial an error, the PC Engines imager doesn't partition the drive. I had to make sure it was MBR and FAT32 before starting.
https://docs.microsoft.com/en-us/windows-server/storage/disk-management/change-a-gpt-disk-into-an-mbr-disk
Then ran the PC Engines installer for TinyCore
https://pcengines.ch/howto.htm#TinyCoreLinux
Then loaded the latest APU firmware image onto it. (apu2 v4.15.0.3)
https://pcengines.github.io/

I booted, and ran the flashrom command with no issues.
flashrom -w /media/TINYCORE/apu2_v4.15.0.3.rom -p internal

After that, the etcher created USB with GPT for OPNsense 22.1 worked! No fancy kernel boot parameters required. It didn't take forever to boot either.

=== Conclusion ===
THOU SHALT UPDATE THINE FIRMWARE!

PC Engines is really awesome for continuing to support their systems after all these years. The fact that I could get a firmware that was released just 12 days ago, is mindblowing! I'm really happy with this hardware and the continued support. I'm also happy that it solved the problems. I will definitely be paying attention to firmware updates from now on.

Sorry to drag this up. But I've solved my problem.

It seems that something in my config was incompatible. I finally took the time to do a fresh install and reconfigure everything from scratch. This solved the problem.

I don't know what about the configs are different/incompatible. I can compare them line-by-line, but I don't think this is a very useful exercise at this point.

I will note that NordVPN has updated their tutorial on how to get their VPN running on OPNsense, which is great.

I use a pihole DNS on my local network, and so I don't have unbound running on OPNsense. I also had to disable DNSSEC in pihole to get queries to go through the VPN, because NordVPN is doing forwarding on their end (thus breaking DNSSEC).

I hope this helps someone if they experience the same issue. Not an idea solution, but it's what I had to do.