IPv6 WAN Failover - NPT Help

[ciaduck]

I'm trying to figure out how to get IPv6 failover working on OPNsense.
My network is configured for dual stack 4 and 6.

WAN1 is Comcast, set to DHCP6 with working prefix deligation, LAN is set to track interface, and IPv6 is working fine.
Clients using SLAAC, and RA is configured "Unmanaged" for the same. I run pi-hole for DNS and local name resolution. All works fine.

WAN2 is t-mobile on an LTE modem, IPv6 on the modem works. The WAN2 interface is set to DHCP6 and getting assigned a /64. Interestingly this connection is also creating a opt5_stf interface, which I've left unassigned, because I don't know what it is.

None of the clients seem to pick up the WAN2 address space. I assume that is because track interface will only use RA for the default gateway. Do I need to set up more RA for another route/subnet?

I have a gateway group for both IPv4 gateways, and IPv6 gateways. The IPv4 failover works fine (no surprise). IPv6 is unable to route once WAN1 goes down.
I can't figure out NPT, the documentation is hazy to me after having configured a few things I thought would work.
pfsense docs mention the target should be /64 (but not the delegated address). How do I know which address that is?
https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
opnsense docs mention nothing of the sort.
https://docs.opnsense.org/manual/nptv6.html#nptv6

What am I supposed to put in NPT config to get this to work? Where do I find the correct information if the interface address isn't correct? Do I get it from upstream on the modem?

Thanks

[ciaduck]

Attempting to verify things, I tried to have opnsense ping from the WAN2 interface. No luck. I tried to ping both the address obtained by the LTE modem (netgear lm1200) and the opnsense interface address. Neither work.

I must not have the correct IPv6 config on the WAN2 interface. I'll try switching it from DHCPv6 to SLAAC, and maybe figure out what t-mobile or the mvno is using for IPv6.

I'm not sure what addresses the modem provides. The status page suggests it obtains a /64, but I don't know for sure. The opnsense interface address and the obtained modem address have the same first 64 bits. So it seems to at least be picking up the network.

I'm still trying to figure out why I get a 6to4 stf interface. Reading up on it here:
https://man.freebsd.org/cgi/man.cgi?query=if_stf&sektion=4&format=html

EDIT: The address acquired by the modem appears to be dynamic. The first 2 quartets are the same, but the third is different.

[Maurice]

The LTE modem almost certainly uses SLAAC, not DHCPv6. The easiest way to test whether the WAN2 IPv6 connection works at all is to configure gateway monitoring with a public monitor IP.

A single LAN interface can only track a single WAN interface, so you'll indeed need NPT for WAN2.

But NPT only works with a static internal prefix. You'll have to configure the LAN interface with a static address - choose a /64 from the prefix delegated by Comcast. Of course this only works if the delegated prefix is somewhat static. If it changes frequently, you're pretty much out of luck.

The NPT configuration is straight forward: Interface is WAN2 and internal prefix is your static LAN prefix (/64). You don't need to specify an external prefix, the WAN2 prefix (/64 advertised by the LTE modem) will be used automatically.
You'll also have to install and configure the Ndproxy plugin to be able to use the WAN2 /64 prefix: Uplink interface is WAN2, downlink MAC address is the WAN2 MAC address and uplink IPv6 address is the WAN2 gateway address (the LTE modem's link-local address).


Cheers
Maurice

[elyl]

It seems you're trying to do something similar to me, but are a bit further along.  I am also trying to have WAN failover to a T-mobile modem, but I get very patchy IPv4/v6 connectivity on the T-mobile side, in fact it rarely works.

What modem are you using to connect (I'm using Quectel RM520 in an ethernet enclosure) and do you have it set up in 'bridge' mode?  How is your WAN2 interface configured (DCHP/DHCPv6)?
I get a 192.0.0.2/27 address for IPv4 from the modem, and sometimes I eventually get an IPv6 address, but my gateway monitoring has 100% loss on the IPv4, and it doesn't even try to ping the IPv6 monitoring address.

[ciaduck]

Maurice thank you for the recommendation to use the gateway monitor to try and determine if the interface config is valid. It's a great time saver.

Unfortunately I've not been able to figure this out. The gateway just will not respond to IPv6 ICMP, no matter what I do.

The WAN2 side is a Netgear LM1200 with t-mobile data sim, in bridge mode. It get's an address in the same /64 as the router interface, but it is a different address. I attempted to configure ndproxy, but that hasn't enabled the interface to work. I will note OPNsense is grabbing a /64 over SLAAC, and the ndproxy config recommends getting a /128.

I'm not sure I can make this work with the current hardware. I may need a different LTE modem.

Some extra info, which may not help:
LTE Modem IPv6
2607:XXXX:YYYY:9f26:70d6:0665:7552:90fd
OPNsense IPv6
2607:XXXX:YYYY:9f26:20d:b9ff:fe45:c4b8/64

WAN2 Interface configuation
IPv4 DHCP
IPv6 SLAAC
Promiscuous mode CHECKED

I've also tried DHCPv6, which picks up the same address as SLAAC.

ndproxy config:
uplink = WAN2
Downlink MAC = WAN2 MAC
uplink address = Link-Local WAN2 GW address


I appreciate the help. I just can't figure it out. I may have to abandon this and try making a 6to4 tunnel on the working v4 address instead.


elyl, your issue of intermittent connection seems more related to location and signal strength, not necessarily OPNsense. I would recommend configuring your WAN interfaces under DHCP client configuration to reject leases from the service IP, for my netgear modem that's 192.168.5.1. This will help avoid picking up a private 192.168.0.0/16 address on the WAN side, instead of a real address. Install the unifi wifiman app on a phone, and start surveying where the best cell signal is. Put your LTE modem in that location. If you can't put it there, run LTE TS9 antennae.

[Maurice]

For gateway monitoring, do not ping the gateway address, but explicitly specify a public IP address ("Monitor IP") in the gateway's settings. Make sure to specify an address which is known to respond to pings.

NPT and ndproxy are only relevant for the LAN clients. It is not required for OPNsense's own IPv6 Internet access. So make sure gateway monitoring works before working on NPT and ndproxy.

Enabling DHCPv6 doesn't disable SLAAC. If you don't get an additional address with DHCPv6, you should use the SLAAC setting.

[ciaduck]

Thanks for the reply.

The Monitor IP on the Gateway is set to quad 9 (2620:fe::fe) which does respond on WAN1. It doesn't seem to matter what I set this to, I get no ping response outside of the actual opnsense interface address at 2607::

I'm not sure if it has something to do with the service/ISP side, or if it's the hardware, but it doesn't seem to want to work at all. It could be that the "data sim" in the LTE modem is restricted by T-Mobile somehow. Which is funny because IPv4 will NAT through it all day long.

I really appreciate your help and insight. I wouldn't have known about ndproxy otherwise.

Hopefully this post helps others in the future.

If I solve this in the future, I'll update with my findings, but I'm not sure I can fix it, and it will require much deeper troubleshooting.

[Maurice]

Did you test whether IPv6 Internet connectivity works when connecting the LM1200 directly to a different device, like a PC?

[ciaduck]

I didn't. I will try that.

One thing I also failed to disclose is that the WAN interface is heading into a switch for VLANs. So actually, it's VWAN1 and VWAN2. I'm not sure how this is impacting broadcasts, but as far as the upstream modem devices go, they are untagged. The Opnsense side is a "trunk" port with tags for the 2 WAN VLANs.

I did assign the regular WAN interface, but it is not configured due to the VWANs being what I'm interested in.

I'll try directly connecting my old "networking" laptop that I use for this kind of thing.

Again, thank you.

[elyl]

elyl, your issue of intermittent connection seems more related to location and signal strength, not necessarily OPNsense. I would recommend configuring your WAN interfaces under DHCP client configuration to reject leases from the service IP, for my netgear modem that's 192.168.5.1. This will help avoid picking up a private 192.168.0.0/16 address on the WAN side, instead of a real address. Install the unifi wifiman app on a phone, and start surveying where the best cell signal is. Put your LTE modem in that location. If you can't put it there, run LTE TS9 antennae.

Issue is definitely not signal strength, I can connect into the 5G modem itself and see good signal.  I am only getting a 192.0.0.2 address given to me by the modem, which is CGNAT (i.e. not 192.168 local IP).  Are you saying you are getting an actual public IPv4 address on T-Mobile?  Are you in the US?

[ciaduck]

Yes, I'm in the US. I'm not sure if it's using CGNAT, but it probably is. My IPv6 test results show a different public IPv4 than the modem.

I just tested plugging directly into a windows 10 laptop.

Client was assigned both an IPv6 address and a temporary in the same prefix.

I was able to successfully ping 2620:fe::fe

I can try and get a screenshot and include the ipconfig info. I'm not certain yet why opnsense isn't working.

Another interesting observation is that the DNS servers assigned by the modem are
fd00:976a::9,10

I'm not sure why an ISP would use public ULA address. I'm guessing it's the Netgear providing these.

Let me know what other data I should collect.

Thanks!

[Maurice]

I was able to successfully ping 2620:fe::fe

Then gateway monitoring should "just work" in OPNsense.

Let me know what other data I should collect.

I'd start with connecting the LM1200 to a fresh OPNsense test system, then go from there.

[ciaduck]

If finally works.

Seems to have been something wrong with the interface config.
I unchecked block privates and block bogons, and it's working now.
I'll probably reenable to bogons block at some point if it doesn't break it, but at least it's up and gateway ping is finally responding.

I'm guessing it uses some private local address for something to do with RA and SLAAC from the LM1200.

The final interface config for "VWAN2" is:
Enable [X]
Block private networks [_]
Block bogon networks [_]
IPv4 Configuration Type [DHCP]
IPv6 Configuration Type [SLAAC]
Reject Leases From [192.168.5.1]
Override MTU [X]

Everything else is unset or left to defaults.

I'm sure that I can get NPT working at this point.

Thanks for the support Maurice.