Messages

Just one final update. Sorry to bump this thread. But I wanted to get the info out there.

I've simply decided to ignore having any IPv6 on the backup LTE via SLAAC. This has simplified the gateway monitoring, routes, and switching.

I've noticed that part of the issue with the LTE modem is the ISP will drop the IPv6 connection if any traffic originates from it that isn't from the assigned /64 SLAAC address. This is part of why NPT wasn't great. I could set up NDP in the future to provide addresses for the SLAAC interface, but that just doesn't seem worth it for the few hours per month the backup is in play.

The prefix issue still exists, but is far less catastrophic to the network. The short times appear to help, but I still notice some Windows clients getting stuck with the old prefix and therefore unable to route IPv6 traffic after the primary WAN is restored. I could probably detect and solve this problem with a script, but I'm pretty busy and my family probably doesn't have the patience for that.

Thanks for the great product and support.

Thanks for the reply. There were a few changes I made and things appear to be more stable. I'll have to wait a bit longer to be sure. Over the last few months I've been having to power cycle the router to fix a "split brain" situation with the networks.

The 2 things that seem to have made a difference.

1 - In verifying my settings, I didn't have any DNS server set in [system] > [settings] > [general]. I've corrected this and set them to the same servers as I have in the gateway monitoring.

2 - I've set more aggressive timeouts for RADV. I'm now using:

Code Select Expand

Minimum Interval = 10
Maximum Interval = 30
AdvPreferredLifetime = 60
AdvRouteLifetime = 90
I've disabled any settings for NPTv6 from the [Firewall] > [NAT], because of my GUA prefix changing. Thanks for your feedback about ULAs. I know there are issues with dual stack networks, and it seems this would be one of those cases where using ULA for NPT would simply result in no IPv6 going out the secondary WAN due to "happy eyeballs" and IPv4 preference.

I'll look into NAT66. I'm also researching using a reserved GUA (like 2000:db8:: ) for NPT, but this would be a hack.

Thanks for the time and feedback.

I'm having some issues with multi-wan failover using IPv6.

WAN is DHCPv6

WAN2 is set to SLAAC via LTE Modem, I'm not as concerned that this doesn't seem to work for ipv6 at the moment. I've been able to get things to work with NPT, but I think I will assign dedicated NAT addresses in the future, because NPT needs to also be updated every time the WAN prefix changes.

LAN is set to "Track Interface"

I'm using radv "Router Advertisements" in the services.
It set to "Assisted" with "Automatic" source address.
I've not set any advanced options, everything is default.

I tested my failover by unplugging the cable from my cable modem. When service was restored, the gateway monitoring functioned, and fail to LTE was fine. Once I plugged it back in, I noticed a lot of delay trying to get to test-ipv6.com

I had the same issue on my phone. I cycled the wifi connection on and off and it solved it.

I can see from a windows client that I still have the old prefix/address.

What can I do to solve this issue? I'd like to have clients properly deprecate/abandon an old address when the WAN flaps.

Here is an example output from ifconfig. The c881 is the new address, and the c800 is the old one.

Code Select Expand

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : home.arpa
   IPv6 Address. . . . . . . . . . . : 2601:281:c881:fb80:1a7a:5927:4cd4:e21b
   IPv6 Address. . . . . . . . . . . : 2601:281:c800:3910:3e2f:a436:d203:d072
   Temporary IPv6 Address. . . . . . : 2601:281:c800:3910:a512:d226:8873:46cb
   Temporary IPv6 Address. . . . . . : 2601:281:c881:fb80:a857:a7c4:21fe:3929
   Link-local IPv6 Address . . . . . : fe80::d1fd:217e:6ec2:961%25
   IPv4 Address. . . . . . . . . . . : 192.168.1.161
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.1
Should I set radv lifetimes to something more aggressive than the defaults?

I haven't opened a bug ticket, but I took the time to write a custom monit rule file. It wasn't difficult. This suffices for now.

Code Select Expand

root@OPNsense:~ # cat /usr/local/etc/monit.opnsense.d/unbound.conf
check program unbound_servfail with path "/usr/bin/drill google.com" timeout 60 seconds
        start program = "/usr/local/sbin/configctl unbound start" with timeout 30 seconds
        stop program = "/usr/local/sbin/configctl unbound stop"
        if content = "SERVFAIL" then alert
        if content = "SERVFAIL" then restart
        if 5 restarts within 5 cycles then unmonitor
Thanks!

https://imgur.com/a/opnsense-test-content-not-allowed-RrRdwPn
When clicking "save" I get the error on the field. "Test DNS_SERVFAIL with type File Content not allowed for this service type"

I had a lot of frustrations with pihole as DNS and DHCP. It didn't really support IPv6 very well, and trying to write rules in opnsense was a nightmare.

I eventually removed pihole entirely. I use Unbound with the hagezi Multi PRO and Threat Intel blocklists. I forward to Dnsmasq which takes care of DHCP and local DNS for both IPv4 and IPv6.

I'm trying to write a set of monit instructions to restart unbound if I start to see "SERVFAIL".
This condition seems to happen in my environment when there are WAN issues or there is a Multi-WAN failover.

It appears that "custom" creates a program service checker, but I'm unable to use the "content" keyword to check the output. The "content" rule type is giving an error that the service is not of type "file".

Monit docs say "content" is supported by the "program" service.
https://mmonit.com/monit/documentation/monit.html#PROGRAM-OUTPUT-CONTENT-TEST

In monit, I have a "service" called "dns_resolve_failure" that has the following config:

Code Select Expand

Type   Custom
Path   "/usr/bin/drill google.com"
Start  "/usr/local/sbin/configctl unbound start"
Stop   "/usr/local/sbin/configctl unbound stop"
Tests  "Nothing Selected"
I have a "service test" called "DNS_SERVFAIL" that looks like this:

Code Select Expand

Condition:  content = "SERVFAIL"
Action:     Restart
The services works fine, I can see the output of drill, but I'm unable to assign a "content" test to it.

I'm running OPNsense 25.7.11_9 currently.

I guess I could set up an "advanced" script in "/usr/local/etc/monit.opnsense.d/", but I was hoping I could do it through the GUI.

If finally works.

Seems to have been something wrong with the interface config.
I unchecked block privates and block bogons, and it's working now.
I'll probably reenable to bogons block at some point if it doesn't break it, but at least it's up and gateway ping is finally responding.

I'm guessing it uses some private local address for something to do with RA and SLAAC from the LM1200.

The final interface config for "VWAN2" is:
Enable [X]
Block private networks [_]
Block bogon networks [_]
IPv4 Configuration Type [DHCP]
IPv6 Configuration Type [SLAAC]
Reject Leases From [192.168.5.1]
Override MTU [X]

Everything else is unset or left to defaults.

I'm sure that I can get NPT working at this point.

Thanks for the support Maurice.

Yes, I'm in the US. I'm not sure if it's using CGNAT, but it probably is. My IPv6 test results show a different public IPv4 than the modem.

I just tested plugging directly into a windows 10 laptop.

Client was assigned both an IPv6 address and a temporary in the same prefix.

I was able to successfully ping 2620:fe::fe

I can try and get a screenshot and include the ipconfig info. I'm not certain yet why opnsense isn't working.

Another interesting observation is that the DNS servers assigned by the modem are
fd00:976a::9,10

I'm not sure why an ISP would use public ULA address. I'm guessing it's the Netgear providing these.

Let me know what other data I should collect.

Thanks!

I didn't. I will try that.

One thing I also failed to disclose is that the WAN interface is heading into a switch for VLANs. So actually, it's VWAN1 and VWAN2. I'm not sure how this is impacting broadcasts, but as far as the upstream modem devices go, they are untagged. The Opnsense side is a "trunk" port with tags for the 2 WAN VLANs.

I did assign the regular WAN interface, but it is not configured due to the VWANs being what I'm interested in.

I'll try directly connecting my old "networking" laptop that I use for this kind of thing.

Again, thank you.

Thanks for the reply.

The Monitor IP on the Gateway is set to quad 9 (2620:fe::fe) which does respond on WAN1. It doesn't seem to matter what I set this to, I get no ping response outside of the actual opnsense interface address at 2607::

I'm not sure if it has something to do with the service/ISP side, or if it's the hardware, but it doesn't seem to want to work at all. It could be that the "data sim" in the LTE modem is restricted by T-Mobile somehow. Which is funny because IPv4 will NAT through it all day long.

I really appreciate your help and insight. I wouldn't have known about ndproxy otherwise.

Hopefully this post helps others in the future.

If I solve this in the future, I'll update with my findings, but I'm not sure I can fix it, and it will require much deeper troubleshooting.

Maurice thank you for the recommendation to use the gateway monitor to try and determine if the interface config is valid. It's a great time saver.

Unfortunately I've not been able to figure this out. The gateway just will not respond to IPv6 ICMP, no matter what I do.

The WAN2 side is a Netgear LM1200 with t-mobile data sim, in bridge mode. It get's an address in the same /64 as the router interface, but it is a different address. I attempted to configure ndproxy, but that hasn't enabled the interface to work. I will note OPNsense is grabbing a /64 over SLAAC, and the ndproxy config recommends getting a /128.

I'm not sure I can make this work with the current hardware. I may need a different LTE modem.

Some extra info, which may not help:
LTE Modem IPv6
2607:XXXX:YYYY:9f26:70d6:0665:7552:90fd
OPNsense IPv6
2607:XXXX:YYYY:9f26:20d:b9ff:fe45:c4b8/64

WAN2 Interface configuation
IPv4 DHCP
IPv6 SLAAC
Promiscuous mode CHECKED

I've also tried DHCPv6, which picks up the same address as SLAAC.

ndproxy config:
uplink = WAN2
Downlink MAC = WAN2 MAC
uplink address = Link-Local WAN2 GW address


I appreciate the help. I just can't figure it out. I may have to abandon this and try making a 6to4 tunnel on the working v4 address instead.


elyl, your issue of intermittent connection seems more related to location and signal strength, not necessarily OPNsense. I would recommend configuring your WAN interfaces under DHCP client configuration to reject leases from the service IP, for my netgear modem that's 192.168.5.1. This will help avoid picking up a private 192.168.0.0/16 address on the WAN side, instead of a real address. Install the unifi wifiman app on a phone, and start surveying where the best cell signal is. Put your LTE modem in that location. If you can't put it there, run LTE TS9 antennae.

Attempting to verify things, I tried to have opnsense ping from the WAN2 interface. No luck. I tried to ping both the address obtained by the LTE modem (netgear lm1200) and the opnsense interface address. Neither work.

I must not have the correct IPv6 config on the WAN2 interface. I'll try switching it from DHCPv6 to SLAAC, and maybe figure out what t-mobile or the mvno is using for IPv6.

I'm not sure what addresses the modem provides. The status page suggests it obtains a /64, but I don't know for sure. The opnsense interface address and the obtained modem address have the same first 64 bits. So it seems to at least be picking up the network.

I'm still trying to figure out why I get a 6to4 stf interface. Reading up on it here:
https://man.freebsd.org/cgi/man.cgi?query=if_stf&sektion=4&format=html

EDIT: The address acquired by the modem appears to be dynamic. The first 2 quartets are the same, but the third is different.

I'm trying to figure out how to get IPv6 failover working on OPNsense.
My network is configured for dual stack 4 and 6.

WAN1 is Comcast, set to DHCP6 with working prefix deligation, LAN is set to track interface, and IPv6 is working fine.
Clients using SLAAC, and RA is configured "Unmanaged" for the same. I run pi-hole for DNS and local name resolution. All works fine.

WAN2 is t-mobile on an LTE modem, IPv6 on the modem works. The WAN2 interface is set to DHCP6 and getting assigned a /64. Interestingly this connection is also creating a opt5_stf interface, which I've left unassigned, because I don't know what it is.

None of the clients seem to pick up the WAN2 address space. I assume that is because track interface will only use RA for the default gateway. Do I need to set up more RA for another route/subnet?

I have a gateway group for both IPv4 gateways, and IPv6 gateways. The IPv4 failover works fine (no surprise). IPv6 is unable to route once WAN1 goes down.
I can't figure out NPT, the documentation is hazy to me after having configured a few things I thought would work.
pfsense docs mention the target should be /64 (but not the delegated address). How do I know which address that is?
https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
opnsense docs mention nothing of the sort.
https://docs.opnsense.org/manual/nptv6.html#nptv6

What am I supposed to put in NPT config to get this to work? Where do I find the correct information if the interface address isn't correct? Do I get it from upstream on the modem?

Thanks

It really bothers me the answer is "Don't do this" when it could just as easily be "Let's fix it."

In the spirit of open source, I've decided to take it to the FreeBSD kernel people and see if we can get something going.

I guess I'll update this topic if there ever is a fix. My experience with open source projects has been hit or miss though. So we will see if I can get any traction.