Messages

I haven't opened a bug ticket, but I took the time to write a custom monit rule file. It wasn't difficult. This suffices for now.

Code Select Expand

root@OPNsense:~ # cat /usr/local/etc/monit.opnsense.d/unbound.conf
check program unbound_servfail with path "/usr/bin/drill google.com" timeout 60 seconds
        start program = "/usr/local/sbin/configctl unbound start" with timeout 30 seconds
        stop program = "/usr/local/sbin/configctl unbound stop"
        if content = "SERVFAIL" then alert
        if content = "SERVFAIL" then restart
        if 5 restarts within 5 cycles then unmonitor
Thanks!

https://imgur.com/a/opnsense-test-content-not-allowed-RrRdwPn
When clicking "save" I get the error on the field. "Test DNS_SERVFAIL with type File Content not allowed for this service type"

I had a lot of frustrations with pihole as DNS and DHCP. It didn't really support IPv6 very well, and trying to write rules in opnsense was a nightmare.

I eventually removed pihole entirely. I use Unbound with the hagezi Multi PRO and Threat Intel blocklists. I forward to Dnsmasq which takes care of DHCP and local DNS for both IPv4 and IPv6.

I'm trying to write a set of monit instructions to restart unbound if I start to see "SERVFAIL".
This condition seems to happen in my environment when there are WAN issues or there is a Multi-WAN failover.

It appears that "custom" creates a program service checker, but I'm unable to use the "content" keyword to check the output. The "content" rule type is giving an error that the service is not of type "file".

Monit docs say "content" is supported by the "program" service.
https://mmonit.com/monit/documentation/monit.html#PROGRAM-OUTPUT-CONTENT-TEST

In monit, I have a "service" called "dns_resolve_failure" that has the following config:

Code Select Expand

Type   Custom
Path   "/usr/bin/drill google.com"
Start  "/usr/local/sbin/configctl unbound start"
Stop   "/usr/local/sbin/configctl unbound stop"
Tests  "Nothing Selected"
I have a "service test" called "DNS_SERVFAIL" that looks like this:

Code Select Expand

Condition:  content = "SERVFAIL"
Action:     Restart
The services works fine, I can see the output of drill, but I'm unable to assign a "content" test to it.

I'm running OPNsense 25.7.11_9 currently.

I guess I could set up an "advanced" script in "/usr/local/etc/monit.opnsense.d/", but I was hoping I could do it through the GUI.

If finally works.

Seems to have been something wrong with the interface config.
I unchecked block privates and block bogons, and it's working now.
I'll probably reenable to bogons block at some point if it doesn't break it, but at least it's up and gateway ping is finally responding.

I'm guessing it uses some private local address for something to do with RA and SLAAC from the LM1200.

The final interface config for "VWAN2" is:
Enable [X]
Block private networks [_]
Block bogon networks [_]
IPv4 Configuration Type [DHCP]
IPv6 Configuration Type [SLAAC]
Reject Leases From [192.168.5.1]
Override MTU [X]

Everything else is unset or left to defaults.

I'm sure that I can get NPT working at this point.

Thanks for the support Maurice.

Yes, I'm in the US. I'm not sure if it's using CGNAT, but it probably is. My IPv6 test results show a different public IPv4 than the modem.

I just tested plugging directly into a windows 10 laptop.

Client was assigned both an IPv6 address and a temporary in the same prefix.

I was able to successfully ping 2620:fe::fe

I can try and get a screenshot and include the ipconfig info. I'm not certain yet why opnsense isn't working.

Another interesting observation is that the DNS servers assigned by the modem are
fd00:976a::9,10

I'm not sure why an ISP would use public ULA address. I'm guessing it's the Netgear providing these.

Let me know what other data I should collect.

Thanks!

I didn't. I will try that.

One thing I also failed to disclose is that the WAN interface is heading into a switch for VLANs. So actually, it's VWAN1 and VWAN2. I'm not sure how this is impacting broadcasts, but as far as the upstream modem devices go, they are untagged. The Opnsense side is a "trunk" port with tags for the 2 WAN VLANs.

I did assign the regular WAN interface, but it is not configured due to the VWANs being what I'm interested in.

I'll try directly connecting my old "networking" laptop that I use for this kind of thing.

Again, thank you.

Thanks for the reply.

The Monitor IP on the Gateway is set to quad 9 (2620:fe::fe) which does respond on WAN1. It doesn't seem to matter what I set this to, I get no ping response outside of the actual opnsense interface address at 2607::

I'm not sure if it has something to do with the service/ISP side, or if it's the hardware, but it doesn't seem to want to work at all. It could be that the "data sim" in the LTE modem is restricted by T-Mobile somehow. Which is funny because IPv4 will NAT through it all day long.

I really appreciate your help and insight. I wouldn't have known about ndproxy otherwise.

Hopefully this post helps others in the future.

If I solve this in the future, I'll update with my findings, but I'm not sure I can fix it, and it will require much deeper troubleshooting.

Maurice thank you for the recommendation to use the gateway monitor to try and determine if the interface config is valid. It's a great time saver.

Unfortunately I've not been able to figure this out. The gateway just will not respond to IPv6 ICMP, no matter what I do.

The WAN2 side is a Netgear LM1200 with t-mobile data sim, in bridge mode. It get's an address in the same /64 as the router interface, but it is a different address. I attempted to configure ndproxy, but that hasn't enabled the interface to work. I will note OPNsense is grabbing a /64 over SLAAC, and the ndproxy config recommends getting a /128.

I'm not sure I can make this work with the current hardware. I may need a different LTE modem.

Some extra info, which may not help:
LTE Modem IPv6
2607:XXXX:YYYY:9f26:70d6:0665:7552:90fd
OPNsense IPv6
2607:XXXX:YYYY:9f26:20d:b9ff:fe45:c4b8/64

WAN2 Interface configuation
IPv4 DHCP
IPv6 SLAAC
Promiscuous mode CHECKED

I've also tried DHCPv6, which picks up the same address as SLAAC.

ndproxy config:
uplink = WAN2
Downlink MAC = WAN2 MAC
uplink address = Link-Local WAN2 GW address


I appreciate the help. I just can't figure it out. I may have to abandon this and try making a 6to4 tunnel on the working v4 address instead.


elyl, your issue of intermittent connection seems more related to location and signal strength, not necessarily OPNsense. I would recommend configuring your WAN interfaces under DHCP client configuration to reject leases from the service IP, for my netgear modem that's 192.168.5.1. This will help avoid picking up a private 192.168.0.0/16 address on the WAN side, instead of a real address. Install the unifi wifiman app on a phone, and start surveying where the best cell signal is. Put your LTE modem in that location. If you can't put it there, run LTE TS9 antennae.

Attempting to verify things, I tried to have opnsense ping from the WAN2 interface. No luck. I tried to ping both the address obtained by the LTE modem (netgear lm1200) and the opnsense interface address. Neither work.

I must not have the correct IPv6 config on the WAN2 interface. I'll try switching it from DHCPv6 to SLAAC, and maybe figure out what t-mobile or the mvno is using for IPv6.

I'm not sure what addresses the modem provides. The status page suggests it obtains a /64, but I don't know for sure. The opnsense interface address and the obtained modem address have the same first 64 bits. So it seems to at least be picking up the network.

I'm still trying to figure out why I get a 6to4 stf interface. Reading up on it here:
https://man.freebsd.org/cgi/man.cgi?query=if_stf&sektion=4&format=html

EDIT: The address acquired by the modem appears to be dynamic. The first 2 quartets are the same, but the third is different.

I'm trying to figure out how to get IPv6 failover working on OPNsense.
My network is configured for dual stack 4 and 6.

WAN1 is Comcast, set to DHCP6 with working prefix deligation, LAN is set to track interface, and IPv6 is working fine.
Clients using SLAAC, and RA is configured "Unmanaged" for the same. I run pi-hole for DNS and local name resolution. All works fine.

WAN2 is t-mobile on an LTE modem, IPv6 on the modem works. The WAN2 interface is set to DHCP6 and getting assigned a /64. Interestingly this connection is also creating a opt5_stf interface, which I've left unassigned, because I don't know what it is.

None of the clients seem to pick up the WAN2 address space. I assume that is because track interface will only use RA for the default gateway. Do I need to set up more RA for another route/subnet?

I have a gateway group for both IPv4 gateways, and IPv6 gateways. The IPv4 failover works fine (no surprise). IPv6 is unable to route once WAN1 goes down.
I can't figure out NPT, the documentation is hazy to me after having configured a few things I thought would work.
pfsense docs mention the target should be /64 (but not the delegated address). How do I know which address that is?
https://docs.netgate.com/pfsense/en/latest/recipes/multiwan-ipv6.html
opnsense docs mention nothing of the sort.
https://docs.opnsense.org/manual/nptv6.html#nptv6

What am I supposed to put in NPT config to get this to work? Where do I find the correct information if the interface address isn't correct? Do I get it from upstream on the modem?

Thanks

It really bothers me the answer is "Don't do this" when it could just as easily be "Let's fix it."

In the spirit of open source, I've decided to take it to the FreeBSD kernel people and see if we can get something going.

I guess I'll update this topic if there ever is a fix. My experience with open source projects has been hit or miss though. So we will see if I can get any traction.

Seems like it IS a FreeBSD issue. And seems like all USB Ethernet devices seem to be broken on a kernel level.

So much for my devd fencing idea.

I just found this bug report via reddit:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252165

The reporter outlines the exact symptom.

https://www.reddit.com/r/OPNsenseFirewall/comments/12kv3ns/random_ue0_link_state_change_to_down_ue_link/

And with the duration of this bug, and the current status of the bug report, I have no hope that this will ever be fixed. Time to find a different (more expensive) solution. *sigh*

So I think I've figured out what the problem is, but I have no idea how to actually fix it.

netwait didn't seem to matter or make any difference. I tried adding it to the boot config:

Code Select Expand


root@OPNsense:~ # cat /usr/local/etc/rc.syshook.d/early/21-netwait

echo "NETWAIT"
netwait_enable="YES"
netwait_timeout="10"
netwait_ip="${defaultrouter}"
netwait_if="ue0"


On to my diagnosis/findings

Steps to reproduce:
Plug in USB NIC and uplink.
Configure/Enable interface in the GUI.
Watch the flapping in /var/log/system/latest.log

What I think is actually going on:
I think devd is somehow "fencing" on the USB NIC device/category, being both an ethernet NIC, and a USB. This will happen when the interface is configured in the GUI, but may be subverted by booting with the interface config already enabled. However hotplugging the USB also seems to cause the issue. (I actually had it resolve the issue once as well, the interface was "ON", and I unplugged and replugged it, but this doesn't always work.)

This appears to be an issue with OPNsense, and not FreeBSD. I was able to get into the CLI and manually run

Code Select Expand


root@OPNsense:~ # ifconfig ue0 down
root@OPNsense:~ # ifconfig ue0 up
root@OPNsense:~ # dhclient -d ue0
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 48.X.X.X
bound to 48.X.X.X -- renewal in 21600 seconds.
^C
root@OPNsense:~ # ifconfig ue0
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: WAN2 (opt3)
        options=80008<VLAN_MTU,LINKSTATE>
        ether 20:7b:d2:d8:29:56
        inet 48.X.X.X netmask 0xffffff00 broadcast 48.255.255.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Clearly manually configuring the interface from the shell works. FreeBSD seems to be happy about the NIC.

This also does not cause the flapping the the logs. Something about OPNsense devd is causing it to constantly up/down/configure/reconfigure the interface. It will do this in an endless loop.

Not sure where to go next. Maybe I'll get on github with a bug report?

Thanks.

Thanks, but it's not heat. Both of these devices have aluminum casings and feel fine to the touch.

I did find this on the FreeBSD forums. I'll try it out when I have time later.
https://forums.freebsd.org/threads/kernel-upgrade-from-12-2-stable-to-12-4-release-sshd-could-not-bind.90114/#post-620622

Is there a recommended method for settings? Is it appropriate to modify /etc/rc.conf ?