Messages

I seem to have similar issues. The firewall seems to be still up & running, but it seems to shut out everything. The issue reminds me of the "new" startup behavior with divert-to rules: all traffic is dropped until the Suricata service is up & running. But this is happening after a day of uptime and the service (probably) up. In the suricata logs I found these errors:

Error
suricata
[100216] <Error> -- thread W-8000 failed

Warning
suricata
[101690] <Warning> -- Write to ipfw divert socket failed: No buffer space available


I'm not sure what buffer space ran out. mbufs seemed to be fine when checking the health graph in reporting. I'm running with kern.ipc.nmbclusters = 1000000

Unfortunately I just upgraded the system on the weekend from the rock solid 25.7.11. I also did the rules migration and migrated Suricata to the new divert-to functionality. So many moving parts changed in just a few days.
To me the problem "feels" to be firewall related so my first mitigation attempt is to revert the divert-to changes back to netmap for now.

I'm using a Protectli FW2B on CoreBoot with an Intel Celeron J3060

Thanks everybody for the useful input.

I want to address a few questions and comments.

Wouldn't it still be better to run dual stack for a few more years until IPv6 penetration reached the last isolated IPv4-only services? Dual stack is well documented and just works for most setups without much oddities.

Im just curious why running IPv6 only with a translation to IPv4 is even worth it. I don't negate the hobbyist aspect of it though.

Though if the tools do not offer what you need, why not wait for a while longer?

Well, yes and no.
I'm currently running Dual Stack and yes it works. I have 2 issues with DS:

• Running DS is much more complicated than just running IPv6 only. IPv6-only is kinda simple really. Running DS is also much more complicated than just running IPv4 only. IPv4 is what most (hobby) admins are grown up with and are using since day 1.
  But running both IPv4 and IPv6 in DS requires extra knowledge and efforts. And I feel that the DS overhead has become a boon of IPv6 adoption by now. Most hobby admins probably stick with IPv4 coz that's what they know - unless they are enthusiasts.
• Running DS is no longer that easy. Many ISPs do no longer offer DS, because they lack the IPv4 addresses. Handing out precious IPv4 to retail customers is getting rare and is only an option to old "ISP nobility" which has been around since the early days of the internet and thus owning enough IPv4 address space. Most newer ISPs just cannot provide IPv4 to retail customers. It's not even an option, if you are willing to pay extra. What you get instead is DS-Lite. DS-Lite gives you a proper global IPv6 prefix and a CGNAT IPv4 address in 100.64.0.0/16 space. This private IPv4 is then NATed by the ISP with NAT44 to a few public IPv4 addresses it owns. CGNAT is the worst IMHO. Coming from DS, switching to DS-Lite is a big step backwards. Personally I think it might be more beneficial to "flight forward" in this situation, forget about the crappy CGNAT IPv4 and just use IPv6 only. I'd happily trade CGNAT for NAT64

But I'd say it's up to the MNOs to add IPv6 support to their ePDGs.

Yes, I agree that would be the best solution. But I believe there must be some obstacle preventing it even though they try to adopt IPv6 fast. I do not know much about IPSec, but I heard them say "Keep your IPv4 and your IPv6 IPSec separate". If this is true, then they need to run the entire ePDG tunneling twice over. Yet another example of why DS is cumbersome and avoided.

I've experimented with WiFi Calling over NAT64 in the past and never got it to work. If anyone had success with this (with any NAT64), I'd be interested.

Ok, that's a bummer. I was assuming the native NAT64 solutions may support ESP. Which NAT64 implementation were you using?

Again, I do not know much about IPSec, but in ESP tunnel mode I believe there should be 2 IP headers with addresses to be exchanged. An inner one of the original packet that is to be tunneled, and an outer one that is added by IPSec for the sake of encapsulation and transport. I'd assume that the outer header is typically NATed ok, but the inner one is probably omitted, if there is no explicit support for ESP.

Hi,

I'm currently experimenting with IPv6-only networks. I might soon get a DS-Lite connection and consider finally getting rid of IPv4 (except for some niches) and run an IPv6-preferred network.
For testing I have NAT64 set up with the Tayga plugin and DNS64 deployed via Unbound. This setup works ok. The fact that many clients on Android, iOS and recent Windows support 464XLAT by now helps to mitigate issues with IPv4 literals that can not be addressed by DNS64. DHCP option 108 is already available and PREF64 support is on the way (I think radvd 2.20 supports it already, only UI support is missing). All things considered, I believe NAT64 could be a viable IPv4 transitioning solution. There is really only one single issue with my tests: IPSec

Why is IPSec important? Well, other than the fact that most corporate VPN solutions use IPSec, the main reason IMHO is VOWIFI aka Wifi Calling. VOWIFI allows handheld devices to extend the mobile network via a WIFI network. This is achieved by establishing an IPSec connection to an Evolved Packet Data Gateway (ePDG) of the mobile provider via the internet. This essentially brings mobile services to where no mobile reception is. I'm sure most of you know Wifi Calling. IMHO a killer feature not to be missed.
The IPSec mode used by the VPN connection to the ePDG is tunnel mode with the ESP protocol and UDP encapsulation (NAT-T). Unfortunately, the ePDGs seem to be using IPv4 exclusively (see https://www.netify.ai/resources/mobile-gateways). I do not know why there is still no support for IPv6, even though mobile providers feel the pain of IPv4 address space running out. I also do not know of any plans to change that in the near future. So one can assume Wifi Calling with IPv6 will require NAT64 for the time being.

Now, the good thing is, NAT64 in general should be able to support IPSec with the ESP protocol (unlike IPSec with the AH protocol). However, NAT64 via Tayga does not. In my tests I was not able to use IPSec and in extension Wifi Calling. As far as I can tell Tayga simply does not have support for protocol 50 (ESP) - only 1 (ICMP), 6 (TCP) and 17 (UDP). I believe support for rewriting ESP headers could just be added, but it's not there for now.

TL;DR: NAT64 via Tayga is incomplete.

Which brings me to the actual question: What is the current state of "native" NAT64 support on OpnSense? Are there any plans to support NAT64 besides the Tayga plugin? I could not find anything in the roadmap about it.
I'd assume using Jool is out of the question being a Linux kernel module.
FreeBSDs own IPFW has support for NAT64. The original PF on OpenBSD has support for NAT64, too. The FreeBSD port of PF however does not have NAT64 support AFAIK, because it was forked before NAT64 support was added.
PFSense seems to have gotten native support for NAT64 recently, but I do not know how it is implemented there.

AFAIK it uses an IPSsc VPN. Check UDP ports 500 and 4500.

iXSystems has released a new TrueNAS version 13.0-U6.2 [1] with the FreeBSD fix back ported to 13.0 [2]


[1] https://www.truenas.com/docs/core/13.0/gettingstarted/corereleasenotes/#130-u62
[2] https://ixsystems.atlassian.net/jira/software/c/projects/NAS/issues/NAS-129828?jql=project%20%3D%20%22NAS%22%20ORDER%20BY%20created%20DESC

IPv6 is not CGN'ed, coz unlike IPv4, there is no shortage of IPv6 address blocks.

Obviously I do not have many Apple devices in my network then :D

Thanks Patrick. I do understand that use case, but in such a case you could explicitly configure the client resolver to make use of the ndots option. The ndots option in the resolve.conf[1] defines the dots threshold for appending the search domain. The default is 1 meaning "if there are less than 1 dots, i.e. none, then append the search domain". In a scenario like yours, setting ndots throughout the network to 2 should help. Do Macs have a resolve.conf? For Windows there's a registry entry to configure[2] the dots threshold.

Still, that does not help to explain the invalid queries. The nts.ntstime.de query was not done by an Apple device. It was done by my OpnSense instance, which should use the ndots default value of 1.

Code Select Expand


#cat /etc/resolv.conf
domain home.lan
nameserver 127.0.0.1
search home.lan


[1] https://man.freebsd.org/cgi/man.cgi?resolv.conf
[2] https://admx.help/?Category=Windows_10_2016&Policy=Microsoft.Policies.DNSClient::DNS_AppendToMultiLabelName

EDIT: inline links changed coz not working

Hi,

maybe you can help me understand search domains. I don't seem to get it.

Here's the relevant description of my setup:

• I'm using OpnSense and have a home domain home.lan (I explicitly tried to avoid .local used by mDNS to not get into some sort of trouble)
• OpnSense hosts a network-facing, resolving and caching Unbound instance.
• The Unbound instance handles all the resolving of dynamically assigned devices for home.lan.
• The Unbound instance delegates resolving of all statically assigned devices to a local Bind instance via a stub zone.
• The local Bind instance is private and provides home.lan records to Unbound for all devices with static IPs.
• DHCP is providing the search domain home.lan via option 119.
• DHCP is providing the domain home.lan via option 15.
• Devices with static IPs have the search domain and domain manually set to home.lan (that includes the OpnSense instance)

What is a search domain?
As I understand it, search domain is a convenience feature that "interpolates" host names into FQDNs. It tries to detect, if a given domain string is a proper FQDN or just a host name. In the latter case it creates a FQDN by appending the search domain. This turns a mere host name into something known to DNS that can actually be used for addressing.
For example: Entering the host name opnsense into a ping will "interpolate" into opnsense.home.lan which is a proper domain name known to DNS.

Code Select Expand

#ping -v opnsense
ping: sock4.fd: 3 (socktype: SOCK_RAW), sock6.fd: 4 (socktype: SOCK_RAW), hints.ai_family: AF_UNSPEC

ai->ai_family: AF_INET, ai->ai_canonname: 'opnsense.home.lan'
PING opnsense.home.lan (172.22.24.254) 56(84) bytes of data.
64 bytes from 172.22.24.254: icmp_seq=1 ident=30591 ttl=63 time=0.757 ms
64 bytes from 172.22.24.254: icmp_seq=2 ident=30591 ttl=63 time=0.658 ms
64 bytes from 172.22.24.254: icmp_seq=3 ident=30591 ttl=63 time=0.693 ms
^C
--- opnsense.home.lan ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2032ms
rtt min/avg/max/mdev = 0.658/0.702/0.757/0.041 ms

In my case the A record of opnsense.home.lan is provided by the private Bind instance to the Unbound instance, which will cache it and provide it to the requesting client. All good and well.

Now, my issue is with the host name detection mechanism of the search domain feature. By default the search domain is appended, if there are no dots in the given domain string. The reasoning being: No dots means host name (cf. host name opnsense vs. domain name google.com).
However, that detection does not seem to work well. I can see multiple instances in which nonsensical DNS queries were created by appending the search domain. Here's an example from a time server lookup of my Chrony instance trying to lookup nts.ntstime.de.home.lan:



Can anyone explain, why I see such invalid search-domain-appended queries from clients? Why is the search domain appended to the proper domain nts.ntstime.de in this instance?

Next time check in System > Routes > Status, if you have a default route set

If you suspect a bad unbound config, did you run unbound-checkconf? (https://unbound.docs.nlnetlabs.nl/en/latest/manpages/unbound-checkconf.html)

Intrusion detection systems need to track the flows. If you do address translation then sources or targets of flows are rewritten. The original flow is terminated and replaced. Intrusion detection systems typically only see one leg of the entire communication. Either the original flow leg or the replaced, new flow leg. But in either case they keep on missing half of what's going on.
Feel free to read the documentation for details. It's all there right in the "Choosing an interface" chapter: https://docs.opnsense.org/manual/ips.html#choosing-an-interface".

PS: There is a reason why many admins hate NAT. You have to jump a lot of hoops and deal with heaps of BS just to keep using the old IPv4 address.

Also NAT and intrusion detection systems are no friends.

Unfortunately, CAKE is not available for Opnsense.
The best we got is FQ-CoDel. It needs a bit of setup, though. Obviously you need to set the pipe bandwidth. It should be sufficient to set it to a value 10% less than the line rate. If your line rate is varying, then try to choose a conservative average rate in the beginning.
In order to get good results, you may also need to set the FQ-CoDel quantum value to your WAN MTU. You can get it from your default route in System > Routes > Status (assuming Path MTU Discovery is working).
Also, setting the FQ-CoDel target for very slow (like mobile) or very fast connections (like fiber) maybe be needed. In order to obtain a target value for your WAN connection, you can use mtr(1). Make sure the line is unused and let it run for a while to get a decent sample count. Use the average ping of the first hop as your target value.

(1) mtr -bz -i 5 -o LSD__NAW__M_X__ -L 55555 1.1

Interesting. I can remember an instance when Unbound was entirely broken. Not only just not reporting, but also not responding. Restarting the Unbound service via UI did not work and repeatedly failed with a service timeout error.
So, I used the CLI and I did a ps aux | grep unbound and killed all Unbound processes manually. Interestingly there were multiple flock processes which I killall'ed. Once all processes were killed, I could start Unbound in the UI, everything was fine again.
So, I indeed believe there may be an edge case in which the lock handling with flock does not work. I guess there should not ever be multiple flock processes.

similar thread: https://forum.opnsense.org/index.php?topic=40509.0