Messages

The dual stack stuff needs more overhead and therefore reduces the usable MTU size. ry reducing it to 1300 or even smaller. DS-lite and non working VPN connections are a very common issue. Reducing MTU is the best bet to solve these issues.

ZeroTier modifies the Ethernet address of the device on its own.

It has to. Each device in a ZT network has its own MAC which is calculated from the member id of that device. This address does not change as long as the member id doesn't change which it only does if someone manually resets the member id and therefore makes it a new device to ZT.

ZT needs its own MAC because it works as a SDWAN switch and needs arp to function.

That answers the question why nothing arrived at the firewall. It was just impossible to send Ethernet frames to the ZT network member MAC from OPNsense.

I have no way of testing this

I can set up a public ZT network for you to play with, just drop me a line

so someone with the setup please take a closer look at ifconfig in the working and non-working case.

Just the ZT part:

Non working:

Code Select Expand


REDACTED: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 5000 mtu 2800
description: ZeroTier (opt2)
options=80000<LINKSTATE>
ether 58:9c:fc:10:92:2f
inet 172.27.8.25 netmask 0xffff0000 broadcast 172.27.255.255
inet6 fe80::5a9c:ffff:ffff:ffff%REDACTED prefixlen 64 scopeid 0x7
groups: tap
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 66352



Working after applying the patch:

Code Select Expand


REDACTED: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 5000 mtu 2800
description: ZeroTier (opt2)
options=80000<LINKSTATE>
ether 7a:fd:ba:es:1f:1c
hwaddr 58:9c:fc:10:92:2f
inet 172.27.8.25 netmask 0xffff0000 broadcast 172.27.255.255
inet6 fe80::5a9c:ffff:ffff:ffff%REDACTED prefixlen 64 scopeid 0x7
groups: tap
media: Ethernet 1000baseT <full-duplex>
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Opened by PID 61635


My assumption is still that this is true for assigned ZeroTier interfaces, but maybe I missed someone confirming that.

The ZT networks are assigned to an interface in my case, yes.

And is this an IPv4 or IPv6 tunnel?

IPv4 in the tunnel

# opnsense-patch 1dba25fed8

Applied to 24.7.4 and ZT is working again after a reboot.

> Reverting to 24.7.3_1 restores ZT connectivity immediately.

Someone will tell me what this means? Full revert? Still leaves the question if this is a kernel or core issue...

It means the whole system was reverted to 24.7.3_1.

There have been issues in the past, mostly because of routing issues within OPNsense – I guess – in all cases the device was connected properly to ZT was able to see all neighbours. But no comms to OPNsense. A zt leave and zt join fixed it (or removing the checkmark for the network in the gui).

This time it's different. Everything works with 24.7.3_1 (and with many releases prior, too. The above issue was last seen sometimes earlier this year). After installing 24.7.4 ZT no longer works.

A zt leave and zt join don't solve it anymore, no comms.

The firewall rules to allow data flow from the ZT network to other networks or the firewall itself don't show any states in inspect mode.

Reverting to 24.7.3_1 restores ZT connectivity immediately.

Question is: how to hunt down "which change, which component update of this affects ZeroTier operation"? Any directions on where to start?

Ah. My mistake. I got a bit confused because it looks different from 24.1 and it shows less information in the list view. And it seemed as if the remove did nothing. It does.

Remove works. No errors :-)

Could it be that this didn't make it into the 24.7.3 (or not fully)?

I tried an API request and the response from trust/crl/search doesn't contain a UUID like cert/search or ca/search.

So I assume that the UI doesn't get a UUID, too and therefore cannot remove the CRL entry.

Deleting a revocation under "System: Trust: Revocation" trows error "Endpoint not found". Happens in 24.7_x and 24.7.1.

Certs have been revoked sometimes with 24.1 or 23.x – so it might be something that happens to older entries only. Have no newer revocations.

Putting two 10/1 connections on one tier will *not* create one single connection out of both. They are still completely separate. The router/firewall just distributes connections across both. It's no aggregations, this would need some support on the remote side to enable such feature.

I would not expect a full 20/2 in this scenario.

You can pull one WAN down and check the speed again, then repeat with the other WAN down.

But don't forget that PPPoE has an absurdly high overhead which will significantly increase the system load. So depending on the used hardware, the system might as well reach some bottlenecks.

Regarding your IPTV, it's normally never using both connections at the same time and is only routed via one WAN (because the packets cannot be combined without the previous mentioned aggregation support on the ISP side).

This behavior can not be turned off and I suspect it is the same on XenServer.

Maybe I completely misunderstood the problem here but sure... you can not only change the MAC to your liking, the MAC is fixed.

I've attached a screenshot from XenOrchestra.

Hi!

Maybe I'm mistaken but... Sending a request to API endpoint kea/dhcpv4/getSubnet responds with subnet details like DNS servers, routers and NTP. But the subnets configured are not part of the response.

I guess the field "subnet" should contain the CIDR, but it doesn't.

Not sure if this is a known issue, I couldn't find anything in the forum.

Screenshots attached.

OPNsense version is 24.1.2_1

And a small addition: the pools section is empty, too. And if there are more than one subnet defined, only the first one is in the API response.

Take your time, no need to hurry :-)

I installed the patch and the lease duplicates are gone.

After each reboot of a client, Kea does see that the system did reboot and that it tries to get its lease back. So far so good.

But each of these result in a new lease (for the same IP and MAC) being shown in the Leases DHCPv4 section.

Screenshots attached.

Is that expected behaviour?

Well, that is not how static leases work. Their nature is that these are leases which just never time out and my will guess is that 95% of these reside inside the DHCP range because all the addresses outside are normally left free for devices which cannot retrieve a DHCP address or have to set as static in the device itself.

But... And that's a me-problem, I was under the impression that OPNsense is not using the two years ago EoLed ISC DHCP anymore. That's planned for 24.1 as I found out. Then things are expected to work better as Kea can keep track of static leases.

There's a ton of other weird DHCP things going on which I think are also related to the ISC implementation used. One of them is the DHCP lacking of keeping track of IPs given to the same device if it's part of multiple VLANs (like APs do).

Coming from the Vyatta based EdgeOS platform from Ubiquiti which uses the ISC DHCP as well I have to say that I never encountered these issues in 10+ years (well, it can keep track of devices residing in multiple VLANs, too). So it must be something related to how both pfSense and OPNsense work internally – or it's even older and is something from back in the Monowall days.

Anyhow... Version 24.1 is around the corner so let's wait for it.