OPNsense Forum
English Forums => High availability => Topic started by: bimbar on August 24, 2021, 09:43:04 am
-
Scenario is 2 firewalls behind a router that hands out prefixes via DHCPv6-PD.
Since the firewalls are effectively standalone, they get different prefixes, which they in turn hand out to the LAN.
Now the problem is that each client gets the 2 prefixes, and gets 2 default routes but does not typically associate the prefix to the default router, so that for example clients are trying to communicate with the prefix from fw1 over the gateway fw2, which the firewall drops.
I can prioritize the default gateways, but then it still can happen that a client chooses the wrong prefix for the assigned gateway.
So, how can that be solved? I could go stateful with dhcpv6, but then each client only gets one IP and redundancy is lost.
-
Same problem here. I could work around that by allowing as last rule on the LAN interface all IPv6 traffic instead of allowing only traffict originating from LAN net.
Another problem: for IPv4 I can define a virtual IP which is "shared" between the two firewalls. What about IPv6? It appears that with prefix delegation each firewall has to use its own IPv6 adresses. So it's not possible to assign an IPv6 adress to a domain in order to reach a server behind the firewall. Either the domain points to the address of the first or to the address of the second firewall?
Where am I wrong here?
-
As I run it now, I have internal ULA addresses and NAT on WAN. The internal side is done with CARP fe80::1 and this is advertised via RA (there is a PR in github that allows to select the RA SRC address).
I have one external service with dyndns, but that's not redundant.
There is a heap of RFCs I read that basically mean that in the end it's not possible because the end devices do not cleanly implement said RFCs (source and nexthop selection). They don't even respect the RA priorities.
So that's not perfect. Maybe best to stick to IPv4 or use something like cloudflare for internal services.
-
@bimbar, do I understand correctly that "internally" you only work with ULA addresses?
Regarding the external service, I also have no solution yet. But since sooner or later I may loose my public IPv4 access with IPv6 remaining as only access possibility from outside, I have rent a virtual server which has a public IPv4 address and forwards any request e.g. on port 443 via IPv6 to my firewall (using 6tunnel).
I consider that on this virtual server I may run a script which tests accessability of firewall 1 and firewall 2 using the IPv6 addresses of the firewalls each published via DynDNS to respective Domains. In case connection is lost to one of the Domains (i.e. firewalls), the script just instructs 6tunnel to use the other domain for forwarding requests.
-
Explicit Routes should not be required.
If you want to do outgoing NAT, the Firewalls should also request an address, not only a prefix.
I do work with ULA addresses only on LAN.
-
ok, you write
(there is a PR in github that allows to select the RA SRC address)
. What are you referring to?
-
https://github.com/opnsense/core/pull/5185
-
I've given up on it. IPv6 NAT stops working intermittently (and who can blame it, it's not supposed to be used), and everything other scenario doesn't work either.
So, if you don't have a static prefix and want to have redundant firewalls, don't bother.
I have disabled IPv6 on one of the firewalls, created an interface group of my internal interfaces, and added a floating rule in the style of "allow ifgroup_internal -> IPv6 -> !ifgroup_internal, allow ifgroup_internal -> IPv6 ->This Firewall", which is the best I can come up with right now.
-
I would like to get rid of my frustration here and above all save a lot of people life time that they could waste with Opnsense, IPv6 and CARP. It is definitely the case that Opnsense does not run with Carp IPv6. It's a pity that this is not admitted officially.
Any attempt to implement Carp with IPv6 fails. For example, if you define a virtual interface fd00:..., it will be used instead of the additionally available global IPv6 address 2004:... . and the stupid system tries to transport all packets via fd00 out to the WAN. The solution is to manually change the order of the IPv6 addresses after each reboot. Disable Carp IPv6, apply, re-enable, reboot RA.
If you are lucky, NAT IPv6 to a fd00 address will work. This MAY work for a while. Eventually, however, it will stop working and Opnsese routes fd00 packets into nirvana. I'm really fed up with IPv6 and Opnsense. And I suspect more will have this problem here in Germany, as our providers are moving more and more to only allow accessibility from the WAN via IPv6.
-
Sorry, but CARP as an isolated issue works as it should in 21.7.5.
We have a redundant setup with a static prefix and a global unicast CARP address for IPv6 on the WAN interface. On the LAN network we also have global unicast addresses for the individual firewalls, but a link local CARP address as the default gateway for all internal systems.
The internal link local CARP including correct router advertisments was introduced in 21.7.5.
Kind regards,
Patrick
-
We discussed this via PM and I still think that opnsense works for that, if you have a static GUA.
I think your setup is still not quite right.
-
Sorry, but CARP as an isolated issue works as it should in 21.7.5.
...which means that besides CARP something like NAT for IPv6 to a link local address fd00... does not work? Correct? At least that is what I'm experiencing.
We have a redundant setup with [...] a link local CARP address as the default gateway for all internal systems.
In other words, you define an fe80... CARP address on e.g. the LAN interface and additionally (or instead?) of the normal fe80 gateway you define the fe80-Carp address as gateway for this interface (Gateway-Interface = LAN)? Correct?
-
...which means that besides CARP something like NAT for IPv6 to a link local address fd00... does not work? Correct?
Possibly.
We have a redundant setup with [...] a link local CARP address as the default gateway for all internal systems.
In other words, you define an fe80... CARP address on e.g. the LAN interface and additionally (or instead?) of the normal fe80 gateway you define the fe80-Carp address as gateway for this interface (Gateway-Interface = LAN)? Correct?
Yes - if I understand you correctly ;)
See screenshots for details.
-
And additionally you have to define a further Gateway (System-Gateways-Single) or do you keep the "normal" IPv6 Gateway there?
-
Why would you define a gateway on LAN?
On WAN I have static GUA on both firewalls and another static GUA as CARP. This is used as the gateway to my network on the outside.
I don't NAT. I was just referring to the fact that CARP does work, IMHO.
-
Does not work - I have no IPv6 internet connection on computers in the LAN in case the following is used:
- Virtual IP Carp LAN fe80::2:2/64
- Router Advertisements LAN unmanaged, Priority High, RA Interface LAN_VIP6 (fe80::2:2)
- WAN_DHCP6 Gateway with address of the Fritzbox (router) on the WAN side
Screenshots attached. As soon as I deactivate the virtual CARP and use RA Interface "LAN dynamic", IPv6 connection to WAN is possible.
-
Running 21.7.5? This is a new feature.
-
OPNsense 21.7.5-amd64
Windows reports as gateway fe80::2:2%8, whereas in the interface overview of Opnsense the address is fe80::2:2/64
-
fe80::2:2%8 is correct. There is no prefix length on a gateway. The "%8" is the scope ID, because link local addresses are all the same across all interfaces.
It should be the interface number 8 of your Windows, whatever that means. Here in my home network the entry of e.g. my Nextcloud jail which uses SLAAC is:
fe80::3eec:efff:fe00:5430%epair0b
So the scope is the "epair0b" interface.
So at least the gateway advertisements seem to be correct. Now check the GUA or ULA address of your Windows, and use e.g. tcpdump on the OPNsense to check what happens to outbound packets.
-
Windows GUA and ULA seems to be correct. I did a ping -6 google.de on windows, which resulted in a timeout. TCPDUMP results in:
GUA_LAN_Opnsense: 2a02:810b:....
Opnsense LAN fe80 address: fe80::2e0:4cff:fe68:337c
Windows GUA: 2a02:810b:c03f:..f27f
Interface Capture output
WAN
igb0 21:29:04.489739 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) 2a02:810b:... > 2a03:4000:2:ee9:68bb:31ff:fecd:dd60: [icmp6 sum ok] ICMP6, echo request, seq 156
WAN
igb0 21:29:04.504490 IP6 (flowlabel 0x244af, hlim 55, next-header ICMPv6 (58) payload length: 8) 2a03:4000:2:ee9:68bb:31ff:fecd:dd60 > 2a02:810b:...: [icmp6 sum ok] ICMP6, echo reply, seq 156
WAN
igb0 21:29:05.490666 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) 2a02:810b:... > 2a03:4000:2:ee9:68bb:31ff:fecd:dd60: [icmp6 sum ok] ICMP6, echo request, seq 157
WAN
igb0 21:29:05.506616 IP6 (flowlabel 0x244af, hlim 55, next-header ICMPv6 (58) payload length: 8) 2a03:4000:2:ee9:68bb:31ff:fecd:dd60 > 2a02:810b:...: [icmp6 sum ok] ICMP6, echo reply, seq 157
WAN
igb0 21:29:06.491739 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 8) 2a02:810b:... > 2a03:4000:2:ee9:68bb:31ff:fecd:dd60: [icmp6 sum ok] ICMP6, echo request, seq 158
LAN
igb1 21:29:06.772091 IP6 (hlim 128, next-header ICMPv6 (58) payload length: 40) 2a02:810b:c03f:..f27f > 2a00:1450:4001:811::2003: [icmp6 sum ok] ICMP6, echo request, seq 1779
LAN
igb1 21:29:06.791941 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::2e0:4cff:fe68:337c > ff02::1:fff6:f27f: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a02:810b:c03f:..f27f
LAN
igb1 source link-address option (1), length 8 (1): 00:e0:4c:68:33:7c
LAN
igb1 21:29:07.791570 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::2e0:4cff:fe68:337c > ff02::1:fff6:f27f: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a02:810b:c03f:..f27f
LAN
igb1 source link-address option (1), length 8 (1): 00:e0:4c:68:33:7c
LAN
igb1 21:29:08.806593 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::2e0:4cff:fe68:337c > ff02::1:fff6:f27f: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2a02:810b:c03f:..f27f
LAN
igb1 source link-address option (1), length 8 (1): 00:e0:4c:68:33:7c
LAN
igb1 21:29:10.103587 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::dcdf:3597:709c:b416 > fe80::2:2: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::2:2
LAN
igb1 source link-address option (1), length 8 (1): 34:2e:b7:a8:4a:6c
First, I have no idea what 2a03:4000:2:ee9:68bb:31ff:fecd:dd60 is. This is not on my system. Second, Opnsense does not report the used fe80::2e0:4cff:fe68:337c in its overview of the interfaces. However, I can see it in the opnsense terminal window with ifconfig. Third, while playing around the gateway pointing from WAN to the fe80 address of my Fritzbox router got disabled. And voila, packets passed to my windows computer. ::) Toggling the Gateway on/off did not make any difference. However, after a reboot no IPv6 ping possible any more. Even with IPv6 Gateway enabled or disabled. So this was only a single occurence for whatever reason.
Finally, after a few minutes of doing nothing, ping6 suddenly started to work again on one LAN computer, while on the other LAN computer it didn't work. Completely unpredictible.
-
Do both OPNsense systems have the same GUA prefix on LAN? If not I guess that is the problem here. Can't you set a static prefix and use NAT in that case?
And please permit me to question the business case - you have a dynamic/changing prefix from your provider and want to run a HA pair? Why not get a static /56 or similar?
-
Indeed, the prefixes are different. I use track interface on both systems, both either with IPv6 Prefix ID 0x0 or (I also tried) one with 0x0 and the other one with 0x1.
And please permit me to question the business case - you have a dynamic/changing prefix from your provider and want to run a HA pair? Why not get a static /56 or similar?
The provider does not guarantee that the assigned prefix is static. Even though it appears to be static, they may change it from one day to the other. So, this is nothing I can rely on.
As far as I understand, each client can get multiple GUA adresses. In case the goal is to firstly maintain IPv6 connectivity (even with interruption of connections in case of a failure of one firewall), is there some more easy approach how to ensure that the clients are using IPv6 connectivity (and existing GUA) of the other firewall?
-
Indeed, the prefixes are different. I use track interface on both systems, both either with IPv6 Prefix ID 0x0 or (I also tried) one with 0x0 and the other one with 0x1.
And please permit me to question the business case - you have a dynamic/changing prefix from your provider and want to run a HA pair? Why not get a static /56 or similar?
The provider does not guarantee that the assigned prefix is static. Even though it appears to be static, they may change it from one day to the other. So, this is nothing I can rely on.
As far as I understand, each client can get multiple GUA adresses. In case the goal is to firstly maintain IPv6 connectivity (even with interruption of connections in case of a failure of one firewall), is there some more easy approach how to ensure that the clients are using IPv6 connectivity (and existing GUA) of the other firewall?
Clients with multiple GUAs on multiple gateways does not work. That's a client problem.
-
Idea:
- instead of PD just get one IPv6 address per firewall via DHCPv6 on WAN
- use statically configured ULA on LAN (or a "known available" GUA
- use link local CARP address on LAN as discussed
- use NAT on WAN or possibly NPT (I have not tried this, yet, but if NPT works, that would be a next-to-perfect solution
I hope you don't have inbound connections to systems on LAN - in that case I really cannot see a way to implement that with a changing prefix and HA.
HTH,
Patrick
-
use statically configured ULA on LAN (or a "known available" GUA
This means, that for the LAN Interface IPv6 is set to static and e.g. fd00::100 ?
use NAT on WAN or possibly NPT (I have not tried this, yet, but if NPT works, that would be a next-to-perfect solution
I assume that NPT requires an external IPv6 Prefix, which however in the present case is not static. So, NPT should not work.
Regarding NAT on WAN - in case this works at all, which is questionable (see discussion above), what would be correct here? Just WAN NAT IPv6 with Source address "LAN" and Translation / target "Interface address"?
-
NAT66 on WAN does not work well, see https://github.com/opnsense/core/issues/5325 .
Dynamic NPTv6 does not (yet?) exist, see https://github.com/opnsense/core/issues/5284 .
So you're pretty much stuck where I am. I have now track interface on just one firewall, no redundancy for v6.
-
...which means that in case
- you are running services for inbound connections on the LAN
- and you are not provided with a public IPv4 address
- and you are not provided with a static IPv6 prefix
high availabililty is not possible with Opnsense. After failover you can still communicate from LAN to WAN (via IPv4), but not vice versa since Ipv6 is not available here.
A heretical question: Can pfsense do it better? ::)
Nevertheless, I just tested NAT. Setup
nat on igb0 inet6 from (igb1:network) to any -> (igb0:0) port 1024:65535(igb0=WAN, igb1=LAN)
ping google.de:
WAN
igb0 14:16:53.140591 IP6 2a02:810b:c03f:fe20:... > 2a03:4000:2:ee9:68bb:31ff:fecd:dd60: ICMP6, echo request, seq 2423, length 8
WAN
igb0 14:16:53.156001 IP6 2a03:4000:2:ee9:68bb:31ff:fecd:dd60 > 2a02:810b:c03f:fe20:...: ICMP6, echo reply, seq 2423, length 8
LAN
igb1 14:16:55.239746 IP6 fd00::..:66fb > 2a00:1450:4001:828::2003: ICMP6, echo request, seq 9, length 40
LAN
igb1 14:16:59.960756 IP6 fe80::..:62d6 > fe80::2:2: ICMP6, neighbor solicitation, who has fe80::2:2, length 32
NAT works, the WAN packets originate from the correct WAN interface GUA. On LAN, both addresses correspond to the windows machine performing the ping. However, the ping results in a timeout.
-
Essentially both OPNsense and pfSense are fancy UIs for FreeBSD and pf. I run NAT66 without problems, but, again, with a static address. I doubt any of this can really be improved in the short term given how IPv6 and pf both work.
Can't you get a static prefix if you have HA requirements? I tend to think HA --> mission critical business. So the budget for a real Internet connection should not be hard to justify.
-
Assuming I get the static address, what would be the setup? What is to be set on Wan, Lan?
-
WAN: static GUA on both nodes, another GUA as CARP, provider uses that as the gateway to your e.g. /56.
LAN: static GUA on both nodes, link local as CARP, CARP address for RA. Possibly an additional GUA as another CARP address if you have statically configured systems that don't use RA. Or services like DNS, NTP, ... that need a GUA on internal networks.
DMZ 1, 2, 3, ... like LAN.
Working flawlessly here.
-
...and of course, the static GUA being used is different for each node, right?
-
Yes, of yourse. No different from IPv4 here. One address per node, one shared CARP address. The only new feature in 21.7.5 is that you can use a link local CARP address for router advertisements, now. In earlier versions each node used its own LLA - which resulted in two gateways active at the client system.
Combine that with clients not really following router priorities ... but you've been there IIRC.
-
Sure :) and RA unmanaged?
-
Yes.
-
What is the IPv6 Gateway on the second node? At the moment I use the fe80:: address of the router on the WAN to which Opnsense is connected (as I did before using HA). But the second node shows the gateway as being offline?
I have to correct myself, I use the fe80 address as mentioned, it is reported to be offline, but in fact, it is working? On failover it immediatelly goes online... Strange?
-
Same here - hadn't noticed. Possibly it is trying to use the CARP address to ping the GW ...
-
Everything seems to work now. Please note that I had to set RA to stateless or assisted. Otherwise Android devices won't be able to use IPv6.
-
Another finding regarding Dynamic DNS: In case I have WAN DHCPv6 as well as a WAN CARP Address configured, Dynamic DNS will public the Carp address.
Is there a way to teach Dynamic DNS the address to use?
-
Same here - hadn't noticed. Possibly it is trying to use the CARP address to ping the GW ...
For me the IPv6 gateway shows down on both master and backup as dpinger is binding to the WAN interface address rather than the CARP VIP. As I only have a single GUA for the WAN link, the WAN interface addresses are ULA (I.e. not in the same subnet as the VIP):
root@router-nuc:~ # ps x | grep dpinger
40628 - Is 0:00.02 /usr/local/bin/dpinger -f -S -r 0 -i WAN_GWv6 -B fd00:1234:5678:90ab::5 -p /var/run/dpinger_WAN_GWv6.pid -u /var/run/dpinger_WAN_GWv6.sock -C /usr/local/etc/rc.syshook monitor -s
On the other hand the IPv4 gateway monitor binds to the VIP and it works.
From what I can see, determination of the address to bind to is done in dpinger_configure_do(). For IPv4, it iterates through the interface IPs until it finds one in the same subnet as the monitor address, so in my case it finds the VIP address. For IPv6, if the monitor address isn't an LLA it uses interfaces_primary_address6() to get the bind address which basically finds the first valid address which isn't an alias. It doesn't do the subnet check.