IPv6 working inconsistently, strange firewall behaviour

[andreaslink]

I have a problem getting IPv6 up and running in a stable way. I'm connected via FTTH behind a FritzBox (FB) and running IPv4 fine. Since some days my f***** provider forced me silently behind a Carrier Gate NAT but added - at least - a public IPv6 within the same change now. So I got a common /56 to use on my own, which is assigned towards my FB. Within the FB IPv6 setup I activated "DNS-Server und IPv6-Präfix (IA_PD) zuweisen" aka allowing it to share (parts of) the /56 via DHCPv6 further on with other routers in the LAN.

Behind the FB I've my OPNsense (OPNsense 20.7.1-amd64) running, where all my clients are, so I'm routing from my computer (Ubuntu 20.04 Linux) via OPNsense via FB into the Web. I know I'm double NATted within IPv4 (with CGNAT I guess even three times, but who cares...).  This forced me now to get deeper into IPv6 again so I also activated IPv6 within OPNsense according to common descriptions and examples in the web.

This means, what I did so far - beside the FB setup as explained before:

• Activated IPv6 witin OPNsense
• Set "IPv6 Configuration Type" on WAN (bce0) IF to DHCPv6
• Set within the basic "DHCPv6 client configuration":
• Request only an IPv6 prefix --> true
• Prefix delegation size --> 60 (As I got a /56 and I just wanted to have "some" (4 Bytes aka 16) subnets available on OPNsense (some more I can experiment on another router later)
• Send IPv6 prefix hint --> true
• Use IPv4 connectivity --> false
• Use VLAN priority --> Disabled
• On the LAN interface (bce1) I defined "IPv6 Configuration Type" as "Track Interface"
• Deactivated "Block private networks" as well as "Block bogon networks" on LAN IF (as the LAN behind the FB obviously falls under these rules)
• Under "Track IPv6 Interface" I set the value "WAN" for parameter "IPv6 Interface"
• Left the "IPv6 Prefix ID" unchanged at 0
• "Manual configuration" I left "false"
• Setup a Firewall rule to allow all ICMPv6 travel IN from WAN as well as for LAN (to cover all IPv6 ping and MTU-size requirements etc.)

With this setup, LAN got a decent IPv6 assigned from the FB as well as the /60. So this works fine and all clients within the OPNsense LAN got IPv6 addresses from the first subnet assigned. This looked OK so far and as far as I can evaluate.

Problem:
Now we come to the problem, sometimes I can ping the web and sometimes and I can only ping the OPNsense firewall from my computer. Then OPNsense cannot ping the FB via fe80 addr and then it works out of sudden. I would swear, I did not change the setup, I either rebooted or I only "refreshed" an interface, like e.g. WAN interface by pressing safe/apply without any changes and then it worked - sometimes. I'm annoyed as I cannot find a pattern for "sometimes" and my setup is fairly common with all I googled so far. So why does the behaviour flip so much? When it works it usually works really long. Last time I destroyed and reset me setup again when we had the Cloudfare aka transit provider issue in the internet as I first thought it was me here at home - bad timing .

First there was no chance for the OPNsense to ping the FB, but then I read somewhere else that OPNsense is usually not following the entries in the routing table but enforces a GW per IF. So activated the option "Disable force gateway" on 'Firewall --> Settings --> Advanced' which leads to the route table to be used/evaluated. Right after this it worked out of the box, but I was never able to reproduce this step.

I then restored everything from a working backup, did the steps again and got another behaviour. Today I stopped experimenting and turned IPv6 of again on LAN so far as many website did not work with IPv6 not being correctly routed into the web.

I also had sometimes the impression that I made changes, which did not really have an effect or immediate impact. Then I thought I have it, restarted the firewall and after the boot it did not work out again or was behaving differently again, while it was working fine before I rebooted.

Solution approaches:
Beside many changes in the setup and some reboots I focussed closer on the firewall log. I added a logging IPv6 rule allowing all in via LAN, I named it "Default allow LAN IPv6 to any rule" and it allows any IPv6 protocol from everywhere to eveywhere in LAN, so more or less opening up the firewall. Just to be sure I also added the same rule for "out" on LAN in case I need to secure the way back and log it. Similar rules are defined for WAN, so I had a quite oben IPv6 net allowing all between OPNsense-LAN and Fritzbox-LAN aka OPNsense-WAN.

On my machine (ending on ::14cf) I have a client running which constanly tries to reach a secured (port tcp 443) web service on a server in the web (ending with ::59) and as you can see, sometimes packages are blocked randomly by the default deny rule, even though I explicitly opened all ports here. ICMPv6 is working fine as well. So even though the firewall was "mainly green", in this case the connection was not working - but traceroute6 (UDP) showed a perfect route from start to end - so routing in general seems to work correctly.




I have further screenshots, where I also show the details of the rules, while kicking in:

Block-Details:


Accept-Details:


And it does not matter, when I change the service or the target in the web, I can always monitor the same behaviour.
BTW, traceroute6 is always sucvcessful and correct. So the route as such seems to be fully correct. DNS works perfect and route is correct, but the real tcp content is not routed as it should.


What I might need to add is my local machines routing table for IPv6 (parts):

route -n6:

Kernel-IPv6-Routentabelle
Destination                    Next Hop                   Flag Met Ref Use If
::1/128                        ::                         U    256 2     0 lo
2a02:xxx:xxxx:fcf0::/64        ::                         Ue   100 9     0 enp5s0_vlan3
fe80::/64                      ::                         U    256 10     0 enp5s0_vlan3
::/0                           fe80::221:5eff:fec8:be8a   UGe  100 9     0 enp5s0_vlan3
::1/128                        ::                         Un   0   11     0 lo
2a02:xxx:xxxx:fcf0::14cf/128   ::                         Un   0   10     0 enp5s0_vlan3
2a02:xxx:xxxx:fcf0:82ee:73ff:fe28:2165/128 ::                         Un   0   4     0 enp5s0_vlan3
fe80::82ee:73ff:fe28:2165/128  ::                         Un   0   9     0 enp5s0_vlan3
ff00::/8                       ::                         U    256 11     0 enp5s0_vlan3
::/0                           ::                         !n   -1  1     0 lo


With:
fe80::221:5eff:fec8:be8a --> Being my OPNsense LAN link local address

And with "ip a s" showing (parts):
enp5s0_vlan3@enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 80:ee:73:28:21:65 brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.10/24 brd 192.168.42.255 scope global enp5s0_vlan3
       valid_lft forever preferred_lft forever
    inet6 2a02:xxx:xxxx:fcf0::14cf/128 scope global dynamic noprefixroute
       valid_lft 4094sec preferred_lft 435sec
    inet6 2a02:xxx:xxxx:fcf0:82ee:73ff:fe28:2165/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 85754sec preferred_lft 13754sec
    inet6 fe80::82ee:73ff:fe28:2165/64 scope link
       valid_lft forever preferred_lft forever


I have no clue, why there is this random behaviour of OPNsense with IPv6 while IPv4 is running like a charm. I cannot find a pattern or where to work on, were is the problem or is it a software issue?
Looking very much for anyones smart input or thoughts or own experience, after I have invested days now and a lot of googleing!
Thanks in advance!

PS: And thanks for reading until here . This description became more and more now .

[andreaslink]

Looong story I entered here, and today I updated to the freshly released OPNsense 20.7.2-amd64 and since then I did all the setup again and it works right out of the box. So this was the same last time, meaning I'm not trusting it yet . But I'm monitoring it...

[andreaslink]

Problem is still not solved . Today I removed "netdata" as this was causing thousand of these error log entries after a short while:

netmap_transmit bce0 drop mbuf that needs checksum offload

So the log get's cleaner, but no idea how to progress and solve or analyse this IPv6 issue .

[bobm]

I'm new to opnsense and I don't know what I'm doing but I would try to turn off hardware network features, especially LRO:

https://docs.opnsense.org/manual/interfaces_settings.html

[andreaslink]

Thanks bobm, very good hint, but this I already worked through (also a very recommended requirement for "Intrusion Detection" to be started/used).
So I have all three hooks set, so all of these three are disabled:
- Hardware CRC
- Hardware TSO
- Hardware LRO

[ischilling]

I do have the same issue - here it is even getting worse thanks to Multi-WAN, which is obviously an additional problem. A backup connection seems not to be part of the concepts in OPNsense if it comes to IPv6 - works however great in IPv4...

When I used pfSense before I decided to go to OPNsense due to its, from my point of view, better security policy when it comes to updates, etc., I was warned that - whysoever - IPv6 is in many cases a problem with OPNsense. And I must say, indeed, it seems to be still a problem.

I tried everything here, also I am in the situation to have different opportunities to test:
* Single WAN with Vodafone Cable in Germany
* Multi-WAN with Vodafone Cable and T-DSL business in Germany
* Single WAN within a Hetzner DC on different system configurations (e.g. HW and SW)
* Multi-WAN with COLT and Telekom (both fibre) in Germany

Where I use Multi-WAN, to figure out the issues, I tried a single-WAN configuration and found that it is more likely to work with Telekom and Colt than with Vodafone Cable. The ladder is due to some configuration obstacles with Vodafone Cable installations. The Single WAN Vodafone Cable works, however meanwhile stable and the Multi-WAN Vodafone Cable is at least steady providing IPv6/56 as well.

As here described, IPv6 works always from the FW itself, except for Hetzner where a number of more configuration things had to happen and some problems between COLT and OPNsense machines, with same HW the things here worked however like a charme with PF as FW system....

The routing of packages from the internal network to external network(s) in IPv6 is like playing Roulette, it works from time to time without clear evidence why, it is independent whether I
* reboot the FW
* restart only the services
* clear all states
* or do all this together 

I also tried at all places just Single-WAN, single internal network. This worked with Telekom connections like a charme, multiple internal networks as well. All testes with IPv4/IPv6 lazy networking rule (all allowed).

Exchanging the Telekom connection with any other connection, works in so far from time to time if a reboot isn't happening... a reboot is like 'rien ne va plus' in Roulette... the outcome is as with Roulette, you seldom win.

So I investigated the Telekom connection and tried to figure out whether there are 'significant' differences in the protocols - not at all there is something I could find; which doesn't say anything since I am far, far away from being a networking or firewall expert.

Same configuration works with pfSense as expected; both OPNsense and pfSense where tested with most recent updates, versions etc...

I had such issues with pfSense a year ago, including some security related issues - which where the reason to give OPNsense a chance before going to the 'established' Enterprise stuff, and costs... and security issues,....

So I still hope that OPNsense get's the issue solved.

I am happy to provide any information, configuration etc. needed - even test access for one location for the developers - to finally get the IPv6 issue solved.

[gpb]

This *might* be related as an acknowledged problem.  It may be causing different issues for different setups.  For me, I get no ipv6 on boot until resaving both my WAN and LAN interfaces.  From there forward I'm ok, except radvd does not seem to reply to solicitations.

https://github.com/opnsense/core/issues/4338