Strange IPv6 behavior after update

[mahescho]

Hi,

to day I revived my new OPNsense appliance. I set it up with IPv4 and IPv6 and everything worked as expected. The I've updated it to the current version 18.1.13_1-amd64 and IPv6 stopped working. I found that just no IPv6 packets left the appliance. I've seen arrive my ICMP echo requests but no answers. IMHO The packets got dropped. I had to remove the IPv6 address from the WAN interface and the re add it to make it work again.

Just want to let the devs know about this strange behavior.

Matthias

[franco]

Hi Matthias,

It could be related due to ISP oddities: the DHCPv6 advanced options given by OPNsense are plenty for that reason and should be double-checked with the recommendations for your ISP. We're also working on different DUID support schemes and it'll take some time to get everything into the release and documented accordingly.

18.1.13_1 is not the latest version anymore. The point release is 18.7.2 now.


Cheers,
Franco

[mahescho]

Hi Franco,

In my setup I use a Microtik router to connect to my ISP. There I get a /29 IPv4 net and a /48 IPv6 net. I do not delegate subnets right now I just use static configurations of subnets and routes. So I use a static IPv6 setup on the firewall.

Now I've upgraded from 18.1-amd64 to 18.7-amd64 and it's all the same as soon as I reboot. I've to disable IPv6 on the WAN interface and re enable it to get back IPv6 connectivity. Wen I check the configuration on the shell with "netstat -rn" and "netstat -in" every thing looks fine but it just does not work. When I monitor the WAN interface with "tcpdump" while I'm pinging I don't see a single packet. I think it's some kind of problem in conjunction with "pf". It looks linke all outgoing packets get droppe silently. When I ping the firewall from the router and monitor the WAN interface with "tcpdump" I can see the ICMP echo request bur no replies.

In addition there is a problem with the dashboard. I'm currently pinging a v4 and v6 destination from the firewall without any packet loss and the status of the two gateways on the dashboard flaps every now and then from green to red and vice versa.

One last thing: "ntpd" does not survive the IPv6 reconfiguration. I've to start it afterwards.

Currently the firewall is not in production use. I'm just testing and tying to get familiar with the system. I've no IPv6 on the LAN interface so I can't say if it's affected too.

cheers
Matthias

[bimmerdriver]

After I updated my system, the Windows 10 computer I'm using on it had no IPv6 address. I had to disable and enable the adapter, then everything was normal.

[mahescho]

Well, now I'm on 18.7.2 and it's all the same as described above ..

Any suggestions?

Suggestion to make life easier: The default net mask for IPv6 should (IMO) be 64 and not 128 for Interfaces.

[Droppie391]

Just noticed the same problem here, found that the deafult route was pointing to the wan interface in stead of the pppoe interface. fe80::.....%igb0 in stead of fe80::.....%pppoe0
changeing the default route on the console fixed it temporarily:

route -6 delete default
route -6 add default fe80::.....%pppoe0

this may not survive a re-init of the pppoe session !!!

[Droppie391]

some more info from the system log when the pppoe restarted this morning (by the way im on 18.7.2 since yday)

Sep 10 05:00:05 xxxxxx opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'pppoe0'
Sep 10 05:00:05 xxxxxx opnsense: /usr/local/etc/rc.newwanip: On (IP address: AA.BB.CC.DD) (interface: MTKOM[wan]) (real interface: pppoe0).
Sep 10 05:00:06 xxxxxx opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'wan'
Sep 10 05:00:06 xxxxxx opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Sep 10 05:00:06 xxxxxx opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to wan
Sep 10 05:00:06 xxxxxx opnsense: /usr/local/etc/rc.newwanip: ROUTING: setting IPv4 default route to ISP_GW
Sep 10 05:00:06 xxxxxx opnsense: /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway 'ISP-GW'
Sep 10 05:00:06 xxxxxx opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
S

[Droppie391]

looking at function system_routing_configure in system.inc, it looks like there is no check if the wan is connected to a pppoe interface. the deafult route is here set to the wan interface, which is what i observe, and the Log-message matches this as well (ROUTING: skipping IPv6 default route).

[mahescho]

I'm not using PPPoE. My IPv6 setup is static on a ethernet interface. Lately I found, that my fist guess, that it has to be a PF issue seems to be right. The only thing I need to do to get it working again is a "pfctl -d" and then "pfctl -e" ...

[franco]

Hi guys,

I'm doing thorough audit of the interface code since about 2 weeks and what is described here from different angles seems to be what's going on: when IPv4 and IPv6 devices are not the same network device that's where certain reloads go south. But to be fair, this behaviour isn't new at all. And there is no quick fix so the target for correcting these issues will be 19.1 with smaller confirmed fixes being moved to 18.7.x subsequently.

Important things to note when reporting such issues for completeness: setup type of IPv4 and IPv6 and possible advanced IPv6 settings set and the ISP involved. Plus the last version where this worked fine 100% (if applicable).


Thanks,
Franco

[Droppie391]

Hi Franco, fair enough.

These are our settings:

ISP: Titan Networks
ipv4 config type: PPPOE (static ip address)
ipv6 config type: DHCPv6  ( /48 range w/o dns nor ip addresses)
Request only an IPv6 prefix = activated
Send SOLICIT direct = activated
NO ipv6 static address on wan side

we are happily working with LL addresses over the pppoe connection
ipv6 routing to our dedicated range is handled by the ISP

Things were alright until the upgrade from 18.1.13 to 18.7.1, then the pppoe-loop effect came to us 2 times in 3 days (before 18.7 we never noticed this).

for completeness, we have 2 OPNsenses in HA where we manually have to switch the pppoe link on failure of
the master box. we would love to have a solution here..

[mahescho]

Hi Franco,

what do you mean with "not the same network device"?

My WAN interface is igb2. On this interface IPv4 and IPv6 is configured ... a dual stack setup. Both, IPv4 and IPv6 are static configurations.

cheers
Matthias

[franco]

@Droppie391: try this patch...

https://github.com/opnsense/core/commit/5811fe53

# opnsense-patch 5811fe53

@mahescho: does the following command fix your issue temporarily?

# /usr/local/etc/rc.filter_configure


Cheers,
Franco

[mahescho]

Hi Fanco,

/usr/local/etc/rc.filter_configure does NOT fix my problem.

Code: [Select]

root@OPNsense:~ # ping6 google.de
PING6(56=40+8+8 bytes) 2001:1a50:5105:1::193 --> 2a00:1450:4001:818::2003
^C
--- google.de ping6 statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # /usr/local/etc/rc.filter_configure
Configuring firewall.......done.
root@OPNsense:~ # ping6 google.de
PING6(56=40+8+8 bytes) 2001:1a50:5105:1::193 --> 2a00:1450:4001:818::2003
^C
--- google.de ping6 statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@OPNsense:~ # pfctl -d
pf disabled
root@OPNsense:~ # pfctl -e
pf enabled
root@OPNsense:~ # ping6 google.de
PING6(56=40+8+8 bytes) 2001:1a50:5105:1::193 --> 2a00:1450:4001:818::2003
16 bytes from 2a00:1450:4001:818::2003, icmp_seq=0 hlim=56 time=14.594 ms
16 bytes from 2a00:1450:4001:818::2003, icmp_seq=1 hlim=56 time=14.431 ms
^C
--- google.de ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 14.431/14.512/14.594/0.081 ms

cheers
Matthias

[franco]

Hi Matthias,

This is strange. Filter configuration includes reloading the filter and would be expected to unblock operation. Disabling and enabling the filter on the other hand should have no visible effect but it looks like it has.

Toggling the filter will remove it from the kernel hooks and push it back it so that would somehow indicate something is blocking flow inside a kernel facility?

Only guessing at this moment in the hopes to get one step closer through discussion or more diagnostics...


Cheers,
Franco