Messages

Running Opnsense virtualized on proxmox for past many years. Currently on 24.7
It has run extremely stable for this entire duration.

A month ago or so, Internet stopped working. When I checked the console, It had this message repeating in a loop

Code Select Expand

[udp_inpcb] kern.ipc.maxsockets limit reached
Restarting the VM fixed it.

This morning I encountered the same problem again.
This particular error message doesnt seem to return any pertinent google results either.
ANy pointers on what possible troubleshooting I should look at?

Upgraded to 23.7.4 last night and realized this morning that my primary WAN (PPPoE) is showing offline on ipv4 although online on ipv6. ipv4 connections from clients were also being routed via backup WAN.

After spending a couple hours trying to troubleshoot, performed a opnsense-revert to 23.7.1 and that fixed it immediately

Just thought of sharing this here in case anyone else runs into this issue

I had been running opnsense virtualized for the past several months and finally decided it deserves a dedicated hardware unit of its own.

I decided to redeploy an old HP SFF with i5-6500t that was being used as a desktop for this.
The hardware HP 800G4 with i5-6500t / 8GB RAB and a 256GB SATA disk.

While running windows , the unit used to idle at 14-16W most of the time (as measured by a reasonably accurate kill-a-watt meter on the wall plug) 

After installing opnsense 23.7, i enabled Power-D and set states to adaptive.
However the unit now seems to draw 28-35W all the time.
The use case is for a home and the CPU utilization is 3-5% most of the time which is about the same as windows idle.
There are no CPU intensive packages on the system and IDS etc is disabled.
The only additional plugin installed is wireguard although no inbound VPN connections were on during testing.

Power is expensive in the region I live in (about 20 US cents per kwh) and any opportunity to optimize it is welcome.
Even technically, I am surprised why FreeBSD/ Opnsense is drawing 2X power for the same CPU utilization.

Any suggestions or pointers?

You should definitely keep "request prefix only" in your settings.

I'd still hope you could install the dhcp6c test version

# opnsense-revert -z dhcp6c

and disable prevent release to see if that makes a difference after the next boot and subsequent disconnect.


Cheers,
Franco

Sure, can try that out this weekend.
Currently it's been 2+ days and connectivity haas been rock stable but It may be worth pointing out a couple of things

a) I have a virtualized opnsense install on proxmox and the adapters use VirtIO - not sure if that could be one of the underlying reasons

b) One of the things I had done on my previous install was installing sensei.
Looking through the forums, it seems there are other users who have also had issues with a combination of sensei with  ipv6.
At the time of testing, even though sensei was disabled but the fact that LAN client could not even ping the valid ipv6 addr of the LAN interface makes me suspect that the presence of sensei could be one of the underlying reasons (or ot could be a complete red herring too)
Either way, the fresh install does not have sensei and  I have now added every other plugin I had previously.

This weekend, I will backup the opnsense VM and re-add sensei to see if it breaks connectivity

On an unrelated note, now that ipv6 seems to be running stable, I added a pass ICMP rule on the primary WAN interface to the LAN network alias as I had read somewhere that ipv6 relies a lot on certain kind of ICMP requests.
Now it could be a placebo but post the change, the general(and subjective)  internet responsiveness seems to have improved.

Thanks @Cyberturtle and @Franco

Based on your hints, I think I may have figured it out.
There was way too much finagling I had done on the opnsense router to fix this and I figured a fresh start may be the right way to do it.


A fresh install + setting the Prevent release and only request prefix flag to enabled, things seem to be fine now..

I recently setup an OPNsense router on my side. I have read somewhere that some ISP require the option ,,Only request a prefix". If I got it right it's necessary as some ISP provide the prefix by DHCPv6 and the WAN address via SLAAC. Maybe you could try this option?

I'll try the new version 23.1.7. with the DHCP6c patch as soon as I have some time. Currently my IPv6 seems stable as there is no nightly ASSIA forcing a reconnect.

Thanks - I set the WAN to only request a prefix and that seems to have fixed the /128 issue
The WAN side is now getting a /64 address and prefix
The LAN interface also has a /64 prefix

Here is where it gets weird

LAN clients are also getting a /64 prefix but cannot ping6 even the LAN router addres, let alone a WAN address

eg Client 1 has obtained 2401:4900:1c02:1045:x:x:x:4c50/64
LAN Inteface on router is 2401:4900:1c02:1045:x:x:x:963e/64

but ping6 or traceroute6 from client 1 to LAN interface results in the following that does not proceed any further

Code Select Expand

abhinav@Abhinavs-MacBook-Air ~ % ping6 2401:4900:1c02:1045:x:x:x:963e
PING6(56=40+8+8 bytes) 2401:4900:1c02:1045:x:x:x:4c50 --> 2401:4900:1c02:1045:x:x:x:963e


Code Select Expand

traceroute6 to 2401:4900:1c02:1045:x:x:x:963e (2401:4900:1c02:1045:x:x:x:963e) from 2401:4900:1c02:1045:x:x:x:4c50, 64 hops max, 12 byte packets
1  2401:4900:1c02:1045:x:x:x:4c50  3033.698 ms !A  3033.286 ms !A  3033.940 ms !A

There might be changes coming to dhcp6c soon. We can try one more thing if you are able to reproduce this at will. Would appreciate the help here!

This might fix it too.. https://github.com/opnsense/dhcp6c/commit/db9f45927


Cheers,
Franco

So there is more to this issue it seems :(

After changing the release flag, the connectivity on the WAN side remained stable for another hour or so after which clients started to fail to route via ipv6.

Clients had a valid ipv6 address but ping6/tracroute6  could not find a valid route.
So I checked further and (not sure if this is relevant- i am not an expert on ipv6), the WAN IP had a /128 prefix now.

The delegated prefix was /64 (see attached screenshot)

Again, I would have normally blamed the ISP for this but I decided to test further and plugged in an old TP Link ER-605 router bridged to the ISP equipment.
With the ER605, I was able to instantly obtain a xxxx/64 address for the router as well as the LAN SLAAC+Stateless DHCP mode selected) and everything is working as expected.

The ipv6 connectivity was also immediate as against a reasonably long wait time on opnsense after pppoe completion and ipv4 up.

Not quite sure what could be causing this :|
I can't really use the ER605 as a replacement for opnsense for many reasons - not to mention that it does not have any firewall for v6

There might be changes coming to dhcp6c soon. We can try one more thing if you are able to reproduce this at will. Would appreciate the help here!

This might fix it too.. https://github.com/opnsense/dhcp6c/commit/db9f45927


Cheers,
Franco

Thanks for looking into this @franco
Don't think I am able to reproduce this at will as so far , the drops and subsequent non recovery have been seemingly random.

I have enabled the "Prevent Release" flag and its been just over an hour with no drops.
The ppp log also seems to be blissfully static.
Will monitor it overnight and report back if there are any drops or issues!
Happy to help whichever way I can.
Let me monitor the connection

I'd try to set "Prevent release" on Interfaces: Settings page to see if that at least keeps connectivity.

But it looks like a daily disconnect?

"dhcp6c RELEASE on pppoe0" only happens once as far as I can see.

I also assume you have "Use IPv4 connectivity" set for IPv6 on the WAN? And IPv6 mode is set to DHCPv6?


Cheers,
Franco

Thanks - let me try enabling prevent release and see if that helps.
Its not exactly a daily disconnect. I have had instances where it has dropped within an hour or two on multiple occasions on the same day .
And sometimes it sticks for several hours.
What I pasted was a single snippet for ease of reading.

And yes, ipv6 on the WAN is via IPv4 connection and IPv6 is set to DHCPv6

Running OPNsense 23.1.7_3-amd64 on a multi WAN failover setup.

The primary connection is a gigabit fiber over PPPoE (ISP - Airtel )
With the ISP provided equipment in routing mode, both ipv4 and ipv6 are normally 100% stable with no reboots etc required for months.

With the ISP equipment set to bridging mode, opnsense obtains ipv4 and ipv6 as expected on reboot.
All clients get an ipv6 address as expected.

However after anywhere between an hour to several hours, the ipv6 connectivity on WAN drops and requires a pppoe restart.

Looking at the Logs, the one place i could find something of relevance was under System>Logs>General
dhclient seems to be generating at new resolv.conf every hour.
A few hours after last restart - I see the following in the log

Code Select Expand

2023-05-10T10:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T09:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T08:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T07:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T06:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T05:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.63.255'

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.63.255'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.63.255'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T05:31:37
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:37
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6

[b][color=red]2023-05-10T05:31:36
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:36
Notice
dhcp6c
dhcp6c REQUEST on pppoe0 - running newipv6

2023-05-10T05:31:34
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:34
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6[/color]

[color=red]2023-05-10T05:31:34
Notice
dhcp6c
RTSOLD script - Sending SIGHUP to dhcp6c[/color]

2023-05-10T04:41:11
Notice
dhclient
Creating resolv.conf
[/b]
2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.x.255'

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.x.255'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.x.255'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T03:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: Chose to bind AIRTEL_PPPOE on 122.169.x.x since we could not find a proper match.

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: plugins_configure monitor (execute task : dpinger_configure_do())

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: plugins_configure monitor ()

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'


In particular, this part is where the failure seems to occur

Code Select Expand

2023-05-10T05:31:37
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:37
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6

2023-05-10T05:31:36
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:36
Notice
dhcp6c
dhcp6c REQUEST on pppoe0 - running newipv6

2023-05-10T05:31:34
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:34
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6

2023-05-10T05:31:34
Notice
dhcp6c
RTSOLD script - Sending SIGHUP to dhcp6c

2023-05-10T04:41:11
Notice
dhclient
Creating resolv.conf

Any pointers what settings should I be looking at ?
I would have normally attributed this to the ISP but if I use the ISP router, ipv6 remains rock solid