Messages

1

24.7 Production Series / error: "udp_inpcb kern.ipc.maxsockets limit reached" after few days to few weeks

« on: October 30, 2024, 08:55:25 am »

Running Opnsense virtualized on proxmox for past many years. Currently on 24.7
It has run extremely stable for this entire duration.

A month ago or so, Internet stopped working. When I checked the console, It had this message repeating in a loop

Code: [Select]

[udp_inpcb] kern.ipc.maxsockets limit reachedRestarting the VM fixed it.

This morning I encountered the same problem again.
This particular error message doesnt seem to return any pertinent google results either.
ANy pointers on what possible troubleshooting I should look at?

2

24.1 Legacy Series / Re: Opnsense (Virtualized) - very high CPU utilization (System Interrupt)

« on: April 17, 2024, 06:09:57 pm »

@Taomyn

Here is the conf file.

 i did have opnsense running for 2 years without glitches on proxmox on my primary home server .
I later shifted it to a separate baremetal installation on a different spare machine but found managing / maintaining difficult so reverted it back to a virtualised install on the same spare machine

Code: [Select]

root@pve:/etc/pve/qemu-server# cat 100.conf
agent: 1
boot: order=virtio0;ide2;net0
cores: 6
cpu: host
ide2: local:iso/OPNsense-24.1-dvd-amd64.iso,media=cdrom,size=1936730K
memory: 8000
meta: creation-qemu=8.1.5,ctime=1712431571
name: opnsense
net0: virtio=BC:24:11:F5:75:A7,bridge=vmbr0,firewall=1
net1: virtio=BC:24:11:B5:8C:CF,bridge=vmbr1,firewall=1
net2: virtio=BC:24:11:C9:2A:EF,bridge=vmbr2,firewall=1
numa: 0
onboot: 1
ostype: other
scsihw: virtio-scsi-single
smbios1: uuid=2976059b-de19-4aa8-bac4-0405ac2b8bc2
sockets: 1
startup: order=1
vga: qxl
virtio0: local-lvm:vm-100-disk-0,iothread=1,size=120G


3

24.1 Legacy Series / Opnsense (Virtualized) - very high CPU utilization (System Interrupt)

« on: April 16, 2024, 10:02:51 pm »

Environment: Running opnsense 24.1.5_3 virtualised on a proxmox 8.1.4 server (i5 9th gen)
NIC: 3X Intel Pro in VirtIO mode (2 WAN, 1 LAN) 1X I219-KM and 2X 82571EB
WAN: 2X WAN in LB/Failover mode - Both PPPoE over fiber (ISP ONU in bridged mode), 1000/500 and 200/100

Issue: System works fine for several hours. After some time though (could be several hours or more), the host (Promox) will start showing high CPU utilisation on opnsense even when idle (20-25%)

Top shows 16-17% utilized by System Interrupts (this is almost exactly equal to 1 of the 6 cores) , user /system utilisation is normal (see screenshot)

Opnsense System> Diagnostics > Activities shows 1 CPU core (core 4) 99-100% utilised by [intr{swi4: clock (0)}]

Any other info: Not sure if relevant but dmesg shows multiple message similar to :
"cannot forward from fe80::ce4:35d8:2fcc:872c to ff02::fb nxt 44 received on vtnet0"

A reboot of the opnsense VM fixes the issue till it recurs again after a seemingly random period of time.

I understand this may not be adequate information to help resolve the problem.
However what I was hoping for is to understand what the probable causes for this could be and what should be the next set of logical tests to run so as to arrive at the potential root cause .
TIA

4

23.7 Legacy Series / opnsense 23.7.4 - pppoe ipv4 offline. Revert to 23.7.1 fixes it

« on: September 24, 2023, 01:10:58 pm »

Upgraded to 23.7.4 last night and realized this morning that my primary WAN (PPPoE) is showing offline on ipv4 although online on ipv6. ipv4 connections from clients were also being routed via backup WAN.

After spending a couple hours trying to troubleshoot, performed a opnsense-revert to 23.7.1 and that fixed it immediately

Just thought of sharing this here in case anyone else runs into this issue

5

23.7 Legacy Series / Opnsense idle power consumption 2X that of windows for the same hardware?

« on: September 11, 2023, 08:43:10 am »

I had been running opnsense virtualized for the past several months and finally decided it deserves a dedicated hardware unit of its own.

I decided to redeploy an old HP SFF with i5-6500t that was being used as a desktop for this.
The hardware HP 800G4 with i5-6500t / 8GB RAB and a 256GB SATA disk.

While running windows , the unit used to idle at 14-16W most of the time (as measured by a reasonably accurate kill-a-watt meter on the wall plug) 

After installing opnsense 23.7, i enabled Power-D and set states to adaptive.
However the unit now seems to draw 28-35W all the time.
The use case is for a home and the CPU utilization is 3-5% most of the time which is about the same as windows idle.
There are no CPU intensive packages on the system and IDS etc is disabled.
The only additional plugin installed is wireguard although no inbound VPN connections were on during testing.
 
Power is expensive in the region I live in (about 20 US cents per kwh) and any opportunity to optimize it is welcome.
Even technically, I am surprised why FreeBSD/ Opnsense is drawing 2X power for the same CPU utilization.

Any suggestions or pointers?

6

23.1 Legacy Series / Re: ipv6 connectivity getting dropped after few hours

« on: May 13, 2023, 11:20:14 pm »

You should definitely keep "request prefix only" in your settings.

I'd still hope you could install the dhcp6c test version

# opnsense-revert -z dhcp6c

and disable prevent release to see if that makes a difference after the next boot and subsequent disconnect.


Cheers,
Franco

Sure, can try that out this weekend.
Currently it's been 2+ days and connectivity haas been rock stable but It may be worth pointing out a couple of things

a) I have a virtualized opnsense install on proxmox and the adapters use VirtIO - not sure if that could be one of the underlying reasons

b) One of the things I had done on my previous install was installing sensei.
Looking through the forums, it seems there are other users who have also had issues with a combination of sensei with  ipv6.
At the time of testing, even though sensei was disabled but the fact that LAN client could not even ping the valid ipv6 addr of the LAN interface makes me suspect that the presence of sensei could be one of the underlying reasons (or ot could be a complete red herring too)
Either way, the fresh install does not have sensei and  I have now added every other plugin I had previously.

This weekend, I will backup the opnsense VM and re-add sensei to see if it breaks connectivity

On an unrelated note, now that ipv6 seems to be running stable, I added a pass ICMP rule on the primary WAN interface to the LAN network alias as I had read somewhere that ipv6 relies a lot on certain kind of ICMP requests.
Now it could be a placebo but post the change, the general(and subjective)  internet responsiveness seems to have improved.

7

23.1 Legacy Series / Re: ipv6 connectivity getting dropped after few hours

« on: May 11, 2023, 07:06:31 am »

Thanks @Cyberturtle and @Franco

Based on your hints, I think I may have figured it out.
There was way too much finagling I had done on the opnsense router to fix this and I figured a fresh start may be the right way to do it.


A fresh install + setting the Prevent release and only request prefix flag to enabled, things seem to be fine now..

8

23.1 Legacy Series / Re: ipv6 connectivity getting dropped after few hours

« on: May 10, 2023, 11:25:08 pm »

I recently setup an OPNsense router on my side. I have read somewhere that some ISP require the option „Only request a prefix“. If I got it right it’s necessary as some ISP provide the prefix by DHCPv6 and the WAN address via SLAAC. Maybe you could try this option?

I’ll try the new version 23.1.7. with the DHCP6c patch as soon as I have some time. Currently my IPv6 seems stable as there is no nightly ASSIA forcing a reconnect.

Thanks - I set the WAN to only request a prefix and that seems to have fixed the /128 issue
The WAN side is now getting a /64 address and prefix
The LAN interface also has a /64 prefix

Here is where it gets weird

LAN clients are also getting a /64 prefix but cannot ping6 even the LAN router addres, let alone a WAN address

eg Client 1 has obtained 2401:4900:1c02:1045:x:x:x:4c50/64
LAN Inteface on router is 2401:4900:1c02:1045:x:x:x:963e/64

but ping6 or traceroute6 from client 1 to LAN interface results in the following that does not proceed any further

Code: [Select]

abhinav@Abhinavs-MacBook-Air ~ % ping6 2401:4900:1c02:1045:x:x:x:963e
PING6(56=40+8+8 bytes) 2401:4900:1c02:1045:x:x:x:4c50 --> 2401:4900:1c02:1045:x:x:x:963e


Code: [Select]

traceroute6 to 2401:4900:1c02:1045:x:x:x:963e (2401:4900:1c02:1045:x:x:x:963e) from 2401:4900:1c02:1045:x:x:x:4c50, 64 hops max, 12 byte packets
 1  2401:4900:1c02:1045:x:x:x:4c50  3033.698 ms !A  3033.286 ms !A  3033.940 ms !A

9

23.1 Legacy Series / Re: ipv6 connectivity getting dropped after few hours

« on: May 10, 2023, 10:10:07 pm »

There might be changes coming to dhcp6c soon. We can try one more thing if you are able to reproduce this at will. Would appreciate the help here!

This might fix it too.. https://github.com/opnsense/dhcp6c/commit/db9f45927


Cheers,
Franco

So there is more to this issue it seems

After changing the release flag, the connectivity on the WAN side remained stable for another hour or so after which clients started to fail to route via ipv6.

Clients had a valid ipv6 address but ping6/tracroute6  could not find a valid route.
So I checked further and (not sure if this is relevant- i am not an expert on ipv6), the WAN IP had a /128 prefix now.

The delegated prefix was /64 (see attached screenshot)

Again, I would have normally blamed the ISP for this but I decided to test further and plugged in an old TP Link ER-605 router bridged to the ISP equipment.
With the ER605, I was able to instantly obtain a xxxx/64 address for the router as well as the LAN SLAAC+Stateless DHCP mode selected) and everything is working as expected.

The ipv6 connectivity was also immediate as against a reasonably long wait time on opnsense after pppoe completion and ipv4 up.

Not quite sure what could be causing this :|
I can't really use the ER605 as a replacement for opnsense for many reasons - not to mention that it does not have any firewall for v6

10

23.1 Legacy Series / Re: ipv6 connectivity getting dropped after few hours

« on: May 10, 2023, 05:39:25 pm »

There might be changes coming to dhcp6c soon. We can try one more thing if you are able to reproduce this at will. Would appreciate the help here!

This might fix it too.. https://github.com/opnsense/dhcp6c/commit/db9f45927


Cheers,
Franco

Thanks for looking into this @franco
Don't think I am able to reproduce this at will as so far , the drops and subsequent non recovery have been seemingly random.

I have enabled the "Prevent Release" flag and its been just over an hour with no drops.
The ppp log also seems to be blissfully static.
Will monitor it overnight and report back if there are any drops or issues!
Happy to help whichever way I can.
Let me monitor the connection

11

23.1 Legacy Series / Re: ipv6 connectivity getting dropped after few hours

« on: May 10, 2023, 11:51:02 am »

I'd try to set "Prevent release" on Interfaces: Settings page to see if that at least keeps connectivity.

But it looks like a daily disconnect?

"dhcp6c RELEASE on pppoe0" only happens once as far as I can see.

I also assume you have "Use IPv4 connectivity" set for IPv6 on the WAN? And IPv6 mode is set to DHCPv6?


Cheers,
Franco

Thanks - let me try enabling prevent release and see if that helps.
Its not exactly a daily disconnect. I have had instances where it has dropped within an hour or two on multiple occasions on the same day .
And sometimes it sticks for several hours.
What I pasted was a single snippet for ease of reading.

And yes, ipv6 on the WAN is via IPv4 connection and IPv6 is set to DHCPv6

12

23.1 Legacy Series / ipv6 connectivity getting dropped after few hours

« on: May 10, 2023, 09:11:11 am »

Running OPNsense 23.1.7_3-amd64 on a multi WAN failover setup.

The primary connection is a gigabit fiber over PPPoE (ISP - Airtel )
With the ISP provided equipment in routing mode, both ipv4 and ipv6 are normally 100% stable with no reboots etc required for months.

With the ISP equipment set to bridging mode, opnsense obtains ipv4 and ipv6 as expected on reboot.
All clients get an ipv6 address as expected.

However after anywhere between an hour to several hours, the ipv6 connectivity on WAN drops and requires a pppoe restart.

Looking at the Logs, the one place i could find something of relevance was under System>Logs>General
dhclient seems to be generating at new resolv.conf every hour.
A few hours after last restart - I see the following in the log

Code: [Select]

2023-05-10T10:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T09:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T08:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T07:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T06:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T05:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.63.255'

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T05:33:47
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.63.255'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.63.255'

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T05:31:41
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T05:31:37
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:37
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6

[b][color=red]2023-05-10T05:31:36
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:36
Notice
dhcp6c
dhcp6c REQUEST on pppoe0 - running newipv6

2023-05-10T05:31:34
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:34
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6[/color]

[color=red]2023-05-10T05:31:34
Notice
dhcp6c
RTSOLD script - Sending SIGHUP to dhcp6c[/color]

2023-05-10T04:41:11
Notice
dhclient
Creating resolv.conf
[/b]
2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.x.255'

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T04:37:56
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.x.255'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet6 default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: keeping current inet default gateway '122.169.x.255'

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2

2023-05-10T04:36:49
Notice
opnsense
/usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults

2023-05-10T03:41:11
Notice
dhclient
Creating resolv.conf

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: Chose to bind AIRTEL_PPPOE on 122.169.x.x since we could not find a proper match.

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: plugins_configure monitor (execute task : dpinger_configure_do())

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: plugins_configure monitor ()

2023-05-10T03:25:27
Notice
opnsense
/system_gateways.php: ROUTING: keeping current inet6 default gateway 'fe80::c242:d0ff:fe94:814e%pppoe0'


In particular, this part is where the failure seems to occur

Code: [Select]

2023-05-10T05:31:37
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:37
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6

2023-05-10T05:31:36
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:36
Notice
dhcp6c
dhcp6c REQUEST on pppoe0 - running newipv6

2023-05-10T05:31:34
Notice
opnsense
/usr/local/etc/rc.newwanipv6: Failed to detect IP for Airtel[opt2]

2023-05-10T05:31:34
Notice
dhcp6c
dhcp6c RELEASE on pppoe0 - running newipv6

2023-05-10T05:31:34
Notice
dhcp6c
RTSOLD script - Sending SIGHUP to dhcp6c

2023-05-10T04:41:11
Notice
dhclient
Creating resolv.conf
Any pointers what settings should I be looking at ?
I would have normally attributed this to the ISP but if I use the ISP router, ipv6 remains rock solid