Messages

Sorry for the multiple updates: Had problems pretty much immediately.

I've tried `sysctl kern.timecounter.hardware=ACPI-fast` and got the issue within 5 minutes of this change. In fact, multiple times, same frequency as before, so I'll revert...

Differences I can think of apart from the config above is the direct WAN connection instead of the intermediate router - but this was working before.

Just to leave an update: It took me longer than I wanted to get back to this. No issues experienced since my last update with WAN in DHCP, I just now went back to PPPoE. These are the only changes I did as well as a couple of reboots of both the VM and the proxmox host (which I had tried before):

Code Select Expand

--- /conf/backup/config-1790233266.5738.xml    2026-09-24 08:01:06.576665000 +0100
+++ /conf/backup/config-1790612178.7565.xml    2026-09-28 17:16:18.760095000 +0100
@@ -322,6 +322,14 @@
       <type>none</type>
       <virtual>1</virtual>
     </lo0>
+    <wan>
+      <if>vtnet4</if>
+      <descr>WANETH</descr>
+      <enable>1</enable>
+      <lock>1</lock>
+      <spoofmac/>
+      <mtu>1508</mtu>
+    </wan>
     <opt1>
       <if>bridge0</if>
       <descr>LAN</descr>
@@ -391,34 +399,46 @@
       <type>group</type>
     </local_intf>
     <opt8>
-      <if>vtnet4</if>
+      <if>pppoe0</if>
       <descr>WAN</descr>
       <enable>1</enable>
       <lock>1</lock>
       <spoofmac/>
       <blockpriv>1</blockpriv>
       <blockbogons>1</blockbogons>
-      <mtu>1500</mtu>
-      <ipaddr>dhcp</ipaddr>
-      <dhcphostname/>
-      <alias-address/>
-      <alias-subnet>32</alias-subnet>
-      <dhcprejectfrom/>
-      <adv_dhcp_pt_timeout/>
-      <adv_dhcp_pt_retry/>
-      <adv_dhcp_pt_select_timeout/>
-      <adv_dhcp_pt_reboot/>
-      <adv_dhcp_pt_backoff_cutoff/>
-      <adv_dhcp_pt_initial_interval/>
-      <adv_dhcp_pt_values>SavedCfg</adv_dhcp_pt_values>
-      <adv_dhcp_send_options/>
-      <adv_dhcp_request_options/>
-      <adv_dhcp_required_options/>
-      <adv_dhcp_option_modifiers/>
-      <adv_dhcp_config_advanced/>
-      <adv_dhcp_config_file_override/>
-      <adv_dhcp_config_file_override_path/>
-      <ipaddrv6>slaac</ipaddrv6>
+      <mtu>1508</mtu>
+      <ipaddr>pppoe</ipaddr>
+      <ipaddrv6>dhcp6</ipaddrv6>
+      <dhcp6-ia-pd-len>8</dhcp6-ia-pd-len>
+      <dhcp6-ia-pd-send-hint>1</dhcp6-ia-pd-send-hint>
+      <dhcp6prefixonly>1</dhcp6prefixonly>
+      <adv_dhcp6_interface_statement_send_options/>
+      <adv_dhcp6_interface_statement_request_options/>
+      <adv_dhcp6_interface_statement_information_only_enable/>
+      <adv_dhcp6_interface_statement_script/>
+      <adv_dhcp6_id_assoc_statement_address_enable/>
+      <adv_dhcp6_id_assoc_statement_address/>
+      <adv_dhcp6_id_assoc_statement_address_id/>
+      <adv_dhcp6_id_assoc_statement_address_pltime/>
+      <adv_dhcp6_id_assoc_statement_address_vltime/>
+      <adv_dhcp6_id_assoc_statement_prefix_enable/>
+      <adv_dhcp6_id_assoc_statement_prefix/>
+      <adv_dhcp6_id_assoc_statement_prefix_id/>
+      <adv_dhcp6_id_assoc_statement_prefix_pltime/>
+      <adv_dhcp6_id_assoc_statement_prefix_vltime/>
+      <adv_dhcp6_prefix_interface_statement_sla_len/>
+      <adv_dhcp6_authentication_statement_authname/>
+      <adv_dhcp6_authentication_statement_protocol/>
+      <adv_dhcp6_authentication_statement_algorithm/>
+      <adv_dhcp6_authentication_statement_rdm/>
+      <adv_dhcp6_key_info_statement_keyname/>
+      <adv_dhcp6_key_info_statement_realm/>
+      <adv_dhcp6_key_info_statement_keyid/>
+      <adv_dhcp6_key_info_statement_secret/>
+      <adv_dhcp6_key_info_statement_expire/>
+      <adv_dhcp6_config_advanced/>
+      <adv_dhcp6_config_file_override/>
+      <adv_dhcp6_config_file_override_path/>
     </opt8>
     <opt9>
       <if>vlan02</if>
@@ -481,8 +501,8 @@
   </ntpd>
   <revision>
     <username>[___myusername___]</username>
-    <description>/interfaces.php made changes</description>
-    <time>1790233266.57</time>
+    <description>/api/core/backup/revert_backup/config-1790202624.039.xml made changes</description>
+    <time>1790612178.76</time>
   </revision>
   <OPNsense>
     <wireguard>

If it breaks I'll continue debugging with the suggestion above, I'm unsure if the PPPoE/DHCP change (and lack of issues) gives any interesting data point. If it doesn't break further I'll also leave an update.

Thanks!

I haven't yet tried your suggestions - I had to get reliable internet for the day so I've switched to my ISP router. Interesting things happened though.

I switched my opnsense WAN from PPPoE to DHCP and connected it to my ISP router. And it's been working again since then (the opnsense connection that is - I kept everything running). I'll give it one day or so (same time as before) to see if it breaks again or if PPPoE is a factor.

The changes I did was change the WAN interface, reduce the MTU from 1508 to 1500 and I had a physical (I called it WANETH) interface for the physical (well, later virtualised) interface to also just set the MTU to 1508 which I removed for now. Unrelated but since it may not be very orthodox I did test this MTU and it was working without breaking packets.

I'll try what you suggest either when this fails with DHCP or I'll simply go back to PPPoE to try tomorrow or on the weekend worst case.

Thanks again!

Thanks, here's the result of those:

Code Select Expand

root@OPNsense:~ # sysctl kern.timecounter.hardware
kern.timecounter.hardware: kvmclock
root@OPNsense:~ # sysctl kern.timecounter.choice
kern.timecounter.choice: TSC(-100) i8254(0) ACPI-fast(900) kvmclock(975) dummy(-1000000)
root@OPNsense:~ # sysctl kern.timecounter.fast_gettime
kern.timecounter.fast_gettime: 1
root@OPNsense:~ #
root@OPNsense:~ # sysctl kern.eventtimer.timer
kern.eventtimer.timer: LAPIC
root@OPNsense:~ # sysctl kern.eventtimer.choice
kern.eventtimer.choice: LAPIC(600) i8254(100) RTC(0)
root@OPNsense:~ # sysctl kern.eventtimer.periodic
kern.eventtimer.periodic: 0
root@OPNsense:~ #
root@OPNsense:~ # sysctl kern.eventtimer.et.LAPIC.frequency
kern.eventtimer.et.LAPIC.frequency: 500003658
root@OPNsense:~ # sysctl kern.eventtimer.et.LAPIC.quality
kern.eventtimer.et.LAPIC.quality: 600


No, they don't coincide with snapshots or backups of this or other VM/LXC in the system. I can't say they never match as I do have 15min snapshots  (zfs) and not very precise detection of this, but the distribution is random, I have plenty of examples falling out of the snapshot/backup timestamps. It's like 5 events per hour, sometime less sometimes more.


Timers do seem relevant though - I'm parroting ChatGPT a bit, but it helped me build some dtrace analysis and (I'll quote a paragraph):

...
More importantly, the fourth capture did work, and it contains exactly the evidence we were looking for. There are substantial samples in:

Code Select Expand

tcp_timer_rexmt
tcp_timer_enter
softclock_call_cc
softclock_thread
and also a large amount of networking processing around tcp_input_with_port, IPv6, netgraph, VirtIO, etc.

So the earlier clock observation was real: at least during this event, softclock_thread is spending time processing TCP retransmission timers. That is a significant narrowing of the problem.
...


EDIT: taking a hint from the linked post thread:

Code Select Expand

root@OPNsense:~ # vmstat -i | grep timer
cpu0:timer                       8789276        829
cpu1:timer                        312606         29
cpu2:timer                        316936         30
cpu3:timer                        281377         27


Unfortunately I need to go back on my words: while everything was good for 1 day the problem came back again.

The only thing I did to make it better were a few restarts of the opnsense VM and removing the passthrough from the VM.


I've tried another reboot (of opnsense), but even immediately after that the issue repeated itself.


I noticed a post above about PPP(oE) and my WAN is a PPPoE connection (no VLAN) in case that makes a difference, but I get the feeling the post might be some unrelated boot issue?

I'll try to dig a bit further, but any pointer on the best way to go about identifying the problematic process/thread/interrupt sources would be helpful.

Thanks!

Not sure if it counts as "solved", but setting up as virtio doesn't cause the issue anymore! The intel drivers might have brought something finicky, but for virtualisation this works! Thanks again!

Thanks meyergru, perhaps not a great one, but when I set things up I followed some friends setup that had used passthrough.
Perhaps silly, but I was also learning my steps and didn't want to accidentally give my proxmox an IP/access via WAN.


Regardless, just to acknowledge the answer and I just ported everything to virtio.
Let me monitor for a while. The first 15min look good, there's very minor peaks, on the healthy side (5% as opposed to 25% of my total CPUs), but sometimes I had 20-30min between peaks so let me give it a few hours.

Hi,

Long time lurker (thanks everyone, very useful posts), first time poster.

I'm having an issue since the 15th/16th of September that results in momentary connection drops.

Context:
I use opnsense as a firewall in my homelab, I was away for some time and on my return I immediately started experiencing occasional internet drops for 8-10sec.
OPNsense is virtualised in proxmox with the NICs (I226-V) passthroughed directly to the VM. CPU is in host mode (4 vCPUs, physical CPU is an N150).
On debug, I saw proxmox CPU usage history with higher spikes since the aforementioned dates - so I suspect that's when it started.

Current observations:
If I monitor the ping to 8.8.8.8 or 9.9.9.9 (in fact I used both simultaneously) - I sometimes get spikes of 5-10sec. During this time, I simultaneously looked at CPU usage and the system usage spikes. ChatGPT got involved from here to help debug and may be taking me in wrong directions - it did help spot that clock{clock (0)} consumes 100% of 1 CPU over this time, but I'm not sure if this by itself is very valuable.



I'm trying to progress from here but I feel like I'm getting into the weeds of the kernel's working. Is there a recommended debug strategy for this? Since the dates coincide with the latest updates was any known change introduced? The only thing that stood out (disclaimer disclaimer: helped by ChatGPT to spot it) was "o src: iflib: assorted upstream patches from stable/15" but since it was working before it would imply a regression.

In the meanwhile I'll keep trying to break down this clock thread that spikes.

Many thanks!