Strange hiccup in flows

[utahbmxer]

Hi

Seeing some hiccups/blips in connections every 20-30 seconds, more noticeable with UDP traffic such as video conference calls, and some games running on my server in another VLAN.  All games from LAN to server in DMZ experience the blip at the same time.

Used iperf as client on OPNsense via SSH in a few different scenarios and the firewall is the only common piece.  Tests out LAN interface, and WAN (over ipsec to Azure) show same exact behavior.  They are sharing a dual port GbE Intel card, gonna try and swap it when I can find time.

PktCaps from the firewall only show one packet out of order, so it's like things hum a long fine, then queue up and burst out (see attachment.  No re-transmissions or ZeroWindows that I can see.

Any diagnostics that would show a hardware or software issue?  I checked Interfaces > Overview and don't see any errors on the interfaces in question, however enc0 for the ipsec stuff doesn't show?

Thanks!

[tuto2]

Hi,

Since you're able to SSH in, a few tools might help you exclude some stuff.

Try 'netstat -s' for a complete overview of the state of the protocols. 'netstat -Q' shows queue behaviour. Check the manual page for more options. It might also be worthwile to run an iperf session and monitor 'top -CHS' (or just plain top for user processes) and see if any one user/kernel process is taking up a lot of resources.

Cheers,
Stephan 

[utahbmxer]

Looks like I've narrowed it down to at least one, if not two python processes which spike and then a hiccup.   The first one I found was flowd_aggregate.py.  I have stopped the netflow service.

Issue still happens and when it does, /usr/local/bin/python3 /usr/local/opnsense/scripts/filter/update_tables.py spikes.

Going to remove my aliases which resolve various youtube names (moved this to pi-hole anyway) and see if the issue goes away.  I did notice there was a few names that were not resolving anymore, maybe related?

[utahbmxer]

Started digging around and found I have a GEO alias that isn't populating with entries.  I removed it and things seem to have cleared up.  When I add it back in as a test (only includes United States) and it sits and spins and then errors with: "Cannot allocate memory. [UnitedStates]"

I have another Alias which includes the United States and Ireland and it has 469829 prefixes, doesn't seem to have issues.  I set tunables net.pf.request_maxcount to 3000000 before trying to create the new Alias, but now it's back at 1000000.  ??