[SOLVED] Intermittent network something's!

[stuwolf]

Hi Everyone,

New OPNsense user here,

So I believe I am going a little nutts, I have an issue that I can not nail down and I "think" its related to OPNsense but honestly I am not sure. I have not been able to workout exactly what is causing the problem nor can I easily reproduce it when I want to as it happens intermittently.

First a little background, Recently ( in the last two months ) I moved from pfsense to OPNsense. The reason for the move was a few small things like I wanted to use wireguard, I like the interface more, pfsense has kind of a bad reputation and on top of that they are now moving to new pro plan stuff. All that being said however pfsense for the most part worked and these issues started showing up after the switch.

The problem -

As I said above I cannot nail it down fully, originally I thought it was a wireless issue even though it was is the same setup with pfsense. However when the issue started happening via a wired connection I started to look elsewhere. The problem seems to happen if I am trying to access data locally or externally. It feels like something is interrupting flow for just a short enough time to Bork things but not enough to easily discover what is going on.

Two main things keep happening

1. I use an AlwaysON MS VPN connection to work (I heard that its a bit temperamental on unstable networks). I get disconnected from the VPN randomly sometimes every few mins other times ever few hours. ( Both on wired and wireless ) I have confirmed by using my Mobile as a hotspot its not my work equipment as I can stay connected with no issues that way.

2. Chromecast's and Nvidia Shield. When playing Media locally from Plex I sometimes get error's saying that my connection to the server is not fast enough and the media will stop playing. If I restart immediately playing the media it then continues to play with out issues for a while. I have also noticed it says now and then ( Not every time ) that I have lost my internet connection as well.

When it comes to normal usage I have only noticed that I sometimes get lag spikes when trying to access things on the network. Be that on my local desktop or via a mobile device.

I did a dig around the logs and could not find anything that stood out but maybe I was looking in the wrong spot or for the wrong thing? Also for the fun of it I setup continuous pings both from my desktop and my work laptop.

These pings reached out to the local gateway of the VLAN (OPNSense box), Local gateway provided to my cable modem and to Google.. From my local PC nothing really showed up when I would get a connection drop apart from an increase in response time from 1-2ms to 200ms+ to the OPNsense box for a few then back to normal.

I tested the same on the work PC and it was the same but once I lost VPN I lost all network access (apart from to the OPNsense box due to the configuration of my works VPN setup)


And that's basically it. For me the issue is super annoying esp. with my VPN to work as you can imagine. I really like OPNsence and I do not want to switch back so I am looking to take the time to see if I can sort this out.

I would like some advise from someone with more experience with OPNsence on things I can test to see if I can rule out OPNsence but as I said above its the only thing that has changed in my network as of late. Did changing to OPNsence highlight a problem I have had all along with something else? I am not sure but I need to start somewhere.

My Configuration :

Hardware -

Intel(R) Core(TM) i5-4570S CPU @ 2.90GHz (4 cores)
16Gb Ram
128Gb SSD HDD
1Gb Nic Intel 82576 Chip, Dual port, PCIe 2
1Gb Nic Realtek onboard

Physical Layout for networking equipment -
Cable Modem <-> Opnsense 2x1gb <-> Ubiquiti US-24 Unifi Switch <-> 1Gb Ubiquiti UAP-FlexHD-US Access Point

General Networking Setup -

Wan :
Xfinity cable with a SB8200 connected directly to em0 (Realtek) I used the realtek for this as there is no vlans etc. as I had read this was an issue.

Internal Networks :
igb0 - Intel
 Lan(Default), House(VLAN), wireless(VLAN)

igb1 - Intel
 VPN(VLAN), DMZ(VLAN)

Each one is its own Vlan with a /24. I used the two nics originally just to spread load out a bit between the VLANS as there can be a lot of data moving about. Not sure it was worth it but that's how its setup right now and I do tend to max out them out.

VPN (Wireguard)
I have one outgoing Wireguard connection that routes all data from the VPN Vlan.
I also have one incoming Wireguard connection from my Nginx proxy that only has access to the DMZ VLAN

I do have plans to add another outgoing Wireguard connection for everything outgoing not on the VPN VLAN and also more incoming connections for remote access but I have held off until I can work out this issue.

I have 4 plugins installed right now that I am using

os-dmidecode (installed)
os-wireguard (installed)
os-dyndns (installed)
os-speedtest-devel (orphaned)

Everything else is mostly stock standard. I added few rules to allow the flow of traffic between VLANS and some rules to restrict flow from some as well.

I have a port forward rule in place from WAN to internal for Wireguard. I have not setup the other features as I am working to rule out this problem first.

If you have taken the time to read all this thank you, If I do find out what is goin on I will of course post here as well

[stuwolf]

I have managed to find some logs that match up to the drops, I am still looking into it but more or less every time I get the  promiscuous mode enabled/disabled it drops out.

Looks like it could maybe be due to PF reloads,

2021-03-16T10:23:07   kernel   pflog0: promiscuous mode enabled   
2021-03-16T10:23:07   kernel   pflog0: promiscuous mode disabled


2021-03-16T10:23:08   configd.py[86741]   message a837bd4c-4e91-4011-9239-91eefe8549ae [filter.refresh_aliases] returned {"status": "ok"}   
2021-03-16T10:23:07   configd.py[86741]   [7d5585a3-a42b-43b9-8861-d6f7601dc31e] updating dyndns WGMullvadAlpha   

[roninxt]

I also get this and I lose ipv6 connectivity.

[stuwolf]

Update for anyone who is interested,

I took the Realtek NIC out of service and went back to only using the two Intel NIC's This seems to have resolved the problem for me so far There was a lot of chat in the forums with others who had similar issues where their Realtek card was resetting and then causing the PF reload. Since I had the hardware free I though it would be just simpler to disable it since it was onboard and it seems to have worked.

[wernsting]

Very interested!

I’m having similar connectivity issues. It’s really intermittent and I can’t point to anything either, I have a Eglobal Braswell Fanless Mini PC AES-NI Intel N3160/J3160 300M Wifi https://s.click.aliexpress.com/e/lFrVbdYk which contains two Realtek NICs. Unfortunately I can’t disable those nor can I replace them with Intel ones without buying a complete new box. I used to run pfsense on it (2.4.5) without these issues. I don’t want to go back as I much prefer the opnsense interface.

Is this hardware or drivers issue you think? My understanding was that opnsense has replaced the Realtek drivers a while ago in the kernel with better supported ones from Realtek. Just don’t understand how I didn’t appear to have these issues running pfsense before.

Have these issues resolved themselves completely since moving to the Intel NIC? I might have to replace my box if that’s the only solution…

[tessierp]

I have the same issue here. Ever since the last two updates, I've been getting random network disconnects. I am running version OPNsense 21.1.4.

I ran version 20.x for a few months without any issues. I'm using this network card :

https://www.servethehome.com/syba-dual-2-5-gigabit-ethernet-adapter-review/

Works great but like I said, since the last few updates, I've been getting lost packets, random disconnections.

[tessierp]

Looking at the logs I noticed something which I think may be causing the problem but not 100% sure :

2021-04-19T23:43:24   kernel   pflog0: promiscuous mode enabled   
2021-04-19T23:43:24   kernel   pflog0: promiscuous mode disabled

Like I said, I was fine for months and all of a sudden after the last two updates, I've been getting those random disconnects.