Intermittent and transient network errors

[GeoffW]

Regarding the conf file: at first I directly edited /var/unbound/etc/dot.conf but then discovered that was being overwritten (presumably from the GUI config support).  So I browsed the unbound.conf file and found out I could drop my own .conf into /var/unbound/etc/ and it would be picked up, which is what I had done.


I'm running these VMs under VMware Workstation v16 (currently over a Windows 10 host).  I first set this up with pfSense years ago, intending it just for evaluation purposes, but it was convenient and seemed to work stably (and the performance boost over my dedicated IPCop firewall of the time was impressive) and so I've kept it that way (over various OS versions and hardware).  The WAN network adapter is disabled on the host OS, and only selected into the firewall VMs (Bridged).  The LAN adapter is shared with the host in Bridged mode.

I can't see it being a network adapter issue, it's the same adapter definition for all interfaces on both machines ("e1000") and I have not overridden MTU or similar details so they should be the same.  The pfSense firewall VM is running with a single processor and 1GB of RAM (and runs Squid, SquidGuard and even ntopng, so it's really putting in).  The OPNsense firewall has been given 2 processors and 4GB of RAM (no proxy or anything else exciting, so it should be kicking back cooling its heels).

I didn't need any MAC cloning.  I had already configured the Cradle to give me an area for static IPv4 addressing, which is what I've used on the firewalls - and they both use the same address.  This means I can only have one running at a time, but that's true anyway thanks to DHCP etc.  As expected, traceroute shows no difference, 11 hops, identical path.


After all the mucking about, it seemed a good idea to revert back to an early snapshot, which I did.  This had just DHCP and Captive Portal and used ordinary DNS.  This was still getting packet loss - sometimes up to 9%!  I disabled Captive Portal and packet loss seemed to subside (but was still happening), but this may have been just coincidence.

That didn't resolve the issue so I returned to my more fully configured snapshot and updated it to 21.1.2.  I rather like the idea of the "Audit now" options, I did both a security and a health audit - all reported okay.  Then I rebooted to be sure.  Re-tested and the same problem persists.

A few times today my pfSense firewall has reported some packet loss (1-2%) and some long latency times (network is obviously busy, download was slower but uploads still fast), but this has not resulted in the same protocol errors or connection resets, things just went slower - which is what I expect.


I think it's time I accepted defeat.  There's obviously something about this set up that OPNsense doesn't like.  I'll keep the VM around to try again with a future update.

Thanks everyone for your input.

[youngman]

I had similar loss issues a while back and it came down to MTU as someone posted earlier. Just had to put an override number in at the WAN interface and it was all good. No idea why it couldn't auto detect and correct the MTU... I suspect it was ISP related.

If you are monitoring the gateway, are your tolerances set too tightly - causing it to restart itself intermittently?

System: Gateways: Single --> Advanced (perhaps temporarily disable monitoring just to eliminate that possibility?)

[GeoffW]

I had been reluctant to "play with" the MTU size because I'm not enough of an expert to know the consequences of my choices ... but inputting the default value of 1500 seemed safe and easy enough, so I did that on both WAN and LAN interfaces.  No change.

I also tried disabling the gateway monitor.  No change.


Today I was experimenting with pfBlockerNG on my pfSense firewall and I see that when it blocks via DNSBL the result is sometimes a ERR_SSL_PROTOCOL_ERROR.  Of course the difference is that a page refresh in this case keeps blocking persistently.  So the problem on OPNsense is not any blocking rules (because refresh will load things that previously failed), but it does show that DNS issues could result in at least one of the errors I am seeing, although I am less clear how a DNS issue could explain the interrupted GIF loads I saw (presumably a ERR_CONNECTION_RESET).

[thowe]

An MTU of 1500 may be the standard in many Ethernet scenarios. However, especially when transmitting via PPPoE or other tunnels, a lower MTU can be more efficient, since the packets are otherwise fragmented. Depending on the protocol, this is a loss of performance or prevents connections.

That said, I don't think the MTU is the main problem here (if it is a problem at all).

[youngman]

Not suggesting that MTU size is 100% the issue but with a 3G modem I vaguely recall being forced down to ~1370ish to prevent fragmentation. 4G may be similar? Look up MTU ping test - it isn't hard to confirm an appropriate size.

Some programs do not handle fragmentation well (e.g. In my experience Steam will simply refuse to connect to their game controller), others may be unaffected - giving the impression of intermittent errors.

[GeoffW]

The MTU ping test is going to get exactly the same value as what operating system PMTUD gets more dynamically (for TCP anyway).  And the problems I am seeing are on normal browser page loads, no exciting games involved ... but never let it be said I didn't try.

The MTU ping test let through blocks of no more than 1432 bytes.  According to the articles I found I could add 28 bytes to that to get 1460 as the appropriate MTU, but then I thought: if I'm doing this lets push the issue and use 1432.  I could not find an article that was explicit about whether the MTU on a firewall needed to be set on both interfaces, but I assumed that would be best (for MTU to be a problem here we're assuming OPNsense is screwing up the fragmentation process, so let's make sure it never has to fragment by never seeing a big frame).

Change MTU on LAN and WAN and rebooted ("Apply Changes" does not actually update the MTU according to the "Overview" page).  Verified the change had taken (MTU ping test would no longer accept 1432, but would accept 1400 - and presumably 1404 - which matches expectations).

Then tested for the problem:  Still getting some packet loss, but more importantly, I am still getting the same sorts of inconsistent network errors as first described.  (I say "more importantly" because I believe the packet loss is a symptom of the underlying problem, not a cause; as noted earlier, I sometimes see packet loss on the old firewall, especially when the network is very busy, but it only makes things slower, it doesn't cause these weird transient errors.)

Thanks for your suggestion.  It was worth a shot, but I think I have now excluded MTU as the culprit here.