Messages

Thanks for your reply.

I'm pretty sure I must be doing something wrong, because I cannot get any interface filter to work as expected.

As I noted in the OP, Zenarmor insists on using the nic device name, not the other possibilities (DMZ or opt1), so in this case vmx2.  But just in case I've tried all variations.  And just to be sure, I tried setting up a positive (equals) test using LAN, "LAN", 'LAN', lan, "lan", 'lan', vmx1, "vmx1", 'vmx1'

All the "NOT EQUALS" tests I've tried make no difference to the output.  All the EQUALS tests I've tried produce empty results.

I notice that the filter display, after I hit refresh, changes from showing something like: Interface: LAN

to showing: Interface: Lan

And (so on with quotes or not according to the variation I tested).  Is there any chance this is actually doing a case-sensitive match and so getting it wrong on my all upper or all lowercase interface names?  Or is this just a filter display error?

It really seems like this must be a user error, but I cannot work out what I'm doing wrong.  Removing the interface filter brings everything back to what I've described.

Some extra detail...

The "Table of Local Assets" shows IP addresses (untranslated IP addresses) from LAN and DMZ, so this looks right.

The "Table of Remote Hosts" shows only the two default gateway IP addresses (one for the LAN interface and one for the DMZ interface).   Perhaps this is expected, but if so it doesn't seem particularly useful.

I did just realise that using the drilldown and session detail feature from the Top Local Hosts tries to use these as the source address, and since most of these are Internet destination addresses this produces empty results.  (Ditto the Top Remote Hosts searching for destinations using source addresses.)  Which is a relief, it suggests local and remote do mean what I think they mean, and it's ZenArmor that's having an inconceivable moment.


Note that doing a drilldown or session detail from "Top Local Server Ports" searches for a destination port, which makes sense because servers are destinations by definition I would have thought, but of course finds records in which the destination IP is Internet (remote).  "Top Remote Ports" also searches for destination port, which also makes sense, but in my case finds only destination IPs on the LAN or DMZ (for port 53 because of my DoT set up).  But then the filtering from these reports is a bit suspect anyway, since filtering only by destination port is not going to give local vs remote distinction anyway, and that is what I'm seeing when I use this on ports used on both local and remote servers.

I've got a new OPNsense installation, and now I'm trying ZenArmor v1.12.3 in Passive mode.  I'm mostly interested in the session details capability, I don't really want all the extra block/filtering smarts so Passive mode seems the best fit.   Installed using MongoDB.  OPNsense is running as a VM inside ESXi v7 using three physical interfaces I called WAN, LAN, DMZ.  ZenArmor set to monitor LAN and DMZ (but insists on using the underlying interface name vmx1 and vmx2).

Anyway the confusion coming in when I see that "Top Local Hosts" reports mostly Internet names/addresses, although there are a couple of LAN address in there too, while "Top Remote Hosts" is showing only LAN addresses.

Much the same on "Top Local Server Ports" vs "Top Remote Ports".  The "local" is showing stuff I know to be destination ports on the Internet, while the "remote" is showing stuff I'm pretty sure is LAN only.

I have set up to use DNS over TLS with LAN and DMZ port-forward rules to enforce this.  So there is port 53 traffic on the LAN and DMZ interfaces but should only be 853 over the WAN.  Could this be confusing ZenArmor.

Or maybe I am the one who is confused and perhaps "local" and "remote" mean different things in this context?

PS: or is this a duplicate of this question: https://forum.opnsense.org/index.php?topic=31674.0

The MTU ping test is going to get exactly the same value as what operating system PMTUD gets more dynamically (for TCP anyway).  And the problems I am seeing are on normal browser page loads, no exciting games involved ... but never let it be said I didn't try. :)

The MTU ping test let through blocks of no more than 1432 bytes.  According to the articles I found I could add 28 bytes to that to get 1460 as the appropriate MTU, but then I thought: if I'm doing this lets push the issue and use 1432.  I could not find an article that was explicit about whether the MTU on a firewall needed to be set on both interfaces, but I assumed that would be best (for MTU to be a problem here we're assuming OPNsense is screwing up the fragmentation process, so let's make sure it never has to fragment by never seeing a big frame).

Change MTU on LAN and WAN and rebooted ("Apply Changes" does not actually update the MTU according to the "Overview" page).  Verified the change had taken (MTU ping test would no longer accept 1432, but would accept 1400 - and presumably 1404 - which matches expectations).

Then tested for the problem:  Still getting some packet loss, but more importantly, I am still getting the same sorts of inconsistent network errors as first described.  (I say "more importantly" because I believe the packet loss is a symptom of the underlying problem, not a cause; as noted earlier, I sometimes see packet loss on the old firewall, especially when the network is very busy, but it only makes things slower, it doesn't cause these weird transient errors.)

Thanks for your suggestion.  It was worth a shot, but I think I have now excluded MTU as the culprit here.

I had been reluctant to "play with" the MTU size because I'm not enough of an expert to know the consequences of my choices ... but inputting the default value of 1500 seemed safe and easy enough, so I did that on both WAN and LAN interfaces.  No change.

I also tried disabling the gateway monitor.  No change.


Today I was experimenting with pfBlockerNG on my pfSense firewall and I see that when it blocks via DNSBL the result is sometimes a ERR_SSL_PROTOCOL_ERROR.  Of course the difference is that a page refresh in this case keeps blocking persistently.  So the problem on OPNsense is not any blocking rules (because refresh will load things that previously failed), but it does show that DNS issues could result in at least one of the errors I am seeing, although I am less clear how a DNS issue could explain the interrupted GIF loads I saw (presumably a ERR_CONNECTION_RESET).

Regarding the conf file: at first I directly edited /var/unbound/etc/dot.conf but then discovered that was being overwritten (presumably from the GUI config support).  So I browsed the unbound.conf file and found out I could drop my own .conf into /var/unbound/etc/ and it would be picked up, which is what I had done.


I'm running these VMs under VMware Workstation v16 (currently over a Windows 10 host).  I first set this up with pfSense years ago, intending it just for evaluation purposes, but it was convenient and seemed to work stably (and the performance boost over my dedicated IPCop firewall of the time was impressive) and so I've kept it that way (over various OS versions and hardware).  The WAN network adapter is disabled on the host OS, and only selected into the firewall VMs (Bridged).  The LAN adapter is shared with the host in Bridged mode.

I can't see it being a network adapter issue, it's the same adapter definition for all interfaces on both machines ("e1000") and I have not overridden MTU or similar details so they should be the same.  The pfSense firewall VM is running with a single processor and 1GB of RAM (and runs Squid, SquidGuard and even ntopng, so it's really putting in).  The OPNsense firewall has been given 2 processors and 4GB of RAM (no proxy or anything else exciting, so it should be kicking back cooling its heels).

I didn't need any MAC cloning.  I had already configured the Cradle to give me an area for static IPv4 addressing, which is what I've used on the firewalls - and they both use the same address.  This means I can only have one running at a time, but that's true anyway thanks to DHCP etc.  As expected, traceroute shows no difference, 11 hops, identical path.


After all the mucking about, it seemed a good idea to revert back to an early snapshot, which I did.  This had just DHCP and Captive Portal and used ordinary DNS.  This was still getting packet loss - sometimes up to 9%!  I disabled Captive Portal and packet loss seemed to subside (but was still happening), but this may have been just coincidence.

That didn't resolve the issue so I returned to my more fully configured snapshot and updated it to 21.1.2.  I rather like the idea of the "Audit now" options, I did both a security and a health audit - all reported okay.  Then I rebooted to be sure.  Re-tested and the same problem persists.

A few times today my pfSense firewall has reported some packet loss (1-2%) and some long latency times (network is obviously busy, download was slower but uploads still fast), but this has not resulted in the same protocol errors or connection resets, things just went slower - which is what I expect.


I think it's time I accepted defeat.  There's obviously something about this set up that OPNsense doesn't like.  I'll keep the VM around to try again with a future update.

Thanks everyone for your input.

None of the changes I've tried removed the intermittent SSL protocol errors nor the connection reset errors.

Monitoring 1.1.1.1 from the Gateway "Monitor IP" setting showed intermittent periods of packet loss between 1% and 4%.  The RTT was sitting up around 160ms.

Disclaimer: I don't know anything about the following website except that it claims to do what I needed for this test, use at your own risk.  The Packet Loss Test website: https://packetlosstest.com also reports packet loss between 1% and 4% across several tests, and has latency average sitting up around 150ms with a lot calls stretching up above 200ms.


Switching back to my old pfSense firewall, the errors go away as usual.  I set Gateway monitoring to look at 1.1.1.1 (as for OPNsense above) and it reported an RTT of approximately 50ms (on an otherwise idle network, it goes up during speed tests etc.), and I have not seen packet loss go above 0.0%.  Also, the Packet Loss Test website lost only a single packet over several tests and reports latency between 50ms and 80ms across several tests.

...

So something's going on with the WAN connection under OPNsense, but I don't seem to be any closer to working out what.  I thought TCP generally arranged for automatic re-transmission of lost packets, so unless the packet loss rate is very high it should normally be transparent other than the performance loss (no errors like those that I am seeing).


Thank you very much opnfwb and ingof for your suggestions.  I am open to more, but may be slower responding because I really have to catch up on some work that I've been ignoring while trying to get this going.

Thanks heaps for the suggestions and extra detail.

After those changes (directly editing dot.conf with domain name, turn off blocking of private/bogon in WAN, setting up on monitoring of gateway using IP 1.1.1.1), the results are:

I still get SSL protocol and connection reset errors (intermittently and temporarily) as first described.

I saw a lot of packet loss on the gateway monitor during startup (1..4%), and I am seeing some intermittent (but much smaller) bouts of packet loss since.  Still no errors reported on the interface itself.

I've only been running a few minutes after the change so far, so I'll leave it a while longer to see how the packet loss is reported.

Thanks very much for that explanation regarding more secure DoT.  I will give that a try.

I actually like having the block private networks still ticked, since the point is to protect the LAN side even from guests whose devices I do not necessarily trust ... still, I guess I could try it and just see (no guests here now anyway).  Probably this evening, I'll let you know.

You had me wondering for a moment :) so I double checked.  The WAN side definitely only has the one cable and that plugs into the interface for this firewall.

The WAN side is a bit messy (but doesn't change between firewalls).  I'm remote and use a 4G wireless (from Telstra in Australia) broadband connection.  The actual router with the SIM is a small "Hotspot" device, but because I'm not actually mobile I plug it into a NetGear Cradle (with external antenna) that acts as separate router.  So, roughly:

Code Select Expand


   [ LAN ]-----[OPNsense]-------[ Cradle ]-------[ Hotspot ]-------[whatever-Telstra-has]-----{internet}
           LAN         10.x.x.x/24       10.y.y.y/24        10.z.z.z/?
          subnet


The Cradle does have WiFi that guests use.  I have a separate WiFi on the LAN side of the firewall for my own devices.  There is no cross-over using WiFi.

And like I said, none of the WAN side changes.  I'm just sliding out the pfSense VM and sliding in the OPNsense VM.  So I doubt if the mess is relevant and probably just distracts.


Thanks for your response.  If you see something in that mess that might make a difference peculiar to OPNsense then let me know.  Note that I did try ticking/unticking that "Disable Reply-To" Multi-WAN option after reading posts here, but it didn't seem to make any difference either way.

Thanks very much for your response.

For Unbound DNS setup I followed the instructions I found here: https://www.dnsknowledge.com/unbound/opnsense-set-up-and-configure-dns-over-tls-dot/  (including checking the resulting dot.conf file).

As I understand it, with 21.1 you no longer need the Custom Options.  The instructions seemed to work (confirmed traffic on 853 and none on 53).  At one stage I had both CloudFlare and Google DNS servers defined, but then I read that mixed DNS sources can lead SSL protocol errors (although I don't think those articles were talking of this particular situation, I think they were talking about DNS on the browser/client being in conflict with the DNS reported by the network).


None of the interfaces show any errors (both sides reporting 0/0 for in/out errors).  Which I suppose makes my subject line here a bit ambiguous.  The errors are not hard network errors, but something higher level.  Also no blocks or rejects get reported on the Firewall (which I thought might happen after the connection reset errors, but no).

So the only errors (so far) are observed on browser clients (mostly Firefox and Vivaldi, Vivaldi is a Chromium based browser, running on Windows 10 20H2) - although this includes video and music player components that get interrupted with connection resets.

I'm all out of ideas, so I've switched back to the old firewall and the protocol errors and connection resets have stopped.  I think the throughput is a bit slower, but at least its working.  I really like (most of) the interface and reporting features of OPNsense, but not at the expense of core network function.

Maybe I'll come back and try again when you're up to Zealous Zorilla or something.

I'm getting the same thing on my VMware VM console.  As far as I can tell the messages come from running the OPNsense GUI, whether it's the dashboard or something else I'm not sure, the messages seem to come a few in a burst and then some delay - sometimes only a few seconds, sometimes many minutes.

If there is any sort of polling of the VM console it must be built into the VMware host, because it's nothing I'm doing.  And the only thing accessing the OPNsense GUI is my browser.

Just had another glitch on another site that pretty much proves it's not DNS.  A GIF started loading and actually started playing, before suddenly blinking out and being replaced with a broken-link icon.  The console showed it had received an ERR_CONNECTION_RESET error.  There are no network errors reported on the interfaces, the Internet link has been at least as stable as it was before I put in this firewall.

And speaking of broken-link icons, this very page with its list of emojis above the edit area has approximately half of them displayed as broken link icons.  I attach two more screen captures, the display and the console showing the errors.

A few days ago I set up OPNsense v21.1 (and updated to v21.1.1) on a virtual machine (VMware), to experiment with as a replacement for a pfSense installation.  Lots to like after I got my head around the things that are different, but I've got intermittent network errors that I was not getting with the other firewall.

The actual errors vary, but mostly they seem to be either ERR_SSL_PROTOCOL_ERROR, or sometimes ERR_CONNECTION_RESET.

They sometimes manifest in the browser as a page not loading, but more often it is just some of the images on the page that fail to load.  I've attached a screen capture of a browser console log after loading a news website.  (Ignore the last item in the list, that's a real/persistent error.)  Simply reloading the page (or right-click "Load Image") is enough to have the page/image load properly - so the errors are intermittent and transient.

The nature of the problem makes it really hard to analyse - especially since you can't just use a previous page to verify any change, and there is no guarantee the next page/site will see the problem this time around.

I am using Unbound DNS with secure DNS (CloudFlare), but I can't really see that being the problem.  I have disabled IPv6 (I have no use for it on such a simple network so prefer to remove the complication).

I have have Captive Portal turned on, but no proxy.

This is a very small network.  Just half-a-dozen machines connecting out (and some more devices that are intentionally being blocked by the Captive Portal).

Can anyone offer and suggestions on what I might try to resolve this?  I've lost the last three days to my experiments with no change, so I could really use some hints.

Thanks.