Messages

Hi,

How often do you see latency spikes ? There is no time unit indication on the graph you supplied. Also, how many firewall aliases do you have ?

We run many virtualized OPNsense instances on Proxmox too and had to deal with minutely CPU spikes.
See: https://forum.opnsense.org/index.php?topic=17953.msg81563#msg81563

Hope it helps,

--
Marin

Thanks for this!

I suppose this happens because the local DNS daemon (unbound) is not yet available when ipsec interfaces are set  up, as services are started later in the boot process.

One option would be to check the Do not use the local DNS service as a nameserver for this system check box in the System > Settings > General page, and provide at least one DNS resolver in the fields just above. This would allow the box to use a remote DNS resolver for its own needs, and remove the dependency on the local unbound service.

I'll try to implement this on my side too and report the results.

Unfortunately, nothing new on my side either. The boot delay has to do with the setup of the VTI interfaces: disabling the IPsec service has no effect as long as the VTI interfaces still exist. Maybe someone should open an issue on the GitHub tracker ?

Hi,

We experience the very same issue with several OPNsense instances. IPsec VTI configuration takes several minutes to complete. Did you find a solution to this problem ?

Thanks!

This is just awful. Since some months, every couple of updates bring some kind of bug. This, added to the lack of proper release notifications (no mailing list, no GitHub releases, just a forum thread which cancels your subscription on any new release) make OPNsense quite unusable in demanding environments.

We deployed it to power schools and care centers; we've got tens of instances dispatched in many small sites on a wide area. I can't imagine losing access to all those distant sites because someone did not take the time to test the changes to such a critical feature as aliases.

I'm sure someone will soon answer me that we've got no right to complain since this is a free product, that quality assurance has a cost, and we should pay for professional support. OK, seems fair. But giving us nightmares every 2 months is not the best way to engage new customers. Since OPNsense is a free product, we do expect some bugs. We are prepared to handle proxy failures, unbound config errors (only a few weeks ago; another sweet memory), API bugs, et al. But certainly not empty aliases making 80+ instances unreachable in the middle of a week off.

I am on
* OPNsense 21.7.7-amd64
and I am using a Realtek NIC.

After reading through this, I want to make sure I use download the os-realtek-re on beforehand.

How can I do this?

Do installed plugins persist through updates?

Not through major upgrades. Erratum: they should persist between upgrades.

I had just tested. Unfortunately, the configuration does not work (yet). The interface in the VRF can be pinged. But the web GUI of the OPNSense cannot be reached. In the meantime I restarted the web GUI, but without success. I suspect that the URPF mentioned in the other forum post is interfering.

By default, all commands run in the context of VRF 0. If a command is expected to run in another context, it must be prepended with setfib(1). So I suspect that changing the VRF of the GUI interface also requires amending rc and configd scripts.

yep, and if you want to verify upstreams or your upstreams strictly checks SNI headers (like WAP\ADFS do) you will have to make separate upstreams for each.

It seems nginx works quite differently than HAProxy regarding TLS validation. With nginx, the validated SNI is the one set on the upstream, not on the upstream server. HAProxy does exactly the contrary. This comment from the nginx support team might help people dealing with the same problem.

Thank you for your help!

@marinbernard-pep06
hi.

upstream SSL certificate does not match "upstream5959918e46f84fbb8bbf02dc24f2bbc5"

since nginx plugin uses upstreams with uniform names, for verification to work you need specify the name in the "TLS: Servername override" field so that nginx compares the name in the certificate with this name, not with the name of the upstream with UID ("upstream5959918e46f84fbb8bbf02dc24f2bbc5" in your case). should work then.
this is not a bug, this is how it should work imho

Hi,
I don't think so: our config used to work before the update, and stopped working after. Our upstream references several upstream servers, each of them using an individual certificate matching its real host name. Forcing the use of a single SNI would mean re-issuing certificates for all those hosts. This is a no-go for us.

Hi,

We're experimenting another (probably related) issue since the update to 20.7.8. The log states:

Code Select Expand


*1 upstream SSL certificate does not match "upstream5959918e46f84fbb8bbf02dc24f2bbc5" while SSL handshaking to upstream, client: 10.6.97.138, server: server.domain, request: "GET / HTTP/2.0", upstream: "https://10.6.105.115:443/", host: "server.domain:444"


I disabled certificate validation but did not revert to a previous version. Certificate validation is failing even when no specific CA is selected in the UI. The whole TLS chain is already present in OPNsense certificate store. The only constraint we add is chain depth = 2. Removing it does not help either.

Hi,

We've been using OPNsense as a captive portal for some time now, and would like to implement transparent proxying of HTTP(S) traffic to an external proxy server. We already do this on our wired networks, but I'm unsure whether using pf policy routing in tandem with captive portal is a supported scenario.

I found an old forum post about pf bypassing ipfw when policy routing is enabled. Is it still the case on 20.7 ?

Thanks!

Try to monitor system process with top:

• Launch top
• Press "s", then "1", then ENTER to set the refresh interval to 1 second
• Press "a" to show full command strings
• Press "i" to hide idle processes

Then just wait for some spike to happen, and note the python command invocation string.

If the console width is too narrow, either use an ssh / tty connection with a maximized window (run resizewin on tty to have the terminal fit the dimensions of the window).

On my boxes, periodic pf table updates spawn a python3 process every minute. While the cron job is not new, chances are that the CPU overhead caused by the script (update_tables.py) or the python interpreter itself is more important than in previous releases, because it used to remain unnoticed (at least by me) for months until somewhere in the 20.1 branch.

In virtualized environments, these regular CPU spikes put a lot of stress on our hypervisors, which indeed results in latency issues and lost packets (see this post).

Hi,

So far, OPNsense regularly updates pf tables by running a Python script (/usr/local/opnsense/scripts/filter/update_tables.py) as a cron job on every minute. The invocation of the Python interpreter results in a substantial increase of CPU load for a few seconds.

When several OPNsense VMs are hosted on the same hypervisor, with properly configured time synchronization, the current schedule triggers the cron job at the very same time on every VM. This leads to CPU spikes every minute on the host system, as all VM compete for CPU time.

Would it be possible to add a small random delay at each cron job invocation to stagger table updates ? Something like:

Code Select Expand


/bin/sleep $(jot -r 1 1 45) && (/usr/local/bin/flock -n -E 0 -o /tmp/filter_update_tables.lock /usr/local/opnsense/scripts/filter/update_tables.py) > /dev/null


Many thanks,

This would be great!
Many thanks for taking the time to answer.