Messages

Hi all, love my OpnSense. Except lately, its been running out of swap space and crashing. I haven't been able to find a clean and easy way to view all the ram usage per process to track down the cause of the issue. Here's what my system memory overview for the last 20 hours looks like: https://i.imgur.com/N0vb3zP.png (And 60 hours: https://i.imgur.com/6Oa1gbc.png)

I have 4GB of ram installed on this machine. It is running a AMD Ryzen V1000 in a Hyper-V guest VM. (https://www.newegg.com/asrock-4x4-box-r1000v/p/N82E16856158066). Two vNICs installed in the VM.

Looking at `top`, I can't find any way to sort by most memory used (people say to use shift m, i, etc. but none of that works on my shell at least). But I can see that I have multiple nginx processes using 100M+ of ram. Seen here: https://i.imgur.com/0HLgKic.png

Looking at another way to view ram per process, I see that ps lists the %MEM stat and the VSZ stat. If I sort by either of those methods, I still don't know how to track this down to what subsystem of OpnSense is leaking ram and causing my router to require multiple reboots every week for my regular home (<20 devices) network. Shown here: https://i.imgur.com/8oDylu8.png

I'm leaning towards this being an issue with nginx just because I have like 20 HTTP servers (and 20 locations and 20 upstreams and 20 upstream servers) to services in my network and nginx is one of my top memory users.

Does anyone have any tips or tricks for me to figure out exactly what the cause is? If nginx is the root cause, are there any good ways to track down what's up? Looking at the nginx global log, there are a few errors about ssl_early_data not supported and getting ignored. Or a duplicate server name getting ignored. Certainly nothing big that suggests nginx sucking up all my ram.

Thanks all!

Aggggh, I'm going crazy. I'm sure that its something simple, but I can't find it. I'm trying to have an nginx server that forwards requests to a remote path. Example:

Someone accesses https://myhost.mydomain.com and the nginx serves data from https://site.somedomain.com/path

The URL shows only https://myhost.mydomain.com to the user. The issue I'm having is that normally, I just use an upstream server for my requests, which is a domain name or an IP address. No prefix and no path appended. So I can't figure out how to have nginx go get the remote location at the path.

Anyone able to point me to a previous post on this? I'm sure that I'm not the first one to ask, but I don't see anything covering this scenario in my searches or in the documentation: https://docs.opnsense.org/manual/reverse_proxy.html

Actually box is working again on opnsense but version 20.7 with latest updates.
After several boots of 21.1. i left it with 20.7.
So I will have to wait with some hope that newer version will work with this box.

If is there somebody who could help with some advice I would appreciate it  ;)

Sorry, I only tried it with 20.7. Got it to work fine, didn't try newer.

I actually installed Hyper-V on the machine and put OpnSense in a VM on it with 3 cores. It is working fantastic.

AMD Ryzen Embedded... nice i am curious. This will be a wonderful year :)

cheers
till

Amen, @tillsense! I just bought the little brother to this guy and will get my hands on it Friday to try this out, too. Hope it'll detect the dual realtek just fine :)

Where did you buy it? I am looking to go in this direction.

I got mine from Newegg under AMD Ryzen VBOX 1000 or something.

Oh interesting. I didnt think that would be relevant since i dont have that plugin installed even. Just dnssec support enabled and only dns server is 1.1.1.3 (cloudflare family node).

Here are my plugins:


(Note: several of these installed plugins say misconfigured because they were installed before I did the Configuration import from my previously OpnSense router. They all seem to work fine though)

It would make sense if FreeBSD doesn't know how to handle dynamic ram for it to use up all space instead of accepting the host requests to increase its RAM dynamically.

Seems like these started happening after I installed sensei when trying to see if that could help me figure out why Unbound kept crashing. Maybe I'll uninstall it and see if those out of swap errors go away.

I have been experiencing Unbound freezing every 3-7 days roughly since I applied the 20.7.7 update earlier this year. I am currently on 21.7.1. I have not done any pkg revert or additions for unbound.

My unbound config includes several domain overrides and host overrides, but nothing else really.

Today it died again and looking at my system log, I see several hundred lines of getswapspace(\d+): failed like so:

Code Select Expand

2021-08-30T18:34:21 kernel swap_pager_getswapspace(31): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(9): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(18): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(24): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(24): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(16): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:20 kernel pid 35220 (php-cgi), jid 0, uid 0, was killed: out of swap space
2021-08-30T18:34:15 kernel swap_pager_getswapspace(20): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(4): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(18): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(20): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(22): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(25): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(4): failed

I'm running OpnSense in a HyperV vm with dynamic ram set, though, I never see it changing from the 1024 that is set on initial boot in VM manager:


I ended up doing the monit restart solution rather than revert pkg so at least my internet will get back online quick after DNS dies. Hope this info helps someone else in the future.

I'm in for $25. Love my OpnSense (though, I'm having an issue with Unbound DNS crashing right now. Small beans to deal with though)

I ran into issues with LE certs not getting auto renewed and making me revert to emergency admin UI a time or two. Decided to run a single cert just for admin UI and then other certs for the other stuff. While the 2nd one has broken a few times (typically due to me making address changes on my DNS records), my admin UI always gets its cert. :)

At home, I have a 250 u/d FTTH single WAN that is running on an old laptop with AMD A8-4500M. Passmark reports it at:

1710

Single Thread Rating: 902
Cross-Platform Rating: 4146

As you can tell, its a decent bit weaker then either of your options. I am using a single NIC with VLANs for the Lan/Wan going into the router. I'm making heavy use of Nginx to VM's on my NAS. When I upgraded my fiber to 1gbps temporarily, I was hitting around 450Mbps (expected due to the overloaded single NIC).

I don't use traffic shaping. I'd suspect that you'll be fine with either CPU and you should just do whichever one you have less use for.

AMD Ryzen Embedded... nice i am curious. This will be a wonderful year :)

cheers
till

Amen, @tillsense! I just bought the little brother to this guy and will get my hands on it Friday to try this out, too. Hope it'll detect the dual realtek just fine :)

Interesting. I'll have to give that a go with ICAP. Never heard of either.

I'm aware that OpnSense uses its own pkg ... repo... ? that stuff has to get loaded into. Just curious if anyone know what it would take for mitmproxy (https://mitmproxy.org/) to work on opnsense. Not even looking for gui, just the program itself.