Shortly after startup, restart or service bounces I get a HBSD SEGVGUARD error as follows:
[HBSD SEGVGUARD] [filterdns (14248)] Supension expired -> pid: 14248 ppid: p_pax: 0x50<SEGVGUARD, ASLR>
Though I have not discoveed any obvious operational problems associated with this error it is consttant and repeatable. (version i86 17.7.8) Any ideas or recommendations?
Edit: Please note the pid: ##### changes with each instance of the error.
That message would happen because filterdns is crashing often. Can you provide a core dump or steps to reproduce your issue?
The steps to it are pretty straight away. If I do a reload of services (option 11 on the console) the error occurs at: "Configuring WAN interface ..." Likewise on a restart. At any change make in the GUI that has any DNS selection made. For Example, if I am on
System: Settings: General
and toggle the setting; "Allow DNS server list to be overridden by DHCP/PPP on WAN" on or off I get the filter DNS error. Though with that on it will occur seemingly randomly during normal operation (Web browsing, music/video streaming, etc).
I do not have the WAN used in the IDS/IPS. I am not using Unbound DNS.
When you said you are running the i86 version, does that mean i386 (aka, 32-bit)?
At this time, yes I am running at 32 bits. I just downloaded the 64 bit earlier today. But wanted to get past some of this first before upgrading to 64 bits.
It's possible our ASLR is a bit too aggressive on 32-bit. I would suggest trying 64-bit. Below is a more detailed explanation as to why. If you don't understand, no worries. However, I'd just like to take the opportunity to document why it might be an issue.
The VDSO must lie between the top of the stack and the kernel virtual address space. Given that the 32-bit address space is extremely limited, it's possible that trying to find a mapping for the VDSO is failing, causing issues and resulting in crashes. Our SEGVGUARD implementation is a deterrent for ASLR bruteforcing, preventing frequently-crashing applications from being restarted too quickly (similar in concept to adding a delay each time someone enters the wrong password on a login prompt.)
In general, ASLR on 32-bit is rather weak. Like mentioned above, there simply isn't enough bits to randomize effectively. This isn't a problem with HardenedBSD's implementation; rather, it's a weakness inherent in the architecture. This isn't an issue with 64-bit systems as the virtual address space is sufficiently large.
To see if ASLR is the issue, you could add the following line to /boot/loader.conf.local:
hardening.pax.aslr.status=1
That will effectively disable ASLR on your system. If you still see the issue, please let me know and I'll spin up a virtualized 32-bit OPNsense installation to see if I can reproduce the issue.
Thank you. I will be setting up a 64 bit install, hopefully this weekend. If that cures it , great. If not I will let you know.
Regards,
Stefan
Cool. By switching to 64-bit, you'll also gain SafeStack, an exploit mitigation that helps with buffer overflows. OPNsense is the first and currently only firewall shipping with SafeStack.
Edited to add: Like I said in my previous post, if you have issues with 64-bit, I'd be happy to help find the underlying issue and fix it. ASLR simply is rather limited on 32-bit systems and frankly not worth it on 32-bit systems.
.
Upgrade to 64 did not go well. The install failed after it could not create the local user.
What hardware are we talking about?
HP Pavilion CPU: Intel(R) Pentium(R) CPU G620 @ 2.60GHz (2594.16-MHz 686-class CPU), 2 CPUs,
1 package(s) x 2 core(s).
4 Gb RAM
Intel Dual Port Pro/1000
500 GB Segate Drive
One other issue I found was that installing the amd64 did not wipe the boot sector and the i386 Bootloader is still there. Even after aggressive formatting of the drive, the amd64 can't overwrite the i386 bootloader. I have since tried to reinstall the i386 version and it too fails.
I tried cleaning the drive with the FreeBSD installer, didn't work. Then gave the Debian installer a try, same problem. I am borrowing a Windows 10 installer later today and will try that (I do not have any Windows products so I needed borrow some).
So the good news is that I have an amd64 install. Abet via a MBR setup.
A giant step forward. Thank you.
After a reboot this morning the filterdns issue again reared it's ugly head. But, confirmed it is a 32 bit issue.
At the login prompt:
[HBSD SEGVGUARD] [filterdns (79625)] Suspension expired.
->pid: 79652 ppid: 1 p_pax: 0x450<SEGVGUARD,ASLR,DISALLOWMAP32BIT>
During the reboot there were huge filterlog dumps to the console screen.
After second reboot. NTP and Suricata service starts are "deferred" requiring manual start.
"Starting NTP Service...deferred"
"Starting suricata...deferred"
Hi Stefan,
NTP being deferred is normal, Suricata should not be able to print "deferred", because NTP is the only thing that does that as far as I know. :)
G620 is around 2011 - 2013, it should run amd64, but if it's not it (or the mainboard) may be damaged... it's hard to tell.
Filterdns is an old daemon that resolves host aliases to IP addresses for firewall operation. How many aliases do you have in terms of hosts in them?
From what I can see being added by Ad in the development version, filterdns will be removed for this particular use case with 18.1.
Cheers,
Franco
Well then I look forward to 18.1!
I have 31 aliases. Though the worst offenders are trouble no matter how you add then into the filter.
I use feeds where ever possible but a few of the biggest trouble makers mutate daily, so to speak. For example Tor exits and Linode (both are common visitors to my systems and websites). Both have a hand full of ASN's. But each of their ASNs will have 1400+ CIDRs all of which are /29, /30, and /31 networks (note that each those networks only have between 1 and 4 IP addresses each). There are about a dozen major trouble makers running thousands of small networks (/24 or smaller). These are the ones that are hard to handle.
Hi Stefan,
Ok it would make sense that there is considerable pressure on filterdns to keep up to date which may cause this. I'm assuming that when ASLR triggers, it could be a latent bug in the filterdns code. I can ping this thread when we have confidence in the replacement if you are interested in trying the newer model before 18.1 is out officially.
Thank you,
Franco
Thanks Franco,
Yes, I would be interested in being an early adopter. Other software companies even offer "nightly" builds to early adopters, including AutoDesk and an Austrian Mac based rules engine developer. Adding OPN into the fold would be something that I would enjoy doing.
Cheers,
Stefan
Hi Stefan,
We do have a parallel development track and a private nightly build system ( https://nightly.opnsense.org/ ) ... but we are not confident it helps people to upgrade into untested packages and code, so we instead build one development package per release, which has a more consistent state.
Switching is easy:
# opnsense-update -t opnsense-devel
And switching back...
# opnsense-update -t opnsense
From both packages, you can use the latest code safely most of the time also:
# opnsense-code core
# cd /usr/core
# make upgrade
I've added a ping reminder in the ticket for the alias rework for later, see:
https://github.com/opnsense/core/issues/1971
Cheers,
Franco
Thank you, Franco
FYI: The opnsense-devel update going out with 17.7.11 tomorrow will no longer use filterdns at all.
Cheers,
Franco
Fantastic, thank you! I look forward to the update.
17.7.11 the same as 18.1.b_199?
Hi Stefan,
We don't keep track of the pre RC builds, but 17.7.11's development version translates to 18.1.b_273. It just counts the commits on this track. :)
Cheers,
Franco
Okay I'm in sync with things now.
Switched to opnsense-devel. Much improved filter stability.
Now running LibreSSL, showed marked improvement in system wide performance when using high level cryptography. No loss of GUI accessibility.
That's a good start. Thank your for testing! :)
Saw attempted DDOS attack. The system held, no HBSD SEGVGUARD error. Although filterdns dumped to the console screen.
Updating to b_273 and will wait for next attack.
This was a stray filterdns process from the older version it seems. Reboot to be sure, it should never come back.
Cheers,
Franco
Yes, I did. Then updated from b_199 to b_273 and rebooted again. My process monitoring show a nice tight core in b_273. I'm sure it can take a beating and keep on ticking. Well done.
I am experiencing lots of issues like this. Migrating to x86_64 is not an option, since my hardware is an eBox 3310A-L2 (http://www.compactpc.com.tw/product.aspx?act=detail&id=24 (http://www.compactpc.com.tw/product.aspx?act=detail&id=24)).
The message is as follows:
[HBSD SEGVGUARD] [python2.7 (process number varies)] Suspension expired.
-> pid: 76923 ppid: 168 p_pax: 0x90<SEGVGUARD,NOALSR>
I already disabled ASLR, as stated before in this post.
Any ideas why python2.7 is crashing? the box has 478MB RAM free, enabling swap does not help, swap is at 0% use all times.