OPNsense Forum

Archive => 17.7 Legacy Series => Topic started by: Stefan on November 29, 2017, 07:38:09 pm

Title: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on November 29, 2017, 07:38:09 pm
Shortly after startup, restart or service bounces I get a HBSD SEGVGUARD error as follows:

[HBSD SEGVGUARD] [filterdns (14248)] Supension expired -> pid: 14248 ppid: p_pax: 0x50<SEGVGUARD, ASLR>

Though I have not discoveed any obvious operational problems associated with this error it is consttant and repeatable.  (version i86 17.7.8) Any ideas or recommendations?

Edit: Please note the pid: ##### changes with each instance of the error.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: lattera on November 30, 2017, 01:13:40 am
That message would happen because filterdns is crashing often. Can you provide a core dump or steps to reproduce your issue?
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on November 30, 2017, 03:23:41 am
The steps to it are pretty straight away. If I do a reload of services (option 11 on the console) the error occurs at: "Configuring WAN interface ..." Likewise on a restart. At any change make in the GUI that has any DNS selection made. For Example, if I am on
   
System: Settings: General

and toggle the setting; "Allow DNS server list to be overridden by DHCP/PPP on WAN" on or off I get the filter DNS error. Though with that on it will occur seemingly randomly during normal operation (Web browsing, music/video streaming, etc).


I do not have the WAN used in the IDS/IPS. I am not using Unbound DNS.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: lattera on November 30, 2017, 03:27:43 am
When you said you are running the i86 version, does that mean i386 (aka, 32-bit)?
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on November 30, 2017, 04:40:13 am
At this time, yes I am running at 32 bits. I just downloaded the 64 bit earlier today. But wanted to get past some of this first before upgrading to 64 bits.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: lattera on November 30, 2017, 05:21:42 am
It's possible our ASLR is a bit too aggressive on 32-bit. I would suggest trying 64-bit. Below is a more detailed explanation as to why. If you don't understand, no worries. However, I'd just like to take the opportunity to document why it might be an issue.

The VDSO must lie between the top of the stack and the kernel virtual address space. Given that the 32-bit address space is extremely limited, it's possible that trying to find a mapping for the VDSO is failing, causing issues and resulting in crashes. Our SEGVGUARD implementation is a deterrent for ASLR bruteforcing, preventing frequently-crashing applications from being restarted too quickly (similar in concept to adding a delay each time someone enters the wrong password on a login prompt.)

In general, ASLR on 32-bit is rather weak. Like mentioned above, there simply isn't enough bits to randomize effectively. This isn't a problem with HardenedBSD's implementation; rather, it's a weakness inherent in the architecture. This isn't an issue with 64-bit systems as the virtual address space is sufficiently large.

To see if ASLR is the issue, you could add the following line to /boot/loader.conf.local:

hardening.pax.aslr.status=1

That will effectively disable ASLR on your system. If you still see the issue, please let me know and I'll spin up a virtualized 32-bit OPNsense installation to see if I can reproduce the issue.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on November 30, 2017, 10:10:55 pm
Thank you. I will be setting up a 64 bit install, hopefully this weekend. If that cures it , great. If not I will let you know.

Regards,
Stefan
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: lattera on December 01, 2017, 02:15:55 pm
Cool. By switching to 64-bit, you'll also gain SafeStack, an exploit mitigation that helps with buffer overflows. OPNsense is the first and currently only firewall shipping with SafeStack.

Edited to add: Like I said in my previous post, if you have issues with 64-bit, I'd be happy to help find the underlying issue and fix it. ASLR simply is rather limited on 32-bit systems and frankly not worth it on 32-bit systems.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 03, 2017, 06:00:25 am
.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 04, 2017, 04:45:17 am
Upgrade to 64 did not go well. The install failed after it could not create the local user.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 04, 2017, 04:51:49 pm
What hardware are we talking about?
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 04, 2017, 05:11:53 pm
HP Pavilion CPU: Intel(R) Pentium(R) CPU G620 @ 2.60GHz (2594.16-MHz 686-class CPU), 2 CPUs,
1 package(s) x 2 core(s).

4 Gb RAM

Intel Dual Port Pro/1000

500 GB Segate Drive

One other issue I found was that installing the amd64 did not wipe the boot sector and the i386 Bootloader is still there. Even after aggressive formatting of the drive, the amd64 can't overwrite the i386 bootloader. I have since tried to reinstall the i386 version and it too fails.

I tried cleaning the drive with the FreeBSD installer, didn't work. Then gave the Debian installer a try, same problem. I am borrowing a Windows 10 installer later today and will try that (I do not have any Windows products so I needed borrow some).

Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 05, 2017, 06:57:38 am
So the good news is that I have an amd64 install. Abet via a MBR setup.

A giant step forward. Thank you.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 05, 2017, 05:30:27 pm
After a reboot this morning the filterdns issue again reared it's ugly head. But, confirmed it is a 32 bit issue.

At the login prompt:

[HBSD SEGVGUARD] [filterdns (79625)] Suspension expired.
->pid: 79652 ppid: 1 p_pax: 0x450<SEGVGUARD,ASLR,DISALLOWMAP32BIT>

During the reboot there were huge filterlog dumps to the console screen.

 

Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 05, 2017, 09:20:24 pm
After second reboot. NTP and Suricata service starts are "deferred" requiring manual start.

"Starting NTP Service...deferred"

"Starting suricata...deferred"
 

Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 06, 2017, 06:58:00 am
Hi Stefan,

NTP being deferred is normal, Suricata should not be able to print "deferred", because NTP is the only thing that does that as far as I know. :)

G620 is around 2011 - 2013, it should run amd64, but if it's not it (or the mainboard) may be damaged... it's hard to tell.

Filterdns is an old daemon that resolves host aliases to IP addresses for firewall operation. How many aliases do you have in terms of hosts in them?

From what I can see being added by Ad in the development version, filterdns will be removed for this particular use case with 18.1.


Cheers,
Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 06, 2017, 03:28:57 pm
Well then I look forward to 18.1!

Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 06, 2017, 03:56:00 pm
I have 31 aliases. Though the worst offenders are trouble no matter how you add then into the filter.

I use feeds where ever possible but a few of the biggest trouble makers mutate daily, so to speak. For example Tor exits and Linode (both are common visitors to my systems and websites). Both have a hand full of ASN's. But each of their ASNs will have 1400+ CIDRs all of which are /29, /30, and /31 networks (note that each those networks only have between 1 and 4 IP addresses each). There are about a dozen major trouble makers running thousands of small networks (/24 or smaller). These are the ones that are hard to handle.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 07, 2017, 03:51:19 pm
Hi Stefan,

Ok it would make sense that there is considerable pressure on filterdns to keep up to date which may cause this. I'm assuming that when ASLR triggers, it could be a latent bug in the filterdns code. I can ping this thread when we have confidence in the replacement if you are interested in trying the newer model before 18.1 is out officially.


Thank you,
Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 07, 2017, 04:28:00 pm
Thanks Franco,

Yes, I would be interested in being an early adopter. Other software companies even offer "nightly" builds to early adopters, including AutoDesk and an Austrian Mac based rules engine developer. Adding OPN into the fold would be something that I would enjoy doing.

Cheers,
Stefan

Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 07, 2017, 05:34:57 pm
Hi Stefan,

We do have a parallel development track and a private nightly build system ( https://nightly.opnsense.org/ ) ... but we are not confident it helps people to upgrade into untested packages and code, so we instead build one development package per release, which has a more consistent state.

Switching is easy:

# opnsense-update -t opnsense-devel

And switching back...

# opnsense-update -t opnsense

From both packages, you can use the latest code safely most of the time also:

# opnsense-code core
# cd /usr/core
# make upgrade

I've added a ping reminder in the ticket for the alias rework for later, see:

https://github.com/opnsense/core/issues/1971


Cheers,
Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 07, 2017, 06:09:57 pm
Thank you, Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 19, 2017, 07:45:47 am
FYI: The opnsense-devel update going out with 17.7.11 tomorrow will no longer use filterdns at all.


Cheers,
Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 19, 2017, 02:34:09 pm
Fantastic, thank you! I look forward to the update.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 19, 2017, 07:59:11 pm
17.7.11  the same as 18.1.b_199?
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 19, 2017, 08:36:18 pm
Hi Stefan,

We don't keep track of the pre RC builds, but 17.7.11's development version translates to 18.1.b_273. It just counts the commits on this track. :)


Cheers,
Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 19, 2017, 10:33:40 pm
Okay I'm in sync with things now.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 20, 2017, 03:55:38 pm
Switched to opnsense-devel. Much improved filter stability.

Now running LibreSSL, showed marked improvement in system wide performance when using high level cryptography. No loss of GUI accessibility.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 20, 2017, 03:59:12 pm
That's a good start. Thank your for testing! :)
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 20, 2017, 10:42:12 pm
Saw attempted DDOS attack. The system held, no HBSD SEGVGUARD error. Although filterdns dumped to the console screen.

Updating to b_273 and will wait for next attack.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: franco on December 21, 2017, 05:00:19 pm
This was a stray filterdns process from the older version it seems. Reboot to be sure, it should never come back.


Cheers,
Franco
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: Stefan on December 21, 2017, 05:45:15 pm
Yes, I did. Then updated from b_199 to b_273 and rebooted again. My process monitoring show a nice tight core in b_273. I'm sure it can take a beating and keep on ticking. Well done.
Title: Re: HBSD SEGVGUARD errors on filterdns
Post by: luzemario on January 18, 2018, 10:39:41 pm
I am experiencing lots of issues like this. Migrating to x86_64 is not an option, since my hardware is an eBox 3310A-L2 (http://www.compactpc.com.tw/product.aspx?act=detail&id=24 (http://www.compactpc.com.tw/product.aspx?act=detail&id=24)).

The message is as follows:

Code: [Select]
[HBSD SEGVGUARD] [python2.7 (process number varies)] Suspension expired.
 -> pid: 76923 ppid: 168 p_pax: 0x90<SEGVGUARD,NOALSR>

I already disabled ASLR, as stated before in this post.

Any ideas why python2.7 is crashing? the box has 478MB RAM free, enabling swap does not help, swap is at 0% use all times.