OPNsense Forum

Archive => 17.1 Legacy Series => Topic started by: franco on March 16, 2017, 09:00:49 pm

Title: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 16, 2017, 09:00:49 pm
Hi all,

Shawn has been working on finalising the introduction of SafeStack to our base system binaries:

Quote
SafeStack is an instrumentation pass that protects programs against attacks based on stack buffer overflows, without introducing any measurable performance overhead. It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack.

via: http://releases.llvm.org/3.8.1/tools/docs/SafeStack.html

On the kernel side, there have been numerous reports of intermittent IPsec traffic loss that affects IPv4 TCP and is caused by the packet filter dropping connections because they do not behave according to normal TCP traffic. Upon further digging, this seems to be caused by a problem in FreeBSD 11.0 IPsec input handling.

You can test both of these changes by switching your 17.1.3 installation to the new base/kernel:

# opnsense-update -bkr 17.1.3-next
# /usr/local/etc/rc.reboot

Note that the manually installed base/kernel will be overwritten when 17.1.4 is released. Both patches are likely to land in this next release. We are actively looking for feedback on these and if they make a difference for you, both bad or good.

Both changes have been tested internally. The risk of breakage is minimal. If you need to go back, simply type:

# opnsense-update -bk
# /usr/local/etc/rc.reboot

Looking forward to hearing your feedback! :)


Cheers,
Franco
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: lattera on March 16, 2017, 09:23:13 pm
SafeStack is working for me on my production OPNsense installation.
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: ejprice on March 17, 2017, 02:48:08 am
SafeStack appears to be working for me as well. I don't have IPsec configured so I can't test that.
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: weust on March 17, 2017, 08:31:32 am
Made a Snapshot and updated. And now we wait.
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: gregober on March 17, 2017, 08:36:19 am
I have installed It on my home router (APU 1D4).
PPPoE + DynDNS (using FreeDNS) + IPSec

No problem, simply had to also upgrade another router to 17.1.3 to have IPSec working again.
Can't really tell if this is because of the upgrade or not. Just can report that It has been working again after reboot of remote node (using 17.1.3 standard).

Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 17, 2017, 05:49:43 pm
Hi,

install, reboot ok.
Code: [Select]
FreeBSD 11.0-RELEASE-p8 #0 abe907c58(stable/17.1): Wed Mar 15 02:19:04 CET 2017
cheers till
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: Andreas on March 17, 2017, 07:12:30 pm
still no functional ipsec.. still it hangs in the firewall..
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 18, 2017, 05:28:24 pm
till no functional ipsec.. still it hangs in the firewall..

Look, this is very misleading and unfriendly. This post talks about IPsec connection instability, not a magical fix for a problem you don't provide reference.

It's already caused other users to assume worse, so I would like to warn you about doing this again.

Please provide a reference to your issue, be it other forum posts or GitHub.

Please try to reproduce this with a FreeBSD 11.0 kernel. If the same problem appears we have to look there, not here. Let me know if you want to try this...


Cheers,
Franco
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 29, 2017, 07:31:33 pm
Hi Franco,

update to 17.1.4 don't work?

Code: [Select]
The operation will free 20 MiB.
pkg-static: Cannot delete vital package: opnsense!
pkg-static: If you are sure you want to remove opnsense,
pkg-static: unset the 'vital' flag with: pkg set -v 0 opnsense

cheers till
Title: Re: [CALL FOR TESTING: amd64] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 29, 2017, 07:36:36 pm
Your mirror may be out of sync. This is a bad sign. Whatever you do, do not run that proposed command, it comes from the FreeBSD package manager and it will cause the GUI to be removed on the next upgrade it if is faulty again.

Which mirror do you currently use? Did you switch from LibreSSL to OpenSSL or vice versa?
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 29, 2017, 07:37:52 pm
If you could send this output if it keeps happening (it will only print the upgrade details, but do nothing):

# pkg upgrade -n
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 29, 2017, 07:51:09 pm
# pkg upgrade -n

Code: [Select]
pkg upgrade -n
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (38 candidates): 100%
Processing candidates (38 candidates): 100%
Checking integrity... done (1 conflicting)
  - py27-setuptools-32.1.0_1 conflicts with py27-setuptools27-32.1.0 on /usr/local/bin/easy_install
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 39 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
py27-sqlite3-2.7.13_7
opnsense-17.1.3
py27-setuptools27-32.1.0

New packages to be INSTALLED:
py27-setuptools: 32.1.0_1

Installed packages to be UPGRADED:
squid: 3.5.24 -> 3.5.24_2
png: 1.6.28 -> 1.6.29
pkgconf: 1.3.0_3 -> 1.3.0,1
php70-zlib: 7.0.16 -> 7.0.17
php70-xml: 7.0.16 -> 7.0.17
php70-sqlite3: 7.0.16 -> 7.0.17
php70-sockets: 7.0.16 -> 7.0.17
php70-simplexml: 7.0.16 -> 7.0.17
php70-session: 7.0.16 -> 7.0.17
php70-pdo: 7.0.16 -> 7.0.17
php70-openssl: 7.0.16 -> 7.0.17
php70-mcrypt: 7.0.16 -> 7.0.17
php70-ldap: 7.0.16 -> 7.0.17
php70-json: 7.0.16 -> 7.0.17
php70-hash: 7.0.16 -> 7.0.17
php70-gettext: 7.0.16 -> 7.0.17
php70-filter: 7.0.16 -> 7.0.17
php70-dom: 7.0.16 -> 7.0.17
php70-curl: 7.0.16 -> 7.0.17
php70-ctype: 7.0.16 -> 7.0.17
php70: 7.0.16 -> 7.0.17
opnsense-update: 17.1.3 -> 17.1.4
opnsense-lang: 17.1.3 -> 17.1.4
ntp: 4.2.8p9_4 -> 4.2.8p10_2
lzo2: 2.09 -> 2.10_1
git: 2.11.1 -> 2.12.1

Installed packages to be REINSTALLED:
py27-ujson-1.35 (direct dependency changed: py27-setuptools)
py27-requests-2.11.1 (direct dependency changed: py27-setuptools)
py27-pytz-2016.10,1 (direct dependency changed: py27-setuptools)
py27-netaddr-0.7.18 (direct dependency changed: py27-setuptools)
py27-MarkupSafe-1.0 (direct dependency changed: py27-setuptools)
py27-Jinja2-2.8 (direct dependency changed: py27-Babel)
py27-Babel-2.3.4 (direct dependency changed: py27-setuptools)
openvpn23-2.3.14_1 (options changed)
dnsmasq-2.76,1 (options changed)

Number of packages to be removed: 3
Number of packages to be installed: 1
Number of packages to be upgraded: 26
Number of packages to be reinstalled: 9

The operation will free 20 MiB.
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 29, 2017, 08:03:52 pm
Did you ever build anything from ports or install from extra packages?
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 30, 2017, 08:12:55 am
Hi Franco,

yes a while ago on the apu runs squidanalyzer and squidview (only install) but all updates run so far.

On the console is shown the following: (reboot don't help)
Code: [Select]
The operation will free 20 MiB.
pkg-static: Cannot delete vital package: opnsense!
pkg-static: If you are sure you want to remove opnsense,
pkg-static: unset the 'vital' flag with: pkg set -v 0 opnsense
A firmware update is currently in progress.

cheers till
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 30, 2017, 08:22:03 am
her the screen try first update yesterday...install the main 17.1.3 kernel !?
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 30, 2017, 10:17:41 am
opnsense-update wasn't updated so it's still on 17.1.3 and since you installed the test kernel which is 17.1.3-next it tried to reapply the known good one.

It's not unfixable, yet I think something changed in FreeBSD with the python setup tools package that is causing this.

First step is to identify the extra packages you installed:

# pkg query -e '%a = 0' %o

Next step would be to remove the offending packages, then the update should be fixed. Last step is putting your extra packages back based on the latest ports code (we may have to add them to our mirror packages to make sure future upgrades work ok for you).
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 30, 2017, 10:36:31 am
Ok here the output

# pkg query -e '%a = 0' %o

Code: [Select]
devel/git
www/lightsquid
editors/nano
opnsense/opnsense
opnsense/os-smart
ports-mgmt/pkg
lang/python27
www/squid
www/squidanalyzer
www/squidview
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 30, 2017, 11:10:53 am
# pkg delete squidanalyzer squidview
# pkg autoremove

This should bring the system back into an upgradeable state, double-check with upgrade -n again. If it doesn't want to remove "opnsense-17.1.3" package you can go for it. :)
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: tillsense on March 30, 2017, 01:47:30 pm
Hi Franco,

has unfortunately not worked. after many months of the dev apu and then these lines: :o
Code: [Select]
FreeBSD repository update completed. 25859 packages processed.
pkg-static: No packages available to install matching 'opnsense' have been found in the repositories

it's time for new ways....
Code: [Select]
pkg install ca_root_nss
fetch https://raw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh
sh ./opnsense-bootstrap.sh

Reboot.....Done....Perfect  :)

thanks for your help
cheers till
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: franco on March 30, 2017, 04:55:26 pm
Hi Till,

Funny, ok. Just for future endeavours... "opnsense-bootstrap" is installed by default. :D


Cheers,
Franco
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: dragon2611 on April 01, 2017, 02:22:34 am
Upgraded from 17.3 to 17.4 on the one location where I still use IPSEC on opnsense, was able to disable the quick match/floating rules and it looks like the site2site VPN tunnels may be passing traffic now, but I'd like to give it a couple days before I say for sure that it's fixed.

I'll need to Give it a couple days to be sure

For Reference that opnSense VM has the following IPSEC tunnels at the moment

IKEv1 tunnel + NAT-T from a CHR (Virtual) RouterOS (need to move to IKE2, I think when i set it up IKE2 support in routerOS was just out and didn't seem to want to play nice with opnSense)

IKEv2 tunnel from a ubnt Edge Router

IKEv2 tunnel from an Routerboard RB750Gr3 (RouterOS)
Title: Re: [MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts
Post by: ristridin on May 31, 2017, 11:09:45 pm
Hi,

I´m on 17.1.7 and IPSec traffic is only working if any - any is enabled.
As soon as any source or destination ip is inserted, all ipsec traffic get´s blocked by default any block rule.

Anything I can provide in order to help?

cheers!