[MERGED INTO 17.1.4] SafeStack base and IPsec IPv4 TCP connection aborts

[franco]

Hi all,

Shawn has been working on finalising the introduction of SafeStack to our base system binaries:

SafeStack is an instrumentation pass that protects programs against attacks based on stack buffer overflows, without introducing any measurable performance overhead. It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack.

via: http://releases.llvm.org/3.8.1/tools/docs/SafeStack.html

On the kernel side, there have been numerous reports of intermittent IPsec traffic loss that affects IPv4 TCP and is caused by the packet filter dropping connections because they do not behave according to normal TCP traffic. Upon further digging, this seems to be caused by a problem in FreeBSD 11.0 IPsec input handling.

You can test both of these changes by switching your 17.1.3 installation to the new base/kernel:

# opnsense-update -bkr 17.1.3-next
# /usr/local/etc/rc.reboot

Note that the manually installed base/kernel will be overwritten when 17.1.4 is released. Both patches are likely to land in this next release. We are actively looking for feedback on these and if they make a difference for you, both bad or good.

Both changes have been tested internally. The risk of breakage is minimal. If you need to go back, simply type:

# opnsense-update -bk
# /usr/local/etc/rc.reboot

Looking forward to hearing your feedback! :)


Cheers,
Franco

[lattera]

SafeStack is working for me on my production OPNsense installation.

[ejprice]

SafeStack appears to be working for me as well. I don't have IPsec configured so I can't test that.

[weust]

Made a Snapshot and updated. And now we wait.

[gregober]

I have installed It on my home router (APU 1D4).
PPPoE + DynDNS (using FreeDNS) + IPSec

No problem, simply had to also upgrade another router to 17.1.3 to have IPSec working again.
Can't really tell if this is because of the upgrade or not. Just can report that It has been working again after reboot of remote node (using 17.1.3 standard).

[tillsense]

Hi,

install, reboot ok.

Code Select Expand

FreeBSD 11.0-RELEASE-p8 #0 abe907c58(stable/17.1): Wed Mar 15 02:19:04 CET 2017

cheers till

[Andreas]

still no functional ipsec.. still it hangs in the firewall..

[franco]

till no functional ipsec.. still it hangs in the firewall..

Look, this is very misleading and unfriendly. This post talks about IPsec connection instability, not a magical fix for a problem you don't provide reference.

It's already caused other users to assume worse, so I would like to warn you about doing this again.

Please provide a reference to your issue, be it other forum posts or GitHub.

Please try to reproduce this with a FreeBSD 11.0 kernel. If the same problem appears we have to look there, not here. Let me know if you want to try this...


Cheers,
Franco

[tillsense]

Hi Franco,

update to 17.1.4 don't work?

Code Select Expand

The operation will free 20 MiB.
pkg-static: Cannot delete vital package: opnsense!
pkg-static: If you are sure you want to remove opnsense,
pkg-static: unset the 'vital' flag with: pkg set -v 0 opnsense


cheers till

[franco]

Your mirror may be out of sync. This is a bad sign. Whatever you do, do not run that proposed command, it comes from the FreeBSD package manager and it will cause the GUI to be removed on the next upgrade it if is faulty again.

Which mirror do you currently use? Did you switch from LibreSSL to OpenSSL or vice versa?

[franco]

If you could send this output if it keeps happening (it will only print the upgrade details, but do nothing):

# pkg upgrade -n

[tillsense]

# pkg upgrade -n

Code Select Expand


pkg upgrade -n
Updating OPNsense repository catalogue...
OPNsense repository is up-to-date.
All repositories are up-to-date.
Checking for upgrades (38 candidates): 100%
Processing candidates (38 candidates): 100%
Checking integrity... done (1 conflicting)
  - py27-setuptools-32.1.0_1 conflicts with py27-setuptools27-32.1.0 on /usr/local/bin/easy_install
Cannot solve problem using SAT solver, trying another plan
Checking integrity... done (0 conflicting)
The following 39 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
py27-sqlite3-2.7.13_7
opnsense-17.1.3
py27-setuptools27-32.1.0

New packages to be INSTALLED:
py27-setuptools: 32.1.0_1

Installed packages to be UPGRADED:
squid: 3.5.24 -> 3.5.24_2
png: 1.6.28 -> 1.6.29
pkgconf: 1.3.0_3 -> 1.3.0,1
php70-zlib: 7.0.16 -> 7.0.17
php70-xml: 7.0.16 -> 7.0.17
php70-sqlite3: 7.0.16 -> 7.0.17
php70-sockets: 7.0.16 -> 7.0.17
php70-simplexml: 7.0.16 -> 7.0.17
php70-session: 7.0.16 -> 7.0.17
php70-pdo: 7.0.16 -> 7.0.17
php70-openssl: 7.0.16 -> 7.0.17
php70-mcrypt: 7.0.16 -> 7.0.17
php70-ldap: 7.0.16 -> 7.0.17
php70-json: 7.0.16 -> 7.0.17
php70-hash: 7.0.16 -> 7.0.17
php70-gettext: 7.0.16 -> 7.0.17
php70-filter: 7.0.16 -> 7.0.17
php70-dom: 7.0.16 -> 7.0.17
php70-curl: 7.0.16 -> 7.0.17
php70-ctype: 7.0.16 -> 7.0.17
php70: 7.0.16 -> 7.0.17
opnsense-update: 17.1.3 -> 17.1.4
opnsense-lang: 17.1.3 -> 17.1.4
ntp: 4.2.8p9_4 -> 4.2.8p10_2
lzo2: 2.09 -> 2.10_1
git: 2.11.1 -> 2.12.1

Installed packages to be REINSTALLED:
py27-ujson-1.35 (direct dependency changed: py27-setuptools)
py27-requests-2.11.1 (direct dependency changed: py27-setuptools)
py27-pytz-2016.10,1 (direct dependency changed: py27-setuptools)
py27-netaddr-0.7.18 (direct dependency changed: py27-setuptools)
py27-MarkupSafe-1.0 (direct dependency changed: py27-setuptools)
py27-Jinja2-2.8 (direct dependency changed: py27-Babel)
py27-Babel-2.3.4 (direct dependency changed: py27-setuptools)
openvpn23-2.3.14_1 (options changed)
dnsmasq-2.76,1 (options changed)

Number of packages to be removed: 3
Number of packages to be installed: 1
Number of packages to be upgraded: 26
Number of packages to be reinstalled: 9

The operation will free 20 MiB.


[franco]

Did you ever build anything from ports or install from extra packages?

[tillsense]

Hi Franco,

yes a while ago on the apu runs squidanalyzer and squidview (only install) but all updates run so far.

On the console is shown the following: (reboot don't help)

Code Select Expand


The operation will free 20 MiB.
pkg-static: Cannot delete vital package: opnsense!
pkg-static: If you are sure you want to remove opnsense,
pkg-static: unset the 'vital' flag with: pkg set -v 0 opnsense
A firmware update is currently in progress.


cheers till

[tillsense]

her the screen try first update yesterday...install the main 17.1.3 kernel !?