Jails and privilege isolation

[segfault]

Hello everyone,

I'd like to know if the Opnsense community has discussed before the tradeoffs and potential security improvements of running as many of its services and daemons within https://wiki.freebsd.org/Jails.

In relation to this topic, I'd like to ask what's the rationale for running certain daemons as root when they don't need to. Notable examples are:

- lighttpd
- php-cgi
- suricata
- openvpn

Thank you

[AdSchellevis]

Hi segfault,

I don't think there has been a lot of discussions about jails in the forum, but at a first glance I expect they would add quite some complexity to the firewall (in terms of configuration and routing data in/out all different jails).
Ideas are always welcome, but wrapping services into jails is not in our plans at the moment.

About some of the services which still run as root, I agree we should look into those if possible (php for one isn't possible yet because of the legacy code still in there).

Best regards,

Ad

[segfault]

IMO, the complexity added by isolating different daemons in jails is offset by the security benefits it brings.

The reduction of privileges by running processes as non-root is a good first step in that direction. Things are not secure right now running as root. How can I help move this forward?

[AdSchellevis]

The easiest way is probably to describe the change you propose and how it should work for the user and system underneath (no impact, minimum impact, needed transition path), keep things small and simple.

Next you can create pull requests to support your idea's with code here https://github.com/opnsense/core/.

I don't have a lot of time at the moment, but if you keep the steps small and easy to understand, I can review the changes and check for impact.

The best starting point are the services like suricata, our new style code (using configd) doesn't require the http server / php to run as root, but migrating all legacy code is still quite some more work.
Check our architecture overview for guidelines (https://docs.opnsense.org/development/architecture.html).

Thanks for your help!

[franco]

This is problematic, stemming from a long project history all the way back to m0n0wall.

The truth is: the web server runs as root because it needs to run privileged operations.

pfSense added a background daemon, which was tailored for a few simple "fire and forget" reload tasks (its name was "check_reload_status"). This daemon wasn't suitable for general purpose privilege separation and was written in C which made it difficult to extend.

In the spirit of what Chris Buechler called "NIH syndrome" we, or rather Ad, went ahead and built a general purpose privilege separation / background daemon dubbed "configd", which was written in Python and has since gained a lot of privileged operations.

The MVC parts use this exclusively. That's good.

What's not so good is that the static PHP pages still have a hardcoded backend that does a lot of privileged operations, which voids the premise of running lighttpd as non-root or in a jail. In fact, man-years can easily be wasted there for things that make sense, but in no way align with project goals, required work elsewhere and "state of the art".

We've done a lot of work towards privilege separation, but we're only half-way there. What we need is actionable ideas, help in coding and testing and time to get there.


Cheers,
Franco

[segfault]

Since the web interface can be setup to listen only on localhost and accessible via an SSH tunnel, I find it less worrisome than for other network services that must be open to potential hostile networks, like the Internet.

If ways can't be found to reduce the required privileges of such network exposed services, isolating them with Jails seems like a solution worth exploring.

The firewall is only going to protect us from packets that aren't supposed to get through. Once they're in, we need to rely on other security mechanisms. Running so many of the daemons as root is a sure fire way to get easily hacked sooner rather than later.

[franco]

Hi segfault,

Okay, then we have a general direction. 

Unbound and dhcpd are chrooted, dnsmasq drops privileges.

I think squid also runs as squid/squid.

Here is a report for strongswan. It's not impossible, but I'm cautious about "cannot add a route properly [...] potentially be fixed by using scripting hooks in StrongSwan via some sudo work" and what it means to find all bugs related to it.

https://github.com/opnsense/core/issues/1103

Last big daemon is Suricata, not sure how it is set up. There's more minor ones, tough.

Where should we proceed?


Cheers,
Franco

[Beeblebrox]

My $0.02 here and hope I'm not necro posting.

1. First off, a solid firewall configuration is going to add more security than any other measure. That's been done on OPNs, so services not exposed to the WAN are not the primary concern (unless the sysadmin does something stupid like offer dhcp to WAN)

2. Any service directly exposed to WAN should be jailed, period. That means DNS (unbound + dnscrypt-proxy), Squid, Privoxy, IDS/IPS. This implies that dnsmasq may not be a wise choice since it bundles dns+dhcp, but I don't know enough about dnsmasq to claim anything.
I'd include any service doing dirty work to the list to be jailed; the likes of CamAV/Symantec, or whatever risky service is being considered.

3. The bad news: I've found that PF cannot filter on a per-alias basis. For example, if the Privoxy jail is on VLAN1 with alias <IP>/32 and we wish to filter that specific IP, PF can't do it. I've yet to look into IPFW which was claimed to have this capability.

4. The difficulty for OPNs web-gui would come from any ifconfig change as far as I can tell. If VLAN or jail IP settings were to be changed down the road or if iniital setup was not very well thought through, I assume web-gui could have difficulties with shuffling ip addresses and firewall rules. A dedicated subnet for jails and default VLAN creation could facilitate this problem (ex 192.168.10.0/26) or some other potentially unused subnet.

5. Otherwise creating and configuring jails is pretty easy specially with qjail or ezjail (which is a single script btw). I just copied the script and config file to my box and created my jails. web-gui would have to have some kind of interface to set which services should be enabled for the particular jail - or just expose <jail>/etc/rc.conf

6. Consider ZFS (not advised on 32Bit) with snapshot rollback feature. Create a pristine ZFS snapshot at first start of jail. The snapshot is periodically called for rollback to restore the jail to pristine. Any undetected compromise of jail is thus wiped out. You mount /usr/local as "nullfs -ro" to <jail>/usr/local so that package upgrades are low maintenance and individual jails do not need package maintenance. If world gets updated, jails will require an etcupdate run to bring them to same level as host.

HTH.

[anoncat]

How would a jail work on the edge with an IDS/IPS? Jails are virtual instances which means your pulling away from bare metal. Nice in theory and practice on all the other services you mentioned but an IDS/IPS needs the absolute raw packet to be effect at filtering. Which is a tiny part of the reason checksums are disabled on the NIC you host the listening interface on. Running unprivileged would be amazing as IDS exploits while few and far between do exist it would be nice to have at least a simple first step. No idea how to even start when it requires control over network devices which is reserved most times for OS system level.

There are very small changes to some packets when running as a VM vs bare metal from a few test I've done over the years. Not a huge amount but the same exploit to both machines have tiny difference on some packets which has a potential to be used by people far smarter then me. Trust me a monkey with a rusty nail is pretty close comparison.

ZFS on  firewall... I love ZFS. Long live the ZFS NAS! I hate ZFS pre-req's. Last thing anyone wants is a small appliance with memory being taken up for storage when it should be used for session states and IDS/IPS rules. I don't know about you but having a small appliance or machine with 2GB ram is enough for 25 devices at home to all talk with no problems that's with torrents running. Now if the installer detects the device it's being added on has 32+GB ram you have my support. At that point reporting actually matters and you have a much larger user base behind to protect and worry about.

For the web interface part I'm all ears, can you make it happen? Hey smart dev's, OP needs to start write a little code don't you think :-).

[Beeblebrox]

* Good point on IDS/IPS, apparently I had not thunk it through. OP's question included openvpn (which would/should normally be placed in a jail) but IDS/IPS layer on an openvpn jail is beyond my knowledge.

* ZFS: The user should be able to choose the FS (UFS/ZFS) at install stage - problem solved. I agree with the "small box requirements", in which case user would choose UFS. On amd64 ZFS, unless you intend to have hundreds of datasets 4GB is sufficient btw.

* I would suggest the first service to be moved to a jail would be Squid rather than the web-gui, unless some users intend to expose web-gui to the WAN. Also remember that HardenedBSD which OPNsense partially (for the time being) uses is one step ahead of FreeBSD when it comes to privilege escalation problems.

* I already have, with a minor hiccup,  (unbound-DNSSEC + dnscrypt-proxy), searx & ClamAV running in separate jails on a 32Bit 2GB box.

Regards.