Messages

1

Tutorials and FAQs / Re: HOWTO for installing a jail under OPNsense

« on: April 18, 2022, 07:35:08 pm »

Thanks for writing this up! It was super useful as a baseline to get to the setup that I wanted. Info on my changes in a second, but first a very important and very subtle bug in this setup. For whatever reason, OPNsense doesn't load any devfs rules on boot, which means that the mount.devfs directive gives full access to all devices in devfs -- allowing root on the jail to, for example, have raw disk access. This kind of defeats the purpose of having a jail! The fix is to make sure that, at some point, you run service devfs start which will load the default rules; once those are loaded, the default jail configuration will pick them up and use a reasonable set of defaults (ruleset 4, which hides a lot of things but exposes a set of "safe" devices which is enough for most systems). In your setup, you can probably stick it in either one of your syshook scripts (or make a new one); in my setup, since I don't need any syshook setups, I just stuck it in exec.prepare which is a little silly, you only need to run it once per boot but it's harmless to re-run, but lets me keep all of my conf in one place.

(It's maybe a bug that OPNsense doesn't do this by default, and maybe a bug that the FreeBSD base doesn't complain about jails trying to use a non-existent devfs ruleset? I dunno.)


Okay, so what I did differently. Gonna try to explain what I was trying to do and why I needed to set things up this way; my jail.conf is below.

For one, I'm using a read-only base system modified with symlinks into a read-write mount, nullfs mounted on each other to make the jail root -- this is exactly the setup in the "thin jails" section of this post which you helpfully linked. I named things slightly differently, and used exec.X directives to mount/unmount instead of fstabs (since those can substitute $name more easily) but the idea and mechanics are exactly the same as in that post.

But more interesting for this crowd is that I wanted my jails to live in their own LAN/subnet -- basically, have an interface dedicated to them in OPNsense like my other LANs and VLANs. Then I can set special firewall rules for my set of jails as a unit, e.g., firewall them off from both my main networks but also my guest networks. The way it works is pretty simple: I created a new bridge interface in OPNsense but with no interfaces added into the bridge, and then my exec directives create and configure epair interfaces and add them to the bridge and the jail's VNET as appropriate.

Only a couple of tricks to that. The first is that, for whatever reason, the epair interface on the host, which I add to the bridge, also needs an IP assigned. My understanding was that interfaces joined to a bridge don't/shouldn't have their on IPs, but things didn't seem to work without doing that.

The biggest trick is how to get OPNsense to create an empty bridge. Not only will the UI not let you do it, the low-level config code skips any empty bridges, and even contains a great comment calling out that this might not be ideal. But it also skips adding any invalid member interfaces -- but only after creating the bridge. So what I did was trick OPNsense: I created the bridge in the UI, selecting any interface as a member so it would let me save. (Make sure to enable the box for a link-local address too otherwise IPv6 RA won't make it through to autoconfigure the default route.) Then I manually edited /conf/config.xml -- I found my bridge in the config (inside a <bridged> block), and then changed the members to <members>invalid</members> -- can use anything there as long as it doesn't match the name of another interface you have. Reload or reboot, and there you have your empty bridge. You can assign it in the UI, set up DHCP etc as any other interface.


So let me sum up and actually show my config. Important points if you want to do what I did:

• Whether you follow ajm's guide or my adaptations, you should almost certainly make sure service devfs start gets run at some point!
• Follow the linked guide to set up zfs datasets and null mounts.
• Create an empty bridge by manually editing config.xml to set the bridge members to a non-empty set of invalid interfaces. Make sure to enable the box for a link-local address on the bridge before you do this.

OK, now my config. In OPNsense, my jail/bridge0 LAN is set up on 10.0.9.0/24, with OPNsense itself living at 10.0.9.1; for IPv6 my ISP routes 2001:db8:1234:5670::/60 to me, and the jail LAN is set to "track interface" with a prefix ID of 9 (i.e., 2001:db8:1234:5679::/64 is my jail LAN).

/etc/jail.conf

Code: [Select]

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;

mount.devfs;
exec.prepare += "service devfs start";

host.hostname = "${name}.localdomain";
path = "/jails/roots/$name";

exec.prepare += "mount -t nullfs -o ro /jails/base /jails/roots/$name";
exec.prepare += "mount -t nullfs /jails/$name /jails/roots/$name/rw";
exec.release += "umount /jails/roots/$name/rw";
exec.release += "sleep 5 && umount /jails/roots/$name";

vnet;
vnet.interface = "${if}b";
exec.prepare += "ifconfig ${if} create";
exec.prepare += "ifconfig bridge0 addm ${if}a";
exec.prepare += "ifconfig ${if}a inet ${haddr}/24";
exec.prepare += "ifconfig ${if}a inet6 2001:db8:1234:5679::${haddr}";
exec.start += "ifconfig ${if}b inet ${addr}/24";
exec.start += "ifconfig ${if}b inet6 accept_rtadv";
exec.start += "ifconfig ${if}b inet6 2001:db8:1234:5679::${addr}";
exec.start += "route add default 10.0.9.1";
exec.prestop += "ifconfig ${if}b -vnet $name";
exec.release += "ifconfig ${if}a destroy";

alcatraz {
        $if = "epair101";
        $haddr = "10.0.9.10";
        $addr = "10.0.9.11";
}

/conf/config.xml snippet showing the bridge (only manual edit was to the <members>)

Code: [Select]

  <bridges>
    <bridged>
      <linklocal>1</linklocal>
      <descr>JailBridge</descr>
      <maxaddr/>
      <timeout/>
      <bridgeif>bridge0</bridgeif>
      <maxage/>
      <fwdelay/>
      <hellotime/>
      <priority/>
      <proto>rstp</proto>
      <holdcnt/>
      <members>invalid</members>
      <ifpriority/>
      <ifpathcost/>
    </bridged>
  </bridges>

zfs list (basically same as FreeBSD guide with things renamed to make more sense to me)

Code: [Select]

NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot               2.25G  11.3G       96K  /zroot
zroot/ROOT           904M  11.3G       96K  none
zroot/ROOT/default   903M  11.3G      903M  /
zroot/jails         1.35G  11.3G      132K  /jails
zroot/jails/base     464M  11.3G      464M  /jails/base
zroot/jails/skel    4.41M  11.3G     4.33M  /jails/skel
zroot/jails/alcatraz 910M  11.3G      876M  /jails/alcatraz
zroot/tmp            728K  11.3G      728K  /tmp
zroot/usr            384K  11.3G       96K  /usr
zroot/usr/home        96K  11.3G       96K  /usr/home
zroot/usr/ports       96K  11.3G       96K  /usr/ports
zroot/usr/src         96K  11.3G       96K  /usr/src
zroot/var           5.95M  11.3G       96K  /var
zroot/var/audit       96K  11.3G       96K  /var/audit
zroot/var/crash       96K  11.3G       96K  /var/crash
zroot/var/log       5.45M  11.3G     5.45M  /var/log
zroot/var/mail       120K  11.3G      120K  /var/mail
zroot/var/tmp         96K  11.3G       96K  /var/tmp