OPNsense in a jail on a FreeBSD host?

[nxg]

Is there a way of installing and running OPNsense within a jail on a standard FreeBSD host?  And if there is, is that a supported mode, rather than a works-by-accident configuration?

It's looking very much as if the answer to the question is 'no'; is that right?  That means I can rule it out of my considerations, but it would be good to get that confirmed in words of one syllable.

I can see why one might not want to do this (for a firewall, a standalone box is mentally tidy, if nothing else), and I can see why this might be hard in principle (pf really wants to run in the host, so at the very least the jailed OPNsense would have to have some suitably tunneled way of communicating with the host).  But it would potentially be useful for experiments, and for the phase of the moon when I think 'the more in jails the better' rather than the phase where I think 'let's have lots of little boxes'.

I see a 2017 post (see [1]) which is titled 'Can I install Opnsense in an existing FreeBSD installation ?', but while that answer reiterates/reassures/illustrates that OPNsense is indeed layered on top of FreeBSD, it doesn't say 'jail' anywhere.  Also, the page [3] makes it clear that opensense-bootstrap is for converting a FreeBSD install into an OPNsense one.

There's a 2016 discussion [2] about jails, but that's really about whether more of OPNsense's services can or should be put in jails.

The 'Virtual and cloud' documentation [3] mentions various virtualisation options.  The absence of the word 'jail' on this page is probably telling me what the answer is here.

Best wishes,

Norman


[1] https://forum.opnsense.org/index.php?topic=4472.0
[2] https://forum.opnsense.org/index.php?topic=4078.msg22706
[3] https://docs.opnsense.org/manual/virtuals.html

[chemlud]

imho jails share the same kernel,

https://en.wikipedia.org/wiki/FreeBSD_jail

while opnsense is an OS on its own with custom kernel. dunno if BSD has some virtualization suitable for such a case, like KVM or alike...

bhyve?

https://forum.opnsense.org/index.php?topic=32813.0

[Patrick M. Hausen]

I have a dev/test system running TrueNAS CORE 13.3 (FreeBSD 13.3) and OPNsense in a bhyve VM with two network interfaces via PCIe pass through. Works splendidly.

[nxg]

Yes, jails do typically share the kernel – that's what makes them lightweight.  So running OPNsense as a complete OS would indeed require something like bhyve.

But OPNsense is to a substantial extent a layer on top of FreeBSD (someone will surely shout if I'm muddled about this), so given a suitable FreeBSD install, there will (?) be some set of installed userland software which would turn a FreeBSD install into a OPNsense install (this, as far as I understand it, is what opnsense-update does).  And walling different userlands off from each other is to some extent what jails are for.

I'm guessing, though, that opnsense-update won't work in a jail, simply because the jail won't have the right degree of access to (inter alia) the pf firewall in the host.

(To be clear, I'm running FreeBSD anyway, and it looks like running a full OPNsense install virtualised under bhyve is the next thing to explore, but I'm just trying to confirm I should rule out opnsense-in-jail as a more lightweight alternative).

[Patrick M. Hausen]

For one pf does work in a jail. Second you can pass network interfaces into a jail without bridging.

But I am also quite sure it would require massive changed to the software (OPNsense) itself to make it work.

[feld]

There are changes to the OpnSense kernel that are not going to be in your FreeBSD kernel. Whether or not they are important is a different discussion, but it may cause issues.

If you really want to run this on the same hardware it should be run in a bhyve VM.

[Patrick M. Hausen]

I have this system up and running - works incredibly well. If that is worth the money that I put into it is debatable - I sort of got carried away. But here is a "server grade" ECC memory, mirrored boot, mirrored storage pool implementation of

• TrueNAS CORE
• OPNsense (VM)
• Windows 10 (VM)
• TrueCommand (VM)
• Observium (jail)
• Grafana (jail)

in a system with a footprint of a sheet of paper.

Code: [Select]

┌──────────────────────────────────────────────────────────┐
│                                                          │
│                           TrueNAS CORE                   │
│          ┌────────────────────────────────────────────┐  │
│          │                                            │  │
│          │                           OPNsense VM      │  │
│          │       ┌ ─ ─ ─ ─ ─ ┐   ┌──────────────────┐ │  │
│          │    ┌───────────┐      │                  │ │  │
│          │ ┌──┴────────┐  │  │   │  LAN        WAN  │ │  │
│          │ │           │  │      │┌─────┐    ┌─────┐│ │  │
│          │ │ VMs/jails │  │  │   ││ ix0 │    │ ix1 ││ │  │
│          │ │           │  ├ ─    │└─────┘    └─────┘│ │  │
│          │ │           ├──┘      │   ▲          ▲   │ │  │
│          │ └────────┬──┘         └───┼──────────┼───┘ │  │
│          │          │                │          │     │  │
│          │          │                │   PCIe   │     │  │
│          │          │                │   pass   │     │  │
│          │ ┌────────┴─────────┐      │   thru   │     │  │
│          │ │                  │      │          │     │  │
│          │ │     bridge0      │      │          │     │  │
│  ┌────┐  │ │┌─────┐    ┌─────┐│   ┌──┴──┐    ┌──┴──┐  │  │
│  │IPMI├──┼─┼┤ ix0 │    │ ix1 ││   │ ix2 │    │ ix3 │  │  │
│  └────┘  │ │└──┬──┘    └──┬──┘│   └──┬──┘    └──┬──┘  │  │
│   .102   │ └───┼──────────┼───┘      │          │     │  │
│          └─────┼──────────┼──────────┼──────────┼─────┘  │
│                │          │.2        │.1        │        │
│                ▼          └──────────┘          ▼        │
│                                                          │
│            to laptop      172.31.0.0/24     to uplink    │
│                                                          │
│                                                          │
│  Mobile Lab                                              │
│  ----------                                              │
│  Supermicro A2SDi-4C-HLN4F                               │
│  Supermicro SC-101F                                      │
│                                                          │
└──────────────────────────────────────────────────────────┘

[nxg]

Thanks, everyone.  This is all most informative.

Reading over the posts here, I think I'm in danger of making things a little more complicated for myself than I need to at this point (it's good to learn two new things at once; three starts to get confusing...).  But I'm reassured that OPNsense+bhyve+jail configurations are possible and reasonable, in various combinations, so that can come back on the menu later.

Best wishes,

Norman

[zeon]

Hello
I came across this post just now. Interesting that somebody else is also looking at jailed version of OPNsense.
I have mostly positive expirience running opnsense inside FreeBSD jail. There are some little thing that are either not working or woking with a little bit of work. Again, mostly all functionality is there. Ask me questions on this matter if interested