Topics

I've searched the docs and forum without a result.

How can I specify the unix UID of accounts created via 'System: Access: Users' ?

The default seems to be to start numbering them from 2000, I'd like to start from 1000, or simply specifiy the UID at account creation.

[HOWTO for installing a jail under OPNsense]

WRT: https://forum.opnsense.org/index.php?topic=26724

15-02-22: First version
16-02-22: Added a note on use of 'freebsd-update' inside the jail.

HOWTO for installing a jail under OPNsense

This guide describes one approach to the creation of jails running FreeBSD 13.0-RELEASE, under OPNsense 22.1. The example shown, should be adapted as necessary to suit the planned environment.

The example OPNsense host boots from CF, and has a SATA SSD added for the exclusive use of the jails and associated data. It is using ZFS/Boot environments, and the SSD is configured with a ZFS pool.

This way, minimal changes are needed to the OPNsense environment. ZFS is required to permit easy deployment of the jail filesystems using snapshots and clones.

In this scenario, a bridge is required in OPNsense, for the VNET 'epair' interface to be bound. Other methods of networking jails are available, but are outside the scope of this guide.

Note that for installation, the host machine was booted into stock FreeBSD 13.0-RELEASE.
The same procedures should work under OPNsense, but please see the note below regarding 'freebsd-update'.

The following configuration items are referenced:

'tank':
The ZFS pool, mounted at '/tank' in which the jails are created.

'bridge16':
The bridge created in OPNsense, which provides L2/L3 connectivity for the jails.

'epair101':
'epair102':
The VNET epair interface which connects each jail to the bridge.

Jails:
'fserv': a permanent NFS filestore
'mserv': a smart mailhost

These procedures were based on material at:

https://genneko.github.io/playing-with-bsd/system/learning-notes-on-jails/
https://genneko.github.io/playing-with-bsd/networking/freebsd-vlan/
https://clinta.github.io/freebsd-jails-the-hard-way/

DISCLAIMER: This guide may not show the technically optimal solution, and quite possibly could be improved by someone more technically adept than the author. However 'it works for me'.

E&OE, YMMV etc.

Host config file changes

Host: /usr/local/etc/rc.syshook.d/start/11-mount-tank

Purpose: mount ZFS pool 'tank' due to failure of OPN to auto-mount the 2nd. zpool.
Note: must be in running order BEFORE jails are started !

#!/bin/sh
zpool import -f tank

Host: /usr/local/etc/rc.syshook.d/start/12-epair-create

Purpose: create VNET 'epair' interfaces
Note: must be in running order AFTER network interfaces are configured !

#!/bin/sh

# /usr/local/etc/rc.syshook.d/start/12-epair-create
# create required epair interfaces and add as bridge members
#
ifconfig epair101 create
ifconfig epair101b up
ifconfig epair102 create
ifconfig epair102b up
ifconfig bridge16 addm epair101b addm epair102b

Host: /etc/rc.conf

Add:
jail_enable="YES"
jail_list="fserv mserv"

Host: /etc/jail.conf

Code Select Expand


exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

host.hostname = $name;
path = "/tank/jails/$name";
exec.consolelog = "/var/log/jail_${name}_console.log";

vnet;
vnet.interface = $vif;

exec.start += "ifconfig $vif $addr";
exec.start += "route add default $gw";

# workaround
# https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326
exec.prestop += "ifconfig $vif -vnet $name";

fserv { $vif = "epair101a"; $addr = "10.0.16.16/24"; $gw = "10.0.16.4"; }
mserv { $vif = "epair102a"; $addr = "10.0.16.17/24"; $gw = "10.0.16.4"; }


Host commands to set up jails:

Set up ZFS datasets

Note: The SSD had previously been partitioned and labelled using gpart and glabel.

Code Select Expand


root@a-fw:~ # zpool create -m /tank tank /dev/gpt/tank
root@a-fw:~ # zfs set compression=on tank
root@a-fw:~ # zfs set atime=off tank
root@a-fw:~ # zfs create tank/jails
root@a-fw:~ # zfs create tank/jails/base
root@a-fw:~ # zfs create tank/jails/base/13.0


Install base.txz into 'basejail' dataset

Code Select Expand


root@a-fw:~ # mkdir /tank/tmp
root@a-fw:~ # cd /tank/tmp
root@a-fw:/tank/tmp # fetch ftp://ftp.uk.freebsd.org/pub/FreeBSD/releases/amd64/13.0-RELEASE/base.txz
root@a-fw:/tank/tmp # tar -xJpf base.txz -C /tank/jails/base/13.0
root@a-fw:/tank/tmp # cp /etc/localtime /tank/jails/base/13.0/etc
root@a-fw:/tank/tmp # vi /tank/jails/base/13.0/etc/rc.conf

sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
syslogd_flags="-ss"
cron_flags="-J 60"


Copy resolver configuration from the host to the template:

root@a-fw:~ # cp /etc/resolv.conf /tank/jails/base/13.0/etc/

Edit system crontab in the template to disable adjkern

root@a-fw:~ # vi /tank/jails/base/13.0/etc/crontab

      > #1,31 0-5 * * * root adjkerntz -a

Create /etc/periodic.conf to disable some of the predefined scheduled jobs which are not required for jails

root@a-fw:~ # vi /tank/jails/base/13.0/etc/periodic.conf

Code Select Expand


# No output for successful script runs.
daily_show_success="NO"
weekly_show_success="NO"
monthly_show_success="NO"
security_show_success="NO"

# Output to log files which are rotated by default.
daily_output="/var/log/daily.log"
daily_status_security_output="/var/log/daily.log"
weekly_output="/var/log/weekly.log"
weekly_status_security_output="/var/log/weekly.log"
monthly_output="/var/log/monthly.log"
monthly_status_security_output="/var/log/monthly.log"

# No need for those without sendmail
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_mailq_enable="NO"
daily_queuerun_enable="NO"

# Host does those
daily_status_disks_enable="NO"
daily_status_zfs_zpool_list_enable="NO"
daily_status_network_enable="NO"
daily_status_uptime_enable="NO"
daily_ntpd_leapfile_enable="NO"
weekly_locate_enable="NO"
weekly_whatis_enable="NO"
security_status_chksetuid_enable="NO"
security_status_neggrpperm_enable="NO"
security_status_chkuid0_enable="NO"
security_status_ipfwdenied_enable="NO"
security_status_ipfdenied_enable="NO"
security_status_ipfwlimit_enable="NO"
security_status_ipf6denied_enable="NO"
security_status_tcpwrap_enable="NO"


Apply latest FreeBSD patches to base jail

Note: This was performed under stock FreeBSD 13.0-RELEASE, same version as the base.txz
As 'freebsd-update' is not included in OPNsense, this step could be skipped, and the updates carried out within the jails once it is running. Alternatively, it may be an option to copy 'freebsd-update' into OPNsense, and use it with --currently-running 13.0-RELEASE -b <path/to/jail>.

root@a-fw:~ # freebsd-update -b /tank/jails/base/13.0 fetch install

Customize the jails' root shell prompt

root@a-fw:~ # vi /jails/base/13.0/root/.cshrc

   >   # ANSI Color 32 = Green
      set prompt="%{\033[32m%}%B<%n@%m>%b%{\033[0m%}:%~%# "

Create snapshot and clone to a new jail hostname 'fserv'

root@a-fw:~ # zfs snapshot tank/jails/base/13.0@p7
root@a-fw:~ # zfs clone tank/jails/base/13.0@p7 tank/jails/fserv

Test

root@a-fw:~ # service jail start fserv
root@test:~ # jls
   JID  IP Address      Hostname                      Path
     1                  fserv                         /tank/jails/fserv

root@a-fw:~ # jexec 1 /bin/csh

<root@fserv>:/#

DONE !

The jails can now have application packages installed in the normal way. For compiling from source, the relevant 'sets' (.txz's) would need to be installed alongside 'base.txz'.

Post-install updates of the jail

freebsd-update can be used from inside the jail, using the '--currently-running' option so it ignores the OPNsense kernel version:

Code Select Expand


<root@fserv>:/# freebsd-update --currently-running 13.0-RELEASE fetch
Looking up update.FreeBSD.org mirrors... none found.
Fetching metadata signature for 13.0-RELEASE from update.FreeBSD.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p7.


I feel a bit ashamed having to ask this here but I've not found any references to the correct way under OPNsense to mount a second zpool at system start.

The system is already root-on-ZFS, and the second zpool created no problems and was mounted (see output below), however after a reboot its no longer mounted.

I'm unfamiliar with the OPNsense system startup, so unsure as to the 'correct' way to do this. Perhaps a 'syshook' script, or modify /etc/fstab ? Any advice gratefully recieved..

Code Select Expand


root@a-fw:~ # camcontrol devlist
<ULTIMATE CF CARD Ver7.01C>        at scbus0 target 0 lun 0 (pass0,ada0)
<CT2000MX500SSD1 M3CR043>          at scbus1 target 0 lun 0 (pass1,ada1)

root@a-fw:~ # gpart create -s GPT /dev/ada1
ada1 created

root@a-fw:~ # gpart add -t freebsd-zfs -a 4k /dev/ada1
ada1p1 added

root@a-fw:~ # gpart modify -l tank -i 1 /dev/ada1
ada1p1 modified

root@a-fw:~ # gpart show
=>      40  30408256  ada0  GPT  (14G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  26210304     3  freebsd-zfs  (12G)
  30406656      1640        - free -  (820K)

=>        40  3907029088  ada1  GPT  (1.8T)
          40  3907029088     1  freebsd-zfs  (1.8T)

root@a-fw:~ # zpool create -m /tank tank /dev/ada1p1
root@a-fw:~ # zfs set atime=off tank

root@a-fw:~ # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
tank   1.81T   432K  1.81T        -         -     0%     0%  1.00x    ONLINE  -
zroot    12G   805M  11.2G        -         -     2%     6%  1.00x    ONLINE  -

root@a-fw:~ # mount
zroot/ROOT/base-setup on / (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
devfs on /var/dhcpd/dev (devfs)
devfs on /var/unbound/dev (devfs)
tank on /tank (zfs, local, noatime, nfsv4acls)


(Marked as Solved as its not an OPNsense issue but a FreeBSD one and not service-affecting anyway)

On a new install of 22.1, with six igb interfaces, five of which have 2 or 3 VLANs configured, I am seeing non-zero 'output errors'. These are on the VLAN interface not the physical interface.

Pic of Interface Statistics

The error counters are incrementing with traffic.
The counters are remaining at zero, on interfaces WITH a cable connected.
The counters are non-zero, on interfaces WITHOUT cable connected (igb2, igb3).
The counters are remaining at/near zero, on interfaces attached to VLAN's which are currently 'inactive' (no broadcast traffic).

They appear similar to those reported in:

https://github.com/opnsense/src/issues/74

However I'm NOT certain the errors I'm seeing are the same cause. I'm happy to assist with any further system info need to help find a fix. I'll open a new Issue if needed.

I'm working on an OPNsense 22.1 configuration which includes several VLAN trunk ports, which has necessitated the creation of a number of L2-only interfaces, to serve as bridge members. The L3 Interfaces 'proper' are configued on the bridges.

The default OPNsense firewall behaviour is include ALL interfaces. So effectively, traffic ingressing the system is passing though the firewall twice, once through the L2 interface and once through the L3 Interface.

This seems unnecessary and undesirable, so I would prefer to exclude all the L2 interfaces from the firewall.

What's the best way to do this ?

TIA

Re: https://github.com/opnsense/src/tree/stable/13/tools/tools/ath

I need to debug an interface using the 'ath' driver, however the 'athstats' tool is not included in OPNsense 22.1, nor is it available as a package from the stock package repo.

What's the proper way to get this onto OPNsense ? Are there 3rd. party package repos that might have it ?

TIA

For reasons of saving power/rackspace/network ports, I'm considering co-locating a couple of additional services, which aren't available as 'plugins', on the same physical box which will be running OPNsense.

I want to run OPNsense on the metal not as a VM. I don't want to custom compile the packages to run directly on OPNsense. As I already have an investment in jails to run some services elsewhere on the network, I want to look at the option of hosting jails under OPNsense. These would be held on a physically separate zfs pool, dedicated to the jails and data. (OPNsense will of course be under ZFS/BE, on the boot disk.)

I'm not hugely experienced in all this, but my understanding is that so long as I stick with the same ABI OPNsense is using (13.0-STABLE), I should be able to create a base jail using the stock FreeBSD distribution .txz's, and clone my service jail(s) off that. The services would be installed and maintained as stock FreeBSD packages. Does that sound feasible ?

For networking, although the hardware has enough interfaces to give the jail its own physical interface, I would prefer to connect the jails via VNET into OPNsense. I'd be very interested to hear of any experiences with this, and particularly if there are any showstoppers !

Hi, I've checked the 'docs' and searched here for an answers before posting, without success.

I'm assessing OPNsense to move my small business firewall requirements to, I evaluated 21.7 a few weeks ago but hit several faults/issues with it so gave up hoping they'd be fixed in the then upcoming 22.1 release. I've now got 22.1 running and immediately have hit one of the same issues again.

The purpose (I assume) of the dashboard, is to present the sysadmin with quickly digestible status info, so that actions can be taken should anything seem amiss.

Well immediately after installation, the Interface Statistics widget started to report 'Errors Out'. I quickly checked the switch port interface counters and they record ZERO ethernet framing errors.

I've tried clearing all available logs in the GUI, and re-starting, but the 'error' counters are retained across reboots.

So where in the heck is OPNsense getting this information from ? And how can I clear these error counters once the issue has been resolved ? It seems completely pointless reporting information that cannot be acted upon.

TIA, Andy