Messages

[freebsd-boot]

I found this thread to be rather helpful as well.

Updating the EFI loader is done.
But, when updating zfs partitions (freebsd-boot), I encounter this error, and am unsure how to proceed.

Code Select Expand

root@MorikCage:/boot/efi/efi/freebsd %# gpart show -p
=>         3  2000409253    nda0  GPT  (954G)
           3      532480  nda0p1  efi  (260M)
      532483         311  nda0p2  freebsd-boot  (156K)
      532794  1981808640  nda0p3  freebsd-zfs  (945G)
  1982341434    18067822  nda0p4  freebsd-swap  (8.6G)

root@MorikCage:/boot/efi/efi/freebsd %# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0
gpart: /dev/nda0p2: not enough space



[/boot/efi]

Thank you, Patrick. My apologies for sounding dense, but a clarification to below mentioned would be appreciated.

The loader you already have in /boot/loader.efi is the correct one for your version of FreeBSD and ZFS. Copy this to your EFI partition following my instructions in the other thread.

Recommendation was:

Code Select Expand

1. mount -t msdosfs /dev/nda0p1 /mnt
2. mkdir -p /mnt/efi/boot /mnt/efi/freebsd
3. cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
4. cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
5. umount /mnt

I understand that when loading w/ FreeBSD installer CD/DVD, /1/ mounts the disk, and /2/ creates EFI's boot partition for both UEFI and legacy. Existing mount on my node reveals /boot/efi to be mounting /dev/gpt/efifs.

Code Select Expand

root@MorikCage:~ %# mount
zroot/ROOT/24.4.2 on / (zfs, local, nfsv4acls)
devfs on /dev (devfs)
/dev/gpt/efifs on /boot/efi (msdosfs, local)
zroot on /zroot (zfs, local, nfsv4acls)
zroot/tmp on /tmp (zfs, local, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noexec, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noexec, nosuid, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, nosuid, nfsv4acls)
devfs on /var/dhcpd/dev (devfs)
devfs on /var/unbound/dev (devfs)
/usr/local/lib/python3.11 on /var/unbound/usr/local/lib/python3.11 (nullfs, local, read-only)
/lib on /var/unbound/lib (nullfs, local, read-only)

Therefore, am I correct in saying that /boot/efi/efi/boot in the current system equates to /mnt/efi/boot per above. If case be then /2-4/ for me should look like:

Code Select Expand

2'. mkdir -p /boot/efi/efi/freebsd
3'. cp /boot/loader.efi /boot/efi/efi/boot/bootx64.efi
4'. cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi

Then, as to updating legacy boot, the recommendation was:

Code Select Expand

# update legacy boot code
0. gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0

For my case, freebsd-boot is on nda0p2 (based on this):

Code Select Expand

gpart show
=>         3  2000409253  nda0  GPT  (954G)
           3      532480     1  efi  (260M)
      532483         311     2  freebsd-boot  (156K)
      532794  1981808640     3  freebsd-zfs  (945G)
  1982341434    18067822     4  freebsd-swap  (8.6G)

So, command /0/ would be the same for me.

Code Select Expand

# update legacy boot code
0. gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0


Interestingly though, based on the sysctl on [/i](kern.boottrace), I'm guessing that the system is using legacy not EFI for loading stage 1+2 boot? Which would mean that legacy bios update is in need of update for sure?

Code Select Expand

sysctl kern | grep boot
kern.boottime: { sec = 1761201281, usec = 365845 } Wed Oct 22 23:34:41 2025
kern.bootfile: /boot/kernel/kernel
kern.boot_tag: ---<<BOOT>>---
kern.reboot_wait_time: 0
kern.panic_reboot_wait_time: 15
kern.module_path: /boot/kernel;/boot/modules;/boot/dtb;/boot/dtb/overlays
kern.boottrace.table_size: 0
kern.boottrace.shutdown_trace_threshold: 0
kern.boottrace.shutdown_trace: 0
kern.boottrace.enabled: 0
kern.boottrace.shuttrace:
kern.boottrace.runtrace:
kern.boottrace.boottrace:
1 PART nda0p2 159232 512 i 2 o 272631296 ty freebsd-boot xs GPT xt 83bd6b9d-7f41-11dc-be0b-001560b84f0f
2 LABEL gpt/bootfs 159232 512 i 0 o 0
z0xfffff80002818200 [shape=hexagon,label="gpt/bootfs\nr0w0e0\nerr#0\nsector=512\nstripe=0"];
z0xfffff8000386c600 [shape=box,label="DEV\ngpt/bootfs\nr#4"];
    <type>freebsd-boot</type>
    <label>bootfs</label>
  <name>gpt/bootfs</name>
      <name>gpt/bootfs</name>
kern.geom.eli.boot_passcache: 1
kern.vt.kbd_reboot: 1
kern.cam.boot_delay: 0



Hello Patrick et al,

I have a classic RTFM case for which I was hoping to solicit your help. Running OPNsense 25.10_2-amd64 w/ FreeBSD 14.3-RELEASE-p4 on a zfs single disk decision DEC4040 appliance. As part of a recent zpool status check, I encountered a message to upgrade zpool to enable new features. Rather than doing my homework as to whether I need said features or not, I simply issued `zpool upgrade zroot` - a no-no. Now, here we are.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:

NAME        STATE     READ WRITE CKSUM
zroot       ONLINE       0     0     0
  nda0p3    ONLINE       0     0     0

errors: No known data errors


zpool upgrade zroot
This system supports ZFS pool feature flags.

Enabled the following features on 'zroot':
  edonr
  zilsaxattr
  head_errlog
  blake3
  block_cloning
  vdev_zaps_v2

Pool 'zroot' has the bootfs property set, you might need to update
the boot code. See gptzfsboot(8) and loader.efi(8) for details.

I read this post (and also other related ones). As I have yet to reboot the appliance, I had a few questions. My system info can be found below.

Code Select Expand

zpool status
  pool: zroot
 state: ONLINE
config:

NAME        STATE     READ WRITE CKSUM
zroot       ONLINE       0     0     0
  nda0p3    ONLINE       0     0     0

errors: No known data errors

gpart show
=>         3  2000409253  nda0  GPT  (954G)
           3      532480     1  efi  (260M)
      532483         311     2  freebsd-boot  (156K)
      532794  1981808640     3  freebsd-zfs  (945G)
  1982341434    18067822     4  freebsd-swap  (8.6G)


dmesg | grep -i "bios"
[1] smbios0: <System Management BIOS> at iomem 0x7945c000-0x7945c017
[1] smbios0: Entry point: v3 (64-bit), Version: 3.0
[1] usbus0: waiting for BIOS to give up control

dmesg | grep -i "efi"
[1] efirtc0: <EFI Realtime Clock>
[1] efirtc0: registered as a time-of-day clock, resolution 1.000000s


The following efi files exist in /boot:

Code Select Expand

ll /boot/*.efi
-r-xr-xr-x  1 root wheel 157184 Oct  7 04:24 /boot/boot1.efi*
-r-xr-xr-x  1 root wheel 110080 Oct  7 04:24 /boot/gptboot.efi*
-r-xr-xr-x  2 root wheel 658944 Oct  7 04:24 /boot/loader.efi*
-r--r--r--  1 root wheel  13653 Oct  7 04:24 /boot/loader.help.efi
-r-xr-xr-x  1 root wheel 569856 Oct  7 04:24 /boot/loader_4th.efi*
-r-xr-xr-x  1 root wheel 615424 Oct  7 04:24 /boot/loader_ia32.efi*
-r-xr-xr-x  2 root wheel 658944 Oct  7 04:24 /boot/loader_lua.efi*
-r-xr-xr-x  1 root wheel 516096 Oct  7 04:24 /boot/loader_simp.efi*

ll /boot/efi/efi/boot
total 870
-r-xr-xr-x  1 root wheel 890368 Oct 25  2022 bootx64.efi


In reading through Patrick's post (msg#5), the code mentions two files and directories namely

Code Select Expand

# update legacy boot code
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 nda0
# update EFI boot code
mount -t msdosfs /dev/nda0p1 /mnt
mkdir -p /mnt/efi/boot /mnt/efi/freebsd
cp /boot/loader.efi /mnt/efi/boot/bootx64.efi
cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
umount /mnt


Where should I get the Freebsd14.2 loader.efifrom? and which directory should I copy it to?

I found FreeBSD 14.3 image. II use macOS as my primary workstation. So, I doubt that I can mount a UFS system like freebsd onto MacBook (as a volume) or even dd it into a usb disk to then read on macos to peek into /boot folder. My /boot has efi/efi/boot but not /efi/boot /efi/freebsd. Am I doomed?

[How to use environment variables with caddy?]

How to use environment variables with caddy?

I had a need to reuse certain parameters in different reverse proxy handler blocks. The manual way of duplicating the same value in multiple places works ofcourse. It'd be easier to use environment variables. So, I tried add caddy.conf in /usr/local/opnsense/service/conf/configd.conf.d with content like so

Code Select Expand

CADDY_RESOLVERS_DNS_TLS=149.112.112.112 1.1.1.1 8.8.8.8 8.8.4.4

followed by

Code Select Expand

service configd restart
configctl caddy restart
caddy environ

startup fails w/ the environment variable value not found. Perhaps a wiser being could tell me whether there is a trick to this getting the env variables working?

I evaluated it and its possible but very brittle. So it's not going to be included in the plugin.

Thank you for taking the time to look into this matter!

The build will be thinned out soon to only include cloudflare, which will make caddy add-package less prone to fail.

Good to hear! I finally got around to using DNS-01 challenge with Porkbun DNS, so had to rebuild caddy 2.10.1 with porkbun module using xcaddy. go124 upgrade was done implicitly. Slightly painful but works. Thank you for writing such a helpful plugin!

https://github.com/opnsense/plugins/issues/4668

Thank you!

[OPNSense]

Thank you, Cedrik.

OPNSense: You want to do what?
Me: pkg install xcaddy
OPNSense: Did you take me for a fool?
OPNSense:

Code Select Expand

pkg install xcaddy
Updating OPNsense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  249 KiB 255.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 870 packages processed.
All repositories are up to date.
pkg: No packages available to install matching 'xcaddy' have been found in the repositories

Me:

Code Select Expand

sed -in 's/no/yes/'  /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: { enabled: yes }


pkg install xcaddy
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
Updating OPNsense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  249 KiB 255.0kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 870 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
pkg: 1.19.2_5 -> 2.1.2 [FreeBSD]

Number of packages to be upgraded: 1

The process will require 31 MiB more space.
12 MiB to be downloaded.

Proceed with this action? [y/N]: N


OPNSense: Told you, I'll win.
Me: I give up

Also, few minutes later,
Me: having never done ports, why don't we hose our system...

Code Select Expand

pkg install git
...
git clone --depth=1 https://git.FreeBSD.org/ports.git /usr/ports
cd /usr/ports/www/xcaddy
make install clean
....

mkdir -p ~/caddy_build && cd ~/caddy_build
xcaddy build \
  --with github.com/caddyserver/ntlm-transport \
  --with github.com/mholt/caddy-dynamicdns \
  --with github.com/mholt/caddy-l4 \
  --with github.com/mholt/caddy-ratelimit \
  --with github.com/hslatman/caddy-crowdsec-bouncer \
  --with github.com/caddyserver/transform-encoder
...
...
././caddy version
v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=

<< make my changes on crowdsec >>
configctl caddy restart
OK

Phew.. No idea what else did I break, but the feature which I wanted works now. Of course, I do not know how future os-caddy updates will behave. Life is indeed an adventure :-)

Thank you, Cedrik.

Try using xcaddy instead for your personal build.

I presume this'd mean I'd issue xcaddy with all the package names indicated in the previously failed command, like:

Code Select Expand

xcaddy build \
 --with github.com/caddy-dns/scaleway \
 --with github.com/caddy-dns/desec \
 ...
 (+ packages of my interest e.g. crowdsec)

/r
morik
PS: Your efforts in maintaining os-caddy port for OPNsense and other software are very much appreciated!

Hello,

I recently upgraded from Opnsense (FreeBSD 14.2-RELEASE-p2) Business Edition 24.10 --> 25.4, and caddy server stopped working.

Code Select Expand

caddy --version
v2.9.1 h1:OEYiZ7DbCzAWVb6TNEkjRcSCRGHVoZsJinoDR/n9oaY=

I have crowdsec plugin on caddy which I use in Caddyfile for integration w/ crowdsec. So, usually, after major upgrades on Opnsense I end up doing the following:

Code Select Expand

caddy add-package github.com/hslatman/caddy-crowdsec-bouncer github.com/caddyserver/transform-encoder
and off to the races I go. However, this time around, I'm getting a weird 400 error.

Code Select Expand

caddy add-package github.com/hslatman/caddy-crowdsec-bouncer github.com/caddyserver/transform-encoder
2025/05/01 01:17:17.322 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2025/05/01 01:17:17.322 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": ["github.com/caddy-dns/desec@v0.0.0-20240526070323-822a6a2014b2", "github.com/caddy-dns/scaleway@v0.0.0-20231227190624-561fd7f77b1b", "github.com/caddyserver/transform-encoder", "github.com/caddy-dns/namedotcom@v0.1.3-0.20231028060845-b9fae156cd97", "github.com/caddy-dns/ovh@v0.0.3", "github.com/caddy-dns/porkbun@v0.2.1", "github.com/caddy-dns/rfc2136@v0.1.1", "github.com/hslatman/caddy-crowdsec-bouncer", "github.com/caddy-dns/netcup@v0.1.1", "github.com/caddy-dns/vultr@v0.0.0-20230331143537-35618104157e", "github.com/caddyserver/ntlm-transport@v0.1.3-0.20230224201505-e0c1e46a3009", "github.com/caddy-dns/cloudflare@v0.0.0-20240703190432-89f16b99c18e", "github.com/caddy-dns/directadmin@v0.3.1", "github.com/caddy-dns/hetzner@v0.0.2-0.20240820184004-23343c04385f", "github.com/caddy-dns/linode@v0.7.2", "github.com/caddy-dns/namecheap@v0.0.0-20240114194457-7095083a3538", "github.com/caddy-dns/acmedns@v0.3.0", "github.com/caddy-dns/bunny@v0.1.1-0.20240209091254-71ced26b4224", "github.com/caddy-dns/acmeproxy@v1.0.6", "github.com/caddy-dns/infomaniak@v1.0.1", "github.com/caddy-dns/inwx@v0.3.1", "github.com/caddy-dns/mailinabox@v0.0.2-0.20240829173454-39d0e3ce8e25", "github.com/caddy-dns/powerdns@v1.0.1", "github.com/caddy-dns/azure@v0.5.0", "github.com/caddy-dns/gandi@v1.0.4-0.20240531160843-d814cce86812", "github.com/caddy-dns/hexonet@v0.1.0", "github.com/mholt/caddy-ratelimit@v0.1.0", "github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead", "github.com/caddy-dns/dnsmadeeasy@v1.1.3", "github.com/mholt/caddy-dynamicdns@v0.0.0-20241025234131-7c818ab3fc34", "github.com/caddy-dns/duckdns@v0.4.0", "github.com/caddy-dns/ionos@v1.1.0"]}
Error: download failed: download failed: HTTP 400: unable to fulfill download request (id=43358b0e-5041-4319-adac-d96d6a1e570e)


Code Select Expand

caddy upgrade
2025/05/01 01:24:33.309 INFO this executable will be replaced {"path": "/usr/local/bin/caddy"}
2025/05/01 01:24:33.309 INFO requesting build {"os": "freebsd", "arch": "amd64", "packages": ["github.com/caddy-dns/cloudflare@v0.0.0-20240703190432-89f16b99c18e", "github.com/caddy-dns/gandi@v1.0.4-0.20240531160843-d814cce86812", "github.com/caddy-dns/inwx@v0.3.1", "github.com/caddy-dns/acmeproxy@v1.0.6", "github.com/caddy-dns/dnsmadeeasy@v1.1.3", "github.com/caddy-dns/duckdns@v0.4.0", "github.com/caddy-dns/hetzner@v0.0.2-0.20240820184004-23343c04385f", "github.com/caddy-dns/mailinabox@v0.0.2-0.20240829173454-39d0e3ce8e25", "github.com/caddy-dns/namecheap@v0.0.0-20240114194457-7095083a3538", "github.com/mholt/caddy-ratelimit@v0.1.0", "github.com/mholt/caddy-l4@v0.0.0-20250102174933-6e5f5e311ead", "github.com/caddy-dns/bunny@v0.1.1-0.20240209091254-71ced26b4224", "github.com/caddy-dns/directadmin@v0.3.1", "github.com/caddy-dns/linode@v0.7.2", "github.com/caddy-dns/infomaniak@v1.0.1", "github.com/caddy-dns/netcup@v0.1.1", "github.com/caddy-dns/vultr@v0.0.0-20230331143537-35618104157e", "github.com/caddy-dns/acmedns@v0.3.0", "github.com/caddy-dns/azure@v0.5.0", "github.com/caddy-dns/desec@v0.0.0-20240526070323-822a6a2014b2", "github.com/caddy-dns/ovh@v0.0.3", "github.com/caddy-dns/porkbun@v0.2.1", "github.com/caddy-dns/scaleway@v0.0.0-20231227190624-561fd7f77b1b", "github.com/caddyserver/ntlm-transport@v0.1.3-0.20230224201505-e0c1e46a3009", "github.com/caddy-dns/rfc2136@v0.1.1", "github.com/caddy-dns/hexonet@v0.1.0", "github.com/caddy-dns/ionos@v1.1.0", "github.com/caddy-dns/namedotcom@v0.1.3-0.20231028060845-b9fae156cd97", "github.com/caddy-dns/powerdns@v1.0.1", "github.com/mholt/caddy-dynamicdns@v0.0.0-20241025234131-7c818ab3fc34"]}
Error: download failed: download failed: HTTP 400: unable to fulfill download request (id=704dc2db-afa9-4ee4-953a-6ba7ffec9803)

Firewall is not blocking either DNS translation of GitHub or ip connectivity to it. I am wondering if anyone else is having this issue?

If Caddy(Main) and Caddy(Sub) are in a trusted network, you can reverse_proxy between these Caddies without using TLS.

TLS is only important on the way through untrusted networks, e.g. from Client on the Internet to your Caddy(Main).

Caddy(Main) will hold all certificates and terminate tls. The other Caddies would not need to issue any certificates.

So you do not really need an ACME Server or a ACME Challenge redirection. Just use Plaintext.

I understand. Perhaps a better qualification of one of my use-case is in order.

• Telegraf, Influx and Grafana stacks are employed for telemetry. Latter two are web-based interfaces which require explicit certificate configuration in order to use http(s). Telegraf client configurations need root_ca for http(s) and host-specific keys if TLS verification option is enabled.
• Now, first-order question I asked myself was: Do I really need to protect telemetry information from telegram clients to influx db server? Not really : is the honest answer. That said, telemetry information reveals plenty about topology. So, having telemetry data streamed using TLS protection does carry benefits
• The certs (for external domain) will now reside on Caddy(master). With ACME server enabled on Caddy, I can have certbot request the necessary certs and auto-provision (e.g. during cert renewal) certs onto appropriate roles

Reference telegraf client information required:

Code Select Expand

  ## Optional TLS Config for use on HTTP connections.
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

Similar information is required for influx and Grafana.

Regarding the build:

https://caddyserver.com/docs/command-line#caddy-add-package

You can use this to add any package you want from the command line. It will not be persistent though. If the opnsense repo pushes an update at some point you must do it again.

Thank you for the reference. I understand.

The crowdsec usage I referred to earlier is as follows:

Code Select Expand


--> Global block of Caddyfile (info generated using cscli bouncer add
{
crowdsec {
api_url http://<Opnsense_fw>:8080
api_key <valid_key>
ticker_interval 15s
}
}

--> In the site-block of clients
{
route {
# crowdsec based filtering
crowdsec
                ... whatever logic is necessary ...
        }

}


Currently the plugin is rather finished and very specific or overly complicated things will most likely not be added to prevent feature creep.

Thank you for sharing the plugin status and roadmap. I only asked about crowdsec modules (which can be used like above) because a log-based integration of crowdsec is already implemented on your plugin for Opnsense. Addition of the few extra parameters required may increase the hardening posture.[/list]

You could use the HTTP01 challenge redirection on the Caddy Server of OPNsense to the cascaded other Caddy Servers.

I see. I was under the impression that the challenges on cascades servers would work because they get their certs (and therefore a hierarchy) from cascade master. Meaning, Caddy opnsense has, for a given external site, a valid cert. If acme challenge is sent to an internal (hosting an internal site) caddy instance, then the request would not have the same valid verification chain on client-side? Please grant me follow-ups in case I get stuck.


An other unrelated questions for your guidance:
GitHub code for this plugin indicates a custom caddy build (with DNS providers, L4 etc) is being used (120 standard modules, 80 or so optional modules). Would there be steps on how to add modules of interest eg crowdsec? I have used xcaddy in the past to generate custom images on Linux but not on freebsd. I can update this post with a working caddy config where crowdsec together with rate L5-7 rate limiters can take L4-7 information (not just L3 - which is what I think present crowdsec integration provides) to block unwanted behaviors.

[acme_server, tls_internal]

@Cedrik, First off, thank you so very much for creating this (and other) wonder plugins for OPNsense. You make our lives so convenient.

I've read through plugin configuration documentation and through this thread. But, an answer didn't jump out. So, I hope you could excuse me for bothering you with a little issue of mine. TLDR: If os-caddy plugin acts as an ACME server for a site, then a) what would be the ACME url? b) will the location of root ca be /var/db/caddy/data/caddy/certificates/local/<site>?

Presently, I use caddy cascaded in my internal network with a made-up domain name TLS'ed by Caddy. I did so prior to having an externally valid domain name - which I now do. Previously working caddy setup at a high-level was:

Code Select Expand

Caddy master instance (say on internal1.domain)
 {
   acme_server
   tls_internal
  }

Caddy slaves instances (say on internal2.domain)
 https://internal2.domain{
   tls {
     ca https://<internal1.domain>/acme/local/directory
     ca_root <path_to_caddy_master_ca>.pem
   }
  }

Caddy slaves instances (say on internal3.domain)
 https://internal3.domain{
   tls {
     ca https://<internal1.domain>/acme/local/directory
     ca_root <path_to_caddy_master_ca>.pem
   }
  }
... and so on


Now, with os-caddy on Opnsense being available, I'd like to rid of Caddy master on <instance1.domain> and utilize Opnsense caddy plug-in instead. GUI doesn't provide an option to declare acme_server, tls_internal. But inclusion of *.conf on a per-site basis is allowed. So, I added it like so: (reverse proxied external domain to internal2.domain running caddy slave1)

Code Select Expand

# Reverse Proxy Domain: "104d6421-2b1f-407e-af83-f087021ee1b1"
valid.domain {
log {
output file /var/log/caddy/access/104d6421-2b1f-407e-af83-f087021ee1b1.log {
roll_keep_for 10d
}
}
acme_server
tls internal


@08831d42-ad4c-40dc-a7d5-53af52ac6490_validdomain {
not client_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
}
handle @08831d42-ad4c-40dc-a7d5-53af52ac6490_validdomain {
abort
}

handle {
reverse_proxy internal2.domain {
                     transport http {
tls_insecure_skip_verify
}
     header_up Host {http.reverse_proxy.upstream.hostport}
  }
}
}

Then Caddy slaves instances's (internal2.domain) config is changed as:

Code Select Expand

https://internal2.domain{
   tls {
     ca https://<external.domain>/acme/local/directory
     ca_root <path_to_caddy_master_ca copied from /var/db/caddy/data/caddy/certificates/local/internal1.domain>.crt
   }
  }


Unbound resolves external.domain to Opsense's address (192.168.0.1) via host overrides. This is to prevent local request for external domain being handled and served locally without having to go out into the interwebs.

Because Opnsense webGUI is running on 8443, I also tried https://<192.168.0.1>:8443/acme/local/directory but to no avail. ACME server's url appears invalid. I'm unsure how to proceed. Your guidance would be much appreciated.

There should be nothing in need of change or reconfiguration on the FreeBSD side. Are you sure the ports on the new switch are configured for LACP?

Thank you @Patrick. S2 has the same port-channel configuration as S1. Switch-side configuration can be found below. Initial post is updated w/ it as well.

Code Select Expand


interface port-channel9
  switchport mode trunk
  mtu 9216

interface Ethernet1/9/1
  switchport mode trunk
  mtu 9216
  channel-group 9 mode active

interface Ethernet1/9/2
  switchport mode trunk
  mtu 9216
  channel-group 9 mode active


For Cisco NX-OS series switches, mode active engages LACP protocol negotiation from the switch.

[Start status of lagg0]

Hello everyone,
Decisio DEC4040 with 2x25G SFP28 ports (ice0,1) are in a lagg (lagg0) with a switch (S1). Various VLAN subnets and derivative interfaces rules etc depend on lagg0. This lagg is the "main" LAN connection. 100G breakout DAC is used. I need to move the 2 physical connections to a different switch (S2) in the short-term and later in the medium-term split the connections across two switches (S2,3) which are in a vPC configuration. For the short-term move, I have tried the following steps neither of which have yielded a successful outcome:

• Move the 100G cable from S1 to S2. LACP isn't re-established. "No carrier" status is shown on OpnSense
• Move the 100G cable from S1 to S2 after "ifconfig lagg0 down". LACP isn't re-established. "No carrier" status is shown on OpnSense
• Move the 100G cable from S1 to S2 after bringing down all involved interfaces (lagg0,ice0,ice1).LACP isn't re-established. "No carrier" status is shown on OpnSense
• Put new 100G QSFP on S2 + 2x25G SFP28 on DEC4040 and connected with MTP-to-LC breakout cable. Repeat combination of the above steps wrt ifconfig up/down. LACP isn't re-established. "No carrier" status is shown on OpnSense

Start status of lagg0

Code Select Expand


ifconfig -v ice0
ice0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether f4:90:ea:00:9f:72
inet6 fe80::f690:eaff:fe00:a206%ice0 prefixlen 64 scopeid 0x5
media: Ethernet autoselect (25G-AUI <full-duplex>)
status: active
nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
drivername: ice0
plugged: SFP/SFP+/SFP28 25GBASE-CR CA-25G-S (Copper pigtail)
vendor: CISCO-LEONI PN: L45593-D278-B30 SN: LCC2506GADX-CH3 DATE: 2021-02-10
root@MorikCage:~ # ifconfig -v lagg0
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000
description: main_LAGG (opt1)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether f4:90:ea:00:9f:72
hwaddr 00:00:00:00:00:00
inet 192.168.98.1 netmask 0xffffff00 broadcast 192.168.98.255
inet6 fe80::f690:eaff:fe00:9f72%lagg0 prefixlen 64 scopeid 0xd
laggproto lacp lagghash l2,l3,l4
lagg options:
flags=0<>
flowid_shift: 16
lagg statistics:
active ports: 2
flapping: 0
lag id: [(8000,F4-90-EA-00-9F-72,09A8,0000,0000),
(8000,E8-0A-B9-75-49-87,0001,0000,0000)]
laggport: ice0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,F4-90-EA-00-9F-72,09A8,8000,0005),
(8000,E8-0A-B9-75-49-87,0001,8000,01C3)]
laggport: ice1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> state=3d<ACTIVITY,AGGREGATION,SYNC,COLLECTING,DISTRIBUTING>
[(8000,F4-90-EA-00-9F-72,09A8,8000,0006),
(8000,E8-0A-B9-75-49-87,0001,8000,01C4)]
groups: lagg FG_ALL_VLANs FG_CRITICAL_LAN
media: Ethernet autoselect
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: lagg0


S1 (and S2) configs

Code Select Expand


interface port-channel9
  switchport mode trunk
  mtu 9216

interface Ethernet1/9/1
  switchport mode trunk
  mtu 9216
  channel-group 9 mode active

interface Ethernet1/9/2
  switchport mode trunk
  mtu 9216
  channel-group 9 mode active


End status in each of the above cases for when moving lagg0 was

Code Select Expand


ifconfig -vv lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: main_LAGG (opt1)
options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG>
ether f4:90:ea:00:9f:72
hwaddr 00:00:00:00:00:00
inet 192.168.98.1 netmask 0xffffff00 broadcast 192.168.98.255
inet6 fe80::f690:eaff:fe00:9f72%lagg0 prefixlen 64 scopeid 0xd
laggproto lacp lagghash l2,l3,l4
lagg options:
flags=0<>
flowid_shift: 16
lagg statistics:
active ports: 0
flapping: 0
lag id: [(0000,00-00-00-00-00-00,0000,0000,0000),
(0000,00-00-00-00-00-00,0000,0000,0000)]
laggport: ice0 flags=0<> state=41<ACTIVITY,DEFAULTED>
[(8000,F4-90-EA-00-9F-72,8005,8000,0005),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
laggport: ice1 flags=0<> state=41<ACTIVITY,DEFAULTED>
[(8000,F4-90-EA-00-9F-72,8006,8000,0006),
(FFFF,00-00-00-00-00-00,0000,FFFF,0000)]
groups: lagg FG_ALL_VLANs FG_CRITICAL_LAN
media: Ethernet autoselect
status: no carrier
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
drivername: lagg0


I think I may be missing something basic here. When changing lagg endpoints, does FreeBSD require removal and re-addition of the physical interfaces? Or perhaps some another subtlety I may be overlooking?

[gateway]

I have a similar setup. I too didn't see a need to execute the 'policy-based routing' section. Although, I have yet to thoroughly test the failover.

One question I did have on the following:

a question or check of my setup. i recently added a backup internet connection.

the only other adjustments i had to make were:

• any port forwards, i had to add both WAN interfaces to the forward definitions.
• forwarding 80/443 to public for caddy reverse proxy, so had to duplicate that rule on each WAN interface

in rule configuration there's a gateway option whose tooltip says eave as 'default' to use the system routing table. Or choose a gateway to utilize policy based routing. . So, in theory, additional rules definition shouldn't be required?