Messages

Ok, so podman seems to be the issue here. What speaks against using docker?

The same Argument that speaks against changing an entire System / Infrastructure / Set of Scripts / Network Topology / Administration in every Industry or Technical Domain just to address 1 "Edge" / Special Case: it's a big Pain in the Ass ...

So either you need to do a Mixed Setup, possibly requiring an extra IPv4 Public Address (since Podman/Docker will have to use port 80/443 for different Use Cases), or switch everything to Docker (and I almost migrated every Podman Installation to Fedora just because Debian isn't well supported under Podman).

But over the years I also learned that "the grass is NOT greener on the other Side". Whenever you think you switch Solution in order to (try to) fix a Problem, you might (will) also incur in several additional Issues that you didn't have before.

My Netbird host is a Hetzner VPS, ARM64, 2 CPUs, 4 GB of RAM, of which only 1.2GB are used. Postgres as database backend. Can't really see the OOM problems you had.

I have an old Hetzner AMD64 VPS with 2GB or RAM also on Hetzner. They increased the Prices a few Months ago, so I will probably just cancel that and use my own KVM Virtual Machine on the Dedicated Server instead.

EDIT 1: tried your Command after I shut everything else down so that Port 80/443 would be Free ... And after symlinking

# ln -s / usr/bin/podman /usr/bin/docker
#ln -s /usr/bin/podman-compose /usr/bin/docker-compose

Error: no container with name or ID "netbird-quickstart_zdb_1" found: no such container
Error: no container with name or ID "netbird-quickstart_zitadel_1" found: no such container
Error: no container with name or ID "netbird-quickstart_coturn_1" found: no such container
Error: no container with name or ID "netbird-quickstart_management_1" found: no such container
Error: no container with name or ID "netbird-quickstart_relay_1" found: no such container
Error: no container with name or ID "netbird-quickstart_signal_1" found: no such container
Error: no container with name or ID "netbird-quickstart_dashboard_1" found: no such container
netbird-quickstart_caddy_1
Error: no container with ID or name "netbird-quickstart_zitadel_1" found: no such container
Error: no container with ID or name "netbird-quickstart_zdb_1" found: no such container
Error: no container with ID or name "netbird-quickstart_coturn_1" found: no such container
Error: no container with ID or name "netbird-quickstart_management_1" found: no such container
Error: no container with ID or name "netbird-quickstart_relay_1" found: no such container
Error: no container with ID or name "netbird-quickstart_signal_1" found: no such container
Error: no container with ID or name "netbird-quickstart_dashboard_1" found: no such container
netbird-quickstart_caddy_1
537090513c345560782ef175c08e189493932b95de2544738b3c25be008ae775
Error: no container with name or ID "netbird-quickstart_relay_1" found: no such container
Error: no container with name or ID "netbird-quickstart_signal_1" found: no such container
Error: no container with name or ID "netbird-quickstart_coturn_1" found: no such container
Error: no container with name or ID "netbird-quickstart_zitadel_1" found: no such container
Error: no container with name or ID "netbird-quickstart_zdb_1" found: no such container
Error: no container with name or ID "netbird-quickstart_management_1" found: no such container
Error: no container with name or ID "netbird-quickstart_dashboard_1" found: no such container
Error: no container with name or ID "netbird-quickstart_caddy_1" found: no such container
Error: no container with ID or name "netbird-quickstart_zitadel_1" found: no such container
Error: no container with ID or name "netbird-quickstart_zdb_1" found: no such container
Error: no container with ID or name "netbird-quickstart_coturn_1" found: no such container
Error: no container with ID or name "netbird-quickstart_management_1" found: no such container
Error: no container with ID or name "netbird-quickstart_relay_1" found: no such container
Error: no container with ID or name "netbird-quickstart_signal_1" found: no such container
Error: no container with ID or name "netbird-quickstart_dashboard_1" found: no such container
Error: no container with ID or name "netbird-quickstart_caddy_1" found: no such container
Error: no pod with name or ID pod_netbird-quickstart found: no such pod
4834915b5ccc38ab944085421f75e24649a222ac10936b9934503424ae397811
94826dfeff09f96232cb57ee7d3e98bf13555f3126a7ce676d95004e85d1d100
netbird-quickstart_caddy_1
✔ docker.io/netbirdio/dashboard:latest
Trying to pull docker.io/netbirdio/dashboard:latest...
Getting image source signatures
Copying blob f7dab3ab2d6e skipped: already exists 
Copying blob 25d8059c17de done   |
Copying blob ff09aab76d97 done   |
Copying blob e252bd70cdea done   |
Copying blob e9fb81678df7 done   |
Copying blob 78f3aa16cfa5 done   |
Copying blob b6c81a3e8178 done   |
Copying blob 932bd785729d done   |
Copying blob 217c556afd61 done   |
Copying blob f846d527a638 done   |
Copying blob cb7988d44772 done   |
Copying config 5aa906f022 done   |
Writing manifest to image destination
d800170607d2c90e26f03b84e5018a3f6d2510dc11d98dd9e01ddbc8bb590f6d
netbird-quickstart_dashboard_1
3bb334c8ed73d09b4b6d8dc5950116e58eb8bbdccb593979e957aa58334c7111
netbird-quickstart_signal_1
51555daff8ad139dfb116c240b779f41d67ccbbef9732bc670a5c1f5dafb9aa4
netbird-quickstart_relay_1
d4694f689c6e5e43804a9f99e175ae49ac0741cf9185ca797d762a647ae439d2
netbird-quickstart_management_1
9a642849aa9f33ed8836136e2bc9adef3abec013d7070a8e5ce8af4f98048612
netbird-quickstart_coturn_1
Traceback (most recent call last):
  File "/usr/bin/podman-compose", line 33, in <module>
    sys.exit(load_entry_point('podman-compose==1.2.0', 'console_scripts', 'podman-compose')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 3503, in main
    asyncio.run(async_main())
  File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 3499, in async_main
    await podman_compose.run()
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 1742, in run
    retcode = await cmd(self, args)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 2499, in compose_up
    podman_args = await container_to_args(compose, cnt, detached=args.detach)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 1204, in container_to_args
    raise ValueError("'CMD_SHELL' takes a single string after it")
ValueError: 'CMD_SHELL' takes a single string after it

So yeah, not so easy with Podman I guess.

KISS with Wireguard only or Wireguard + Netbird at the "Price" of having a bigger ecosystem that can break more easily ? Uhm ...

I just saw, that I use Netbird with the default IdP Zitadel and not Authentik or Keycloak. Used the provided script and it was running out of the box.
Of course you add another service (at least self hosted if you want), but I think you gain a lot of features, like Zero-Trust for your clients.

Configuring connections to one single hub is fairly easy. If your central WireGuard hub goes down, you're lost, too.

Connecting all the spokes in a peer-to-peer manner is another story, if you have more than 4 spokes: that's 6 spoke connections and one to the hub, with 5 spokes, it's already 10 connections+ the hub.

With Netbird you're able to configure multiple routes to the same destination, if you want. I think OPNsense and Netbird are a perfect match here.

Good to know that's also a Feature Netbird provides  :). If only it would work in my case  :(.

As for Zitadel, that's the third Attempt I did back then on my Hetzner VPS (after Authentik and Keycloak) and it would NOT work at all. Zitadel was such a Memory Hog that I believe it triggered the OOM Killer due to excessive RAM Usage. Anyways, not an Option on a low CPU/RAM VPS. I have a dedicated Server now with several KVM Virtual Machines, so I could try that.

But I really liked Authentik, it's just an absolute PITA to interface with Netbird. And Netbird Debugging / Troubleshooting Capabilities are quite bad in my View, when something does not work (at all), it's not very clear (at least to me) as to why. And when it works, it's probably fine (until it breaks). I never managed to even get something to show up on the Web GUI so it's really frustrating to be honest  :(.

Granted, it could also be due to the Reverse Proxy (Traefik) Setup and possibly some Firewall Rules (I added exceptions based on Netbird specifically mentioning Hetzner Stateless Firewall although that did NOT make any Difference).

As to Wireguard breaking down ... I see that as a MUCH less likely Risk. Yes, it might be more of a PITA to set up Manually 100 Instances of Wireguard (Ironically in my Homelab, Gitlab and Nextcloud kinda forced my Hand on this one, since I HAVE to use NFS since their Update Script doesn't work with Samba/SSHFS Permissions and I don't have the Time to setup a Kerberos server for NFS - so I just do NFSv3 TCP over Wireguard UDP).

But compare generating a Keypair, setting up one small Config file for each Point-to-Point Connection with a System that might very easily break between Updates (either on Netbird side, or on Authentik/Keycloak/Zitadel side). I'd say Wireguard is very Reliable in that Regards.

Netbird should begin having some Consistency in their config File ... Depending on the Guide you Follow some Config/Environment Variables are NETBIRD_AUTH_XXXX and others are AUTH_XXXX and it's not always clear which Direction they are moving towards (I kinda had to duplicate quite a few of them in Order to suppress some Warnings in the Logs, although that did not solve my Problems).

I think Patrick means to use the tools that are already there: WireGuard.

You rent a VPS at your trusted VPS host and let this be the WireGuard hub. All your other locations connect to this hub and traffic is distributed as needed/configured.

I have this currently in place. Some of my locations also have "old" IPsec tunnels between each other. But I want to get rid off those. I could use WireGuard but then I stumbled across Netbird and I'm directly a huge fan of it.

It leverages the idea of Zero Trust, which I definitely prefer as boundaries are vanishing more and more. In a hybrid environment with multi-cloud and multiple On-Prem locations it gives you the best approach to connect everything with each other. And the best part is: the hub concept is only used, when a direct connection is not possible. Otherwise the spokes are connecting directly to each other.

I don't understand why you didn't succeed in getting Netbird up and running. I'm using it with Authentik and used the script that was provided. No issues at all.

Netbird + Authentik: https://github.com/netbirdio/netbird/issues/1684
Netbird + Keycloak: https://github.com/netbirdio/netbird/issues/1715

When the Logs don't say much [very clearly] and the Web UI doesn't even show up (well it shows an empty Page but that's completely useless), then it's a bit difficult to go forward.

I finally managed with Wireguard on its own Manually, so that's Nice. But netbird would be more convenient I guess.

KISS with Wireguard only or Wireguard + Netbird at the "Price" of having a bigger ecosystem that can break more easily ? Uhm ...

I think Patrick means to use the tools that are already there: WireGuard.

You rent a VPS at your trusted VPS host and let this be the WireGuard hub. All your other locations connect to this hub and traffic is distributed as needed/configured.

Exactly.

E.g. in my FreeBSD 14.1 VPS at vulture.com: /usr/local/etc/wireguard/wg0.conf

Code Select Expand


[Interface]
Address = 192.168.254.1/24,2003:a:d59:3840::1/64
PrivateKey = *********
ListenPort = 51820

# Peer 1
[Peer]
PublicKey = *********
AllowedIPs = 192.168.254.2/32,2003:a:d59:3840::2/128

# Peer 2
[Peer]
PublicKey = *********
AllowedIPs = 192.168.254.254/32,2003:a:d59:3840::254/128

[...]


Connect as many peers as you like. If you want to route entire networks to specific peers just add them to the "AllowedIPs" statements. Configure the peers in a matching fashion, done.

I don't know what for one would need a "VPN service". Plus, I don't trust them.

To perform outbound NAT I use /etc/pf.conf:

Code Select Expand


nat on vtnet0 inet from 192.168.254.0/24 to any -> ww.xx.yy.zz
nat on vtnet0 inet6 from 2003:a:d59:3840::/64 to any -> dead:beef:dead:beef:dead:beef:dead:beef

pass all no state


You can add inbound port forwarding or e.g. a Caddy reverse proxy with Letsencrypt as you like.

Yeah I was basically out of Options now that I tried for several Months so I just tried directly with Wireguard from an iPhone and Android Phone to VPN directly to my Home OPNsense (double NAT & Port Forwarding) and set manually the DNS Servers to 192.168.1.xxx etc. After fixing Outbound NAT, it worked quite well (ironically almost better on iPhone compared to Android ...)  :D.

You can do exactly that with a single virtual server at any cloud provider, connect all your locations via e.g. WireGuard and have a self hosted self managed transparent solution. You will have to pay about a fiver per month for that virtual server.

I would never use any "VPN provider" because I care about my privacy.

Hi Patrick. Would you have some specific guide in Mind ?

I attempting to do this since several Months but, even forgetting the "funnel-like" Feature, it always ends up not working as it's supposed to be (or in my Case ... not at all).

- I tried Headscale + Tailscale (basically no support since they claim Headscale is not supported and my Android Phone won't use the Set DNS Servers over VPN).

- I tried Netbird (stuck at installation with either Authentik or Keycloak), never even get to have the Web Panel showing up

I am revisiting an issue I had a few Months ago, this time with IPv6, as opposed to the previous Issue I had with IPv4.

Desktop Main IP: 2XXX:XXXX:XXXX:0001:0000:0000:0003:0066/64

Server Main IP: 2XXX:XXXX:XXXX:0001:0000:0000:0008:0015/64

Containers Dedicated Subnet: 2XXX:XXXX:XXXX:ff15:0000:0000:0000:0000/64

OPNSense has:

• Firewall -> Settings -> Advanced -> [CHECKED]  Bypass firewall rules for traffic on the same interface
  
• Gateway: 2XXX:XXXX:XXXX:0001:0000:0000:0008:0015/64
• Static Routes: 2XXX:XXXX:XXXX:ff15:0000:0000:0000:0000/64 via 2XXX:XXXX:XXXX:0001:0000:0000:0008:0015/64

If I setup the same Static Route on my Desktop, then I can iperf3 -c 2XXX:XXXX:XXXX:ff15:0000:0000:0000:0002 -P 10 and I get something like 500 Mbit/s. Fair enough for a gigabit connection with all Switches etc.

If I go THROUGH OPNSense I get 0.00 bytes/s ... 0.00 bytes/s  ???.

Is there maybe an issue "on the way back" ?

Like: Desktop 2XXX:XXXX:XXXX:0001:0000:0000:0003:0066 -> OPNSense 2XXX:XXXX:XXXX:0001:etc IN -> OPNSense 2XXX:XXXX:XXXX:ff15:etc OUT -> 2XXX:XXXX:XXXX:ff15::0002

But maybe there is no "route back" to the Desktop ? Or if the Connection is successfully established, then the reply Packets would go down that open connection Anyways ?

Nothing "red" in the Firewall Logs.

Hardware NIC Offloading was DISABLED ([CHECKED]), but ENABLED ([UNCHECKED]) did NOT make any difference.

EDIT 1: Iperf3 can go down to full speed (~ 900 Mbit/s) if, DURING THE RUN, I change something in OPNSense -> Firewall -> Advanced -> Settings -> click apply. But starting a new instance of iperf3 -c brings back again the speed to 0.00 bytes/s.

So of Course this is NOT a solution. Neither a workaround.

UDP iperf3 speed is around 1Mbit/s per each parallel thread (-P 10 -> 10 Mbit/s, -P 100 100 Mbit/s) whereas the default TCP iperf3 is 0.00 bytes/s UNLESS I change something in OPNSense -> Firewall -> Advanced -> Settings -> click apply DURING the iperf3 run.

If you are looking for a quicker way to assign multiple IP addresses to a host, check out Ansible or NixOS. Those are a better solution for your use case than tinkering with DHCP IMHO.

Bart...

Well I'd use saltstack if we are talking about going down that path I guess ...

I'm wondering why it's NOT possible to assign multiple IP Addresses to a DHCPv4/DHCPv6 Client using OPNSense (ISC DHCP Server).

For instance, on a Hetzner Server, since a MAC Address is bound to multiple IPs on Hetzner Robot, if everything gets configured via DHCP (Client) on the Server, then the Interface can get as many IPs as it has assigned (in the Rescue System I could get 4 of them). On the Production Server I configure IP Statically (mostly by Choice), so it's not a thing ...

I just wonder why the OPNSense DHCP Server doesn't support this.

Basically:
- Same MAC (DHCPv4 Server) or Same DUID (DHCPv6 Server)
- Different IP Address

Is there a Technical Reason that this is not implemented ?

Use case: I am running Podman (similar to Docker) Containers rootless on a Debian/Fedora Host. The user running the Containers can bind to any Port & IPv6 Address, but the Address must FIRST be assigned to the Host itself (which would be IMHO easier to do via DHCP, rather than having to use `nmcli` in Fedora - `/etc/network/interfaces` in Debian)

The plugin should configure /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml

to use the _ instead of - which is the default value, but not allowed by opnsense.

Can you please run

# configctl crowdsec reconfigure

# tail -f /var/log/configd/latest.log

and see if there's any error?

thanks

Thank you for your Answer.

Here you go:

Code Select Expand

configctl crowdsec reconfigure
OK


Code Select Expand

tail -f /var/log/configd/latest.log
<13>1 2024-06-05T14:42:37+02:00 Router.localdomain configd.py 234 - [meta sequenceId="1"] [b9f126b9-7623-4072-9890-96f072c3d8e0] crowdsec reconfigure
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="2"] [d57dd0fe-b953-4385-96ae-1ec8c01f6d19] Reloading filter
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="3"] [c648db2c-ae47-47a6-9674-c14948d3ba06] request pf current overall table record count and table-entries limit
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="4"] [3bec8a08-36d9-46f4-ab15-bd3111cc8413] list gateways
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="5"] [597c3bf1-f468-4d68-b18a-39e5608a341c] generate template OPNsense/Filter
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="6"] generate template container OPNsense/Filter
<15>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="7"]  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
<15>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="8"]  OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="9"] [e7152f5f-c5b6-481c-b9d5-50aee3779d1d] refresh url table aliases
<14>1 2024-06-05T14:42:41+02:00 Router.localdomain configd.py 234 - [meta sequenceId="10"] message e7152f5f-c5b6-481c-b9d5-50aee3779d1d [] returned b'{"status": "ok"}\n'

Now indeed in OPNSense -> Services -> CrowdSec -> Overview it's indeed better:
Service status: crowdsec [tick / success] - firewall bouncer [tick / success]

But it's still unclear to me why this happens on a stock Install ... and for how long it would even work ???.

I also have a Question ...

I am already managing my Letsencrypt Certificates for my Homelab (and Remote Servers) using a Docker/Podman Container running certbot with DNS Challenge against Cloudflare DNS Hosting (with API Key).

Typically Letsencrypt ACME will complain if you try to obtain an alteready-generated Certificate using another Method (e.g. HTTP-01) or just Certbot running somewhere else.

Isn't there a way to automatically let HAProxy / ACME retrieve the required Certificates from a given Folder ?

I can automatically upload these using SSH/SCP (maybe also Salt/Saltstack using salt-ssh in the Future), I think that would be the easiest (in my case at least).

So basically I upload the required Certificates to e.g. /usr/local/etc/letsencrypt/MYDOMAIN.TLD/{fullchain.pem,privkey.pem,chain.pem,cert.pem}, then HAProxy "just uses them" ?

I am not sure this is the correct Forum/Section for crowdsec. Any Opinion on how to proceed ? I also don't know if this is OPNSense-specific of rather an upstream Issue  :(.

Should I open a BUG Report on the OPNSense Issue Tracker (https://github.com/opnsense/plugins/issues/) ?

I'm on OPN v 22.7 so might not be the right pointer but on it, the table is called crowdsec_blacklists as in your aliases. Seems the code is expecting - instead of _.
Just a guess. Needs crowdsec to advise.

I had the same Impression, but wasn't sure if maybe there is a (uni/bi)directional "_" <-> "-" Conversion happening behind the Scenes, most likely only in one Direction.

I installed (or rather attempted to) Crowdsec on the latest OPNSense Release (with all Updates applied: OPNsense 24.1.8-amd64, FreeBSD 13.2-RELEASE-p11, OpenSSL 3.0.13) according to https://docs.crowdsec.net/docs/getting_started/install_crowdsec_opnsense/.

I also enrolled it to the Crowdsec Console (from SSH-ing into my OPNSense Instance).

However, while the Crowdsec Service appears to work correctly, the Firewall Bouncer dies within a second or so after attempting to be started.

OPNSense -> Services -> CrowdSec -> Overview
Service status: crowdsec [tick / success] - firewall bouncer [cross / fail]

Output of `cscli version`:

Code Select Expand

2024/06/04 17:00:55 version: v1.6.1-freebsd-0746e0c0
2024/06/04 17:00:55 Codename: alphaga
2024/06/04 17:00:55 BuildDate: 2024-05-28_00:23:25
2024/06/04 17:00:55 GoVersion: 1.21.10
2024/06/04 17:00:55 Platform: freebsd
2024/06/04 17:00:55 libre2: C++
2024/06/04 17:00:55 Constraint_parser: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_scenario: >= 1.0, <= 3.0
2024/06/04 17:00:55 Constraint_api: v1
2024/06/04 17:00:55 Constraint_acquis: >= 1.0, < 2.0

According to the logs, it seems one Blacklist doesn't exist. Am I supposed to create it manually (it wasn't in the Tutorial), and if so, how ?

OPNSense -> Firewall -> Aliases show that "crowdsec_blacklists" and "crowdsec6_blacklists" exists.
Note the "_" (underscore) instead of the "-" (dash) which pfctl complains in the logs below.

Output of `cat /var/log/crowdsec-firewall-bouncer.log`

Code Select Expand

time="04-06-2024 16:22:55" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:22:55" level=info msg="backend type : pf"
time="04-06-2024 16:22:55" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:22:55" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:22:55" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:22:55" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:34:42" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:34:42" level=info msg="backend type : pf"
time="04-06-2024 16:34:42" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:34:42" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:34:42" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:34:42" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:43" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:43" level=info msg="backend type : pf"
time="04-06-2024 16:50:43" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:43" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:43" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:43" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:47" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:47" level=info msg="backend type : pf"
time="04-06-2024 16:50:47" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:47" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:47" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:47" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:50:50" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:50:50" level=info msg="backend type : pf"
time="04-06-2024 16:50:50" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:50:50" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:50:50" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:50:50" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:54:03" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:54:03" level=info msg="backend type : pf"
time="04-06-2024 16:54:03" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:54:03" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:54:03" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:54:03" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:04" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:04" level=info msg="backend type : pf"
time="04-06-2024 16:55:04" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:04" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:04" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:04" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"
time="04-06-2024 16:55:06" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"
time="04-06-2024 16:55:06" level=info msg="backend type : pf"
time="04-06-2024 16:55:06" level=info msg="pf table clean-up: /sbin/pfctl -t crowdsec-blacklists -T flush"
time="04-06-2024 16:55:06" level=error msg="Error while flushing table (/sbin/pfctl -t crowdsec-blacklists -T flush): exit status 255 --> pfctl: Table does not exist.\n"
time="04-06-2024 16:55:06" level=info msg="Checking pf table: crowdsec-blacklists"
time="04-06-2024 16:55:06" level=fatal msg="pf init failed: table crowdsec-blacklists doesn't exist"

Actually ... in OPNSense VM both the /64 and the Additional /56 IPv6 Subnet I purchased work correctly, it's just that one Gateway appears down / unpingable (typically /64 Gateway appears down), if the other Gateway is used instead (I guess OPNSense cannot have 2 "default" Gateways, which is fair).

Although it's weird that they don't allow to monitor an External IP when the Gateway isn't being used  ???.

Is this a BUG or a "Feature" ?

I lost a couple of Days trying to understand why I cannot monitor 2 different Google DNS Servers ...

Turns out you need to [UNCHECK] Gateways -> <your Gateway> -> "Disable Host Route", so that traffic to that Monitor IP is forced out through the Specified Gateway. Of course this applies to ALL Traffic (even if you do a "manual" Traceroute etc), but unless you do that, the Gateway will appear down (most likely because the traffic will just go through the other Gateway).