Messages

Well, in these recent Days apparently it stopped working again :(.

I cannot even ping the 2nd Gateway (I previously had a Rule to force a direct Gateway Connection from the LAN Clients, so that I could access from my Desktop PC the Upstream Routers): that seems to not work anymore.

Furthermore of course all Wireguard Clients stop routing completely. Handshake seems to work at least on iPhone, Routing is completely Broken again :(.

Thanks luckylinux

I hade the same problem creating multiple istances in Wireguard with no connection.

I hade connection only on one istance.

I resolve it adding a static route for each wireguards  VPN istances.from System -> Routes -> Contiguration then add my subnet

Best effort Mate.

You solve the issue.

Regards

Glad to hear it helped you.

I still haven't got WAN02 to work to be honest though, so I have still some Troubleshooting to do on my End ...

[Disable reply-to on WAN rules]

Replying to my own Thread.

Solved mainly thanks to @zapotah over IRC Channel for the Routing Part and some Trial-Error on my End for the DNS Part.

List of Changes:

• Uncheck "Disable reply-to on WAN rules" in the Firewall -> Settings -> Advanced Section
• Set the Gateway to "default" (do **NOT** do Policy based Routing using explicit WAN_XX Gateway Selection) in every Rule under Firewall -> Rules -> [Floating], Firewall -> Rules -> [WAN_XX_...] and Firewall -> Rules -> [WG_REMOTE_XXX_...]
• ADD a Static Route for each of the Wireguard Instances under System -> Routes -> Contiguration with Network = 10.8.X.0/24 (Wireguard Network with Private Address Range) and Gateway = WAN_XX (192.168.200.1 for WAN01 / 192.168.205.1 for WAN02)
• Add the 192.168.0.0/20 Network in the AllowedIps Section of the Peer
• iPhone Wireguard App: make sure that the DNS Servers are specified and they are Comma-Separated, NOT **Space** Separated
• iPhone Wireguard App: make sure that the AllowedIPs includes also the 192.168.0.0/20 Target Network
• For Testing using something like (I have it always on) a docker.io/georgyo/ifconfig.io or docker.io/traefik/whoami Container can be useful to check your IP Address (I had to spin a separate Instance up and use Port 8080 to have Direct Access and NOT go through the Caddy Proxy, since the DNS was NOT working until I fixed the DNS Servers that were mistakenly Space-separated)

EDIT 1: It's also working on Android now, as an added Benefit :).

It appears that wireguard connection logging is not existent by design.  see https://forum.opnsense.org/index.php?topic=43997.0

But I would think that some type of basic connection/handshake logging would be possible from within OPNsense because the GUI is able to show the connection status and the last handshake age.

Isn't that simply processing the Output of (assuming wg1 is your Wireguard Interface):

Code Select Expand

wg show wg1 dump
Or possibly just grepping the Human Readable Output

But I guess easiest is for X = 1 ... 8 (depending on which Field you want to analyze)

Code Select Expand

wg show wg1 dump | tail -n1 | awk '{print $X}'

I'm really scratching my head on this one.

Previously I had Wireguard VPN:
- Working very well on my iPhone
- Working very well on my Ubuntu Tablet (not much used though)
- Working VERY DREADFULLY on my Android Phone - even though e.g. Home Assistant Notifications would be sent to the Android Phone, I could never read the Details / open them

Yesterday I had a complete Internet Outage during the upgrade of my OpenWRT Upstream Router. Still unsure what exactly caused it, but basically the OpenWRT Router would NOT route anymore. I later discovered that like half the OpenWRT Services responsible for Routing, DNS, etc got disabled for some weird Reason. I re-enabled, restarted them and rebooted the Router, and that seems to be fixed now.

HOWEVER, in the Attempt to get some Internet back and runnnig yesterday, I connected my Secondary Fiber Connection (which I wanted to setup for a long Time) to the OPNSense Router.
Since then, with Multi-WAN Enabled, and assuming that the Settings are correct in terms of which Gateway to use in which Case, Wireguard VPN is COMPLETELY BROKEN for all Platforms.

No Handshake takes place (or no successfull handshake), I sometimes receive some HomeAssistant Notifications on my iPhone, so not sure what's going on there.

In the Firewall Logs I can see 1 x IN + 1 x OUT Connection occurring as soon as I hit the "Connect" Button on iPhone or I start the VPN Service on my Ubuntu Tablet. Nothing happens though.

After carefully setting up the Routes, at least I got to a Point where the Traffic coming in from one Connection / Upstream Router goes out through the same Connection / Upstream Router.

But it's still NOT handshaking. And still no obvious Stuff that is wrong. There does NOT seem to be anything getting denied in the Logs and I enabled to Log everything as Default).

Network Diagram & Screenshots:
https://imgur.com/a/yEjQs0R

EDIT 1: Excerpt from iPhone Logs

Code Select Expand

2025-06-24 14:32:35.221349: [APP] Tunnel 'RemoteAccess-FIBER01' connection status changed to 'connecting'
2025-06-24 14:32:35.309875: [NET] App version: 1.0.16 (27)
2025-06-24 14:32:35.310085: [NET] Starting tunnel from the OS directly, rather than the app
2025-06-24 14:32:35.333928: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:32:35.334624: [NET] Attaching to interface
2025-06-24 14:32:35.334960: [NET] Routine: encryption worker 1 - started
2025-06-24 14:32:35.334972: [NET] Routine: decryption worker 1 - started
2025-06-24 14:32:35.334994: [NET] UAPI: Updating private key
2025-06-24 14:32:35.334994: [NET] Routine: decryption worker 4 - started
2025-06-24 14:32:35.335012: [NET] Routine: handshake worker 5 - started
2025-06-24 14:32:35.335042: [NET] Routine: encryption worker 3 - started
2025-06-24 14:32:35.335039: [NET] Routine: handshake worker 4 - started
2025-06-24 14:32:35.335104: [NET] Routine: handshake worker 2 - started
2025-06-24 14:32:35.335104: [NET] Routine: decryption worker 3 - started
2025-06-24 14:32:35.335131: [NET] Routine: handshake worker 3 - started
2025-06-24 14:32:35.335135: [NET] Routine: handshake worker 1 - started
2025-06-24 14:32:35.335142: [NET] Routine: event worker - started
2025-06-24 14:32:35.335149: [NET] Routine: encryption worker 5 - started
2025-06-24 14:32:35.335186: [NET] Routine: encryption worker 4 - started
2025-06-24 14:32:35.335229: [NET] Routine: decryption worker 2 - started
2025-06-24 14:32:35.335252: [NET] Routine: decryption worker 5 - started
2025-06-24 14:32:35.335251: [NET] Routine: decryption worker 6 - started
2025-06-24 14:32:35.335269: [NET] Routine: handshake worker 6 - started
2025-06-24 14:32:35.335269: [NET] Routine: encryption worker 2 - started
2025-06-24 14:32:35.335291: [NET] Routine: TUN reader - started
2025-06-24 14:32:35.335305: [NET] Routine: encryption worker 6 - started
2025-06-24 14:32:35.335407: [NET] UAPI: Removing all peers
2025-06-24 14:32:35.335618: [NET] peer(/bTG...Z6TQ) - UAPI: Created
2025-06-24 14:32:35.335648: [NET] peer(/bTG...Z6TQ) - UAPI: Updating preshared key
2025-06-24 14:32:35.335685: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:32:35.335754: [NET] peer(/bTG...Z6TQ) - UAPI: Updating persistent keepalive interval
2025-06-24 14:32:35.335785: [NET] peer(/bTG...Z6TQ) - UAPI: Removing all allowedips
2025-06-24 14:32:35.335821: [NET] peer(/bTG...Z6TQ) - UAPI: Adding allowedip
2025-06-24 14:32:35.335901: [NET] peer(/bTG...Z6TQ) - UAPI: Adding allowedip
2025-06-24 14:32:35.336247: [NET] UDP bind has been updated
2025-06-24 14:32:35.336261: [NET] Routine: receive incoming v4 - started
2025-06-24 14:32:35.336283: [NET] peer(/bTG...Z6TQ) - Starting
2025-06-24 14:32:35.336304: [NET] Routine: receive incoming v6 - started
2025-06-24 14:32:35.336342: [NET] peer(/bTG...Z6TQ) - Sending keepalive packet
2025-06-24 14:32:35.336359: [NET] peer(/bTG...Z6TQ) - Routine: sequential receiver - started
2025-06-24 14:32:35.336401: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:35.336431: [NET] peer(/bTG...Z6TQ) - Routine: sequential sender - started
2025-06-24 14:32:35.337080: [NET] Interface state was Down, requested Up, now Up
2025-06-24 14:32:35.337117: [NET] Device started
2025-06-24 14:32:35.337210: [NET] Tunnel interface is utun5
2025-06-24 14:32:35.337543: [NET] Network change detected with satisfied route and interface order [pdp_ip0]
2025-06-24 14:32:35.337991: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:32:35.338098: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:32:35.338282: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:32:35.338343: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:32:35.338618: [APP] Tunnel 'RemoteAccess-FIBER01' connection status changed to 'connected'
2025-06-24 14:32:35.339078: [NET] Routine: receive incoming v4 - started
2025-06-24 14:32:35.339178: [NET] Routine: receive incoming v6 - started
2025-06-24 14:32:35.339229: [NET] UDP bind has been updated
2025-06-24 14:32:35.347368: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:32:35.348023: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:32:35.348171: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:32:35.348421: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:32:35.348474: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:32:35.348792: [NET] UDP bind has been updated
2025-06-24 14:32:35.348817: [NET] Routine: receive incoming v4 - started
2025-06-24 14:32:35.348859: [NET] Routine: receive incoming v6 - started
2025-06-24 14:32:40.356472: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:32:40.356825: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:45.523502: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 3)
2025-06-24 14:32:45.523828: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:50.756726: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 4)
2025-06-24 14:32:50.756938: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:32:55.971949: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 5)
2025-06-24 14:32:55.972261: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:01.095597: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 6)
2025-06-24 14:33:01.095946: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:06.338416: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 7)
2025-06-24 14:33:06.338771: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:11.374720: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 8)
2025-06-24 14:33:11.375030: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:16.459384: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 9)
2025-06-24 14:33:16.459869: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:21.577206: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 10)
2025-06-24 14:33:21.577559: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:26.771926: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 11)
2025-06-24 14:33:26.772266: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:32.083204: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 12)
2025-06-24 14:33:32.083388: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:37.104242: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 13)
2025-06-24 14:33:37.104530: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:42.333708: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 14)
2025-06-24 14:33:42.334024: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:47.587249: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 15)
2025-06-24 14:33:47.587374: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:52.667419: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 16)
2025-06-24 14:33:52.667739: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:33:57.941159: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 17)
2025-06-24 14:33:57.941498: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:03.070999: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 18)
2025-06-24 14:34:03.071291: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:06.349014: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:06.349536: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:06.349692: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:06.349846: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:06.349972: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:06.350135: [NET] UDP bind has been updated
2025-06-24 14:34:06.350179: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:06.350219: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:08.104404: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:08.104627: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:09.479983: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:09.480848: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:09.481026: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:09.481356: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:09.481408: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:09.481746: [NET] UDP bind has been updated
2025-06-24 14:34:09.481758: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:09.481815: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:13.304960: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:13.305286: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:18.485691: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 3)
2025-06-24 14:34:18.485964: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:23.746238: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 4)
2025-06-24 14:34:23.746510: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:28.898774: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 5)
2025-06-24 14:34:28.899075: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:32.691294: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:32.692257: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:32.692500: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:32.692770: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:32.692825: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:32.693150: [NET] UDP bind has been updated
2025-06-24 14:34:32.693159: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:32.693181: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:34.061434: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:34.061658: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation
2025-06-24 14:34:38.271187: [NET] Network change detected with satisfied route and interface order [pdp_ip0, utun5]
2025-06-24 14:34:38.271585: [NET] DNS64: mapped FIBER01_PUBLIC_IP to itself.
2025-06-24 14:34:38.271675: [NET] peer(/bTG...Z6TQ) - UAPI: Updating endpoint
2025-06-24 14:34:38.271920: [NET] Routine: receive incoming v4 - stopped
2025-06-24 14:34:38.272043: [NET] Routine: receive incoming v6 - stopped
2025-06-24 14:34:38.272310: [NET] UDP bind has been updated
2025-06-24 14:34:38.272334: [NET] Routine: receive incoming v4 - started
2025-06-24 14:34:38.272401: [NET] Routine: receive incoming v6 - started
2025-06-24 14:34:39.095287: [NET] peer(/bTG...Z6TQ) - Handshake did not complete after 5 seconds, retrying (try 2)
2025-06-24 14:34:39.095625: [NET] peer(/bTG...Z6TQ) - Sending handshake initiation

EDIT 2: Added Screenshot showing Gateways (see linked Post since I cannot Post High Resolution Pictures)

EDIT 3: Not sure what is going on. To check that the basic Principle works, I did a basic Setup on my Hetzner Cloud VPS (behind NAT), and there it works as intended. Sure 1 x WAN Connection only instead of 2 (I tried to replicate here in my Homelab by unplugging the 2nd WAN Cable, so that all Traffic is forced through the remaining WAN Interface).

Any Idea ???

EDIT 4: While trying to search I found this Yesterday, not sure if it's really a Thing though.

https://www.reddit.com/r/opnsense/comments/171k5ap/comment/kj85q3f/

Is it possible that the Wireguard Peer Generator is seriously broken ?

EDIT 5: Actually it seems that I am getting the HomeAssistant Notifications only if I am NOT connected to VPN. Which makes the entire thing even more Confusing. Like need to be Disconnected from VPN to receive the Notifications, yet I need to be connected to VPN to be able to read them. What a Mess !

I should probably be able to dig into a previous Backup of OPNSense Configuration XML File, but I cannot see what changed in Terms of Firewall Rules. It should just work ...

Ok, so podman seems to be the issue here. What speaks against using docker?

The same Argument that speaks against changing an entire System / Infrastructure / Set of Scripts / Network Topology / Administration in every Industry or Technical Domain just to address 1 "Edge" / Special Case: it's a big Pain in the Ass ...

So either you need to do a Mixed Setup, possibly requiring an extra IPv4 Public Address (since Podman/Docker will have to use port 80/443 for different Use Cases), or switch everything to Docker (and I almost migrated every Podman Installation to Fedora just because Debian isn't well supported under Podman).

But over the years I also learned that "the grass is NOT greener on the other Side". Whenever you think you switch Solution in order to (try to) fix a Problem, you might (will) also incur in several additional Issues that you didn't have before.

My Netbird host is a Hetzner VPS, ARM64, 2 CPUs, 4 GB of RAM, of which only 1.2GB are used. Postgres as database backend. Can't really see the OOM problems you had.

I have an old Hetzner AMD64 VPS with 2GB or RAM also on Hetzner. They increased the Prices a few Months ago, so I will probably just cancel that and use my own KVM Virtual Machine on the Dedicated Server instead.

EDIT 1: tried your Command after I shut everything else down so that Port 80/443 would be Free ... And after symlinking

# ln -s / usr/bin/podman /usr/bin/docker
#ln -s /usr/bin/podman-compose /usr/bin/docker-compose

Error: no container with name or ID "netbird-quickstart_zdb_1" found: no such container
Error: no container with name or ID "netbird-quickstart_zitadel_1" found: no such container
Error: no container with name or ID "netbird-quickstart_coturn_1" found: no such container
Error: no container with name or ID "netbird-quickstart_management_1" found: no such container
Error: no container with name or ID "netbird-quickstart_relay_1" found: no such container
Error: no container with name or ID "netbird-quickstart_signal_1" found: no such container
Error: no container with name or ID "netbird-quickstart_dashboard_1" found: no such container
netbird-quickstart_caddy_1
Error: no container with ID or name "netbird-quickstart_zitadel_1" found: no such container
Error: no container with ID or name "netbird-quickstart_zdb_1" found: no such container
Error: no container with ID or name "netbird-quickstart_coturn_1" found: no such container
Error: no container with ID or name "netbird-quickstart_management_1" found: no such container
Error: no container with ID or name "netbird-quickstart_relay_1" found: no such container
Error: no container with ID or name "netbird-quickstart_signal_1" found: no such container
Error: no container with ID or name "netbird-quickstart_dashboard_1" found: no such container
netbird-quickstart_caddy_1
537090513c345560782ef175c08e189493932b95de2544738b3c25be008ae775
Error: no container with name or ID "netbird-quickstart_relay_1" found: no such container
Error: no container with name or ID "netbird-quickstart_signal_1" found: no such container
Error: no container with name or ID "netbird-quickstart_coturn_1" found: no such container
Error: no container with name or ID "netbird-quickstart_zitadel_1" found: no such container
Error: no container with name or ID "netbird-quickstart_zdb_1" found: no such container
Error: no container with name or ID "netbird-quickstart_management_1" found: no such container
Error: no container with name or ID "netbird-quickstart_dashboard_1" found: no such container
Error: no container with name or ID "netbird-quickstart_caddy_1" found: no such container
Error: no container with ID or name "netbird-quickstart_zitadel_1" found: no such container
Error: no container with ID or name "netbird-quickstart_zdb_1" found: no such container
Error: no container with ID or name "netbird-quickstart_coturn_1" found: no such container
Error: no container with ID or name "netbird-quickstart_management_1" found: no such container
Error: no container with ID or name "netbird-quickstart_relay_1" found: no such container
Error: no container with ID or name "netbird-quickstart_signal_1" found: no such container
Error: no container with ID or name "netbird-quickstart_dashboard_1" found: no such container
Error: no container with ID or name "netbird-quickstart_caddy_1" found: no such container
Error: no pod with name or ID pod_netbird-quickstart found: no such pod
4834915b5ccc38ab944085421f75e24649a222ac10936b9934503424ae397811
94826dfeff09f96232cb57ee7d3e98bf13555f3126a7ce676d95004e85d1d100
netbird-quickstart_caddy_1
✔ docker.io/netbirdio/dashboard:latest
Trying to pull docker.io/netbirdio/dashboard:latest...
Getting image source signatures
Copying blob f7dab3ab2d6e skipped: already exists 
Copying blob 25d8059c17de done   |
Copying blob ff09aab76d97 done   |
Copying blob e252bd70cdea done   |
Copying blob e9fb81678df7 done   |
Copying blob 78f3aa16cfa5 done   |
Copying blob b6c81a3e8178 done   |
Copying blob 932bd785729d done   |
Copying blob 217c556afd61 done   |
Copying blob f846d527a638 done   |
Copying blob cb7988d44772 done   |
Copying config 5aa906f022 done   |
Writing manifest to image destination
d800170607d2c90e26f03b84e5018a3f6d2510dc11d98dd9e01ddbc8bb590f6d
netbird-quickstart_dashboard_1
3bb334c8ed73d09b4b6d8dc5950116e58eb8bbdccb593979e957aa58334c7111
netbird-quickstart_signal_1
51555daff8ad139dfb116c240b779f41d67ccbbef9732bc670a5c1f5dafb9aa4
netbird-quickstart_relay_1
d4694f689c6e5e43804a9f99e175ae49ac0741cf9185ca797d762a647ae439d2
netbird-quickstart_management_1
9a642849aa9f33ed8836136e2bc9adef3abec013d7070a8e5ce8af4f98048612
netbird-quickstart_coturn_1
Traceback (most recent call last):
  File "/usr/bin/podman-compose", line 33, in <module>
    sys.exit(load_entry_point('podman-compose==1.2.0', 'console_scripts', 'podman-compose')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 3503, in main
    asyncio.run(async_main())
  File "/usr/lib64/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 3499, in async_main
    await podman_compose.run()
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 1742, in run
    retcode = await cmd(self, args)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 2499, in compose_up
    podman_args = await container_to_args(compose, cnt, detached=args.detach)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/podman_compose.py", line 1204, in container_to_args
    raise ValueError("'CMD_SHELL' takes a single string after it")
ValueError: 'CMD_SHELL' takes a single string after it

So yeah, not so easy with Podman I guess.

KISS with Wireguard only or Wireguard + Netbird at the "Price" of having a bigger ecosystem that can break more easily ? Uhm ...

I just saw, that I use Netbird with the default IdP Zitadel and not Authentik or Keycloak. Used the provided script and it was running out of the box.
Of course you add another service (at least self hosted if you want), but I think you gain a lot of features, like Zero-Trust for your clients.

Configuring connections to one single hub is fairly easy. If your central WireGuard hub goes down, you're lost, too.

Connecting all the spokes in a peer-to-peer manner is another story, if you have more than 4 spokes: that's 6 spoke connections and one to the hub, with 5 spokes, it's already 10 connections+ the hub.

With Netbird you're able to configure multiple routes to the same destination, if you want. I think OPNsense and Netbird are a perfect match here.

Good to know that's also a Feature Netbird provides  :). If only it would work in my case  :(.

As for Zitadel, that's the third Attempt I did back then on my Hetzner VPS (after Authentik and Keycloak) and it would NOT work at all. Zitadel was such a Memory Hog that I believe it triggered the OOM Killer due to excessive RAM Usage. Anyways, not an Option on a low CPU/RAM VPS. I have a dedicated Server now with several KVM Virtual Machines, so I could try that.

But I really liked Authentik, it's just an absolute PITA to interface with Netbird. And Netbird Debugging / Troubleshooting Capabilities are quite bad in my View, when something does not work (at all), it's not very clear (at least to me) as to why. And when it works, it's probably fine (until it breaks). I never managed to even get something to show up on the Web GUI so it's really frustrating to be honest  :(.

Granted, it could also be due to the Reverse Proxy (Traefik) Setup and possibly some Firewall Rules (I added exceptions based on Netbird specifically mentioning Hetzner Stateless Firewall although that did NOT make any Difference).

As to Wireguard breaking down ... I see that as a MUCH less likely Risk. Yes, it might be more of a PITA to set up Manually 100 Instances of Wireguard (Ironically in my Homelab, Gitlab and Nextcloud kinda forced my Hand on this one, since I HAVE to use NFS since their Update Script doesn't work with Samba/SSHFS Permissions and I don't have the Time to setup a Kerberos server for NFS - so I just do NFSv3 TCP over Wireguard UDP).

But compare generating a Keypair, setting up one small Config file for each Point-to-Point Connection with a System that might very easily break between Updates (either on Netbird side, or on Authentik/Keycloak/Zitadel side). I'd say Wireguard is very Reliable in that Regards.

Netbird should begin having some Consistency in their config File ... Depending on the Guide you Follow some Config/Environment Variables are NETBIRD_AUTH_XXXX and others are AUTH_XXXX and it's not always clear which Direction they are moving towards (I kinda had to duplicate quite a few of them in Order to suppress some Warnings in the Logs, although that did not solve my Problems).

I think Patrick means to use the tools that are already there: WireGuard.

You rent a VPS at your trusted VPS host and let this be the WireGuard hub. All your other locations connect to this hub and traffic is distributed as needed/configured.

I have this currently in place. Some of my locations also have "old" IPsec tunnels between each other. But I want to get rid off those. I could use WireGuard but then I stumbled across Netbird and I'm directly a huge fan of it.

It leverages the idea of Zero Trust, which I definitely prefer as boundaries are vanishing more and more. In a hybrid environment with multi-cloud and multiple On-Prem locations it gives you the best approach to connect everything with each other. And the best part is: the hub concept is only used, when a direct connection is not possible. Otherwise the spokes are connecting directly to each other.

I don't understand why you didn't succeed in getting Netbird up and running. I'm using it with Authentik and used the script that was provided. No issues at all.

Netbird + Authentik: https://github.com/netbirdio/netbird/issues/1684
Netbird + Keycloak: https://github.com/netbirdio/netbird/issues/1715

When the Logs don't say much [very clearly] and the Web UI doesn't even show up (well it shows an empty Page but that's completely useless), then it's a bit difficult to go forward.

I finally managed with Wireguard on its own Manually, so that's Nice. But netbird would be more convenient I guess.

KISS with Wireguard only or Wireguard + Netbird at the "Price" of having a bigger ecosystem that can break more easily ? Uhm ...

I think Patrick means to use the tools that are already there: WireGuard.

You rent a VPS at your trusted VPS host and let this be the WireGuard hub. All your other locations connect to this hub and traffic is distributed as needed/configured.

Exactly.

E.g. in my FreeBSD 14.1 VPS at vulture.com: /usr/local/etc/wireguard/wg0.conf

Code Select Expand


[Interface]
Address = 192.168.254.1/24,2003:a:d59:3840::1/64
PrivateKey = *********
ListenPort = 51820

# Peer 1
[Peer]
PublicKey = *********
AllowedIPs = 192.168.254.2/32,2003:a:d59:3840::2/128

# Peer 2
[Peer]
PublicKey = *********
AllowedIPs = 192.168.254.254/32,2003:a:d59:3840::254/128

[...]


Connect as many peers as you like. If you want to route entire networks to specific peers just add them to the "AllowedIPs" statements. Configure the peers in a matching fashion, done.

I don't know what for one would need a "VPN service". Plus, I don't trust them.

To perform outbound NAT I use /etc/pf.conf:

Code Select Expand


nat on vtnet0 inet from 192.168.254.0/24 to any -> ww.xx.yy.zz
nat on vtnet0 inet6 from 2003:a:d59:3840::/64 to any -> dead:beef:dead:beef:dead:beef:dead:beef

pass all no state


You can add inbound port forwarding or e.g. a Caddy reverse proxy with Letsencrypt as you like.

Yeah I was basically out of Options now that I tried for several Months so I just tried directly with Wireguard from an iPhone and Android Phone to VPN directly to my Home OPNsense (double NAT & Port Forwarding) and set manually the DNS Servers to 192.168.1.xxx etc. After fixing Outbound NAT, it worked quite well (ironically almost better on iPhone compared to Android ...)  :D.

You can do exactly that with a single virtual server at any cloud provider, connect all your locations via e.g. WireGuard and have a self hosted self managed transparent solution. You will have to pay about a fiver per month for that virtual server.

I would never use any "VPN provider" because I care about my privacy.

Hi Patrick. Would you have some specific guide in Mind ?

I attempting to do this since several Months but, even forgetting the "funnel-like" Feature, it always ends up not working as it's supposed to be (or in my Case ... not at all).

- I tried Headscale + Tailscale (basically no support since they claim Headscale is not supported and my Android Phone won't use the Set DNS Servers over VPN).

- I tried Netbird (stuck at installation with either Authentik or Keycloak), never even get to have the Web Panel showing up

I am revisiting an issue I had a few Months ago, this time with IPv6, as opposed to the previous Issue I had with IPv4.

Desktop Main IP: 2XXX:XXXX:XXXX:0001:0000:0000:0003:0066/64

Server Main IP: 2XXX:XXXX:XXXX:0001:0000:0000:0008:0015/64

Containers Dedicated Subnet: 2XXX:XXXX:XXXX:ff15:0000:0000:0000:0000/64

OPNSense has:

• Firewall -> Settings -> Advanced -> [CHECKED]  Bypass firewall rules for traffic on the same interface
  
• Gateway: 2XXX:XXXX:XXXX:0001:0000:0000:0008:0015/64
• Static Routes: 2XXX:XXXX:XXXX:ff15:0000:0000:0000:0000/64 via 2XXX:XXXX:XXXX:0001:0000:0000:0008:0015/64

If I setup the same Static Route on my Desktop, then I can iperf3 -c 2XXX:XXXX:XXXX:ff15:0000:0000:0000:0002 -P 10 and I get something like 500 Mbit/s. Fair enough for a gigabit connection with all Switches etc.

If I go THROUGH OPNSense I get 0.00 bytes/s ... 0.00 bytes/s  ???.

Is there maybe an issue "on the way back" ?

Like: Desktop 2XXX:XXXX:XXXX:0001:0000:0000:0003:0066 -> OPNSense 2XXX:XXXX:XXXX:0001:etc IN -> OPNSense 2XXX:XXXX:XXXX:ff15:etc OUT -> 2XXX:XXXX:XXXX:ff15::0002

But maybe there is no "route back" to the Desktop ? Or if the Connection is successfully established, then the reply Packets would go down that open connection Anyways ?

Nothing "red" in the Firewall Logs.

Hardware NIC Offloading was DISABLED ([CHECKED]), but ENABLED ([UNCHECKED]) did NOT make any difference.

EDIT 1: Iperf3 can go down to full speed (~ 900 Mbit/s) if, DURING THE RUN, I change something in OPNSense -> Firewall -> Advanced -> Settings -> click apply. But starting a new instance of iperf3 -c brings back again the speed to 0.00 bytes/s.

So of Course this is NOT a solution. Neither a workaround.

UDP iperf3 speed is around 1Mbit/s per each parallel thread (-P 10 -> 10 Mbit/s, -P 100 100 Mbit/s) whereas the default TCP iperf3 is 0.00 bytes/s UNLESS I change something in OPNSense -> Firewall -> Advanced -> Settings -> click apply DURING the iperf3 run.

If you are looking for a quicker way to assign multiple IP addresses to a host, check out Ansible or NixOS. Those are a better solution for your use case than tinkering with DHCP IMHO.

Bart...

Well I'd use saltstack if we are talking about going down that path I guess ...

I'm wondering why it's NOT possible to assign multiple IP Addresses to a DHCPv4/DHCPv6 Client using OPNSense (ISC DHCP Server).

For instance, on a Hetzner Server, since a MAC Address is bound to multiple IPs on Hetzner Robot, if everything gets configured via DHCP (Client) on the Server, then the Interface can get as many IPs as it has assigned (in the Rescue System I could get 4 of them). On the Production Server I configure IP Statically (mostly by Choice), so it's not a thing ...

I just wonder why the OPNSense DHCP Server doesn't support this.

Basically:
- Same MAC (DHCPv4 Server) or Same DUID (DHCPv6 Server)
- Different IP Address

Is there a Technical Reason that this is not implemented ?

Use case: I am running Podman (similar to Docker) Containers rootless on a Debian/Fedora Host. The user running the Containers can bind to any Port & IPv6 Address, but the Address must FIRST be assigned to the Host itself (which would be IMHO easier to do via DHCP, rather than having to use `nmcli` in Fedora - `/etc/network/interfaces` in Debian)

The plugin should configure /usr/local/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml

to use the _ instead of - which is the default value, but not allowed by opnsense.

Can you please run

# configctl crowdsec reconfigure

# tail -f /var/log/configd/latest.log

and see if there's any error?

thanks

Thank you for your Answer.

Here you go:

Code Select Expand

configctl crowdsec reconfigure
OK


Code Select Expand

tail -f /var/log/configd/latest.log
<13>1 2024-06-05T14:42:37+02:00 Router.localdomain configd.py 234 - [meta sequenceId="1"] [b9f126b9-7623-4072-9890-96f072c3d8e0] crowdsec reconfigure
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="2"] [d57dd0fe-b953-4385-96ae-1ec8c01f6d19] Reloading filter
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="3"] [c648db2c-ae47-47a6-9674-c14948d3ba06] request pf current overall table record count and table-entries limit
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="4"] [3bec8a08-36d9-46f4-ab15-bd3111cc8413] list gateways
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="5"] [597c3bf1-f468-4d68-b18a-39e5608a341c] generate template OPNsense/Filter
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="6"] generate template container OPNsense/Filter
<15>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="7"]  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
<15>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="8"]  OPNsense/Filter generated //usr/local/etc/filter_geoip.conf
<13>1 2024-06-05T14:42:38+02:00 Router.localdomain configd.py 234 - [meta sequenceId="9"] [e7152f5f-c5b6-481c-b9d5-50aee3779d1d] refresh url table aliases
<14>1 2024-06-05T14:42:41+02:00 Router.localdomain configd.py 234 - [meta sequenceId="10"] message e7152f5f-c5b6-481c-b9d5-50aee3779d1d [] returned b'{"status": "ok"}\n'

Now indeed in OPNSense -> Services -> CrowdSec -> Overview it's indeed better:
Service status: crowdsec [tick / success] - firewall bouncer [tick / success]

But it's still unclear to me why this happens on a stock Install ... and for how long it would even work ???.